Meeting minutes
Review agenda and next meeting dates
https://
Matt_King: Requests for changes to agenda?
Matt_King: Hearing none, we'll stick with the agenda as planned
Matt_King: Next CG meeting: Wednesday August 13
Matt_King: Next AT Driver Subgroup meeting: Monday August 11
Current status
Matt_King: We currently have 5 plans in draft review and 15 in candidate
Matt_King: Everything that's in the queue now is related to draft
Matt_King: The three that are active are the three that we have on the agenda today
Matt_King: The other two are kind of on hold at the moment
Matt_King: Next up: we have the disclosure (fixes were landed earlier this week)
Matt_King: Tabs with manual activation will come up after that. We're just hoping to get a little experience with the first tabs-related plan, first
Matt_King: No changes with any of the screen reader developers, yet
Vertical Temperature Slider
https://
Matt_King: We have conflicting results with all three screen readers
Matt_King: I didn't raise issues for any of these conflicts, and I don't know if we will need to
Vertical Temperature Slider - JAWS
Matt_King: When I was reviewing the conflicts for JAWS, I noticed that IsaDC and Joe_Humbert recorded different results but that they were using different versions
Matt_King: IsaDC reported an older version
Matt_King: IsaDC perhaps the discrepancy could be resolved if IsaDC updates her software
IsaDC: Ah, because Joe_Humbert gets the min and max, and I didn't
IsaDC: Okay, I will update to the latest version
Matt_King: All of the conflicting results here are in test one, and they're all related to that min and max output
Matt_King: So that seems potentially straightforward
Vertical Temperature Slider - NVDA
Matt_King: I wanted to spend some time on this because for this one (and I think for other conflicts in other test plans) seems to be one of the situations where we would want to use the new feature for reporting that the assertions are untestable
Matt_King: The response you reported indicates that the reading cursor didn't move, which means it wasn't on the appropriate element, so I think it is in line with one of those situations where it's more accurate to mark them as untestable (rather than marking them all as failed)
Matt_King: None of the output indicates that it would be able to pass. For us to be consistent, I think any time that the focus of the reading cursor does not end up on the right element, that we should just mark the assertions as untestable
IsaDC: I updated those, but I need to change the side effects because yes, the cursor didn't move
Joe_Humbert: We have two different unexpected behaviors
Matt_King: Well, it looks here like it wasn't reported as untestable
IsaDC: I did report it as untestable
Joe_Humbert: I did not, though.
Joe_Humbert: I can change mine
Matt_King: Does the conflicts page have a bug where if you mark a test as untestable, it counts it as a failure?
mmoss: If you assign assertion verdicts and then later mark it as untestable, then it can enter a weird state where it is both untestable and failing
Matt_King: That sounds like a real bug!
Joe_Humbert: If I mark it as untestable, it removes the unexpected behaviors. That's a little annoying
Matt_King: How we write the description--we need to think about how it shows up in the report. If the screen reader developer is reading this, the misbehavior should be clear
Matt_King: I think what Joe_Humbert wrote is perfect
Matt_King: So we're aligned there: this should be untestable
Matt_King: And we have an app bug to file
jugglinmike: I'll see to it that someone at Bocoup files this app issue
Joe_Humbert: The second conflict is the same as this one, it just needs to be marked as untestable. Doing that should clear up the conflicts
Joe_Humbert: For this test, though, all the screen reader reports is "blank", so do we need to report some negative side effects?
Joe_Humbert: This is test 3 ("using insert+up arrow in focus mode")
Matt_King: I don't see the conflict; where did it go? It's not on the conflicts page
Matt_King: I think this a clear failure in this case
Joe_Humbert: In that case, what should we put for the negative assertions?
Joe_Humbert: Because technically it doesn't convey that information
Matt_King: Well then I think those should pass
james: I don't think it is appropriate to pass them
Matt_King: Why not? If it says it does not do something, and it doesn't do that thing--then as long as it's on the right element, that's good!
james: the presence of negative assertions that we would have to pass should lead us to mark the test as "untestable"
james: I don't think anyone would say, "it says 'blank' but at least it doesn't..."
Matt_King: The "untestable" feature is for when the behavior blocks the evaluation of the assertion
Joe_Humbert: I guess IsaDC and I just need to know what we need to put here. We all know it isn't working right
james: If it were to pass the other assertions, we wouldn' tbe having this conversation because it actually reported something relevant to the test at hand
Matt_King: It is not reporting the correct information
Joe_Humbert: I have a question here. How is this report of the wrong information different than the test we just discussed?
Joe_Humbert: In the previous test, we pressed a key, and it doesn't move and reports the wrong information
james: It's the not moving that makes the difference
james: You are on the right element, and it did the wrong thing (as opposed to not being on the right element in the first place)
james: To me, it is misleading to say that by not doing anything, the AT did the right thing
Joe_Humbert: It's still failing a couple assertions, so overall, it's still evident that something is wrong
Joe_Humbert: So, fail everything but the negative assertions
Matt_King: Right
Matt_King: If we need more clear guidance, then I'd be happy to write it up
james: I think this resolution is fine for this situation, but I think we'll see this again, and if it concerns a more core behavior, then it will be more difficult for us
Matt_King: So for these negative assertions, it's probably important that they are optional
james: The assertion itself--maybe in the future we would want to be able to mark an individual assertion as untestable
Matt_King: Maybe. That feels complicated.
Matt_King: What was the negative assertion?
IsaDC: minimum value, maximum value, and a numeric value
Matt_King: Oh, this is the value text thing
Matt_King: Right. This is just a weird ARIA thing, where you have two different kinds of values for the same slider, but only one of them is relevant to the user
Matt_King: Those negative assertions are almost a side-effect of the spec being silly
james: I think the spec is off in that it says you should use both
james: I think that's misleading. And there is at least one screen reader that conveys both in a confusing fashion
Matt_King: The more we talk about this, the more I realize that what we've found here is a reasonably bad bug in ARIA itself. It's kind of unfair, in a sense, for the interoperability test to be needing this negative assertion at all
Matt_King: The purpose of a negative assertion is to tell the screen reader not to pay attention to this ARIA bug. At least that's the way I'm looking at it right now
Matt_King: I don't think that we should be over-complicated the interoperabilty test because of the inconsistent specification of ARIA itself
Matt_King: We're saying not to repeat something that shouldn't be there (the spec requires it to be there, but the spec should not require it to be there)
IsaDC: So what do we do here?
Matt_King: We're going to mark the negative assertions as passing
Matt_King: The long-term fix is to get ARIA changed and then to remove the negative assertions
Joe_Humbert: And there's no negative side-effects?
Matt_King: Right. No negative side-effects
Vertical Temperature Slider - VoiceOver
Joe_Humbert: One of the conflicts is the negative assertions. One of you has it passed and one of you has it failed
Matt_King: The VoiceOver cursor doesn't actually move, even though it speaks a different element on the page. But it doesn't actually move
dean: I don't recall exactly what it did, but it did not move
dean: I don't think that counts as "moved in an unexpected manner". I think it's "other"
mmoss: That's the behavior I saw as well, so I marked it as "other"
dean: I've been talking about accordion
Matt_King: We haven't reached that, yet
dean: Ah, sorry! I'll finish the work on vertical temperature slider
Matt_King: In this case, the VoiceOver output is just bizarre. It reads the heading level. I remember trying this out myself. You have the focus on the slider, you press the 'home' key, and for some reason, VoiceOver reads the heading level on the heading that's above the slider
mmoss: I just confirmed: it changes the slider value and reads the heading, but the cursor appears to remain on the slider
Matt_King: Okay, so for the negative assertion, dean was passing it. mmoss was failing it, but it seems like the negative assertion should pass
mmoss: Yes, this speaks to the conversation we were just having with james
mmoss: I couldn't determine that the reading cursor changed until attempting to navigate again after the action
Matt_King: Yeah, this one is sort of a strange one because it's not obvious what's happening
Matt_King: You both recorded the same unexpected behavior: reading cursor changed in an unexpected manner
Matt_King: If we mark it as untestable, I don't know how to fix it with the current bug.
mmoss: I have to mark it as no longer being untestable, then I can change the values, then I can re-mark it as untestable
Matt_King: All assertions should be untestable, but there's no way to un-check the radios in the current UI. So I think we're stuck until we fix the bug
jugglinmike: Didn't we have a discussion as tri-state?
Matt_King: Originally, yes, then we want to checkboxes, then we went to radios
Matt_King: I understand the appeal to tri-state. We implemented "untestable" as a checkbox because we wanted to make sure that all of the assertions were marked as untestable
Matt_King: If you check untestable, we want to make sure that the radios are unchecked
Matt_King: That way we don't have to force the testers to deal with tri-states
Matt_King: If that was the behavior, then there would be no problem, and mmoss would be able to fix his test plan run
Matt_King: So we're stuck with this one on the slider. But if we get a fix for it next week, we can move on in our next meeting
Accordion Test Plan - VoiceOver
Matt_King: I think that all the ones that were "shift+B" or "shift+J", that those were supposed to be untestable, and that's where most of the conflicts lie
dean: I mentioned this in my e-mail. The cursor goes to the form field, and VoiceOver asks "what do you want to put into the form field?"
dean: So my question is: do you make that untestable? If I toggle QuickNav off and then back on, then the command works
mmoss: I always toggle QuickNav off and on after each test. I didn't experience what you've reported, and I think that's why
Matt_King: The command requires you to have Quick Nav on, and if you went into it with Quick Nav off and then turned it on, then mmoss is saying that it works
dean: It worked for me, too, but it didn't work if I started the test with Quick Nav on
Matt_King: I've encountered this in real life, too. It's very frustrating
james: Can you turn autocomplete off in your browser?
Matt_King: For the purpose of this test? I don't think we want to
james: We test screen readers with default settings, but we've never had a case where someone's browser settings has caused conflicts
james: We don't ask testers to reset the settings of their browsers (that would make testing very difficult--logging in again, etc)
Matt_King: I don't know if autocomplate is on or off by default in Safari, but I have some reservations
dean: This is for "navigate backwards to an expanded accordion"
james: VoiceOver is bad a responding to the web page setting focus as opposed to the user setting focus
Matt_King: We might need to raise an issue for this test 8 conflict to decide
Matt_King: For test 8, I want to get multiple peoples' observations
Matt_King: And then, let's talk about all the others
Matt_King: dean, for the remaining accordion tests, can you mark the "shift+J" and others--where the focus wasn't moving--as "untestable"? I think that will resolve most of the conflicts.
Matt_King: We're almost out of time. We can raise an issue and talk more about this next week
Tabs with Automatic Activation
Matt_King: Louis found a bug in the app. I can't submit one of the tests. It only impacts test 13 for JAWS
James: there are details in the issue report
Matt_King: Bocoup expects to deploy a fix next week
ChrisCuellar: It impacts just that one test and only JAWS and VoiceOver
IsaDC: I'll send an e-mail with these details to Joe_Humbert when I have the bots assigned
Hadi: this was one of the most interactive meetings that we've had!
Matt_King: More tests are coming soon. Disclosure could come online this week, and then the tabs with manual activation should come online next week
Matt_King: Thanks, all!