Meeting minutes
Review agenda and next meeting dates
https://
Matt_King: Any requests for changes to the agenda?
Matt_King: Hearing none, we'll use the agenda as planned
Matt_King: Next CG meeting: Wednesday August 27
Matt_King: Next AT Driver Subgroup meeting: Monday September 8
Current status
Matt_King: We have seven plans in draft review right now
Matt_King: Five of them are pretty active, and two of them we have to get back to
Matt_King: I was hoping I would have time to look into the "disclosure navigation menu" to see if there is still anything blocking us, but I don't think we'll get to that today
Matt_King: IsaDC is working on three more for draft review
Matt_King: We might get those mostly ready for testing by the end of the month
Matt_King: That will put us within striking range of having 25 plans in Candidate Review
Matt_King: That could happen in September. It will be a pretty big step up!
Running vertical temperature slider test plan
Matt_King: We've completed both JAWS and NVDA on this one, which is awesome
Matt_King: There are only three conflicts for VoiceOver. These are related to two tests that are untestable
Matt_King: Last week, we were waiting on an app fix (where it would automatically uncheck yes/no radios when we checked "untestable")
howard-e: We released a fix for that
IsaDC: Do Testers have to re-submit results?
mmoss: Because we still have conflicts
mmoss: I marked this as "untestable" originally
mmoss: This was one that I re-ran. We didn't have the "untestable" test run for my initial run
Matt_King: This is for tests number 8 and 9
mmoss: Oh, this might be an issue with Dean's results, but not with mine
IsaDC: Yes, Dean passed an assertion incorrectly
Matt_King: Does this submit?
IsaDC: What happens if you submit it?
Matt_King: On yours, mmoss, it says "other"
mmoss: Right, because it doesn't seem like the cursor is actually changing--it's just reading the wrong thing
mmoss: That's why we changed it to "other". I don't think Dean has done that
Matt_King: I think in the description, we would want to say something like "instead of reading the new slider value, it is reading the heading of the slider"
Matt_King: ...so it's a little more clear exactly what the problem is
Matt_King: But instead of saying "seems to have changed", should we just say, "it's announcing the heading"?
mmoss: Yes, I think that's more clear
Matt_King: Okay, so if you could update your test results
Matt_King: Dean couldn't be here today, but he has observed the same behavior, so I think it's fair to update his results for both test 8 and 9
IsaDC: Okay, I will do that
Matt_King: Great--then, this test plan will be done
Running accordion test plan
Matt_King: This is one that I don't know if we can really talk about. We're all done with this one, as well. JAWS and NVDA are both complete
Matt_King: We have 42 conflicting VoiceOver results
Matt_King: Now that we have the app fix, I think we're good to go here
Matt_King: We just need Dean to go through and correct his results. I guess he didn't get to that, yet
IsaDC: I will e-mail Dean and tell him what we need
Matt_King: Just taking care of the "untestable" stuff should cut down that 42 number down by a lot
Matt_King: And I recall that there were some other changes that he was going to make
Running "Tabs with Automatic Activation" test plan
(This topic is not present in the agenda. The group decided to add this topic on-the-fly)
Matt_King: We made changes to tests 10 through 14. That's eight of the 15 tests changed as a result of last week's discussion
Matt_King: There were conflicts for JAWS, but they could go away because they are in tests that changed
Matt_King: In test 3, the outputs look the same between Hadi and Louis
Hadi: I think Louis was complaining about some redundancy--the word "selected" is repeated
Hadi: I didn't think that was a reason to fail the test
Matt_King: It says "Selected" and then it says "One of four selected"
Matt_King: That is a little strange
Matt_King: I don't know if we should report that or not
Matt_King: I kind of feel like Vispero might want to know about it. It's pretty moderate. But it certainly isn't ideal.
IsaDC: I mean, it is reporting it twice
Matt_King: It's not expected. We say there should not be any moderate side-effects
Matt_King: I think there's no harm in reporting it and then seeing what Vispero says. They've been very responsive. I think this is a judgment call
Hadi: I agree. Let's report it. If they are eager to fix it, then that's good; otherwise, we'll live with it
Matt_King: If we look at a strict reading of our criteria, this test doesn't give any incorrect information, so it doesn't quite meet the bar
Matt_King: We could raise an issue with them but not put it in these reports
Matt_King: If we're sticking to our guns about our definition of what "excess verbosity" is, that's probably the more judicious route
Hadi: Redundancy generally raises my suspicion
Hadi: I think we can report it to them and see
IsaDC: I think they'll appreciate the feedback. It's moderate, after all
Matt_King: I'm fine with leaving it in. If you want to match that in your report Hadi, you can
jugglinmike: This reminds me a little bit of alert. If Vispero wants to keep it, would we add a "MAY" assertion for this?
Matt_King: I can't think of an optional assertion that we could add that would be relevant
Matt_King: We already have an assertion related to the selected state, and it's a MUST
jugglinmike: You could write something like "MAY communicate the selected state twice", if only to commemorate the fact that we observed this, talked about it, and decided to accept it
Matt_King: I don't think there's much room for interoperability improvements here
Matt_King: Okay, so this affects multiple tests
Matt_King: I think all of these are related to the same thing
Hadi: I submitted an issue. Do you see the word "none not selected" when using arrow up on a tab list?
Hadi: The question was confusing because it gave the impression that you should be hearing "not selected"
IsaDC: That was a mistake on our side; we've corrected it, but we haven't added the new test plan into the test plan, yet
Matt_King: So you merged it, but it's not in the queue, yet?
IsaDC: No
Matt_King: Okay, let's do that. It's going to change the whole conflict picture
[Matt_King forces the new version of the test plan into the queue]
Matt_King: Now it says that it's 40% complete for JAWS with two testers. And now it says 9 of Hadi's 15 tests are complete (so we essentially lost 6 from you, though that doesn't mean those tests are fully lost). And we have five conflicts
Matt_King: The conflicts are different because you're results for test 3 need to be revisited
Matt_King: The conflicts for test 5 and 7 are still here, and those are the same
Matt_King: Test 5 has the same kind of verbosity conflicts that we were just talking about a minute ago
Matt_King: Test 7 also has that same thing. They're very consistent here
Matt_King: And also test 8
Matt_King: So I think we've covered that conflict. Hadi, can you go back through tests 1 through 4 (and I'm not sure which others)?
Hadi: Okay!
Matt_King: Then we have two conflicts in NVDA, and this was Joe_Humbert and Louis
Matt_King: Joe_Humbert, a lot of the feedback on this test plan came from you, so hopefully we've addressed everything that you thought was a problem
Matt_King: In terms of conflict, we have test 10, "activating the last tab"...
Joe_Humbert: We didn't get to talk about this last time. The bot had repeated information because the speech viewer was repeating information, but the screen reader was not voicing the repeated information
Joe_Humbert: What the screen reader was saying and what was present in the speech viewer was not matching
Matt_King: Oh my!
Joe_Humbert: Louis isn't here, but I suspect he changed the output. I left mine so that I could bring it to the Community Group
Joe_Humbert: I initially thought this was a problem with the bot, but when I went to run the test manually, I observed the same behavior
James: the bot is collecting the information which is sent to the synthesizer
Joe_Humbert: This occurs on a couple of the tests for "tabs with automatic activation" with NVDA. The collected output was wrong because what was present in the speech viewer did not align with what was voiced by the screen reader
mfairchild: I suspect that NVDA is sending the string to the vocalizer twice
mfairchild: And that the voice interrupts itself
Joe_Humbert: I don't know if this is feedback that we need to send to NVDA
James: You can start to hear the first vocalization before the second interrupts it. It sounds almost as if it is stuttering
Matt_King: Is there a chance that there are two "focus change" events?
James: That's possible
Matt_King: I think we might want to mark this as excess verbosity
Matt_King: Does the dupe show up in Braille? Because you don't get interruptions in Braille
James: so you're suggesting that it is render the Braille twice? One after the other? Or that it would "click" twice in rapid succession?
Matt_King: I don't know
mfairchild: I don't think I would report it if it doesn't impact end users
Matt_King: Yeah. I'm having this question in my head...
IsaDC: It doesn't announce it twice
Matt_King: It introduces a stutter
Joe_Humbert: This is why I captured it. It's weird, and I thought we needed to discuss it
Matt_King: The bot reported it twice because the bot gets it so fast
Joe_Humbert: Do I need to change mine?
Matt_King: I'm thinking about how this works with the automation. If we record it once, and then in the future, the bot spits it out twice, again, we're in a situation where it always conflicts
Matt_King: We don't want to say that the "double output" is good output, but we also don't want to say it's a failure
mfairchild: This is a situation where it would be great to be able to document what went wrong but still test it
Matt_King: In the side effects, we have "SEVERE" and "MODERATE". We could add a new value that we have observed something that doesn't matter
Matt_King: If we put in a severity that's equivalent to "NONE" (and maybe rename from "negative side-effect" to just "side-effect"), then we could report it but avoid failing an assertion
Matt_King: In the mean time, the only question is: do we delete the duplication or not?
IsaDC: I think we should keep it because of the bot
ChrisCuellar: I'm curious about the process. This happens in the VO Bot (I'm discussing this with Corey as we speak).
ChrisCuellar: Let's say the bot reports duplicate announcements (for whatever reason). Joe_Humbert, when you get the bot run reassigned to you, and you're determining the verdict--I would assume you are running the test again and confirming whether you observe the same thing
Joe_Humbert: Correct
ChrisCuellar: So whatever the human tester observes on their machine, that should supersede the bot's output
James: We are observing this. I can hear a stutter
IsaDC: If we use the speech history add-on, it would record this twice, as well
Joe_Humbert: It does. That's why I brought this up
IsaDC: I think this a failure
Matt_King: Okay, but we agree that we would report this as a "NONE" side-effect (instead of "MODERATE") if we could
James: It shouldn't say it twice.
Matt_King: We can file an issue to discuss this further and then in the comments for this moderate negative side-effect, we could include a link to that issue
Joe_Humbert: So, besides changing this from "other" to "verbose output", do I need to change the details?
Matt_King: We could say, "Speech is duplicated but only heard once due to automatic interruption"
Joe_Humbert: Sure, I can do that
James: I'm really curious to how this is expressed on a Braille display
Switch and disclosure
Matt_King: The main thing we need right now is testers
IsaDC: And we have Bot results ready
Joe_Humbert: I can help
Joe_Humbert: You can assign me any screen reader you want
Joe_Humbert: Just know that my new test machine is Windows 11, not Windows 10
IsaDC: I'll assign you to all three, then. Thank you!
mmoss: I volunteer for VoiceOver
IsaDC: Thank you!
Matt_King: Thank you, everybody. We're making some really awesome progress here