Meeting minutes
Update on test plan execution work
Matt_King: The primary goal is to make sure that there aren't any problems, and to make assignments if any volunteers have capacity
https://
James_Scholes: Color viewer slider with JAWS and Firefox - all testing is complete, but there are 7 conflicts
James_Scholes: Isa has been managing this work, but since she's not here today, I can't speak to the status of the conflicts
Matt_King: The site is loading very fast, by the way. Thanks to Alex and Paul from Bocoup!
James_Scholes: VoiceOver for macOS and Chrome - all testing is complete, but there are 4 conflicts
James_Scholes: JAWS and Firefox - Navigation Menu Button - has been assigned to Alyssa, but not started yet
James_Scholes: With NVDA, testing is complete, but there are 7 conflicts. That seems quite high, so we'll look into that
James_Scholes: VoiceOver for macOS and Chrome, awaiting testing from Dylan
James_Scholes: Modal Dialog Example - JAWS and Firefox - testing is almost complete, but Dylan has only reported results for 17 of 18 tests, so I suspect that's a problem with the UI
James_Scholes: NVDA and Firefox - isn't present, so I presume we've published this already
James_Scholes: VoiceOver for macOS and Chrome - awaiting testing from Dylan
Alyssa: I should have the Navigation Menu button with JAWS and Firefox tests done by next week
Matt_King: I think that's as far as we can go with this today
Matt_King: By the time we're ready to start the next Test Plan, we'll have even more available
Justin_Yarbrough: yeah, that's me! I'll be working with Sam_Shaw to get onboarded
mmoss: Me too
Improving new member orientation
Matt_King: Last week, we talked about wiki updates that I made, and we pointed out some obvious gaps and some not-so-obvious (to me) gaps, as well
Matt_King: I don't know if we have an issue to track that work.
Matt_King: I'd like to discuss next steps and figure out who is available to do what
Matt_King: gh-420 seems relevant w3c/
James_Scholes: Assign it to Isa; their GitHub handle IsaDC
Matt_King: I have a "new member welcome kit" on the wiki. It has some links that are missing, but I'm wondering if this group here could advise on the most important next step
<Sam_Shaw> https://
[Matt_King reviews the contents of the page]
Matt_King: I'm a little worried about the currency of the page about running test plans
James_Scholes: Yes, I agree that we need to review that page for accuracy
Matt_King: I'll make a comment about that in gh-420
Matt_King: new members: are there other things that have been hard to learn? Or other roadblocks that you ran into? Anything that's missing from this list?
Alyssa: One thing that confused me when I first started running tests is the terms for "focus mode" and "interaction mode", or "reading mode" and "browse mode". Because the names are different in NVDA and JAWS
James_Scholes: I think that's documented, but the documentation isn't referenced in this welcome kit
mmoss: Also, making a stronger distinction about the use of the term "mode" - whether it's the "Working Mode" or the "screen reader mode"
Matt_King: Apple insists that VoiceOver is "modeless". They have quicknav, but they don't recognize it as a mode.
Matt_King: Nonetheless, I wonder if the test instructions should tell testers to, e.g. "turn on quicknav", where appropriate
Matt_King: This is a bit of a can of worms. I've been thinking. Right now, for we say things like "navigate in reading mode" and "navigate in interaction mode".
Matt_King: What if, instead, we used different verbs?
Matt_King: e.g. what if we say, "focus on" or "move focus to" (for "navigate in interaction mode"), and "browse to" (for "navigate in browse mode")?
James_Scholes: The majority of the reading mode tests that have, say, navigate forward to a button, include the Tab key
Matt_King: Maybe we say, "navigate" and "browse"
Matt_King: Basically, using "browsing" for focus mode and something else for forms mode
Matt_King: If we do that, then we can use the same instructions for JAWS, NVDA, and VoiceOver
Matt_King: Because Testers using VoiceOver would interpret that instruction as a signal about whether or not to have "quick nav" enabled
James_Scholes: I'm not opposed, but I think we need to consider this in the context of specific tests because there always seems to be tests which don't fit the model
James_Scholes: As long as we can find words which don't inadvertently imply something we don't mean
Matt_King: I really like "browse to" for reading mode because it works so well in English. (NVDA happens to use the same term, but I don't mean to give that AT preferential treatment.
Matt_King: It's just an appropriate word which works as both a verb and a adjective
Alyssa: People will come to the project with their own perspective, so I think that any words you choose will require some education
Matt_King: It sounds like there's still apetite for refining our approach here, and that's a good thing for me to know
James_Scholes: I think the change to assertion verdicts that we discussed during the automation meeting on 2023-06-05 is potentially more impactful. I'd like to prioritize discussion and design on that, if possible
Plans for updating app
Matt_King: There are a lot of changes to the app being planned in the background
Matt_King: They will not affect how tests are run, generally speaking
Matt_King: right now, we have an early version of a page for managing the test plans. It's incomplete at the moment
Matt_King: When we're running tests, part of the process is to get feedback from the testers (is a command missing or incorrect? is a whole test missing that you think should be there? etc.)
Matt_King: In draft review, we could end up making changes to the test plan
Matt_King: That's a cue to James_Scholes to make a modification
Matt_King: Then, we'll have two versions of the test plan: the one that you ran, and the one that includes the changes that you requested
Matt_King: We're annotating Test Plans with dates, so it's possible to have multiple versions of the same plan going through the process at the same time
Matt_King: What we don't have in the app right now is the ability to manage these different versions
Matt_King: There's work going on right now; the whole team at Bocoup and PAC is working to make changes to the app in order to bring clarity to the work of managing multiple versions of a test plan
Matt_King: Meanwhile, while we've been working on automation, we've started to reconsider the verdicts of assertions
Matt_King: following a proposal for wording change from jugglinmike, we discussed whether we can/should simplify the states of an assertion
Matt_King: Right now, an assertion can be "supported", "not supported", or "incorrectly supported". We're thinking about reducing the results down to "supported" or "not supported"
Matt_King: One proposal is that the possible assertion verdicts are: "acceptable", "contradictory" and "omitted".
Matt_King: This would help with the kinds of conflicts where Testers have reported both "incorrect output" and "no output"
James_Scholes: When we were testing the "Switch" role, NVDA announced a checkbox. It's not so clear if that's "no output" or "incorrect output"
Matt_King: Or when VoiceOver said simply "toggle button" instead of "toggle button not pressed". Is that "no output" or "incorrect output"?
James_Scholes: neither the three-value or two-value design really provide enough information to understand the cause of failures. In either case, you need to read the AT response to understand what went wrong
mmoss: You've convinced me!
Alyssa: Having just two options like "pass"/"fail" would make using the radio buttons easier as a Tester.
Alyssa: There have been times where I've selected the wrong button and accidentally created conflicts
James_Scholes: let's plan to revisit this next week (we sometimes get more participants on Thursdays than we do on Wednesdays)
James_Scholes: Though there are already some strong arguments in favor of making this simplification