Meeting minutes
Matt_King: Next CG Meeting: September 7
Matt_King: No meeting Sep 13 due to TPAC
Testing pause
Matt_King: Thank you, everyone who's been doing testing!
Matt_King: We now have 10 test plans published, and the Test Queue is currently empty
Matt_King: That's the state that we wanted for the next deployment of ARIA-AT App
Matt_King: We expect to deploy soon, possibly today
Matt_King: There are a couple conditions that need to be met before we resume testing
Matt_King: Most importantly, we need to be able to make changes to the test plan to get consensus from Vispero and Apple to move forward to the "Recommended" phase
Matt_King: e.g. the Toggle Button (for Apple), and the verbosity of commands for e.g. Disclosure (for Vispero)
Matt_King: Lola_Odelola, do we have dates for all the work for Test Format v2?
Lola_Odelola: Not yet--Carmen and howard-e are still working on that
Matt_King: Okay. We at least know that it won't be before September 25
Matt_King: So we're going to have a pause in testing until at least September 25, and it could go even longer than that
Matt_King: We'll be providing regular updates on this through September
Update on coming app changes
Matt_King: Changes are imminent
Matt_King: There are three big projects in the work; they're listed in the agenda
Matt_King: As a shorthand: the MVP App project, the refactor tests project, the automation project
Matt_King: MVP App is all about version control on test plans, and having clarity on version control (which version are you working on, where are the other versions in the process, etc.)
Matt_King: We are close to shipping all the basic requirements on this
Matt_King: Here is a link to the project board containing all the links associated with the MVP App project: https://
Matt_King: Overall, what's on the staging server is pretty amazing
Matt_King: The second project, the refactor tests project, is changing how a test is written so that it can be simpler to read, simpler to understand, more flexible (in terms of a single test for reading and interaction modes)...
Matt_King: ...having three levels of priorities, and being able to exclude some assertions for some commonds (e.g. ignoring assertions for mode change with VoiceOver)
Matt_King: That's more than 90% spec'd out. I still have some "to do" items where details are still being worked out
Matt_King: The third project is bringing automation. Our target for this is the end of September
Matt_King: This is about re-running a test plan that has been run in the past. We can use a machine for this, using the result to see if the "historic" assertion verdict can be re-used
Matt_King: The new system will automatically gather all the AT response for tests that have previously been run by a human. If the AT responses are the same, then it will assign the same verdicts that were assigned by the human
Matt_King: Humans then only have to go and look at the parts where the automation system observed different AT responses
Matt_King: This is initially only for NVDA
Lola_Odelola: I think it makes sense for us to focus on JAWS after that
Lola_Odelola: VoiceOver and Talkback are on the radar, but we may not move on to those until later in the year or next year
James_Scholes: I would lobby for taking on VoiceOver first, before JAWS. We expect JAWS to implement AT Driver sooner than VoiceOver.
James_Scholes: Since we're currently expecting to use the "Generic driver" for both, and since that will be obviated by the "native" implementation from JAWS, then we will likely benefit more from moving forward with VoiceOver next
New assertion priorities
github: w3c/
Matt_King: Previously, we had "required" and "optional", but that didn't line up well with the W3C terminology of "MUST", "SHOULD", and "MAY"
Matt_King: The latest thinking is that we'll start by publishing a W3C note and then maybe one day using that as a basis for a W3C specification.
Matt_King: It's truly test-driven development!
Lola_Odelola: We wanted to understand the usage of MAY. The use of MAY might confuse some users
Lola_Odelola: The presence of MAY might cause folks to assume that other behaviors are not allowed
Matt_King: We might want to be more comprehensive in our assertions
Matt_King: if there are other responses to a specific attribute that are valid, then we might want to include them in the test
Matt_King: So I think actually, that hypotethical interpretation is valid
Matt_King: This places a greater onus on test writers to make sure that the universe of allowable things in included
Matt_King: But we could also write this specification in a way that doesn't exclude other possibilities
Hadi: I appreciate the use of the word "MUST", but I'm concerned about being too strict about specific wording
Matt_King: Agreed. That's why we use words like "convey". So even a "MUST" assertion allows for the kind of variations that you're referencing
Matt_King: MUST is the stuff like name, role, value, and state
Matt_King: Things like communicating change in state (rather than actually changing the state itself) would be "SHOULD"
Matt_King: It could be frustrating or maybe annoying if the AT doesn't do the things it SHOULD do, but the application would still be usable
Matt_King: And "MAY" assertions are essentially saying, "we don't care whether the AT does this"
Matt_King: using "MAY" increases our responsibility for capturing all the potential behaviors, and it also increases our responsibility for documentation (i.e. explaining what it means)
Matt_King: If there are any MAYs listed at all, there will be an implicit expectation that the assertions are exhaustive
James_Scholes: I think that puts an unreasonable burden on the test author--to account for anything a screen reader may do
James_Scholes: If we are tying assertions to any attribute, and some of them have the MAY priority, those are fringe usability benefits
James_Scholes: I don't think it should be the responsibility of the Test Author to enumerate every possible fringe usability benefits
Hadi: What do you folks think about mapping "MUST" "SHOULD" "MAY" to the concept of verbosity?
James_Scholes: I don't think that works for two reasons. First, not all screen readers have the same concept of verbosity (NVDA doesn't have such a concept at all). Secondly, some ATs have made decisions about verbosity which conflict with basic requirements
Alyssa: We test with the most verbose AT settings, right?
James_Scholes: That's right
Matt_King: You're point is well-taken, James_Scholes. In that case, we would need language someplace that explains why some behaviors are recognized with MAY assertions and other behaviors are not mentioned, but they are nonetheless allowed
James_Scholes: I don't want to be in a position where we are claiming that we have thought of every possible innovation which an AT might make
Matt_King: We're going to need clear language around what "MAY" means. They are consensus suggestions as opposed to exclusive lists. We are going to have to write that before we publish anything to APG that includes a "MAY"