Meeting minutes
Matt_King: No CG meeting on July 5 or July 13
Matt_King: I haven't canceled the meeting on the 13th, but I will do that shortly because I'll be out of office that day
Matt_King: Next CG Meeting: July 19 (a Wednesday)
Matt_King: No automation meeting on July 3
Matt_King: Next automation meeting: July 17
Testing for Disclosure Menu
Matt_King: Looks like J is about a third of the way through the testing for NVDA
IsaDC: I'm going to start on that today!
Hadi: I volunteer to for JAWS and Chrome
Matt_King: Thanks!
Open navigation menu issues
Matt_King: Three issues here; they may be resolved, but if so, their resolution isn't documented
Feedback: "Open a menu in interaction mode" (Navigation Menu Button, Test 11) · Issue #960 · w3c/aria-at
github: w3c/
Matt_King: This was related to a conflict during testing, but the conflict appears to have been resolved because we published the report
Matt_King: There are two different outputs recorded here...
IsaDC: I think we solved this two weeks ago
IsaDC: I re-tested, and I got the same results as Alyssa, so I changed them
IsaDC: I'll document that in a comment and then close the issue
Inconsistent announcement of W3C Quick Links Menu "Open a menu in interaction mode" (Navigation Menu Button, Test 11) · Issue #959 · w3c/aria-at
github: w3c/
IsaDC: I can document how this was resolved and close it
Feedback: "Read information about a menu item in interaction mode" (Navigation Menu Button, Test 15) · Issue #843 · w3c/aria-at
github: w3c/
Matt_King: This one is fairly old. It was opened by Roxana from Vispero
IsaDC: I believe we implemented the change being requested--we made the assertion optional
Matt_King: Did we, though? In the current reports, there is still a required assertion which is failing that's related to this
IsaDC: I'm not sure
Matt_King: I think we have to keep this open, until we have further discussion here. I don't think I want to have that discussion without James_Scholes present
[James_Scholes sent his regrets for today's meeting]
Open slider issues
Matt_King: There are two issues that are still open
Matt_King: They're listed in the agenda
IsaDC: Both of them are resolved. I'll post with a note documenting the resolution and then close them
Wording of assertion verdicts
Matt_King: Rethinking the wording for assertion verdicts · Issue #945 · w3c/aria-at
github: w3c/
Matt_King: In this issue, jugglinmike suggests alternative words for the three assertion verdicts
Matt_King: When we discussed this, I talked myself into thinking that there isn't enough of a need to differentiate between kinds of assertion failures
Matt_King: And that it would simplify things meaningfully if we just classified assertions as either "passing" or "failing"
IsaDC: For the output, we will still know if it's "incorrect" or "no output" by judging from the AT Response
Matt_King: Yes, Testers will still report that there was no AT response. Although separately, we need to give Testers a normalize way to describe this (e.g. via a checkbox) rather than inventing their own representation of "no output" each time
jugglinmike: But remember that "No output" as a verdict for an specific assertion is sometimes used even when the AT does respond
jugglinmike: The "no output" assertion verdict is designed to be used when the output does not include information about the information being asserted
Matt_King: But both "no output" and "incorrect output" are both failures, and I don't think tabulating them separately brings enough value to justify the complexity they require
Hadi: Are you suggesting that we remove the ability to describe unexpected responses?
Matt_King: No, we're keeping that
Hadi: I'm concerned that people reviewing the results will not be able to understand why the Tester assigned "fail"
Matt_King: The assertions are granular: there is a separate assertion for each ARIA property and attribute. This means implementers will be able to see precisely what aspect of a test failed
Hadi: Okay, that sounds good. And as long as we're keeping the ability to describe unexpected output with free text, I am supportive of this change
Michael_Fairchild: I support this simplification, as well
JoeHumbert: I do, too. Anything to reduce the number of options that the Testers must choose between
Matt_King: Sounds like we're in agreement. We'll have a separate discussion about when we make that change
Establish explicit test plan life cycle in working mode
<Matt_King> Issue 950: w3c/
jugglinmike: automation can be used to reduce the number of tests that need to be run manually when in the candidate phase. For example, if it can help answer the question "for which tests, has this new version of the AT changed?"
Matt_King: If we have a bot that ran the test and collected a response, and that response is different from the previous version, it seems like we might still want to keep the response recorded by the bot but ask a human to analyze it. We want the human to run the test, but we should keep the response.
jugglinmike: if we have previous results to compare to, is it possible for a test to change while still in the candidate phase? That might invalidate the results.
Matt_King: Yes, the version of the test plan can change, so its important to know that the assertions and commands are the same between comparisons.
jugglinmike: summary: the automated AT response collection system needs to be able to work with test plans in any phase, in order to assist in creating a new test plan run for a change in AT version, Browser version, or both.