19:01:21 <RRSAgent> RRSAgent has joined #aria-at
19:01:25 <RRSAgent> logging to https://www.w3.org/2023/06/29-aria-at-irc
19:01:25 <Zakim> RRSAgent, make logs Public
19:01:26 <Zakim> please title this meeting ("meeting: ..."), Matt_King
19:01:44 <Matt_King> MEETING: ARIA and Assistive Technologies Community Group
19:01:52 <Matt_King> CHAIR: Matt King
19:01:56 <Matt_King> present+
19:02:05 <Matt_King> rrsagent, make minutes
19:02:06 <RRSAgent> I have made the request to generate https://www.w3.org/2023/06/29-aria-at-minutes.html Matt_King
19:03:40 <howard-e> howard-e has joined #aria-at
19:05:22 <boazsender> boazsender has joined #aria-at
19:05:43 <JoeHumbert> JoeHumbert has joined #aria-at
19:06:12 <jugglinmike> present+ jugglinmike
19:06:17 <boazsender> present+
19:06:24 <JoeHumbert> present+
19:09:18 <jugglinmike> scribe+ jugglinmike
19:09:44 <jugglinmike> Matt_King: No CG meeting on July 5 or July 13
19:10:07 <jugglinmike> Matt_King: I haven't canceled the meeting on the 13th, but I will do that shortly because I'll be out of office that day
19:10:25 <jugglinmike> Matt_King: Next CG Meeting: July 19 (a Wednesday)
19:10:33 <jugglinmike> Matt_King: No automation meeting on July 3
19:10:39 <jugglinmike> Matt_King: Next automation meeting: July 17
19:11:01 <jugglinmike> Topic: Testing for Disclosure Menu
19:11:57 <jugglinmike> Matt_King: Looks like J is about a third of the way through the testing for NVDA
19:12:04 <jugglinmike> present+ IsaDC
19:12:11 <jugglinmike> IsaDC: I'm going to start on that today!
19:12:19 <jugglinmike> present+ Hadi
19:13:17 <jugglinmike> Hadi: I volunteer to for JAWS and Chrome
19:13:22 <jugglinmike> Matt_King: Thanks!
19:13:30 <jugglinmike> Topic: Open navigation menu issues
19:13:48 <jugglinmike> Matt_King: Three issues here; they may be resolved, but if so, their resolution isn't documented
19:13:56 <jugglinmike> Subtopic: Feedback: "Open a menu in interaction mode" (Navigation Menu Button, Test 11) · Issue #960 · w3c/aria-at
19:14:05 <jugglinmike> github: https://github.com/w3c/aria-at/issues/960
19:14:27 <jugglinmike> Matt_King: This was related to a conflict during testing, but the conflict appears to have been resolved because we published the report
19:15:25 <jugglinmike> Matt_King: There are two different outputs recorded here...
19:15:55 <jugglinmike> IsaDC: I think we solved this two weeks ago
19:16:45 <jugglinmike> IsaDC: I re-tested, and I got the same results as Alyssa, so I changed them
19:16:57 <jugglinmike> IsaDC: I'll document that in a comment and then close the issue
19:17:44 <jugglinmike> Subtopic: Inconsistent announcement of W3C Quick Links Menu "Open a menu in interaction mode" (Navigation Menu Button, Test 11) · Issue #959 · w3c/aria-at
19:17:51 <jugglinmike> github: https://github.com/w3c/aria-at/issues/959
19:18:05 <jugglinmike> IsaDC: I can document how this was resolved and close it
19:18:20 <jugglinmike> Subtopic: Feedback: "Read information about a menu item in interaction mode" (Navigation Menu Button, Test 15) · Issue #843 · w3c/aria-at
19:18:28 <jugglinmike> github: https://github.com/w3c/aria-at/issues/843
19:18:49 <jugglinmike> Matt_King: This one is fairly old. It was opened by Roxana from Vispero
19:19:17 <jugglinmike> IsaDC: I believe we implemented the change being requested--we made the assertion optional
19:19:48 <jugglinmike> Matt_King: Did we, though? In the current reports, there is still a required assertion which is failing that's related to this
19:20:07 <jugglinmike> IsaDC: I'm not sure
19:20:49 <jugglinmike> Matt_King: I think we have to keep this open, until we have further discussion here. I don't think I want to have that discussion without James_Scholes present
19:21:00 <jugglinmike> [James_Scholes sent his regrets for today's meeting]
19:22:34 <jugglinmike> Topic: Open slider issues
19:22:46 <jugglinmike> Matt_King: There are two issues that are still open
19:22:59 <jugglinmike> Matt_King: They're listed in the agenda
19:23:26 <jugglinmike> IsaDC: Both of them are resolved. I'll post with a note documenting the resolution and then close them
19:23:32 <jugglinmike> Topic: Wording of assertion verdicts
19:23:57 <jugglinmike> Matt_King: Rethinking the wording for assertion verdicts · Issue #945 · w3c/aria-at
19:24:06 <jugglinmike> github: https://github.com/w3c/aria-at/issues/945
19:24:37 <jugglinmike> Matt_King: In this issue, jugglinmike suggests alternative words for the three assertion verdicts
19:25:18 <jugglinmike> Matt_King: When we discussed this, I talked myself into thinking that there isn't enough of a need to differentiate between kinds of assertion failures
19:25:48 <jugglinmike> Matt_King: And that it would simplify things meaningfully if we just classified assertions as either "passing" or "failing"
19:26:34 <jugglinmike> IsaDC: For the output, we will still know if it's "incorrect" or "no output" by judging from the AT Response
19:30:03 <jugglinmike> Matt_King: Yes, Testers will still report that there was no AT response. Although separately, we need to give Testers a normalize way to describe this (e.g. via a checkbox) rather than inventing their own representation of "no output" each time
19:30:41 <jugglinmike> jugglinmike: But remember that "No output" as a verdict for an specific assertion is sometimes used even when the AT does respond
19:31:15 <jugglinmike> jugglinmike: The "no output" assertion verdict is designed to be used when the output does not include information about the information being asserted
19:31:58 <jugglinmike> Matt_King: But both "no output" and "incorrect output" are both failures, and I don't think tabulating them separately brings enough value to justify the complexity they require
19:35:48 <jugglinmike> Hadi: Are you suggesting that we remove the ability to describe unexpected responses?
19:36:00 <jugglinmike> Matt_King: No, we're keeping that
19:36:25 <jugglinmike> Hadi: I'm concerned that people reviewing the results will not be able to understand why the Tester assigned "fail"
19:37:26 <jugglinmike> Matt_King: The assertions are granular: there is a separate assertion for each ARIA property and attribute. This means implementers will be able to see precisely what aspect of a test failed
19:37:57 <jugglinmike> Hadi: Okay, that sounds good. And as long as we're keeping the ability to describe unexpected output with free text, I am supportive of this change
19:38:08 <jugglinmike> present+ Michael_Fairchild
19:38:32 <jugglinmike> Michael_Fairchild: I support this simplification, as well
19:38:38 <jugglinmike> present+ JoeHumbert
19:39:03 <jugglinmike> JoeHumbert: I do, too. Anything to reduce the number of options that the Testers must choose between
19:40:33 <jugglinmike> Matt_King: Sounds like we're in agreement. We'll have a separate discussion about when we make that change
19:41:26 <jugglinmike> Topic: Establish explicit test plan life cycle in working mode
19:41:42 <Matt_King> Issue 950: https://github.com/w3c/aria-at/issues/950
19:41:46 <michael_fairchild> michael_fairchild has joined #aria-at
19:42:04 <michael_fairchild> scribe+
19:44:39 <michael_fairchild> jugglinmike: automation can be used to reduce the number of tests that need to be run manually when in the candidate phase. For example, if it can help answer the question "for which tests, has this new version of the AT changed?"
19:46:01 <michael_fairchild> Matt_King: If we have a bot that ran the test and collected a response, and that response is different from the previous version, it seems like we might still want to keep the response recorded by the bot but ask a human to analyze it. We want the human to run the test, but we should keep the response.
19:50:49 <michael_fairchild> jugglinmike: if we have previous results to compare to, is it possible for a test to change while still in the candidate phase? That might invalidate the results.
19:51:49 <michael_fairchild> Matt_King: Yes, the version of the test plan can change, so its important to know that the assertions and commands are the same between comparisons.
19:57:33 <michael_fairchild> jugglinmike: summary: the automated AT response collection system needs to be able to work with test plans in any phase, in order to assist in creating a new test plan run for a change in AT version, Browser version, or both.
20:10:35 <jugglinmike> Zakim, end the meeting
20:10:35 <Zakim> As of this point the attendees have been Matt_King, jugglinmike, boazsender, JoeHumbert, IsaDC, Hadi, Michael_Fairchild
20:10:37 <Zakim> RRSAgent, please draft minutes
20:10:38 <RRSAgent> I have made the request to generate https://www.w3.org/2023/06/29-aria-at-minutes.html Zakim
20:10:44 <Zakim> I am happy to have been of service, jugglinmike; please remember to excuse RRSAgent.  Goodbye
20:10:45 <Zakim> Zakim has left #aria-at