17:01:05 <RRSAgent> RRSAgent has joined #aria-at
17:01:09 <RRSAgent> logging to https://www.w3.org/2023/08/30-aria-at-irc
17:01:20 <jugglinmike> rrsagent, make logs public
17:01:31 <jugglinmike> Zakim, start the meeting
17:01:31 <Zakim> RRSAgent, make logs Public
17:01:33 <Zakim> please title this meeting ("meeting: ..."), jugglinmike
17:01:54 <jugglinmike> meeting: ARIA and Assistive Technologies Community Group Weekly Teleconference
17:03:42 <murray_moss> murray_moss has joined #aria-at
17:03:54 <murray_moss> present+
17:04:13 <jugglinmike> present+ jugglinmike
17:04:17 <jugglinmike> scribe+ jugglinmike
17:05:03 <Matt_King> Matt_King has joined #aria-at
17:05:09 <JoeHumbert> present+
17:05:14 <howard-e> howard-e has joined #aria-at
17:05:19 <howard-e> present+
17:05:24 <Sam_Shaw> Sam_Shaw has joined #aria-at
17:07:33 <Matt_King> present+
17:08:27 <Sam_Shaw> present+
17:09:22 <Sam_Shaw> present+ James_Scholes
17:11:11 <jugglinmike> Matt_King: Next CG Meeting: September 7
17:11:20 <jugglinmike> Matt_King: No meeting Sep 13 due to TPAC
17:11:30 <jugglinmike> Topic: Testing pause
17:11:49 <jugglinmike> Matt_King: Thank you, everyone who's been doing testing!
17:12:02 <jugglinmike> Matt_King: We now have 10 test plans published, and the Test Queue is currently empty
17:12:16 <jugglinmike> Matt_King: That's the state that we wanted for the next deployment of ARIA-AT App
17:12:43 <jugglinmike> Matt_King: We expect to deploy soon, possibly today
17:12:56 <jugglinmike> Matt_King: There are a couple conditions that need to be met before we resume testing
17:13:37 <jugglinmike> Matt_King: Most importantly, we need to be able to make changes to the test plan to get consensus from Vispero and Apple to move forward to the "Recommended" phase
17:14:23 <jugglinmike> Matt_King: e.g. the Toggle Button (for Apple), and the verbosity of commands for e.g. Disclosure (for Vispero)
17:15:20 <jugglinmike> Matt_King: Lola_Odelola, do we have dates for all the work for Test Format v2?
17:15:26 <jugglinmike> present+ Lola_Odelola
17:15:36 <jugglinmike> Lola_Odelola: Not yet--Carmen and howard-e are still working on that
17:16:01 <jugglinmike> Matt_King: Okay. We at least know that it won't be before September 25
17:16:17 <jugglinmike> Matt_King: So we're going to have a pause in testing until at least September 25, and it could go even longer than that
17:16:46 <jugglinmike> Matt_King: We'll be providing regular updates on this through September
17:17:04 <jugglinmike> Topic: Update on coming app changes
17:17:41 <jugglinmike> Matt_King: Changes are imminent
17:17:51 <jugglinmike> Matt_King: There are three big projects in the work; they're listed in the agenda
17:18:32 <jugglinmike> Matt_King: As a shorthand: the MVP App project, the refactor tests project, the automation project
17:19:22 <jugglinmike> Matt_King: MVP App is all about version control on test plans, and having clarity on version control (which version are you working on, where are the other versions in the process, etc.)
17:19:30 <jugglinmike> Matt_King: We are close to shipping all the basic requirements on this
17:19:58 <jugglinmike> Matt_King: Here is a link to the project board containing all the links associated with the MVP App project: https://github.com/orgs/w3c/projects/21/views/1
17:20:35 <jugglinmike> Matt_King: Overall, what's on the staging server is pretty amazing
17:21:31 <jugglinmike> Matt_King: The second project, the refactor tests project, is changing how a test is written so that it can be simpler to read, simpler to understand, more flexible (in terms of a single test for reading and interaction modes)...
17:22:08 <jugglinmike> Matt_King: ...having three levels of priorities, and being able to exclude some assertions for some commonds (e.g. ignoring assertions for mode change with VoiceOver)
17:22:30 <jugglinmike> Matt_King: That's more than 90% spec'd out. I still have some "to do" items where details are still being worked out
17:23:13 <jugglinmike> Matt_King: The third project is bringing automation. Our target for this is the end of September
17:24:04 <jugglinmike> Matt_King: This is about re-running a test plan that has been run in the past. We can use a machine for this, using the result to see if the "historic" assertion verdict can be re-used
17:25:17 <jugglinmike> Matt_King: The new system will automatically gather all the AT response for tests that have previously been run by a human. If the AT responses are the same, then it will assign the same verdicts that were assigned by the human
17:25:47 <jugglinmike> Matt_King: Humans then only have to go and look at the parts where the automation system observed different AT responses
17:26:12 <jugglinmike> Matt_King: This is initially only for NVDA
17:26:35 <jugglinmike> Lola_Odelola: I think it makes sense for us to focus on JAWS after that
17:28:11 <jugglinmike> Lola_Odelola: VoiceOver and Talkback are on the radar, but we may not move on to those until later in the year or next year
17:33:50 <jugglinmike> James_Scholes: I would lobby for taking on VoiceOver first, before JAWS. We expect JAWS to implement AT Driver sooner than VoiceOver.
17:35:21 <jugglinmike> James_Scholes: Since we're currently expecting to use the "Generic driver" for both, and since that will be obviated by the "native" implementation from JAWS, then we will likely benefit more from moving forward with VoiceOver next
17:35:35 <jugglinmike> Topic: New assertion priorities
17:35:52 <jugglinmike> github: https://github.com/w3c/aria-at-app/issues/737
17:36:21 <jugglinmike> Matt_King: Previously, we had "required" and "optional", but that didn't line up well with the W3C terminology of "MUST", "SHOULD", and "MAY"
17:37:05 <jugglinmike> Matt_King: The latest thinking is that we'll start by publishing a W3C note and then maybe one day using that as a basis for a W3C specification.
17:37:12 <jugglinmike> Matt_King: It's truly test-driven development!
17:39:30 <jugglinmike> Lola_Odelola: We wanted to understand the usage of MAY. The use of MAY might confuse some users
17:39:50 <jugglinmike> Lola_Odelola: The presence of MAY might cause folks to assume that other behaviors are not allowed
17:40:33 <jugglinmike> Matt_King: We might want to be more comprehensive in our assertions
17:40:55 <jugglinmike> Matt_King: if there are other responses to a specific attribute that are valid, then we might want to include them in the test
17:41:08 <jugglinmike> Matt_King: So I think actually, that hypotethical interpretation is valid
17:41:26 <jugglinmike> Matt_King: This places a greater onus on test writers to make sure that the universe of allowable things in included
17:41:39 <jugglinmike> Matt_King: But we could also write this specification in a way that doesn't exclude other possibilities
17:42:04 <jugglinmike> present+ Hadi
17:42:50 <jugglinmike> Hadi: I appreciate the use of the word "MUST", but I'm concerned about being too strict about specific wording
17:43:41 <jugglinmike> Matt_King: Agreed. That's why we use words like "convey". So even a "MUST" assertion allows for the kind of variations that you're referencing
17:44:08 <jugglinmike> Matt_King: MUST is the stuff like name, role, value, and state
17:44:52 <jugglinmike> Matt_King: Things like communicating change in state (rather than actually changing the state itself) would be "SHOULD"
17:45:22 <jugglinmike> Matt_King: It could be frustrating or maybe annoying if the AT doesn't do the things it SHOULD do, but the application would still be usable
17:46:12 <jugglinmike> Matt_King: And "MAY" assertions are essentially saying, "we don't care whether the AT does this"
17:48:06 <jugglinmike> Matt_King: using "MAY" increases our responsibility for capturing all the potential behaviors, and it also increases our responsibility for documentation (i.e. explaining what it means)
17:49:35 <jugglinmike> Matt_King: If there are any MAYs listed at all, there will be an implicit expectation that the assertions are exhaustive
17:50:08 <jugglinmike> James_Scholes: I think that puts an unreasonable burden on the test author--to account for anything a screen reader may do
17:50:48 <jugglinmike> James_Scholes: If we are tying assertions to any attribute, and some of them have the MAY priority, those are fringe usability benefits
17:51:19 <jugglinmike> James_Scholes: I don't think it should be the responsibility of the Test Author to enumerate every possible fringe usability benefits
17:51:52 <jugglinmike> Hadi: What do you folks think about mapping "MUST" "SHOULD" "MAY" to the concept of verbosity?
17:52:45 <jugglinmike> James_Scholes: I don't think that works for two reasons. First, not all screen readers have the same concept of verbosity (NVDA doesn't have such a concept at all). Secondly, some ATs have made decisions about verbosity which conflict with basic requirements
17:52:55 <jugglinmike> present+ Alyssa
17:53:09 <jugglinmike> Alysssa: We test with the most verbose AT settings, right?
17:53:15 <jugglinmike> James_Scholes: That's right
17:53:26 <jongund> jongund has joined #aria-at
17:53:32 <jugglinmike> s/Alysssa/Alyssa/
17:55:48 <jugglinmike> Matt_King: You're point is well-taken, James_Scholes. In that case, we would need language someplace that explains why some behaviors are recognized with MAY assertions and other behaviors are not mentioned, but they are nonetheless allowed
17:57:21 <jugglinmike> James_Scholes: I don't want to be in a position where we are claiming that we have thought of every possible innovation which an AT might make
18:00:56 <jugglinmike> Matt_King: We're going to need clear language around what "MAY" means. They are consensus suggestions as opposed to exclusive lists. We are going to have to write that before we publish anything to APG that includes a "MAY"
18:02:41 <jugglinmike> Zakim, end the meeting
18:02:41 <Zakim> As of this point the attendees have been murray_moss, jugglinmike, JoeHumbert, howard-e, Matt_King, Sam_Shaw, James_Scholes, Lola_Odelola, Hadi, Alyssa
18:02:44 <Zakim> RRSAgent, please draft minutes
18:02:46 <RRSAgent> I have made the request to generate https://www.w3.org/2023/08/30-aria-at-minutes.html Zakim
18:02:53 <Zakim> I am happy to have been of service, jugglinmike; please remember to excuse RRSAgent.  Goodbye
18:02:53 <Zakim> Zakim has left #aria-at
20:01:40 <jongund> jongund has joined #aria-at
21:27:58 <jongund> jongund has joined #aria-at
21:42:04 <jongund> jongund has joined #aria-at
21:43:08 <jongund_> jongund_ has joined #aria-at
21:54:46 <jongund> jongund has joined #aria-at
21:57:03 <jongund__> jongund__ has joined #aria-at
22:19:56 <jongund> jongund has joined #aria-at
22:22:18 <jongund_> jongund_ has joined #aria-at
22:32:20 <jongund> jongund has joined #aria-at
22:34:41 <jongund__> jongund__ has joined #aria-at
22:41:42 <jongund> jongund has joined #aria-at
22:51:37 <jongund_> jongund_ has joined #aria-at
22:54:48 <jongund__> jongund__ has joined #aria-at
23:00:54 <jongund> jongund has joined #aria-at
23:52:56 <jongund> jongund has joined #aria-at