17:02:54 <RRSAgent> RRSAgent has joined #aria-at
17:02:58 <RRSAgent> logging to https://www.w3.org/2024/03/27-aria-at-irc
17:02:59 <jugglinmike> rrsagent, make log public
17:03:07 <jugglinmike> Zakim, start the meeting
17:03:07 <Zakim> RRSAgent, make logs Public
17:03:09 <Zakim> please title this meeting ("meeting: ..."), jugglinmike
17:03:17 <jugglinmike> present+ jugglinmike
17:03:33 <howard-e> howard-e has joined #aria-at
17:03:42 <jugglinmike> present+ howard-e
17:03:49 <jugglinmike> present+ IsaDC
17:04:03 <jugglinmike> present+ LolaOdelola
17:04:15 <Matt_King> Matt_King has joined #aria-at
17:04:24 <webirc54> webirc54 has joined #aria-at
17:04:26 <jugglinmike> present+ Matt_King
17:04:30 <webirc54> present+
17:04:33 <Joe_Humbert> Joe_Humbert has joined #aria-at
17:05:02 <jugglinmike> scribe+ jugglinmike
17:05:03 <murray_moss> present+ murray_moss
17:05:14 <Joe_Humbert> present+
17:05:45 <jugglinmike> meeting: ARIA and Assistive Technologies Community Group Weekly Teleconference
17:05:50 <jugglinmike> topic: Review agenda and next meeting dates
17:06:03 <jugglinmike> Matt_King: Next meeting: Thursday April 4
17:06:19 <jugglinmike> Matt_King: Any requests for changes to the agenda?
17:06:32 <jugglinmike> Matt_King: Hearing none, we'll move forward with the agenda as planned
17:07:08 <jongund> jongund has joined #aria-at
17:07:24 <jugglinmike> Matt_King: The first automation-specific meeting of the year will be on April 8. I will be setting up a W3C calendar event after this call
17:07:35 <jugglinmike> Topic: Upcoming app fix patch
17:07:51 <jugglinmike> Matt_King: There are two bugs that I've asked Carmen and howard-e to prioritize
17:08:13 <jugglinmike> howard-e: Fixes for those two are still on-track to be available on the "staging" server by tomorrow
17:08:30 <jugglinmike> Matt_King: The two bugs are referenced in the agenda, if anyone is interested
17:09:04 <jugglinmike> Topic: Resolve conflicts in toggle button results
17:09:26 <jugglinmike> Matt_King: Once again, I'm going to say "thank you" to jugglinmike and the other folks at Bocoup who worked on automation
17:10:30 <jugglinmike> Matt_King: I was able to use the bot to collect responses from NVDA for these tests, and it goes so fast
17:11:03 <jugglinmike> Matt_King: For NVDA, I found two tests where I got different responses from what IsaDC reported
17:11:30 <jugglinmike> IsaDC: At first, I thought this was a mistake in my results, but I consistently get that for multiple versions of NVDA
17:11:59 <jugglinmike> IsaDC: For me, NVDA says the word "blank" every time--in focus mode. I ran it three times to be sure
17:12:41 <jugglinmike> Matt_King: I didn't make sure I was running exactly the same version of NVDA as the bot
17:12:51 <jugglinmike> Matt_King: ...but I got the same responses at the bot
17:13:18 <jugglinmike> Matt_King: The bot used 2023.3
17:13:39 <jugglinmike> s/bot used/NVDA bot used NVDA version/
17:13:58 <jugglinmike> IsaDC: I used NVDA version 2023.3.4
17:14:34 <jugglinmike> Matt_King: And I'm using NVDA version 2023.3 (followed by some kind of build number that we don't keep track of)
17:15:11 <jugglinmike> Matt_King: I'm curious: is it possible to update the version of NVDA?
17:17:16 <jugglinmike> jugglinmike: It always uses 2023.3 today. As of the deploy scheduled for April 1, it will use the latest version of NVDA which we have packaged for this purpose (that is, that we have built with PAC's plugin)
17:17:46 <jugglinmike> jugglinmike: We do not have AT version selection on the roadmap
17:18:30 <jugglinmike> Matt_King: I think AT version selection will be the next-highest priority feature following support for VoiceOver
17:18:58 <jugglinmike> Matt_King: In the mean time, it sounds like we may be observing a regression in NVDA
17:19:45 <jugglinmike> Topic: Radio test plan
17:20:24 <jugglinmike> Matt_King: For Radio, we already have IsaDC assigned, but we need another person to run the Radio Button Test Plan on Mac
17:20:33 <jugglinmike> murray_moss: I'm happy to volunteer
17:20:39 <jugglinmike> Matt_King: Awesome!
17:23:02 <jugglinmike> Matt_King: the "assign testers" menu has a lot of options, now--a lot of potential testers. It would be nice to have a type-ahead
17:23:14 <jugglinmike> present+ James_Scholes
17:23:33 <jugglinmike> James_Scholes: by the way, focus doesn't enter the menu when you open it
17:23:57 <jugglinmike> Matt_King: For NVDA, we have Alyssa assigned, and we also have the NVDA Bot assigned
17:24:41 <jugglinmike> Matt_King: For Test Admins, there is a button which is labeled "Manage NVDA Bot Run". That opens a dialog which allows you to re-assign the run to yourself
17:26:34 <jugglinmike> Matt_King: I just got a "502 Bad Gateway" error...
17:26:41 <jugglinmike> Matt_King: But I reloaded, and now, it's fine.
17:27:49 <jugglinmike> s/there is a button which/there is a button in the "status" column which/
17:30:07 <jugglinmike> IsaDC: I've re-assigned the bot's run to myself
17:30:26 <jugglinmike> Matt_King: I've scheduled another NVDA Bot run; I'll assign that to Alyssa when it's complete
17:30:36 <jugglinmike> present+ Hadi
17:30:49 <jugglinmike> Hadi: I can run the test with JAWS
17:31:11 <jugglinmike> Matt_King: Great, thanks! I will un-assign Alyssa and assign you
17:31:55 <jugglinmike> Matt_King: Two weeks from today would be April 10. Is that a reasonable target, murray_moss and Hadi?
17:32:02 <jugglinmike> murray_moss: For sure
17:32:08 <jugglinmike> Hadi: Yes
17:32:13 <jugglinmike> Matt_King: And you, IsaDC?
17:32:17 <jugglinmike> IsaDC: Yes, sure
17:32:30 <jugglinmike> Matt_King: Okay! Then we have a plan for Radio
17:32:54 <jugglinmike> Matt_King: Right now, we have limited Bot usage to admins. I'm wondering if we want to make it possible for anybody to run with the bot, as we develop more confidence
17:33:13 <jugglinmike> Matt_King: That doesn't seem like it would be a big change to the user interface...
17:33:28 <jugglinmike> Hadi: Can you explain the new feature a bit more?
17:34:11 <jugglinmike> Matt_King: It runs the Test Plan and it fills in the AT Response text field for every command
17:34:47 <jugglinmike> Matt_King: You still need to run the test to double-check, but it alleviates the need of manually writing the responses into the app
17:35:04 <jugglinmike> Matt_King: It's only available for NVDA at the moment
17:35:15 <jugglinmike> James_Scholes: We can give you a tutorial if you like, Hadi
17:37:34 <jugglinmike> LolaOdelola: Bocoup wrote a blog post on this topic, for those interested https://bocoup.com/blog/aria-automation-launch
17:38:05 <jugglinmike> Matt_King: Is there any reason not to open this feature up to all testers, jugglinmike?
17:38:35 <jugglinmike> jugglinmike: I have some concerns about resource limitations in the GitHub Actions service, but to be honest, we may be risking overrunning those today
17:38:49 <jugglinmike> Matt_King: Let's make a note to investigate that further, then
17:39:04 <jugglinmike> Topic: Proposal to structure test queue primarily by test plan instead of AT/Browser combination
17:39:29 <jugglinmike> Matt_King: Right now, you can use any version of any screen reader you want when you're running the test in "Draft review"
17:39:49 <jugglinmike> Matt_King: As we saw earlier in this meeting, that sometimes causes issues, but that's actually pretty rare (and it's easy to resolve, besides)
17:40:18 <jugglinmike> Matt_King: but when the test plan reaches the "Recommended" phase, that means we'll run it for every new release of a screen reader from that point forward so that we can track interoperability over time
17:40:38 <jugglinmike> Matt_King: that means that when we're generating those reports, we need to be able to say that each is for a specific version of e.g. JAWS
17:40:54 <jugglinmike> Matt_King: So we'll need to be able to run with a specific version of a screen reader
17:41:34 <jugglinmike> Matt_King: but right now, there's nothing to tell Testers when/if they need to use a specific version of a screen reader (or a specific range, e.g. "NVDA 2023.3 or later")
17:42:01 <jugglinmike> Matt_King: That's why we'll be updating the design of the test queue to support this soon
17:42:22 <jugglinmike> Matt_King: While working on the design for this, I realized that the design of the Test Queue is pretty long in the tooth and just doesn't suit the way that we work
17:42:57 <jugglinmike> jugglinmike has joined #aria-at
17:43:32 <jugglinmike> Matt_King: In this agenda item, I'd like to reorganize the Test Queue to be organized by Test Plan, and to list the AT/Browser combos within each of those
17:44:04 <jugglinmike> Matt_King: I think we're all aligned on the basics, and I think that Isaac from Bocoup can go forward on the proposal in the issue
17:44:22 <jugglinmike> Matt_King: The issue is here: https://github.com/w3c/aria-at-app/issues/791
17:44:29 <jugglinmike> howard-e: No questions here
17:44:44 <jugglinmike> Matt_King: Well, if anyone has questions, please feel free to ask in that issue
17:45:23 <jugglinmike> Matt_King: This is in the design stage right now, but because we need this in time for when we have four "Recommended" test plans in April, it's going to come up pretty quickly as something we need to be able to do
17:45:31 <jugglinmike> Matt_King: Hearing no questions, we'll move on!
17:45:41 <jugglinmike> topic: Define assertion verdicts
17:45:50 <jugglinmike> github: https://github.com/w3c/aria-at/issues/1050
17:46:17 <jugglinmike> Matt_King: Vispero believes that it sounds odd to call the result of a "MAY" assertion as a "pass" or a "fail"
17:46:53 <jugglinmike> Matt_King: I think I agree. We're interested in defining some terminology for how we describe when ATs do or don't "check the box" when it comes to "MAY" assertions
17:47:19 <jugglinmike> Matt_King: I've suggested "supported" and "not supported" instead of "passed" and "failed"
17:47:45 <jugglinmike> Matt_King: When it comes to "SHOULD" assertions, it seems like everyone is satisfied with the terms "pass" and "fail"
17:50:44 <jugglinmike> Hadi: That sounds good to me
17:51:12 <jugglinmike> James_Scholes: when it comes to "SHOULD", I find myself flip-flopping between different viewpoints
17:53:03 <Joe_Humbert> Joe_Humbert has joined #aria-at
17:53:22 <jugglinmike> Joe_Humbert: That sounds good to me
17:53:37 <jugglinmike> Murray_Moss: Me, too
17:54:06 <jugglinmike> LolaOdelola: My question is more about the use of "MAY"/"SHOULD"/"MUST" in general, and I know that conversation has already happened, so I don't want to re-tread those points
17:54:42 <jugglinmike> Joe_Humbert: I'd like to get feedback from developers and see if the framing helps them understand how well the different patterns are supported by the different ATs
17:55:04 <jugglinmike> LolaOdelola: Yeah! I also wonder if there's a bit of a conflict between what developers want and what AT vendors want
17:55:30 <jugglinmike> LolaOdela: I also wonder how many developers would even be familiar with these more standards-facing terms like "SHOULD"
17:55:44 <jugglinmike> s/LolaOdela/LolaOdelola/
17:56:25 <jugglinmike> Matt_King: This is really for AT developers. I think web authors are unlikely to drill down into this level of granularity; they will probably be reviewing the test plan reports at the top level
17:57:46 <jugglinmike> Hadi: regarding whether it is geared toward AT developers or web developers: while I see occasionally that some web developers are interested in seeing this level of granularity, I think that overall, they shouldn't have to be concerned about specific features
17:58:11 <jugglinmike> Hadi: I think they should just worry about implementing the APG and considering that as sufficient
17:59:03 <jugglinmike> Joe_Humbert: I agree that would be ideal, but I think that practically speaking, web developers have to be able to prove certain things in certain situations (e.g. when it comes to accessibility compliance)
18:03:44 <jugglinmike8> jugglinmike8 has joined #aria-at
18:04:39 <jugglinmike> Matt_King: We are planning to write a detailed explanation about how to interpret the reports. This will certainly be covered there, too
18:04:57 <jugglinmike> Joe_Humbert: That eases my concerns significantly
18:05:12 <jugglinmike> Zakim, end the meeting
18:05:12 <Zakim> As of this point the attendees have been jugglinmike, howard-e, IsaDC, LolaOdelola, Matt_King, webirc, murray_moss, Joe_Humbert, James_Scholes, Hadi
18:05:14 <Zakim> RRSAgent, please draft minutes
18:05:16 <RRSAgent> I have made the request to generate https://www.w3.org/2024/03/27-aria-at-minutes.html Zakim
18:05:23 <Zakim> I am happy to have been of service, jugglinmike; please remember to excuse RRSAgent.  Goodbye
18:05:23 <Zakim> Zakim has left #aria-at
18:05:24 <jugglinmike> RRSAgent, leave
18:05:24 <RRSAgent> I see no action items