07:52:22 <RRSAgent> RRSAgent has joined #aria-at
07:52:26 <RRSAgent> logging to https://www.w3.org/2025/11/12-aria-at-irc
07:52:26 <breakout-bot> RRSAgent, do not leave
07:52:27 <breakout-bot> RRSAgent, this meeting spans midnight
07:52:27 <breakout-bot> RRSAgent, make logs public
07:52:29 <breakout-bot> Meeting: The Future of Assistive Tech Interoperability with ARIA-AT
07:52:29 <breakout-bot> Chair: Chris Cuellar, Matthew King, Michael Pennisi
07:52:29 <breakout-bot> Agenda: https://github.com/w3c/tpac2025-breakouts/issues/30
07:52:29 <Zakim> Zakim has joined #aria-at
07:52:30 <breakout-bot> Zakim, clear agenda
07:52:30 <Zakim> agenda cleared
07:52:30 <breakout-bot> Zakim, agenda+ Pick a scribe
07:52:31 <Zakim> agendum 1 added
07:52:32 <breakout-bot> Zakim, agenda+ Reminders: code of conduct, health policies, recorded session policy
07:52:32 <Zakim> agendum 2 added
07:52:32 <breakout-bot> Zakim, agenda+ Goal of this session
07:52:33 <Zakim> agendum 3 added
07:52:33 <breakout-bot> Zakim, agenda+ Discussion
07:52:33 <Zakim> agendum 4 added
07:52:33 <breakout-bot> Zakim, agenda+ Next steps / where discussion continues
07:52:35 <Zakim> agendum 5 added
07:52:36 <breakout-bot> Zakim, agenda+ Adjourn / Use IRC command: Zakim, end meeting
07:52:36 <Zakim> agendum 6 added
07:52:36 <breakout-bot> breakout-bot has left #aria-at
08:23:41 <ChrisCuellar> ChrisCuellar has joined #aria-at
08:28:09 <Jamie> Jamie has joined #ARIA-AT
08:28:13 <Matt_King> Matt_King has joined #aria-at
08:30:18 <tantek-projector> tantek-projector has joined #aria-at
08:30:30 <ari> ari has joined #aria-at
08:30:32 <nico> nico has joined #aria-at
08:31:28 <chrisp> chrisp has joined #aria-at
08:31:42 <chrisp> +present
08:32:44 <jugglinmike> jugglinmike has joined #aria-at
08:32:46 <ari> present+
08:33:04 <jugglinmike> present+ jugglinmike
08:33:05 <jugglinmike> scribe+ jugglinmike
08:38:25 <jugglinmike> ChrisCuellar: ARIA-AT is different from other accessibility testing frameworks or platforms that you may have already encountered is that this one is really pushing forward the concept of interoperability within screen readers themselves
08:38:34 <jugglinmike> ChrisCuellar: It's pushing beyond the boundaries of the browser
08:39:07 <jugglinmike> ChrisCuellar: This effort started with the ARIA-AT community group in 2016
08:39:27 <jugglinmike> ChrisCuellar: in that time, the framework has evolved to a high degree of sophistication
08:39:44 <jugglinmike> ChrisCuellar: this involes writing tests, running tests, and even automating test execution
08:39:54 <jugglinmike> ChrisCuellar: We're here to share an overview and give status updates
08:40:23 <jugglinmike> ChrisCuellar: We want to make a deeper dive into the infrastructure--how the tests work, how they are structured, and the underlying methodology
08:40:44 <jugglinmike> ChrisCuellar: And we'd like to give an indication of where the program is headed and share some pointers on how folks can get involved
08:41:09 <jugglinmike> ChrisCuellar: We're sharing a screenshot of the ARIA-AT app which documents support levels for our test plans
08:41:34 <jugglinmike> ChrisCuellar: It features a grid describing screen reader / web browser pairs
08:42:03 <jugglinmike> ChrisCuellar: Initially, we've been testing those pairs' renderings of design patterns from the ARIA Practices Guidelines
08:42:21 <jugglinmike> ChrisCuellar: The goal of the program is to help AT vendors improve interop through testing
08:42:36 <jugglinmike> ChrisCuellar: We've taken a lot of inspiration from the web-platform-tests project
08:43:06 <jugglinmike> ChrisCuellar: Along those lines, we're hoping to get more granular with accessibility-related features
08:44:11 <jugglinmike> Matt_King: This is different from other interop efforts is that normally, you start with a standard and have everyone tested to that standard. Here, there is no standard for how ATs should behave. We're trying to solve that, but not by writing a standard. We're starting with tests as a basis to drive consensus about basic expectations
08:44:19 <berlysia> berlysia has joined #aria-at
08:44:48 <jugglinmike> Matt_King: The biggest value to developers is having the confidence in that the experience you are designing is truly accessible across all platforms
08:45:02 <jugglinmike> ChrisCuellar: There's been a lot of evolution in this project's six-year lifespan
08:45:21 <jugglinmike> ChrisCuellar: We've learned a lot about the difficulty in testing against the accessibility stack
08:45:33 <jugglinmike> ChrisCuellar: We have a concept we've been calling the "four-mile journey"
08:45:57 <jugglinmike> ChrisCuellar: It describes what happens to the code that web authors write in order for it to reach AT users
08:46:28 <jugglinmike> ChrisCuellar: The first mile is about the code as authored by web developers. That's supported by the ARIA Authoring Practices Guide, WCAG, etc.
08:47:18 <jugglinmike> ChrisCuellar: The second mile is the accessibility tree. That's really the territory of browser developers. At this stage, we're able to track interop by web-platform-tests. There's been a lot of innovation in recent years around exposing that tree for testing
08:48:24 <jugglinmike> ChrisCuellar: At the third mile, we have the operating systems' accessibility APIs. Before we reach the screen reader, we have to pass through the operating system. There, we're dealing with AAM tests
08:49:15 <jugglinmike> ChrisCuellar: It gets harder and harder to access each layer we're describing here. But efforts are underway to tap into and to test the accessibility APIs. That's a new frontier for the web-platform-tests
08:49:46 <jugglinmike> ChrisCuellar: The last mile is what we've been talking about--it's where ARIA-AT really lives. It's the behavior of the ATs (e.g. screen readers) themselves
08:50:25 <jugglinmike> ChrisCuellar: We're validating that the various ATs we support (JAWS, NVDA, and VoiceOver at the moment) provide a roughly equivalent experience
08:51:05 <jugglinmike> Matt_King: The idea that the screen readers should behave "more or less the same" is an area that I'm sure many in attendence today will want to interrogate. That's where the ARIA-AT community group spends most of its efforts
08:52:02 <jugglinmike> ChrisCuellar: There are many distinct projects under active development for testing at each of these "miles"
08:52:42 <jugglinmike> ChrisCuellar: Since 2018, we have developed an overall approach to building consensus. I think that's the most unique part of our work. There is so much conversation between testers and AT vendors themselves
08:53:03 <jugglinmike> ChrisCuellar: We have a repeatable, scalable, and automatable test structure
08:53:03 <jugglinmike> ChrisCuellar: We have a testing and reporting platform
08:53:13 <jugglinmike> ChrisCuellar: And we have integrated JAWS, NVDA, and VoiceOver automation
08:53:40 <jugglinmike> Nigel: Is there a reason why TalkBack is not in that list?
08:53:44 <jugglinmike> ChrisCuellar: Why yes there is!
08:53:51 <jugglinmike> ChrisCuellar: It's on the roadmap!
08:54:12 <jugglinmike> ChrisCuellar: Mobile in general is something we started to work on this year, and we made some good progress on Android
08:54:32 <jugglinmike> ChrisCuellar: Likewise, we're also interested in moving beyond the English language
08:54:42 <jugglinmike> Matt_King: Our current scope is limited by resource availabilty, largely
08:55:18 <jugglinmike> Matt_King: Our plans have shrunk over the years. Our plans back in 2018 were much more optimistic about our progress by this point. We wanted more screen reader/browser pairs, etc
08:55:45 <jugglinmike> Matt_King: This work all hinges on the availabilty of automation. Without that, it becomes impossible to keep up with the releases of new versions of platforms
08:56:02 <jugglinmike> Matt_King: So we continued narrowing our scope to a point where we could find succcess given our resources
08:56:36 <jugglinmike> Matt_King: But we've been designing everything to avoid limiting extensions to other kinds of ATs, other languages, etc
08:56:48 <jugglinmike> Nigel: Do ATs have a standard protocol for reporting their state?
08:57:38 <jugglinmike> ChrisCuellar: We're not trying to start with specs and standards--we're backing in to that via testing. However, one area that is under development (one that is powering automation) is a standard protocol for remotely controlling ATs
08:57:56 <jugglinmike> ChrisCuellar: We're calling it AT-Driver, and it's modeled after WebDriver BiDi from the W3C
08:58:21 <chrisp> chrisp has joined #aria-at
08:58:29 <jugglinmike> ChrisCuellar: That's enabling us to do the work driving consensus in AT users' experience
08:59:05 <jugglinmike> Nigel: So you've got some stimulus, you're expecting to observe the behavior, and what are you actually observing?
08:59:14 <jugglinmike> Matt_King: We're at the final level
08:59:21 <jugglinmike> ChrisCuellar: Right, "what is the attached device actually doing?"
08:59:44 <jugglinmike> ChrisCuellar: Initially, it started with pressing keys, but it's starting to evolve into more generic "user intents"
09:00:00 <ChrisCuellar> scribe+
09:00:54 <ChrisCuellar> jugglinmike: We're actually capturing the text being spoken by the screen reader. So it's text data from the screen readers.
09:01:10 <nigel> nigel has joined #aria-at
09:02:15 <ChrisCuellar> ... AT-Driver is implemented as a web driver bidi protocol. It speaks over websockets. We've implemented in NVDA, macOS and in JAWS. As a separate effort, we're hoping that this has other implementations.
09:02:21 <keithamus> keithamus has joined #aria-at
09:02:23 <nigel> present+ Nigel Megitt
09:02:33 <keithamus> keithamus has left #aria-at
09:02:45 <jugglinmike> ChrisCuellar: Yeah! And hearing that, I was wondering: are there other implementations or use-cases that would be valued by folks here?
09:02:54 <jugglinmike> ChrisCuellar: What drew you here to this talk today?
09:03:37 <jugglinmike> florian: The use cases I've had are internationalization-related
09:03:49 <jugglinmike> florian: I've assumed that the well-trodden English paths are the best-tested
09:04:11 <jugglinmike> florian: So I'm concerned with how the accessibility tree is rendered in internationalization contexts
09:04:23 <jugglinmike> ... I'm curious about CJK-related transforms
09:04:38 <jugglinmike> ChrisCuellar: So it would be useful to you to get the final output just to verify?
09:04:53 <jugglinmike> florian: Yeah, in a WPT-style context
09:05:06 <jugglinmike> ... So if implementers really insist on what they're doing, we can reecognize this and have a conversation
09:05:46 <jugglinmike> ... Another case I've wanted this is also related to CJK use-cases. Ruby is an assistive tool for sighted users who, for whatever reason, lack knowledge about the rendered text and need help interpreting it
09:05:59 <jugglinmike> ... Bad information in this context is worse than no information
09:06:47 <jugglinmike> ... There are language-specific things that happen with some technologies where verification is especially important
09:07:17 <jugglinmike> Nigel: One of my use-cases is in an implementation that sends audio description text via an aria-live=polite element
09:07:41 <jugglinmike> ... and there's a related use-case where, if you imagine that you have a video that only has a description and it has hard-of-hearing subtitles
09:08:26 <jugglinmike> ... In the BBC's player, we send the subtitles to the screen reader. Let's say that you have two people watching this video, and one of them can't hear, and one can't see. You're screen reader is on, and your subtitles are on. It would be really good to have a repeatable mechanism to understand the user-experience
09:09:16 <jugglinmike> Yanna: Sometimes, others think about people with physical impairments. Of course we know this work is also about people of advanced age
09:09:27 <jugglinmike> ... We want to be sure that these experiences are designed properly
09:09:52 <jugglinmike> ... Also, my manager asked me to implement something in a website because the target audience is aged above 70 years old.
09:10:06 <jugglinmike> ChrisCuellar: Thanks, everyone!
09:10:18 <jugglinmike> ChrisCuellar: Let's get into how this all works operationally
09:10:24 <jugglinmike> ChrisCuellar: The effort is hosted by the W3C
09:10:35 <jugglinmike> ChrisCuellar: And that's informed a very rigorous process design
09:11:21 <jugglinmike> ChrisCuellar: If you ever join a Community Group call, you'll hear Matt_King assigning Test Plans to testers (here on the slides, we're looking at a "radio group" test plan--specifically one that relies on aria-activedescendent)
09:11:43 <jugglinmike> ChrisCuellar: Generally, we want to have test plans executed by two testers. We're looking to corroborate the results
09:12:32 <jugglinmike> Matt_King: Right. The test plans are authored to be as specific as possible, but there's still plenty of room for people to make mistakes.
09:13:01 <jugglinmike> ChrisCuellar: So, reviewing a test plan like this one I'm sharing for JAWS and Chrome, you can see that there are a lot of instructions.
09:13:37 <jugglinmike> ChrisCuellar: Then, you get a list of different commands--these are steps in the test. You can see that here, we have one command for what happens after you hit the "x" key. Here, it's when JAWS is in a specific mode.
09:13:46 <jugglinmike> ChrisCuellar: Following that, we have the captured output from JAWS
09:14:22 <jugglinmike> ChrisCuellar: We're not just making assertions against the output itself. This is where the role of human testers is critical. There has to be subjective judgements made about the output
09:14:37 <jugglinmike> ChrisCuellar: We have a set of assertions about the output. E.g. "was the list boundary conveyed?"
09:16:08 <jugglinmike> ChrisCuellar: Sometimes, people have different results, and we talk about that in the community group. The bot really helps with the velocity of this task. Today, the human testers work to verify that what the bot reports match their own experience (rather than enter the data manually themselves)
09:17:11 <jugglinmike> Matt_King: The process is that: someone writes the test plan, then at least two people run it. Once we've ironed out the behaviors, we move from "draft review" to a state where we can run the test plan whenever a new version of the AT under test is released
09:18:11 <jugglinmike> Matt_King: Ultimately, this test is kind of defining "what do we mean that the checkbox is supported?" What does that mean in real life? By writing these tests and gaining consensus with screen reader developers, everyone can have a shared understanding about elements in HTML (or role, state, and property in ARIA) means for users
09:18:35 <jugglinmike> ChrisCuellar: All of these get finalized into reports that we published. Those give an overall sense of how the AT/browser combinations are performing
09:19:03 <jugglinmike> florian: This sounds reminiscent of something we used to have in Opera software for visual tests.
09:19:23 <jugglinmike> ... Is this approach something that is or can be integrated with how WPT does tests?
09:19:44 <ChrisCuellar> scribe+
09:21:01 <ChrisCuellar> jugglinmike: In WPT, there are ref tests that are somewhat relevant to this discussion. But the level of fuzziness involved in this kind of testing is different. We have a concept of verdicts in ARIA-AT. It's not enough to say that an assertion is passing or failing. The verdicts are subjective and fallible.
09:21:20 <jugglinmike> florian: In the pre-WPT days at Opera, we had reftests AND visual tests.
09:21:31 <jugglinmike> ... In some cases, a fuzz factor would be sufficient
09:22:10 <jugglinmike> ... I think we had in the visual context where there could be tremendous variability, and it would be obvious to a human if the result was right or wrong, but it would be very difficult to encode that in query
09:22:53 <jugglinmike> ChrisCuellar: I think it might be a non-goal to get this into WPT given the level of infrastructure required to run screen readers seems undesirable for WPT maintainers
09:23:33 <jugglinmike> Matt_King: In the first 1.5 years, we researched what exists already and whether we could fit into off-the-shelf solutions. The result was that we really did need to build a bespoke solution
09:23:44 <jugglinmike> Matt_King: Over the years, we've learned about what we can and cannot abstract
09:24:26 <jugglinmike> Matt_King: Making the decisions about test design--how abstract or concrete to write them. Working with concrete has allowed us, in time, to see the opportunities for abstractions
09:24:46 <jugglinmike> florian: It seems valuable, though I'm sure it would involve a lot of work
09:25:13 <jugglinmike> ... In CSS WG, we did not consult with the people who work on the system to learn what is feasible
09:25:36 <jugglinmike> ... When people implemented our work, they did something bad because there were no tests, and that was bad. Our system may have been great, but they did something else
09:26:12 <jugglinmike> Matt_King: We want people to be able to propose expectations for assistive technologies, place a test in this system, run the tests, and learn about the implementations' behavior
09:26:41 <jugglinmike> florian: I was hoping for a separation between the people writing the implementations and the people writing the tests. That's the WPT parallel I'm interested in
09:27:16 <jugglinmike> Matt_King: Agreed. We think this platform brings a lot of value to the community in terms of moving interoperability forward. We're trying to find the best way to work it into the needs of those working in these spaces
09:27:39 <jugglinmike> Matt_King: Testing new features and experimental implementations--are they potentially able to deliver the value to end-users that we want them to?
09:27:57 <jugglinmike> ChrisCuellar: This slide I'm showing now demonstrates how we're reporting support levels
09:28:43 <jugglinmike> ChrisCuellar: We've been testing against the APG patterns, and we're trying to open the door to writing other kinds of tests
09:28:53 <jugglinmike> Matt_King: The project actually has a lot of different needs
09:29:09 <jugglinmike> Matt_King: One is enlisting people write tests for ARIA and HTML features
09:29:39 <jugglinmike> Matt_King: We also want to make this platform itself work--there is infrastructure, the implementation (three bots at the moment, and a desire for more)
09:30:23 <jugglinmike> Matt_King: If there are people who are passionate about this space and want to see assistive technology interoperabilty become a thing (or you know someone who is, or you know someone who can recruit talent), that would be a big help
09:30:42 <jugglinmike> Matt_King: You can come to us directly, or you can join the Community Group and we'll welcome you there
09:32:33 <jugglinmike> s/Yanna/Chiara/
09:32:46 <jugglinmike> Zakim, end the meeting
09:32:46 <Zakim> As of this point the attendees have been present, ari, jugglinmike, Nigel, Megitt
09:32:48 <Zakim> RRSAgent, please draft minutes
09:32:49 <RRSAgent> I have made the request to generate https://www.w3.org/2025/11/12-aria-at-minutes.html Zakim
09:32:56 <Zakim> I am happy to have been of service, jugglinmike; please remember to excuse RRSAgent.  Goodbye
09:32:56 <Zakim> Zakim has left #aria-at
09:32:59 <jugglinmike> RRSAgent, leave
09:32:59 <RRSAgent> I see no action items