Meeting minutes
<jgraham_> RRSAgent: make minutes
jcraig: Welcome to the automated testing session
… Simon and others kicked off a 2023 interop accessibility investigation
… we've made a whole lot of progress
… I want to share what we've done this year
… integration into the servers and talk about the scope of what's testable
… some new things that people would like to do
… a potential solution we'll be discussing tomorrow with the browser testing and tools group
… webdriver, scheduled for later this week
… any questions?
jcraig: I always like to start with a little bit of background
… I know this is a web group so many of you know this background already
… [sharing slides]
… the web stack, on one side we have the web author, the other we have the end user
… web APIs, html css javascript, then the rendering engine's implementation
… in Safari's case, the implementation inside webkit - webcore
… how pixels are rendered, keyboard and mouse events
… implementations of accessibility differ
… change by platform
… inside the rendering engine there is an accessibility runtime
… it vends properties to a platform-specific context
… that API is accessed by assistive technologies
… screen reader, switch control, etc
… that renders the interface to the end user
… and it's 2-way, the AT will send events back from the user
… the web stack will test things along the way
… many platforms have platform-specific automation
… both Windows and Mac have UIAutomation (though different ones), there are accessibility hooks for that automation
… there is client-side automation
… such as Deque's axe-core
… accessing the DOM, what is available via the DOM
… slides will be shared later
… every engine has it's own specific internal automation
… WebKit has Layout Test
… as of this spring, WebKit had 83k tests
… WPT has 1.8 million tests
… shared between different runtimes, engines
… goal of these tests is a write-once-run-anywhere
… until this year, there was very little testing related to accessibility
… who here has heard of WPT, and Interop 2023
… goal is to achieve interoperability and testing for it in 2023
… this includes flexbox, video codecs, inert, pointer events, accessibility investigation
… in april, we did some infrastructure work to hook into web driver properties
… at TPAC 2019 Chrome and WebKit had implemented, we have a path forward here
… let's see how much we can test
… Firefox and Gecko hadn't implemented the webdriver interface in April, but did in May, but has now
… as of today, we have 620 web accessibility tests running in the CI environments of every major browser engine on the planet
… this is HUGE
… it opens up so many different possibilities for reliability
… we started testing the accessibility names spec (AccName), HTML-AAM, ARIA, we've hit about 600 tests
… thousands of tests that are now unblocked, and we'll likely clear 1000 this year
… we've discovered many bugs, and cleared many of them
… discovered spec ambiguities
… good benefit as well
… one of the differences between this and other tests, there are new incentives to writing these tests
… web developers have little incentive to write browser-specific tests
… it's a lot of work
… learn how to do all these custom specific things that browser engineers should be doing
… write-once-run-anywhere tests are more appealing, they can test issues they run into
… it's using web technologies, HTML and JS, I hope that the web developers in the audience, see this and want to contribute
… you're making the web better
… extend testing beyond role and label
… we have a team of about a dozen from all over: Bocoup, Igalia, Apple, Adobe, Google, Salesforce, etc
… we could use some help
… next steps beyond what we can do this year
… we need to extend our testing
… beyond role and label, there are incubator issues in the AOM incubator (#197 and #203)
… that's the gist of accessibility investigation this year
… show you one more thing [changes shared screen]
<jamesn> WICG/
<jamesn> WICG/
jcraig: [new screen] Scoring criteria for the testing
… we have a list of what we're working on [walks through list]
… there are a number of role tests remaining, and acc name tests
… if there's time at the end, we'll do SVG ARIA and DPUB ARIA
… [WPT dashboard] Label:accessibility
… this is the WPT dashboard showing the accessibility tests
… [grid of different tests]
… failures and different ones in each browsers
jcraig: [return to slides]
… we'd like to extend accessibility testability beyond what we're able to currently test
… 53 ARIA attributes in the editor's draft and role
… how many of those have we been able to test?
Wilco: 3?
jcraig: You knew the answer
… yes, 3 / 53
… we have a lot more to test
… we got 600 tests out of 3 attributes
… all of the remaining ARIA attributes and associated features
… if there are similar host semantics, there is reconiliation of which wins
… ideally we could synthesize associated accessibility actions and events
… there are notable differences
… there are different interoperability issues
… registering to receive notifications, it would be good to test this and intercept in testing
… the accessibility tree one kicked off this project
… People wondered: could we have an interoperable accessibility tree in 2023?
… It won't happen this year, but I think we can work towards it
… it's a worthy test interface, even if we don't end up with an interoperable tree
… we can work towards testability
… we can avoid posting the tests that have irrelevant details
… AOM #197, this is the discussion we've been having
<jamesn> WICG/
jcraig: new test-only web API
… all of the WPT Accessibility stakeholders agree on the following details
… we need test-only web apis
… we agreed that it would be too problematic to open this up to the public for use
… remain test-only for the foreseeable future
… within that test-only API, we need a way to get to an accessibility object and its attributes
… a way to reconcile accessibility objects with elements
… synthesize user events from AT
… as if a user were to press something, click, focus, these events differ
… if ARIA ships aria-actions, that is a different event
… there is no way to test that yet
… a way to register for outbound accessibility notifications
… non goals
… refs to accessible nodes are not expected to be writeable
… we should nto be able to change its state
… not expected to be live
… don't think the implementability will be there
… a snapshot or property bag instead
… refs to accessible node are not expected to be persistent after DOM changes
… no relation to the previous one
… tree identicality, we are not expecting tree interoperability at this point
… platform-specific accessibility APIs are not in scope, yet
… testing accessibility features or ATs is not part of this effort
… we're not automating VoiceOver for example
… another group has a presentation today working on something called AT Driver, there's overlap with some of the people working on the other one, but it's not in scope for this effort
… this portion of the presentation will be more fleshed out tomorrow, when we talk to browser testing tools
… one discussion of what web driver might look like
Matt_King: It's not expected to be live (probably a snapshot object aka "property bag"), I was not following what you meant
jcraig: Thanks, I'm trying my best to avoid words like interface or method, I'm talking about these things conceptually
… sometimes when you explore a new web interface, there are refs or things you can call into
… in the case of WebDriver, it's a server-client relationship where you call out to the backing web server and giving an ID and getting a property back
… it's not like you could observe a change on that element, to do so you'd need to make a request back
aaronlev: You have a test, you get an object, you go to the node interface, test runs, the accessibility interface may destroy the old content, and you'd need to re-request
Matt_King: The objects you're getting are from the platform accessibility tree?
jcraig: No, it's getting the browser's internal representation of the tree, before it makes it out to the platform.
aaronlev: It's not platform specific
jcraig: another example, if you were to hide an element and show it again, the backing accessibility node would likely have a completely new identifier, and bear no relationship to the previous accessibility node that was exposed.
Matt_King: You can't rely on the unseen behaviour of the layer
aaronlev: You can't rely on the underlying DOM
CharlesL: On the last slide, one of the points mentioned platform specific was not going to be test
<aaronlev> ??? = Aaron Leventhal
CharlesL: a chrome implementation, you're not testing windows vs mac
… would the tests really show that?
… for role, you're expecting doc-chapter, depending on mac or windows, those results would be identifiable
… just trying to understand why that would not work?
jcraig: I'll go back to the slide on the web automation stack
… four grey boxes, different types of automation, platform-specific (Mac AX roles), client-side automation (axe-core), in the browser internals it would be the same for both
… the only standard we have is what is the computed ARIA role for this element
… does this element have a role of heading
… but it's different from an AXRole of AXHeading on Mac, or a Windows role of heading in the windows-specific platform accessibility API
… we're testing the web specific portione of the stack, not the platform on
… here is a specific web test, WPT can't do this kind of platform testing well
jgraham: WPT tries to avoid that kind of thing
jcraig: The CI implementation on that is a linux machine, so it's not even running those platform APIs
Wilco: I had 2 questions
… this sounds to me like [?] has with (sorry I missed this)
… have you thought about what ARIA is doing with pulling language or table structure?
jcraig: I hadn't personally thought of the lang attributes
… I know VO will switch languages
<matatk> s/????: We/jgraham: We/
jcraig: it's doing that because there is an attribute for it in the platform accessibility API
… anything with reference to accessibility, and
… repeating the first question, CDP, the Chrome Dev Tools, James Graham, does that align?
… this particular section about the web driver potential extension
… I don't know as much about it as the others
… one recommendation was to put this on the bidi roadmap
jgraham_: I can answer but it might not be worth taking up time
jcraig: We can discuss tomorro
Simon: Web Driver bidi, (missed)
Simon: adding to the language question, the content language, set with the lang attribute
… it's already testable
… it can be tested in WPT today
jcraig: Browser internal accessibility it's going to be platform specific
… you can test it in the browser, but should we test it in accessibility
[cross-chatter]
jamesn: There are bugs and ambiguities to test
jcraig: This will be more fleshed out in the webdriver session tomorrow
… we need either a new method for specific web driver extension
jcraig: Potentially be a web opt-in extension
… because there is so much in accessibility, we don't want to ask web driver to update piecemeal
… we need a method on the existing web driver interface to get a node
… this is a way to reconcile if you have an accessibility object
… if there is a relationship between an object and node we need to get at it
… ways to get an web driver ID from a node
… this is a potential sample JSON object [slide]
… label, role, parent
… ids for each
… checkstate, requiredstate
… way to synthesize an event on an accessibility node
… minimum payload would be the notification type
… focus, press, actions
… open questions
… why return property bag instead of calls per property
… there could be a lot of property values, especially for some tests
… would fit closer with current web driver interface
… are accessibility id refs implementable in an interoperable manner?
… depending on the longevity of the references, some aren't possible yet?
… what about testing accessibility of related native features like `inert`?
… good questions everyone
jgraham_: Make a few comments, first one is that if you have time for discussion, let's make it happen?
[scheduling]
jgraham_: Of the details, you mentioned bidi, a lot of this is a fit for bidi, for the DOM tree, we more accurately represent that in bidi
… it becomes complicated to work with
… I will definitely think about this, classic web driver will be the way to get this working in all browsers
… its' also worth thinking about what you actually want this to look like, then determine protoals
… looks at existing [?] actions
… it moves from point to point, presses, it might fit with that instead of a new endpoint?
jcraig: It's critical we go through the accessibility node
jgraham_: If you want it to be this source, performs this action, it can represent that
jcraig: For bidi vs web driver, I'll do what Sam things is best
jgraham_: Do what's best for browsers, what worries me, if we have this stuff, web authors may use it for their own tests
… but not accessibility, it gives them information that's hard to get other ways
… having things like "is this completely implemented, defined", it could be a risk
… be more specific about what the integration is or browser specific
jcraig: I think we're align on what we want
… webkit has an intermediary accessibility node
… differences would be tree relationships, different parent, but can still get into ut
… if a web author hits that, it may need notes in the TODOs, they can fix their tests
jgraham_: it's definitely difficult if it's different tree structures
jcraig: Anyone jones-ing to write accessibility WPT tests?
[hands!]
Wilco: Question, accessibility conformance testing taskforce
… writes a lot of rules that are picked up in testing, we're having a hard time to track what works where
… this will help us figure that out, and we can help with gaps
jcraig: The arrangement of testing is by spec
… line-by-line we go through the requirements
… it would be good to have tests for WCAg
… the tests are similar if structured differently
… ARIA vs WCAG, we say a lot of bugs in testing
Wilco: I believe that!
jcraig: Thanks everyone for your time, email me!
… contact me via the W3C slack too
Slideset: https://