Meeting minutes
Regular group meeting is tomorrow, 12pm PST
Seth - this is a discussion about moving our manual ARIA-AT tests into automation, via drivers that can automate each AT combo
Seth - majority of time on scope, then logistics. By the end, we hope to have enough feedback to turn our convo into a project plan (sequencing, allocation)
Seth - <intros + Bocoup code of conduct>
Seth - last year, we completed a prototype of automating NVDA. We got 1 test working with 1 AT. Involved figuring out the test format, logistics. It was a success!
Seth - our goal this year is really to scale that up
Goals
Seth - we need to write a draft/initial spec of how these AT drivers should work
Seth - can we also use this work to influence AT vendors
Success Criteria
Seth - Research implementation feasibility on 5 platforms, and implement driver on 2 or more
Seth - Write a draft spec of AT driver protocol
Seth - Unify manual and automated test formats and incorporate test sources beyond APG
Seth - that means ARIA, HTML patterns as well
Seth - Set up automated Test Harness and run one test end-to-end from CI environment to ARIA-AT App
Seth - Get buy in broader stakeholders including AT vendors and all browser vendors
Seth - Plan alignment with web developer tools including WebDriver API semantics and DevTools integration. We're really building a tool that's similar to the WebDriver API. Can we make our tool more accessible to web developers & their current expectations on what an automated tool like this should do
Seth - speaking of, we should think about how a driver API gets built on by something like DevTools
Seth - that's the "what". The "how" is we'd like to center marginalized AT developers, manual testers, web devs, and others as co-designers, mentor 1-2 beginners from under-represented backgrounds in contributing to the AT Automation project
Seth - feedback?
michael_fairchild - how do we measure the success of some of these? At a high level, is Matt_King aligned with this list?
Matt_King - I had same question about metrics, some success criteria are more imp. than others. I also want to make sure we're aligned on prioritization, esp. when it comes to sequencing. Sever dims. to consider — feasibility & scaling up what we've already done (ability to run a whole suite)
Matt_King - also broadly thinking about how we prioritize the desktop OS vs. mobile OSs, where automation will be a lot trickier. I want to get to a place ASAP in the year where people can take our prototypes and prove feasibility in other envs. Like I have a FB product team, would like to prove it w/ them inside our own env.
michael_fairchild - re: "Unify manual and automated test formats and incorporate test sources beyond APG", I think the first part is really important. The last half could be a separate criteria?
Matt_King - agreed, we should be able to treat the source as a black-box. I'd expect that to be managed by the ARIA-AT app
Seth - I agree, this is somewhat of a shared criteria. I don't think it's necessarily a goal to have the driver run a test from another source. It would be that the driver works with open test formats that allow for expansions
Matt_King - this workstream is going to inform ARIA-AT app more than vice versa
michael_fairchild - what does the mentoring aspect look like in practice?
Seth - still working it out, we're (Bocoup) working on a x-work stream technique. This is something on our plate, we don't expect it to create work for anyone else (unless they're interested)
Matt_King - vision is Bocoup plays "teach the teacher" role. Members of the community can use Bocoup created resources/training to repeat outreach. Zooming out - this project can only succeed w/ a lot more people involved
michael_fairchild - are they volunteers, people we're paying?
Seth - paying is on the table. We want people to feel empowered, overcome barriers of W3C process
michael_fairchild - re "Get buy in broader stakeholders including AT vendors and all browser vendors", how do we measure? What level of buy in are we expecting?
Matt_King - gotta figure out success criteria. We want AT venders to actually use our deliverables. Even better, rely on
zcorpan - there are diff. levels of buy-in, but eventually AT & browser vendors should use the tests internally as regression tests, just like they're doing w/ web-platform-tests
michael_fairchild - re "Plan alignment with web developer tools including WebDriver API semantics and DevTools integration", how does DevTools integration work?
Seth - mostly a placeholder, this is more future. We would be successful if we had written down any plan on how we will align the low-level spec w/ a higher level project vision for web devs and end users
zcorpan - I think DevTools is mentioned here b/c they cur. use WebDriver as their API. & at least in some browsers, you can hook up a mobile phone and talk to it from DevTools. So you can debug on your phone through tools on desktop
boazsender - maybe next year, thinking about other ways for rendering sonic interfaces
weston: along the lines of devtools, we've talked about browser pairings, but what about native applicaitons?
Matt_King - not in near term. Certainly something W3C is looking at, we don't have anything like ARIA/APG for native apps
no worries
Matt_king - we do compare ARIA to native HTML and native app implementations. We want to make sure we don't have discrepencies. AT vendors will insist on that though
weston - will AT vendors insist that any drivers they build work with native apps & web?
Matt_King - might happen anyway
zcorpan - even if we only care about browsers, I believe browser vendors would want to automate their chrome as well
Matt_King - need to be careful that consider doesn't create scope creep
Current Priorities
Seth - Recruit and work with co-design experts on automated test format. I think we'll want to do this up-front b/c on the ARIA-AT app side, we'd like to tackle test format changes ASAP
Seth - Implement AT Drivers (NVDA, Narrator, VoiceOver MacOS, VoiceOver iOS, TalkBack)
Matt_King - JAWS instead of Narrator? But if MS wants to provide funding, I'm all for it
weston - FYI JAWS has scripting, where I don't think Narrator does
Matt_King - maybe could use UIA? Someone from MS mentioned it
Seth - work here might include developing a x-AT runtime for Windows
zcorpan - came up working on the prototype, someone suggested that it might be possible to have the same implementation for all screen readers on Windows
Matt_King - Sina had mentioned that for capturing output, the SAPI modules could be useful
Matt_King - I'm less clear about the driving side - how "command X" could be the same
Matt_King - other thing that makes me think it's possible is we now have a std. interface for Braille drivers. Maybe could tap into something like that. Problem is braille and speech are diff, not a good substitute
Simon - Write draft specification for AT Driver protocol & explore integrating with BiDirectional WebDriver Protocol, Unify ARIA-AT and AT Driver test formats
Matt_King - how is WebDriver standardized? Is it W3C?
zcorpan - it's W3C by browser testing & tools WG
<zcorpan> https://
Seth: Align ARIA-AT / AT Automation project governance. A bit fuzzy, but want to make sure automation and ARIA-AT app are aligned
Seth - Create automated test runner, to work in CI. Mostly about putting pieces in place than production-ready. ARIA-AT app will likely need API to submit automated test results to
Seth - Write user documentation and propose features to align with web developer use cases
weston: I have very little idea about how we'd collect output from android/ios. it would be great to have a vague idea of how that would be done
Matt_King - <thinking through how the AT driver would work> both input & output of driver have a dependency on our data model, right?
Seth - the tests we're writing should be able to produce both a manual & automated version (maybe some just manual). That format is the input to the AT automation, then we have the format of assertions/results. Cur. the app displays results in a table for manual tests. We might not need to do much to integrate automated into there. Or automated might need to add data. Design process crosses all work streams
Matt_King - it feels like the input part is way more important than the output. Output can be transformed w/out much rework
Seth - zcorpan has a proposal for automated test format, goal is to describe automated in as much detail as manual
Matt_King - is validating that format pri. 1? Pre-req. to writing driver code?
zcorpan - yes, we should do a research spike to understand the 3 diff. screen readers & understand how they're different
Matt_King - ok so that's even more imp. than validating format of tests
michael_fairchild - +1, my gut concern is that we haven't proved things out on MacOS VoiceOver, and that might inform how we write tests
Matt_King - for VoiceOver, do similar to NVDA. Do whatever hacks you have to do to make it run. Then for JAWS, I wonder if it'll be the hardest. Even though it has scripting... but it's not in a headless way. And there's no JAWS CLI. Ex: no way to start & run script from CLI. Will we have to get support from Vispero? What is the lightest-weight thing they could do? Would they be willing to share source?
michael_fairchild - so highest pri. is research spike, understand VoiceOver MacOS & JAWS. Are mobile SRs in spike?
Matt_King - hard to say... I'm thinking mobile is in next year's scope. I don't want to plow blindly ahead either. Some level of research there, not as far as writing for NVDA
Seth - spend a little time now to save a bunch of time later
Matt_King - yeah let's have intelligent convos with Apple and Google. Let's leave time for those convos
Matt_King - we have a basic approach w/ Apple already, but haven't talked to Vispero. Need to find out what we're up against
zcorpan - for the feasibility spike, I envision learning about the path to automation for a particular SR. Communicate that a vendor needs to build something early
weston: are we distinguishing between mobile emulators vs devices?
Matt_King - no line in the sand yet, we know that there are things you can't test in a simulator. We tell internal devs to use real devices. If our convo were to drive the convo w/ Apple to improve robustness, that would be great
boazsender - is there a risk to deferring the mobile prototype? How much could it impact/churn the standard? I've seen the 4 year process involved w/ updating WebDriver
Matt_King - concerns - there are no API mappings for Android & iOS for ARIA. The issue has been raised with the ARIA WG. I want to make sure that we achieve our goals for the desktop platforms by mid-next years, so we can 100% focus on mobile for 2022-23
Matt_King - agreed it's worth a few weeks of investment now to figure out what we're up against
Matt_King - let's find out what we don't know now, make sure investment roadmap can fully contain mobile. We might need to seek more funding & investment from MS/Google/Apple. If we demonstrate value in the places we know we can (desktop), it will really help
Seth - agreed, the further we get with a draft, the easier it will be to get buy-in
Matt_King - most concerned about getting AT vendors on board. This isn't going to work if they don't participate
Closing
Seth - next steps are to turn this into a fuller <missed it>
Matt_King - need research spike first, prioritize prototyping VoiceOver & JAWS, but we need a research spike on all of them
zcorpan - for test writing maintainability, we want to have a single format that you can generate manual tests & automated tests from. Will req. changes to our cur. format
weston: with how the prototype is written vs how jaws and nvda are implemented, I think there might have to be some timeouts and stuff, so I'm worried about having a single format across tests that need timeouts vs not.
simon: hopefully test authors dont need to care about timeouts
weston: yah
Seth - weston, we can invite you to discussions on the test formats
Simon - logistics, we won't start billable work on this project until late March. Need to describe cadence for checkins
Simon - we'll defer deciding cadence until later
Seth*