18:48:52 RRSAgent has joined #aria-at 18:48:52 logging to https://www.w3.org/2021/02/24-aria-at-irc 18:49:29 MEETING: ARIA And Assistive Technologies Community Group - Automation Workstream 18:49:45 CHAIR: Seth Thompson 18:49:52 rrsagent, make log public 18:50:02 present+ 18:50:08 rrsagent, make minute 18:50:08 I'm logging. I don't understand 'make minute', Matt_King. Try /msg RRSAgent help 18:50:18 rrsagent, make minutes 18:50:18 I have made the request to generate https://www.w3.org/2021/02/24-aria-at-minutes.html Matt_King 18:59:26 michael_fairchild has joined #aria-at 19:01:42 weston has joined #aria-at 19:02:00 boazsender has joined #aria-at 19:02:24 present+ 19:02:39 present+ 19:02:50 present+ 19:03:05 present+ 19:03:23 present+ 19:04:11 scribe: weston 19:04:17 scribenick: weston 19:04:47 Agenda+ Introductions 19:04:48 Regular group meeting is tomorrow, 12pm PST 19:04:58 Agenda+ Goals 19:05:05 Agenda+ Success Criteria 19:05:09 Agenda+ Current Priorities 19:05:21 Agenda+ Logistics 19:05:48 Seth - this is a discussion about moving our manual ARIA-AT tests into automation, via drivers that can automate each AT combo 19:06:30 Seth - majority of time on scope, then logistics. By the end, we hope to have enough feedback to turn our convo into a project plan (sequencing, allocation) 19:08:13 Seth - 19:11:00 Seth - last year, we completed a prototype of automating NVDA. We got 1 test working with 1 AT. Involved figuring out the test format, logistics. It was a success! 19:11:09 Seth - our goal this year is really to scale that up 19:11:20 TOPIC: Goals 19:12:02 Seth - we need to write a draft/initial spec of how these AT drivers should work 19:13:10 Seth - can we also use this work to influence AT vendors 19:13:29 TOPIC: Success Criteria 19:13:48 Seth - Research implementation feasibility on 5 platforms, and implement driver on 2 or more 19:14:10 Seth - Write a draft spec of AT driver protocol 19:14:13 Seth - Unify manual and automated test formats and incorporate test sources beyond APG 19:15:09 Seth - that means ARIA, HTML patterns as well 19:15:49 Seth - Set up automated Test Harness and run one test end-to-end from CI environment to ARIA-AT App 19:16:13 Seth - Get buy in broader stakeholders including AT vendors and all browser vendors 19:16:21 jesdaigle has joined #aria-at 19:16:58 Seth - Plan alignment with web developer tools including WebDriver API semantics and DevTools integration. We're really building a tool that's similar to the WebDriver API. Can we make our tool more accessible to web developers & their current expectations on what an automated tool like this should do 19:17:19 Seth - speaking of, we should think about how a driver API gets built on by something like DevTools 19:18:00 Seth - that's the "what". The "how" is we'd like to center marginalized AT developers, manual testers, web devs, and others as co-designers, mentor 1-2 beginners from under-represented backgrounds in contributing to the AT Automation project 19:18:40 Seth - feedback? 19:19:07 michael_fairchild - how do we measure the success of some of these? At a high level, is Matt_King aligned with this list? 19:20:10 Matt_King - I had same question about metrics, some success criteria are more imp. than others. I also want to make sure we're aligned on prioritization, esp. when it comes to sequencing. Sever dims. to consider — feasibility & scaling up what we've already done (ability to run a whole suite) 19:21:38 Matt_King - also broadly thinking about how we prioritize the desktop OS vs. mobile OSs, where automation will be a lot trickier. I want to get to a place ASAP in the year where people can take our prototypes and prove feasibility in other envs. Like I have a FB product team, would like to prove it w/ them inside our own env. 19:22:19 michael_fairchild - re: "Unify manual and automated test formats and incorporate test sources beyond APG", I think the first part is really important. The last half could be a separate criteria? 19:23:09 Matt_King - agreed, we should be able to treat the source as a black-box. I'd expect that to be managed by the ARIA-AT app 19:23:54 Seth - I agree, this is somewhat of a shared criteria. I don't think it's necessarily a goal to have the driver run a test from another source. It would be that the driver works with open test formats that allow for expansions 19:24:17 Matt_King - this workstream is going to inform ARIA-AT app more than vice versa 19:25:02 michael_fairchild - what does the mentoring aspect look like in practice? 19:25:56 Seth - still working it out, we're (Bocoup) working on a x-work stream technique. This is something on our plate, we don't expect it to create work for anyone else (unless they're interested) 19:27:24 Matt_King - vision is Bocoup plays "teach the teacher" role. Members of the community can use Bocoup created resources/training to repeat outreach. Zooming out - this project can only succeed w/ a lot more people involved 19:27:40 michael_fairchild - are they volunteers, people we're paying? 19:28:29 Seth - paying is on the table. We want people to feel empowered, overcome barriers of W3C process 19:29:21 michael_fairchild - re "Get buy in broader stakeholders including AT vendors and all browser vendors", how do we measure? What level of buy in are we expecting? 19:30:04 Matt_King - gotta figure out success criteria. We want AT venders to actually use our deliverables. Even better, rely on 19:30:57 zcorpan - there are diff. levels of buy-in, but eventually AT & browser vendors should use the tests internally as regression tests, just like they're doing w/ their platform tests 19:31:16 s/their platform tests/web-platform-tests/ 19:31:35 michael_fairchild - re "Plan alignment with web developer tools including WebDriver API semantics and DevTools integration", how does DevTools integration work? 19:32:30 Seth - mostly a placeholder, this is more future. We would be successful if we had written down any plan on how we will align the low-level spec w/ a higher level project vision for web devs and end users 19:34:08 zcorpan - I think DevTools is mentioned here b/c they cur. use WebDriver as their API. & at least in some browsers, you can hook up a mobile phone and talk to it from DevTools. So you can debug on your phone through tools on desktop 19:34:59 boazsender - maybe next year, thinking about other ways for rendering sonic interfaces 19:35:08 scribe: boazsender 19:35:24 weston: along the lines of devtools, we've talked about browser pairings, but what about native applicaitons? 19:35:28 scribe: weston 19:37:08 Matt_King - not in near term. Certainly something W3C is looking at, we don't have anything like ARIA/APG for native apps 19:37:23 no worries 19:38:39 Matt_king - we do compare ARIA to native HTML and native app implementations. We want to make sure we don't have discrepencies. AT vendors will insist on that though 19:38:56 weston - will AT vendors insist that any drivers they build work with native apps & web? 19:39:24 Matt_King - might happen anyway 19:40:09 zcorpan - even if we only care about browsers, I believe browser vendors would want to automate their chrome as well 19:40:27 Matt_King - need to be careful that consider doesn't create scope creep 19:41:23 TOPIC: Current Priorities 19:42:12 Seth - Recruit and work with co-design experts on automated test format. I think we'll want to do this up-front b/c on the ARIA-AT app side, we'd like to tackle test format changes ASAP 19:43:12 Seth - Implement AT Drivers (NVDA, Narrator, VoiceOver MacOS, VoiceOver iOS, TalkBack) 19:43:55 Matt_King - JAWS instead of Narrator? But if MS wants to provide funding, I'm all for it 19:44:29 weston - FYI JAWS has scripting, where I don't think Narrator does 19:44:52 Matt_King - maybe could use UIA? Someone from MS mentioned it 19:45:06 Seth - work here might include developing a x-AT runtime for Windows 19:45:46 zcorpan - came up working on the prototype, someone suggested that it might be possible to have the same implementation for all screen readers on Windows 19:46:04 Matt_King - Sina had mentioned that for capturing output, the SAPI modules could be useful 19:46:38 Matt_King - I'm less clear about the driving side - how "command X" could be the same 19:47:35 Matt_King - other thing that makes me think it's possible is we now have a std. interface for Braille drivers. Maybe could tap into something like that. Problem is braille and speech are diff, not a good substitute 19:48:58 Simon - Write draft specification for AT Driver protocol & explore integrating with BiDirectional WebDriver Protocol, Unify ARIA-AT and AT Driver test formats 19:49:20 Matt_King - how is WebDriver standardized? Is it W3C? 19:49:29 zcorpan - it's W3C by browser testing & tools WG 19:49:43 https://w3c.github.io/webdriver/ 19:50:34 Simon - Align ARIA-AT / AT Automation project governance. A bit fuzzy, but want to make sure automation and ARIA-AT app are aligned 19:50:42 Seth* 19:51:12 s/Simon - /Seth: / 19:51:22 s/Seth*// 19:51:33 Seth - Create automated test runner, to work in CI. Mostly about putting pieces in place than production-ready. ARIA-AT app will likely need API to submit automated test results to 19:51:53 Seth - Write user documentation and propose features to align with web developer use cases 19:52:20 scribe: boazsender 19:52:49 weston: I have very little idea about how we'd collect output from android/ios. it would be great to have a vague idea of how that would be done 19:52:55 scribe: weston 19:54:55 Matt_King - both input & output of driver have a dependency on our data model, right? 19:56:43 Seth - the tests we're writing should be able to produce both a manual & automated version (maybe some just manual). That format is the input to the AT automation, then we have the format of assertions/results. Cur. the app displays results in a table for manual tests. We might not need to do much to integrate automated into there. Or automated might need to add data. Design process crosses all work streams 19:57:12 Matt_King - it feels like the input part is way more important than the output. Output can be transformed w/out much rework 19:58:25 Seth - zcorpan has a proposal for automated test format, goal is to describe automated in as much detail as manual 19:58:57 Matt_King - is validating that format pri. 1? Pre-req. to writing driver code? 19:59:25 zcorpan - yes, we should do a research spike to understand the 3 diff. screen readers & understand how they're different 19:59:37 Matt_King - ok so that's even more imp. than validating format of tests 20:00:04 michael_fairchild - +1, my gut concern is that we haven't proved things out on MacOS VoiceOver, and that might inform how we write tests 20:02:06 Matt_King - for VoiceOver, do similar to NVDA. Do whatever hacks you have to do to make it run. Then for JAWS, I wonder if it'll be the hardest. Even though it has scripting... but it's not in a headless way. And there's no JAWS CLI. Ex: no way to start & run script from CLI. Will we have to get support from Vispero? What is the lightest-weight thing they could do? Would they be willing to share source? 20:02:39 michael_fairchild - so highest pri. is research spike, understand VoiceOver MacOS & JAWS. Are mobile SRs in spike? 20:03:35 Matt_King - hard to say... I'm thinking mobile is in next year's scope. I don't want to plow blindly ahead either. Some level of research there, not as far as writing for NVDA 20:04:01 Seth - spend a little time now to save a bunch of time later 20:04:27 Matt_King - yeah let's have intelligent convos with Apple and Google. Let's leave time for those convos 20:05:03 Matt_King - we have a basic approach w/ Apple already, but haven't talked to Vispero. Need to find out what we're up against 20:05:48 zcorpan - for the feasibility spike, I envision learning about the path to automation for a particular SR. Communicate that a vendor needs to build something early 20:06:08 scribe: boazsender 20:06:23 weston: are we distinguishing between mobile emulators vs devices? 20:06:28 scribe: weston 20:06:31 q+ 20:07:45 Matt_King - no line in the sand yet, we know that there are things you can't test in a simulator. We tell internal devs to use real devices. If our convo were to drive the convo w/ Apple to improve robustness, that would be great 20:08:29 boazsender - is there a risk to deferring the mobile prototype? How much could it impact/churn the standard? I've seen the 4 year process involved w/ updating WebDriver 20:09:58 Matt_King - concerns - there are no API mappings for Android & iOS for ARIA. The issue has been raised with the ARIA WG. I want to make sure that we achieve our goals for the desktop platforms by mid-next years, so we can 100% focus on mobile for 2022-23 20:10:59 Matt_King - agreed it's worth a few weeks of investment now to figure out what we're up against 20:12:10 Matt_King - let's find out what we don't know now, make sure investment roadmap can fully contain mobile. We might need to seek more funding & investment from MS/Google/Apple. If we demonstrate value in the places we know we can (desktop), it will really help 20:18:16 Seth - agreed, the further we get with a draft, the easier it will be to get buy-in 20:20:14 Matt_King - most concerned about getting AT vendors on board. This isn't going to work if they don't participate 20:21:28 TOPIC: Closing 20:21:49 Seth - next steps are to turn this into a fuller 20:22:13 Matt_King - need research spike first, prioritize prototyping VoiceOver & JAWS, but we need a research spike on all of them 20:23:16 zcorpan - for test writing maintainability, we want to have a single format that you can generate manual tests & automated tests from. Will req. changes to our cur. format 20:23:43 scribe: boazsender 20:24:36 weston: with how the prototype is written vs how jaws and nvda are implemented, I think there might have to be some timeouts and stuff, so I'm worried about having a single format across tests that need timeouts vs not. 20:24:57 simon: hopefully test authors dont need to care about timeouts 20:25:00 weston: yah 20:25:10 scribe: weston 20:25:24 Simon - weston, we can invite you to discussions on the test formats 20:25:35 s/Simon/Seth/ 20:26:27 Simon - logistics, we won't start billable work on this project until late March. Need to describe cadence for checkins 20:27:15 Simon - we'll defer deciding cadence until later 20:27:30 Seth* 20:28:39 rrsagent, make minutes 20:28:39 I have made the request to generate https://www.w3.org/2021/02/24-aria-at-minutes.html Matt_King