18:58:43 RRSAgent has joined #aria-at 18:58:47 logging to https://www.w3.org/2023/10/23-aria-at-irc 19:00:09 rrsagent, make log public 19:00:23 Zakim, start the meeting 19:00:23 RRSAgent, make logs Public 19:00:24 please title this meeting ("meeting: ..."), jugglinmike 19:00:41 meeting: Bi-Weekly Meeting of Assistive Technology Automation Subgroup of ARIA-AT Community Group 19:01:02 murray_moss has joined #aria-at 19:01:16 jugglinmike has changed the topic to: 2023-10-23 meeting 19:01:34 Matt_King has joined #aria-at 19:01:47 Sam_Shaw has joined #aria-at 19:01:47 present+ jugglinmike 19:03:55 present+ 19:04:05 present+ 19:04:06 scribe+ 19:04:27 https://www.w3.org/events/meetings/cff6ac42-bdc2-4a38-8993-7dd46e507741/20231023T120000/ 19:06:11 TOPIC: field questions on AT Driver from JAWS maintainers 19:06:23 Folks from Vispero have begun reviewing the AT Driver proposal, and they have some questions they'd like to ask the group. 19:07:07 Present+ Brett Lewis 19:07:21 present+ Glen Gordon 19:07:57 BL: I'm a software engineer at Vispero, we are interested in implementing a version of this driver for JAWS 19:08:40 GG: I've been working on JAWS since 1994, we have reviewed the Go Server and NVDA Add on 19:10:09 present+ Michael Fairchild 19:11:00 BL: First question, is there a newer version of the protocol we should be reviewing? 19:11:04 michael_fairchild has joined #aria-at 19:11:28 present+ 19:11:48 https://w3c.github.io/at-driver/ 19:12:04 jugglinmike: I just pasted a link to the updated reademe 19:12:41 BL: Could you give us an overview of how this process would work, and how would we test? 19:13:05 GG: To test with NVDA today, whats missing from running a test today? What are the steps we would take? 19:13:52 JS: There is a split responsibility here, when PAC implemented the SPEC, there wasn't any additional tooling for testing, or simulation, or running ARIA AT Test. I will hand this to Mike 19:14:21 MK: I will interject, right now the driver doesn't make any assumptions about how you are going to construct your test 19:15:27 MK: Bocoup is developing the middleware that uses the AT Driver to run the test. If you want to recreate what we are doing with Bocoup thats great, but we aren't planning to standardize this. 19:16:06 GG: We want to know when we are done. If we have to create our own tooling, how will we know that this is working? 19:16:39 JS: In order to determine the implementation of the spec is correct, there needs to be a test suite, in terms of I sent this command and I got this response 19:17:32 JS: If that testing exists to put an implementation through its paces, you can be sure its working and complete. I assume you are looking for a some kind of test suite that makes sure all the actions are handled correctly 19:18:10 GG: We can send keystrokes and allow you to collect speech, but that doesn't check if you got all the speech, or the correct speech in the right amount of time? 19:19:02 jugglinmike: Its pretty agnostic about these things, we haven't specified behavior at that granularity. It would be hard to express normative behaviors 19:19:50 jugglinmike: We haven't yet developed conformance tests, but it may be early for that, this is just a proposal, its not on a standards track yet, we aren't there yet but are looking for implementer feedback 19:20:03 jugglinmike: The work that Bocoup is doing is open source 19:21:00 jugglinmike: We have created a harness that is a no JS tool, that consumes tests and speaks this protocol to run them. You could use ARIA AT to validate your test but that's not its intent 19:22:11 MK: The protocol we are talking about is AT Driver. Thats what you would implement in JAWS. Thats the part of the project we want to standardize. The reliability of that protocol is what Glen you are pointing toward. How do we know the speech is complete? 19:22:17 This is the "harness" that Bocoup is developing in service of ARIA-AT: https://github.com/w3c/aria-at-automation-harness 19:22:21 present+ Matt King 19:23:02 JS: We don't specify when a SR is finished speaking, because its never done 19:23:29 JS: The input you send in the form of commands, and the event stream are intentional decoupled. 19:24:15 JS: Its the responsibility of the consuming product to determine when it feels it has received the correct output 19:25:19 GG: At STG, what would you show off? 19:26:01 MK: What we are implementing is when you go to ARIA AT website, there is a page called the test queue, which we can give you access to, which contains runs that are executed by the bot 19:26:24 MK: We want to show, here's how a human runs the test, and how a bot runs the test. Compare the results 19:26:50 MK: The harness piece of it is the piece that runs the test and calls the AT Driver. 19:27:09 MK: Mike they don't have to have our whole server setup to run the ARIA AT Tests right? 19:27:12 jugglinmike: Right 19:27:39 BL: So we need this node.js project/app that generates the commands that are then sent on by the Go server to NVDA itself? 19:27:45 JS: Correct 19:27:58 BL: The the NVDA addon sends back the output 19:28:27 BL: The Go server is just a bridge 19:29:25 GG: If we were to implement something similar, would you be willing to make the Go server changes 19:29:40 JS: We could, but we will need to discuss with Matt first 19:30:02 GG: This kind of seems like we should implement this later. What is the goal of the presentation? 19:31:17 MK: I'm not expecting a terribly involved demo, I really want to show the capabilities and the progress of the project. The biggest value to us, if you have a new version of JAWS coming out in December, we don't have enough human power to rerun all the test for all the test plans and figure out the differences. 19:31:34 MK: We want to be able to re run these tests and get the data out as fast as possilb 19:32:08 BL: The speech output doesn't seem that complicated, we just don't know anything about Go. We don't know about the test harness either 19:32:29 MK: We wouldn't expect you to work with the harness. James how much is the Go server is NVDA specific? 19:33:08 JS: I think whats important to consider is that none of the communication between the GO host and client, is standardized or spec'd out, it only lives on the Go server and NVDA Add on 19:33:24 JS: The Go server is a black box 19:33:52 JS: Essentially you may find it easier to implement the spec into JAWS, rather than using the Go server 19:34:12 JS: Then you don't have to parse out the JAWS output 19:34:36 MK: The Go server lets you restart the Screenreader 19:34:54 JS: Correct, the SC may crash, all of that can communicated to the remote party 19:35:34 JS: It also allows the communication between the Go server and NVDA to change. If we wanted to formalize the communication we could make that change without the user knowing 19:35:48 GG: We want to build as much as we can on what you James have built 19:36:32 JS: When we made the decision to have an opensource server, we intended this extensibility 19:36:59 JS: If you choose to implement the communication between our binary server and your Screen reader, that process isn't documented 19:37:03 GG: Or complicated! 19:37:35 JS: Right, but i dont want to make any assumptions, we have to dicuss with Bocoup how much of our scope should be spent adapting the Go Server 19:37:51 JS: Currently Go Server work is outside our scope 19:38:16 MK: To have that conversation, we would need to know what the extent of changes and timeline 19:38:53 JS: Right, we could also make the server more flexibile, so that the user of the server could provide the commands so its not AT specific 19:39:07 BL: Move it towards a node application 19:39:19 JS: Exactly. That might be easier, it would make it more flexible 19:40:52 jugglinmike: There's also the alternative of forking the Go Server. A lot of the value of a standard is to not have the coordination overhead, like maintenance. Like James is saying there isn't much documentation because there isn't a need to create it. 19:41:23 BL: So you're arguing the SR should replace the Go server 19:41:38 jugglinmike: Sort of, I'm not saying we should get rid of the Go Server. 19:42:04 BL: But moving the onus of documentation to the SR is more plausible to maintain 19:42:28 jugglinmike: That also would imply that the SR maintains it and its more stable over time 19:43:36 jugglinmike: If we extend the Go server to support JAWS, we may have more maintenance long term 19:44:14 MK: If we fork it, then the maintenance is not longer dependent on the supporting other AT 19:46:04 JS: What I would to see, is that each AT that is involved, has its own implementation of the protocol. Each vendor is capable of deciding how to communicate and maintain security. 19:46:54 JS: So that each SR has its own binary that it maintains to make adjustments 19:47:25 JS: we are fully hopeful that in the future that NVAccess will make their own implementation of our ad on 19:47:59 JS: I'd love to see a suite of tools that you can download to use to test any AT 19:54:35 MK: When it comes to anything related to using the protocol, its within the scope of Bocoups work. 19:55:19 MK: All of that work is opensourced. We can discuss implementing modifications that would help you 19:55:48 BL: How would we go about testing our implementation? Do you have a set of cannonical commands? 19:56:00 MK: Mike do you have anything right now you can give them? 19:56:36 jugglinmike: I linked to the harness repo, I want to give a caveat though this has been an internal document so there may be rough edges to this 19:56:44 jugglinmike: Please file any issues you find 19:58:35 jugglinmike: Even with the harness, the determination of "reasonable" will still be on you, because of the generic nature of ARIA AT. 19:59:08 MK: Would it be possible to share the code of known good outputs? 19:59:20 jugglinmike: I don't think so, that more tied to the ARIA AT app 20:00:15 jugglinmike: One way to use ARIA AT outside the project, is a snapshot application to determine changes between versions 20:00:47 jugglinmike: Finally, I want to say because it hasn't come up, we are very much inspired by the Webdriver project, all its tooling and history 20:01:34 jugglinmike: Its where we get alot of our language, why we use web sockets, and alot of our decisions are based on Web driver 20:01:46 GG: Thank you! 20:02:32 MK: Okay we will find the best win win for you 20:03:21 MK: It would be great to present at STG, but would be amazing is if we had a tool to run test with when JAWS 2024 is released 20:03:23 rrsagent, make minutes 20:03:24 I have made the request to generate https://www.w3.org/2023/10/23-aria-at-minutes.html Sam_Shaw 20:06:44 Matt_King_ has joined #aria-at 20:53:37 jugglinmike has joined #aria-at 20:54:24 jugglinmike has changed the topic to: