19:00:21 <RRSAgent> RRSAgent has joined #aria-at
19:00:25 <RRSAgent> logging to https://www.w3.org/2023/08/28-aria-at-irc
19:00:45 <jugglinmike> Topic: Bi-Weekly Meeting of Assistive Technology Automation Subgroup of ARIA-AT Community Group
19:01:20 <jugglinmike> rrsagent, make logs public
19:01:31 <jugglinmike> Zakim, start the meeting
19:01:31 <Zakim> RRSAgent, make logs Public
19:01:32 <Zakim> please title this meeting ("meeting: ..."), jugglinmike
19:02:12 <Sam_Shaw> Sam_Shaw has joined #aria-at
19:02:14 <jugglinmike> Meeting: Bi-Weekly Meeting of Assistive Technology Automation Subgroup of ARIA-AT Community Group
19:04:22 <murray_moss> murray_moss has joined #aria-at
19:05:02 <murray_moss> present+
19:05:55 <michael_fairchild> michael_fairchild has joined #aria-at
19:06:47 <Sam_Shaw> present+ James_Scholes
19:06:52 <Sam_Shaw> present+
19:07:13 <mzgoddard> mzgoddard has joined #aria-at
19:07:25 <jugglinmike> present+ jugglinmike
19:07:27 <Sam_Shaw> scribe+
19:09:41 <Sam_Shaw> TOPIC: Update on the design and implementation of ARIA-AT Automation
19:11:16 <Sam_Shaw> present+ Matt_King
19:12:11 <Sam_Shaw> jugglinmike: I can give an update on the Design Implementation. The User interface design is being led by Izak which we will dicuss later
19:13:33 <Sam_Shaw> jugglinmike: First we need to decide where this work is going to be done. In the design documents we refer to this work as a AT collection jobs. We need to decide what kind of computer will run these jobs and where will it live?
19:13:50 <Sam_Shaw> jugglinmike: The GitHub action service has all of capabilities to run the actions we need
19:14:43 <Sam_Shaw> jugglinmike: We have in mind a solution that will create work orders by pushing to a repo. We can utilize the system that is designed around PRs
19:15:12 <Sam_Shaw> jugglinmike: We are considering this because it limits the CG from having to manage this service
19:15:51 <Sam_Shaw> jugglinmike: This avoids having the manage updates to machine, and instead are performed by GitHub/Microsoft. We also avoid having to know how to manage a server
19:16:18 <Sam_Shaw> MK: Two questions, for the implementation alternative voice, this is using the NVDA plugin that Job wrote correct?
19:16:26 <Sam_Shaw> jugglinmike: No it is using the generic driver
19:16:45 <Sam_Shaw> jugglinmike: It using microsoft speech api
19:17:04 <Sam_Shaw> jugglinmike: the microsoft sdk to simulate key presses
19:17:21 <Sam_Shaw> MK: Since we have the NVDA plugin ready, why are we starting with the Generic Driver?
19:18:00 <Sam_Shaw> jugglinmike: The driver is something we have built and mainintaned at Bocoup, we already have alot of the work that we needed to get that running done already
19:18:21 <Sam_Shaw> MK: Okay well we have to figure out how the NVDA driver fits into the schedule
19:18:35 <Sam_Shaw> MK: Also the Generic driver has a few shortcomings
19:19:23 <Sam_Shaw> jugglinmike: I understood that Job no longer works at PAC though
19:19:32 <Sam_Shaw> MK: That doesn't mean that PAC will stop working on it
19:20:33 <Sam_Shaw> jugglinmike: I wasn't anticipating the merits of a full task test suite. We can look into using the NVDA add on PAC developed
19:21:05 <Sam_Shaw> MK: The goal of the project is to use many AT's
19:21:46 <Sam_Shaw> JS: As Matt mentioned PAC's work on NVDA is stoping because Jobs doesn't work here. PAC has implemented the first four milestones of the spec, so in our minds that work is complete
19:22:29 <Sam_Shaw> JS: This isn't a separate tool, its an implementation of the Spec, that is built into NVDA instead of the system voice. This isn't radically different than what already exists
19:22:44 <Sam_Shaw> michael_fairchild: Regardless we still need to decide on infrastructure right?
19:24:37 <Sam_Shaw> MK: Yes, I'm hessitant to build much infrastrcuture around the generic driver, unless its the only way forward. If we have to wait a couple years for Apple to develop something than that might be a time to work on the generic driver
19:25:19 <Sam_Shaw> MK: Thats good about the GitHub option, we can approach Microsoft about a longer term support of the project. I wonder about the risks of the GitHub approach
19:26:16 <Sam_Shaw> jugglinmike: Okay I'm not prepared to have a infrastructure design discussion today, but I'm happy to set that up in the future. If  we want to open this discussion up I'm more than happy to do that
19:26:33 <Sam_Shaw> MK: I don't know if we need to be involved at that granular level, Just wondering about the risks
19:27:36 <Sam_Shaw> jugglinmike: Rather than having a server that we know is idle, we are constrained by the same way any CICD user would be. There's a risk we have more work that GitHub can process immediately, so we run up against creating queues
19:27:52 <Sam_Shaw> Z: Would we need a server set up outside GitHub to send requests?
19:27:55 <Sam_Shaw> jugglinmike: Yes
19:28:13 <Sam_Shaw> jugglinmike: Here is a link to the design document
19:28:49 <Matt_King> Matt_King has joined #aria-at
19:29:08 <Sam_Shaw> jugglinmike: With this strategy we want something that can manage jobs.
19:29:17 <Sam_Shaw> We want a consistent interface for the app to work with
19:29:41 <Sam_Shaw> Z: GitHub still has a six hour limit on a job on the runner
19:29:56 <Sam_Shaw> Z: You can have multiple runners running simultaneously
19:30:17 <Sam_Shaw> MK: So those kinds of concerns are not a problem when we are manually kicking off every single automated job
19:30:24 <Sam_Shaw> MK: Our jobs wont be that long
19:31:06 <Sam_Shaw> MK: We dont even have that many to run right now, however in the future we are going to be in a place where we will be automatically running a lot of jobs, like when a new version of JAWS is released
19:31:55 <Sam_Shaw> MK: This approach seems fine to me
19:32:08 <Sam_Shaw> JS: I like the idea alot, especially someone else managing the service
19:32:54 <Sam_Shaw> JS: Capacity is an issue, GitHub will only allow us so much. If we are still running a server to send the instructions, I'm not sure what the maintenance cost is saving is
19:33:11 <Sam_Shaw> michael_fairchild: Is there anything in the terms of service that would prevent us from using this?
19:33:15 <Sam_Shaw> jugglinmike: I can look into it
19:34:01 <Sam_Shaw> Z: I would expect there isn't an issue, but there may be some limitations we need to consider. There are also limits on a public repo with how many jobs will run simultaneously.
19:34:13 <Sam_Shaw> Z: There are less mac runners than windows or linux
19:34:32 <Sam_Shaw> Z: That won't stop us, but could be helpful to know.
19:36:05 <Sam_Shaw> Z: If we do run jobs in parallel what are the limitations from GitHub?
19:36:17 <Sam_Shaw> JS: For the free plan its 20, 5 for MAc
19:36:29 <Sam_Shaw> MK: I'm not sure if we are in the free tier
19:37:09 <Sam_Shaw> JS: The max runners you can have at any time is 5 even with a paid service. The only way you can have more is by hosting your own instance or having a Git Hub organization
19:37:23 <Sam_Shaw> MK: For our current needs we're only running one test plan at a time
19:38:52 <Sam_Shaw> MK: Right now an admin will run a kick off a plan manually. I think the GitHub Actions approach is a good find, we need a way to move forward to create this UI now
19:39:06 <Sam_Shaw> MK: Then we can rerun plans when new version of AT get released
19:39:18 <Sam_Shaw> JS: In terms of the terms of use, thats a legal question
19:40:17 <Sam_Shaw> JS: If its a separate repo as was suggested, it may conflict
19:40:34 <Sam_Shaw> MK: Ahh yes the dummy repo would have to be directly associated with the project
19:41:00 <Sam_Shaw> MK: Lets not try to parse the terms of use right now. I would recommend if there is someone at Bocoup that can review this, have them take a look.
19:41:56 <Sam_Shaw> MK: As ARIA At is in the w3c org, this could be a W3c repo which would would have to run it by them, same case for bocoup
19:45:26 <Sam_Shaw> JS: For NVDA, my understanding is that any system running automated tests should be written against the spec, outside the operations
19:46:24 <Sam_Shaw> TOPIC: Design discussion: automation UI in ARIA-AT App
19:46:30 <Sam_Shaw> Issue 732
19:46:40 <jugglinmike> https://github.com/w3c/aria-at-app/issues/732
19:47:07 <Sam_Shaw> jugglinmike: This has some mockups about the UI from Isaac
19:47:20 <Sam_Shaw> jugglinmike: We've started the discussion, Matt has given us some feedback
19:47:32 <Sam_Shaw> jugglinmike: Do folks need some time to review this?
19:47:56 <Sam_Shaw> jugglinmike: Right now we are getting into the extent we are reflecting the AT collection system as another user.
19:50:47 <Sam_Shaw> MK: Isaac initial proposal was that you assign it to a bot and that kicks off the process
19:52:10 <Sam_Shaw> jugglinmike: What is the name of the thing that is evaluating the test plan run?
19:52:57 <Sam_Shaw> JS: Do we have a term for this? We are gathering results and assigning verdicts
19:53:14 <Sam_Shaw> MK: The thing can sit in the queue and have states, partially done, complete
19:53:27 <Sam_Shaw> MK: When a report is complete we can mark it as final
19:55:21 <Sam_Shaw> MK: Ahh yes the report is amalgamation of test plan runs by different people.
19:55:47 <Sam_Shaw> MK: So we will have a primary run
19:55:55 <Sam_Shaw> MK: So what is the thing that is assigned to someone
19:56:08 <Sam_Shaw> JS: Just because some is assigned, it doesn't become a test plan run until they run it
19:56:23 <Sam_Shaw> MK: Thats a implementation details, as to when the test plan is created in the database
19:56:35 <Sam_Shaw> MK: What do we can the unassigned thing? An entry in the test queue
19:57:04 <Sam_Shaw> MK: That in effect is just app specific, an implementation detail, what we call the thing that isn't assigned, the thing that is assigned we call that test plan run
19:57:17 <Sam_Shaw> MK: You can call it whatever as the working mode doesn't care
19:57:20 <Sam_Shaw> jugglinmike: Okay
19:57:58 <Sam_Shaw> jugglinmike: I will say in passing, but it may seem unintuitive for people to see a test queue that isn't a queue of tests.
19:58:14 <Sam_Shaw> jugglinmike: We almost want to call it a test queue entry queue
19:58:42 <Sam_Shaw> MK: The test queue is just the work that needs to be done, the name isn't confusing anyone
19:58:54 <Sam_Shaw> jugglinmike: Its informing how we discuss the design
19:59:12 <Sam_Shaw> jugglinmike: so naming does mater in communciation
20:00:10 <Sam_Shaw> JS: My concern is that we are doing this frequently, but the people who are using the app regularly the terms make sense. We spend alot of time in these calls naming things that already exist, I'm not sure how this helps
20:00:18 <Sam_Shaw> jugglinmike: These are things that don't exist yet
20:00:47 <Sam_Shaw> MK: i'm not sure it needs a name, when we say add to queue, its a test plan with a combo of AT and browser, its basically an unassigned test plan run
20:01:07 <Sam_Shaw> MK: As soon as you assign someone its an assigned test plan run
20:01:16 <Sam_Shaw> JS: But you only have one entry in the queue
20:01:20 <Sam_Shaw> MK: Yes
20:01:45 <Sam_Shaw> MK: In the design are you good with the idea in the reports dialog how to reassign a test to the bot?
20:02:01 <Sam_Shaw> jugglinmike: This is what I'm getting at, to treat the bot like a user, there are two steps
20:02:09 <Sam_Shaw> jugglinmike: create test plan run and assign it
20:02:41 <Sam_Shaw> jugglinmike: if we always create a unassigned test plan run, then assign it, it simpifies
20:03:03 <Sam_Shaw> MK: Thats fine, it just would save the admin some time to be able to assign the test plan run when creating it
20:03:23 <Sam_Shaw> JS: But if a human needed to re add the test to the queue, you would need to add it and assign it
20:05:30 <Sam_Shaw> JS: Personally, I dont think it would be to much to create the test and then assign it in an additional step
20:05:45 <Sam_Shaw> JS: Assuming there are no conflicts, do we just republish it?
20:06:58 <murray_moss> murray_moss has left #aria-at
20:09:20 <Sam_Shaw> rrsagent, make minutes
20:09:22 <RRSAgent> I have made the request to generate https://www.w3.org/2023/08/28-aria-at-minutes.html Sam_Shaw