Meeting minutes
Bi-Weekly Meeting of Assistive Technology Automation Subgroup of ARIA-AT Community Group
Update on the design and implementation of ARIA-AT Automation
jugglinmike: I can give an update on the Design Implementation. The User interface design is being led by Izak which we will dicuss later
jugglinmike: First we need to decide where this work is going to be done. In the design documents we refer to this work as a AT collection jobs. We need to decide what kind of computer will run these jobs and where will it live?
jugglinmike: The GitHub action service has all of capabilities to run the actions we need
jugglinmike: We have in mind a solution that will create work orders by pushing to a repo. We can utilize the system that is designed around PRs
jugglinmike: We are considering this because it limits the CG from having to manage this service
jugglinmike: This avoids having the manage updates to machine, and instead are performed by GitHub/Microsoft. We also avoid having to know how to manage a server
MK: Two questions, for the implementation alternative voice, this is using the NVDA plugin that Job wrote correct?
jugglinmike: No it is using the generic driver
jugglinmike: It using microsoft speech api
jugglinmike: the microsoft sdk to simulate key presses
MK: Since we have the NVDA plugin ready, why are we starting with the Generic Driver?
jugglinmike: The driver is something we have built and mainintaned at Bocoup, we already have alot of the work that we needed to get that running done already
MK: Okay well we have to figure out how the NVDA driver fits into the schedule
MK: Also the Generic driver has a few shortcomings
jugglinmike: I understood that Job no longer works at PAC though
MK: That doesn't mean that PAC will stop working on it
jugglinmike: I wasn't anticipating the merits of a full task test suite. We can look into using the NVDA add on PAC developed
MK: The goal of the project is to use many AT's
JS: As Matt mentioned PAC's work on NVDA is stoping because Jobs doesn't work here. PAC has implemented the first four milestones of the spec, so in our minds that work is complete
JS: This isn't a separate tool, its an implementation of the Spec, that is built into NVDA instead of the system voice. This isn't radically different than what already exists
michael_fairchild: Regardless we still need to decide on infrastructure right?
MK: Yes, I'm hessitant to build much infrastrcuture around the generic driver, unless its the only way forward. If we have to wait a couple years for Apple to develop something than that might be a time to work on the generic driver
MK: Thats good about the GitHub option, we can approach Microsoft about a longer term support of the project. I wonder about the risks of the GitHub approach
jugglinmike: Okay I'm not prepared to have a infrastructure design discussion today, but I'm happy to set that up in the future. If we want to open this discussion up I'm more than happy to do that
MK: I don't know if we need to be involved at that granular level, Just wondering about the risks
jugglinmike: Rather than having a server that we know is idle, we are constrained by the same way any CICD user would be. There's a risk we have more work that GitHub can process immediately, so we run up against creating queues
Z: Would we need a server set up outside GitHub to send requests?
jugglinmike: Yes
jugglinmike: Here is a link to the design document
jugglinmike: With this strategy we want something that can manage jobs.
We want a consistent interface for the app to work with
Z: GitHub still has a six hour limit on a job on the runner
Z: You can have multiple runners running simultaneously
MK: So those kinds of concerns are not a problem when we are manually kicking off every single automated job
MK: Our jobs wont be that long
MK: We dont even have that many to run right now, however in the future we are going to be in a place where we will be automatically running a lot of jobs, like when a new version of JAWS is released
MK: This approach seems fine to me
JS: I like the idea alot, especially someone else managing the service
JS: Capacity is an issue, GitHub will only allow us so much. If we are still running a server to send the instructions, I'm not sure what the maintenance cost is saving is
michael_fairchild: Is there anything in the terms of service that would prevent us from using this?
jugglinmike: I can look into it
Z: I would expect there isn't an issue, but there may be some limitations we need to consider. There are also limits on a public repo with how many jobs will run simultaneously.
Z: There are less mac runners than windows or linux
Z: That won't stop us, but could be helpful to know.
Z: If we do run jobs in parallel what are the limitations from GitHub?
JS: For the free plan its 20, 5 for MAc
MK: I'm not sure if we are in the free tier
JS: The max runners you can have at any time is 5 even with a paid service. The only way you can have more is by hosting your own instance or having a Git Hub organization
MK: For our current needs we're only running one test plan at a time
MK: Right now an admin will run a kick off a plan manually. I think the GitHub Actions approach is a good find, we need a way to move forward to create this UI now
MK: Then we can rerun plans when new version of AT get released
JS: In terms of the terms of use, thats a legal question
JS: If its a separate repo as was suggested, it may conflict
MK: Ahh yes the dummy repo would have to be directly associated with the project
MK: Lets not try to parse the terms of use right now. I would recommend if there is someone at Bocoup that can review this, have them take a look.
MK: As ARIA At is in the w3c org, this could be a W3c repo which would would have to run it by them, same case for bocoup
JS: For NVDA, my understanding is that any system running automated tests should be written against the spec, outside the operations
Design discussion: automation UI in ARIA-AT App
Issue 732
<jugglinmike> w3c/
jugglinmike: This has some mockups about the UI from Isaac
jugglinmike: We've started the discussion, Matt has given us some feedback
jugglinmike: Do folks need some time to review this?
jugglinmike: Right now we are getting into the extent we are reflecting the AT collection system as another user.
MK: Isaac initial proposal was that you assign it to a bot and that kicks off the process
jugglinmike: What is the name of the thing that is evaluating the test plan run?
JS: Do we have a term for this? We are gathering results and assigning verdicts
MK: The thing can sit in the queue and have states, partially done, complete
MK: When a report is complete we can mark it as final
MK: Ahh yes the report is amalgamation of test plan runs by different people.
MK: So we will have a primary run
MK: So what is the thing that is assigned to someone
JS: Just because some is assigned, it doesn't become a test plan run until they run it
MK: Thats a implementation details, as to when the test plan is created in the database
MK: What do we can the unassigned thing? An entry in the test queue
MK: That in effect is just app specific, an implementation detail, what we call the thing that isn't assigned, the thing that is assigned we call that test plan run
MK: You can call it whatever as the working mode doesn't care
jugglinmike: Okay
jugglinmike: I will say in passing, but it may seem unintuitive for people to see a test queue that isn't a queue of tests.
jugglinmike: We almost want to call it a test queue entry queue
MK: The test queue is just the work that needs to be done, the name isn't confusing anyone
jugglinmike: Its informing how we discuss the design
jugglinmike: so naming does mater in communciation
JS: My concern is that we are doing this frequently, but the people who are using the app regularly the terms make sense. We spend alot of time in these calls naming things that already exist, I'm not sure how this helps
jugglinmike: These are things that don't exist yet
MK: i'm not sure it needs a name, when we say add to queue, its a test plan with a combo of AT and browser, its basically an unassigned test plan run
MK: As soon as you assign someone its an assigned test plan run
JS: But you only have one entry in the queue
MK: Yes
MK: In the design are you good with the idea in the reports dialog how to reassign a test to the bot?
jugglinmike: This is what I'm getting at, to treat the bot like a user, there are two steps
jugglinmike: create test plan run and assign it
jugglinmike: if we always create a unassigned test plan run, then assign it, it simpifies
MK: Thats fine, it just would save the admin some time to be able to assign the test plan run when creating it
JS: But if a human needed to re add the test to the queue, you would need to add it and assign it
JS: Personally, I dont think it would be to much to create the test and then assign it in an additional step
JS: Assuming there are no conflicts, do we just republish it?