19:02:55 <RRSAgent> RRSAgent has joined #aria-at
19:02:59 <RRSAgent> logging to https://www.w3.org/2025/04/14-aria-at-irc
19:07:10 <ChrisCuellar> Zakim, start the meeting
19:08:08 <ChrisCuellar> Next meeting is Monday, May 12
19:08:19 <ChrisCuellar> Next BTT meeting is May 14
19:08:32 <ChrisCuellar> First agenda item is maintenance of NVDA server
19:08:40 <ChrisCuellar> jugglinmike present+
19:09:31 <ChrisCuellar> jugglinmike: PAC wants to continue maintaining the NVDA server. They want to continue reviewing patches from Bocoup. We'll proceed with their suggestion.
19:10:06 <ChrisCuellar> Matt_King_: What's the primary impact for efficiency here?
19:10:29 <ChrisCuellar> jugglinmike: Working in Golang is the main efficiency here
19:10:44 <ChrisCuellar> Matt_King_: How high of a bar is working in Golang?
19:11:42 <ChrisCuellar> jugglinmike: We think we can timebox the high-priority issues here to 2 days
19:13:04 <ChrisCuellar> But long-term working in Golang may be a bigger lift for the future, bigger updates to the NVDA server. Also, it may impact future bug triage.
19:14:42 <ChrisCuellar> Matt_King_: I guess at the scale of the things we're talking about now, this shouldn't be a big concern. But for a long-term strategy, it sounds like there may be differences of opinion between Bocoup and PAC on how to build implementations of AT-Driver
19:15:03 <ChrisCuellar> jugglinmike: A more direct conversation on this may be more helpful for navigating that.
19:16:12 <ChrisCuellar> Matt_King_: Let's have a long-term strategy sync for AT-Driver implementations in the future. Right now, let's prioritize the mobile testing work and getting the Vispero implementation out.
19:16:33 <Carmen> Carmen has joined #aria-at
19:16:43 <ChrisCuellar> jugglinmike: How Vispero moves forward will also affect the direction of that conversation.
19:18:06 <ChrisCuellar> Matt_King_: Are we good in terms of decisions today? We will continue to do the NVDA work in the current repo in Go. At some point in the future, we'll have a conversation to review the learnings from the first couple years of AT Driver spec development and determine the best path forward.
19:18:27 <ChrisCuellar> Next topic: Collecting VO hints from the macOS AT driver server
19:19:21 <ChrisCuellar> jugglinmike: Bocoup is starting to collect stability reports in our testing automation infrastructure. We've had issues with consistency capturing VO utterances because of the indeterminacy of VO hints.
19:19:50 <ChrisCuellar> We're experimenting with adding more delays to our automation processes.
19:20:17 <ChrisCuellar> Matt_King_: The most rigorous method would be to wait until we get confirmation from VO that utterances are complete.
19:20:51 <ChrisCuellar> jugglinmike: We're raising this, because adding more delays to the bot will affect production runs of the bot.
19:21:48 <ChrisCuellar> Matt_King_: What are the current delays?
19:22:08 <ChrisCuellar> ChrisCuellar: We want to experiment with up 10 seconds or longer.
19:22:41 <ChrisCuellar> Matt_King_: We can probably get some get kind of spec info about VO Hint Timing from Apple
19:23:10 <ChrisCuellar> Matt_King_: Are there sources of latency that are different in CI from real life?
19:24:17 <ChrisCuellar> Maybe in Appescript for example?
19:24:25 <ChrisCuellar> jugglinmike: It's conceivable.
19:24:57 <ChrisCuellar> Most of my suspicion around differences in latency have to do with lower resources on CI machines.
19:25:18 <ChrisCuellar> Matt_King_: I certainly run into issues with VO not responding in real life.
19:25:53 <ChrisCuellar> jugglinmike: Really high delays will help us determine that.
19:26:21 <ChrisCuellar> Matt_King_: So we have a 5 second delay after every command?
19:27:27 <ChrisCuellar> jugglinmike: Correct. But we're not trying to change the amount of time in the production bot. We just want to experiment with how timing impacts consistency?
19:27:59 <ChrisCuellar> Matt_King_: In terms of AT-Driver, we should be figuring how to write the spec such that we can have performant automated testing
19:29:24 <ChrisCuellar> jugglinmike: Matt, you've previously suggested that ATs should describe the sources for the utterances. That could be a solution. I would want to discuss this with AT vendors. In any case, we will need an evented system of some kind to account for things like live regions, etc.
19:29:49 <ChrisCuellar> Matt_King_: It's a tricky question for sure. But we need to figure out how to make this kind of testing scalable.
19:33:22 <ChrisCuellar> jugglinmike: We've also been talking about annotating hint text. There might be runs with hint text enabled and with it disabled.
19:35:51 <ChrisCuellar> Matt_King_: We could probably just get the info about which text is a hint or not from a single test plan run. We don't need to determine that with every test plan run. Although, we might need to re-annotate that with each new VO version.
19:36:17 <ChrisCuellar> Next topic: Continued evaluation of project's reliance on Github Actions
19:36:54 <ChrisCuellar> jugglinmike: We're still not totally conclusive that Github's open-source policies will suit our needs. But we are pretty sure that it will be sufficient.
19:37:44 <ChrisCuellar> Separately, we've also tested possible limitations on macOS images and we didn't run into any unexpected throttling or limits.
19:38:12 <ChrisCuellar> I just want to temperature-check that everyone is still comfortable with where we're at?
19:38:58 <ChrisCuellar> Since we're still authoring test plans, it's hard to know for sure how much resources we'll eventually need for the testing infrastructure.
19:39:36 <ChrisCuellar> Matt_King_: The testing scope is inherently indeterminate, but we can make some confident extrapolations for the next 3-5 years.
19:40:12 <ChrisCuellar> But when do we want to add anything more to control for this kind of risk. I don't know what level of risk we're currently operating with.
19:41:25 <ChrisCuellar> What could possibly go wrong here? And what would we have to do to adapt to it? For example, if there were new specific limits imposed? We might be able qualify for possible exceptions to certain limits. There's too many hypotheticals right now.
19:41:43 <ChrisCuellar> Michael_Fairchild_: I agree.
19:42:12 <ChrisCuellar> jugglinmike: Is this something we need to address now? Or are folks comfortable with the status quo here?
19:44:20 <ChrisCuellar> Matt_King_: My gut reaction is to document the risk in a meaningful way that could be understood at a program-level for each of the ARIA-AT participants. Like documenting our known dependencies and requirements. I don't see any need to prioritize this more highly right now outside of documenting the possible risks.
19:45:01 <ChrisCuellar> Do we have a way of estimating or tracking our current usage?
19:45:13 <ChrisCuellar> jugglinmike: That only seems to be documented for paid plans.
19:45:24 <ChrisCuellar> We might have to build our own metering solution.
19:46:15 <ChrisCuellar> Matt_King_: Maybe we should have some operational metrics somewhere. Maybe in the app. It could be as simple as understanding how many bot run, or how many commands are run over a period of time.
19:47:06 <ChrisCuellar> jugglinmike: We can get this information from Github currently, but we it might be more manual to tabulate the info altogether.
19:47:30 <ChrisCuellar> Matt_King_: I still would like to have this info somewhere in the app. Especially for how long the bots are running.
19:48:35 <jugglinmike> scribe+ jugglinmike
19:48:51 <jugglinmike> ChrisCuellar: We've want to track how long the bots run in the consistency reports
19:49:02 <jugglinmike> ChrisCuellar: This conversation is making me think that data may be helpful in the app itself
19:49:23 <jugglinmike> ChrisCuellar: Which is to say, some of this may overlap with the app in ways we haven't originally planned
19:49:38 <jugglinmike> ChrisCuellar: Or, how much is an app concern versus a reliability concern for automation?
19:49:55 <jugglinmike> ChrisCuellar: We can think about this as Corey is building out the enhancements to the reports
19:50:13 <jugglinmike> present+ jugglinmike
19:50:19 <jugglinmike> present+ ChrisCuellar
19:50:29 <jugglinmike> present+ Matt_King_
19:51:18 <ChrisCuellar> scribe+ ChrisCuellar
19:51:59 <jugglinmike1> jugglinmike1 has joined #aria-at
19:52:35 <ChrisCuellar> Matt_King_: Having more info in an audit log, could help us tell more meaningful stories to communicate impact of the program.
19:53:11 <ChrisCuellar> In any case, we know we want to document the overall risks of our current infrastructure setup for now.
19:53:25 <ChrisCuellar> jugglinmike: That's it for now!
19:54:24 <jugglinmike1> RRSAgent, make logs public
19:55:33 <jugglinmike1> meeting: ARIA-AT AT Driver Subgroup Monthly Teleconference
19:58:31 <ChrisCuellar> RRSAgent, leave
19:58:31 <RRSAgent> I see no action items
20:00:01 <RRSAgent> RRSAgent has joined #aria-at
20:00:01 <RRSAgent> logging to https://www.w3.org/2025/04/14-aria-at-irc
20:00:29 <ChrisCuellar> RRSAgent, draft the minutes
20:00:30 <RRSAgent> I have made the request to generate https://www.w3.org/2025/04/14-aria-at-minutes.html ChrisCuellar
20:00:52 <ChrisCuellar> RRSAgent, make minutes public
20:00:52 <RRSAgent> I'm logging. I don't understand 'make minutes public', ChrisCuellar.  Try /msg RRSAgent help
20:01:08 <ChrisCuellar> RRSAgent, make logs public
20:09:41 <ChrisCuellar> ChrisCuellar has joined #aria-at