Meeting minutes
Review agenda and next meeting dates
https://
Matt_King: can we add something about BTT?
jugglinmike: Sure
jugglinmike: Next "automation subgroup" meeting: Monday, March 10, 2025
jugglinmike: Next Browser Testing and Tools meeting: Wednesday, February 12, 2025
This week's BTT meeting
Matt_King: I think we don't want to indicate that we would use DOM terminology because that probably wouldn't work
Matt_King: We might use terms from CORE AAM, though they may need modification
jugglinmike: Yeah, if we focus on CORE AAM, we might not need to worry about the DOM or the browsing session etc
Matt_King: I think it just comes down to what is intelligible for the implementers...
Updating AT-Driver in advance of new implementation
jugglinmike: There will be new folks coming onto AT-Driver. I'm concerned about correctness and stability of it. Specifically, I'm concerned about re-structuring "sendText" to be one of many future "userIntents".
That change in the specification would kick off implementation work for the AT vendors. We backed this changed out at the time because it didn't seem worth it at the time for implementers
Matt_King: If the change is only structural, what does that mean for the VO and NVDA bots?
jugglinmike: Right now those implementations are using "sendText" for all use cases
Matt_King: so it's the structure of the API calls
jugglinmike: Right, this is currently a very uninteresting structural change. But with more users potentially coming on, we might need to return to this.
Matt_King: Other implementors might be concerned about implementing code with lots of moving spec changes.
If we can prevent that, that would be helpful.
jugglinmike: It sounds like we need to move forward then with open PR, which will update AT-driver and then necessitate updates for implementors
Matt_King: I'm aligned with it. Are we confident that this is the right structure? This all comes from conversations at TPAC about how to deal with form input.
jugglinmike: Yes, and it also re-orients AT Driver to have an unbounded set of user intent that implementors may or may not decide to implement.
Matt_King: I've been wanting us to in that direction for a long time. It gives implementors a lot of flexibility and deals with Apple's security concerns.
jugglinmike: Ok, I will prioritize this this week. I can update the spec and work on making sure the implementations are caught up with the spec change
Matt_King: A question for Michael Fairchild, will this be suitable for UA? If an AT chooses to implement keyboard commands directly, that means they might not use all the different user intents, correct?
jugglinmike: Practically, that's correct. But implementations that only rely on keyboard commands, that wouldn't be cross-platform.
Matt_King: But if there are more user intents implemented across AT's, then it becomes easier to write cross-platform tests. We've always had a concern if screen readers behave the same way for a command like "Next Heading". There's always a chance that there may be differences between an AT API vs actual user experience. Most testers would care more about user experience.
jugglinmike: Since it's all a black box, there's still a certain amount of trust involved. We're using a protocol to simulate user behaviors at the end of the day.
Matt_King: How does all sound to you, Michael Fairchild?
Micheal_Fairchild: I don't have any strong opinions on any of this. I would feel more comfortable working with the keyboard commands. But I need to go back and review on this. Can you remind me about why we switched to a focus on User Intents?
Why does this need to be required for all ATs? Can we give preference for keyboard commands?
Matt_King: I don't think that will fly in BTT. They will favor approaches that are as cross-platform as possible.
jugglinmike: When we move toward this model of user intents, everything will be a user intent. "Send Keys" will be an Intent and it will be the first (and for now only) User Intent.
So there isn't really a difference between keyboard commands and user intents.
Michael Fairchild: Will there be any intents that will be required?
jugglinmike: All intents will be optional
Michael Fairchild: I'm ok with that. That does seem to give implementors a lot of flexibility.
Matt_King: If an implementor only wanted to use "Send Keys" that would be fine, tho again not great from a cross-platform perspective.
jugglinmike: Ok this helps me prioritize upcoming work.
Understanding the risks behind delays for MacOS releases
jugglinmike: In the ARIA-AT CG, we recognize that the Github automation system doesn't fully support MacOS 15. So there's some delay for us in running automated VO tests using 15.
I don't think this is that dire of a problem. We can still test the latest version of VO. We can also still use the VO Bot. The issue currently is that VO Testers have to be aware that they're working the automated output from an older version of VO. This isn't great, but it is better than nothing.
Most of the automated text that a tester will be provided will still be the same.
There may be differences in what they actually experience when running the tests and they may have to be manually editing the text, which is an inconvenience.
I think that even if this doesn't get resolved asap, I think we're still ok.
Matt_King: We have multiple use cases for automation. One use case is to help collect output from the screen reader so that a manual tester can make decisions about it (and they don't have to manually copy/paste). A bigger, more important use case is that we need to compare test plan results across versions of an AT when a new AT version comes out (regression testing). We can't do this manually.
I think this is a critical problem.
ChrisCuellar: Can we get more collaboration from Apple on this problem?
ChrisCuellar: A lot of the VO settings changed and are sandboxed now
ChrisCuellar: I wonder if there was a way that we had more of a heads up--that we knew that was coming--we might have been able to prepare
ChrisCuellar: The kinds of change we saw with macOS 15 are probably not of the same kind that we should expect to see with subsequent released of the OS
ChrisCuellar: For instance, part of the problem has to do with our ability to configure VoiceOver in the way that we need to. That's currently an unsolved problem because of where some of the preferences got moved in macOS. I'm still wrapping my head around the problem, but in general, the settings changes were significant
ChrisCuellar: macOS 15 is indeed available in GitHub Actions, but it isn't usable for use cases like ours and like GuidePup's
Matt_King: Is this something GitHub can resolve, or are they blocked by Apple?
ChrisCuellar: That's unclear
Matt_King: It feels like, before we went to Apple, we'd have to get something definitive from GitHub
ChrisCuellar: In this situation, if we somehow knew ahead of time that Apple was going to change how user settings for VoiceOver are made available for scripting--then, we would have been ahead of this problem
ChrisCuellar: I think the problem is less about how quickly GitHub can get macOS available in their CI, and more about support for VoiceOver automation
ChrisCuellar: It's a niche problem that GitHub may not have been aware of when they were first making macOS 15 available on GitHub Actions
jugglinmike: I wasn't trying to de-prioritize this, but I do want to say that we should definitely address this before MacOS 16 comes out later this year. That's especially important for the second use case that Matt mentioned.
Matt_King: We already have that problem now. Most of our reports are out of date currently. We don't have a lot of results for MacOS 15
jugglinmike: Isee
s /Isee / I see/
jugglinmike: I understand this problem a lot better now so that's helpful. If there's no other questions, we'll close the meeting now. Thanks everyone!