Meeting minutes
Discussion of restrictions and keypress simulation on macOS
Mike: Two meetings ago we talked about the motivation behind Apple's reluctance to allow simulated arbetrary key presses.
James mentioned remote support as a precedent for allowing this.
So, wondering why our use case is different.
James: If Apple is building this into VO.
It is reasonable that they want to respect user privacy and security.
It should not be something that people can be tricked into overriding.
It is more reasonable for a 3rd party to expect users to turn off security and privacy.
For example, the 3rd parties the build remote access tools like Remote Incident Manager.
Michael: do you have to disable security for it to work?
James: There are a lot of permissions you have to allow in order for Remote Incident Manager to work.
MK: My question is since apple has been so firm about this, I'm wondering is our goal to figure out a way to move forward without allowing arbitrary key presses and still have what we need for web based product testingg?
MK: Or do we want to engage Apple in discussions that have the goal of getting some kind of modification to their stance such that the spec could provide a pathway for this. We would negotiate the obstacles in the pathway
JS: I would ask a higher level question of what is our goal?
JS: If the goal is to have apple build support or all of a subset of the spec into the platform, then naturally what they think of the security is paramount
JS: Or we move forward with the facilities that exist today
JS: What is the aim? Apple aren't saying the facilities exist to inject keypresses, but we also know that apple isn't willing to build something into apple that allows users to use them
MK: In the next year, since there are a lot of capabilities built into the platform and VO supports those, we could simulate AT driver by building a native MAC OS application that makes it look like there is integrated support
JS: If we did that I would recommend we not use apple script
JS: The support that apple are alluding to with controlling VO with apple script, do not include arbitrary key presses'. The problem is that support doesn't work very well
MK: So we would be providing interoperability based on what apple says it can do it
JS: Right
MK: We could build a native MAC app that makes it look like VO supports AT Driver, because remote incident manager exists, is our confidence level that we could build such an app very high?
JS: My confidence level is high
MF: Mine as well
MK: Building this for iOS would be very different
MK: If one of our goals is having AT Driver as a standard is to enable testing not just for ARIA AT but for other projects, the implementation details on how its implement like requiring a 3rd party library, creates some challenges in maintenance long term
MK: I think we need to find out what it takes to get more direct support for AT Driver, and not just for AT Driver which is where it gets complicated in the long term
JS: To go back to your pathways Matt, that the spec could endorse levels of support, like keyboard or headings, its up to an implementer to decide what they support?
MK: Yes
MK: I was thinking there is a feature flag that says the things you must support, and flags other things you are optional to support
JS: The Key press could be removed from the spec to support Screen reader automation. With browsers, with web driver there are certain things you cannot do
JS: The simple act of opening a browser, configuring the browser, closing the browser, that is seperate
MK: Mike can you explain that dividing line?
jugglinmike: Yep, Web driver classic has the concept of sessions, that have capabilities, from the local end you send a request to create a remote session, the remote end provides a browser
jugglinmike: This is to support testing farms, so the remote end might not be creating the session on the same machine
jugglinmike: you can put constraints like headless, or use firefox. Thats also the case in webdriver bi di, which is the new version built on top of sockets
JS: So the browser doesn't host the remote end?
jugglinmike: right
MK: So Who provides the code that sets up that remote? Individual developers like google or mozilla? Or with tools like celinium?
jugglinmike: The code that starts the process is written by Google and Mozilla for themselves. Like what PAC built with Go, it speaks a proprietary protocol, theres a facade
JS: So your're saying browser stack may have implemented their own facade that can spin up their own sessions
jugglinmike: Correct, but they would still call out to the gecko driver
MK: So if you want to be able to spin up chrome or web kit, or gecko, they each have their own tool to spin up webdriver, but to they have their own abstraction of the layer that allows you to spin up any of those 3, or is that outside the scope of the web driver?
jugglinmike: I'm not sure if I would say neither or both. When you spin up a session, you decide what kind of browser you spin up that would reject commands from other browsers
JS: But web driver the protocol only specifies how I must respond, with a session or saying I can't do that
jugglinmike: you mean like what it does to start a process?
JS: yes
jugglinmike: Thats correct
jugglinmike: I'm glad we are talking about this, because are use is a little distinct so far as what we are testing is a singleton, it will be an uphill battle to support multiple instances, like running JAWs twice on one hose
MK: Well I do it
JS: Okay well you shouldn't
jugglinmike: I'm just saying that this concept of a session is less relevant to us
jugglinmike: but we have this concept of intermediary nodes, and service providers like browserstack
MK: I would argue that its not a singleton because we had a case where users would use zoom and JAWS
MK: It does seem the session stuff in that use case
JS: Okay so I was wrong in suggesting the process management is outside the browser
MK: So Thinking about goals and pathways, in the near term, I'm not sure what the long term risks are, but in the near term, the ARIA AT taking responsibility that Apple is taking a position that if we want Automation to be able to AT Driver than we need t build a open source application that is a AT Driver provider
MK: Whether or not it could be scaled to other platforms is an open question
MK: In regards to our inclusion in BTT, We do need to figure out what language would get us around Apples objections
MK: But we've kicked this down the road as we are not on their roadmap in 2024
MK: I had envisioned in the spec that we would have user intents, not part of the spec but referenced by the spec, we have a intent mapping document that lists all possible user intents and the keyboard mappings
MK: That would bridge intent and keyboard
JS: I think that its important that not every AT maps its events to a key
JS: For some AT the primary way to use them isn't keyboard based, like for voice commands, the intent focus could allow developers to optional support intents and not keyboard, or keyboard and not intents
MK: Mike are we answering the questions that need to be answered?
jugglinmike: Yes I think so. We don't need apples support to move forward in a direction we are all comfortable with
jugglinmike: we have a clear vision of how this will evolve over time. I think this make sense, particularly in regards the the not keyboard based AT, it makes sense
JS: We are balancing the spec and the practical side of the project
MK: I think if we can understand the direction of AT Driver from the perspective of BTT, instead of ARIA AT
jugglinmike: I'm think this motivates more imediate work toward session work in the protocol, to signal more strongly how we expect to support other AT's over time
MK: It seems like some of the objections of having AT Driver in BTT was the platform independence, the way the tests are written, certain aspects of those cant be written in a platform independent way, I think moving toward intents move us in the right direction
MK: Mike can you test platform specific capabilities?
JS: What capabilities?
MK: you couldn't test anyting in window's that relies on a next item capability
JS: Wouldn't that be the case where some intents are supported and some are not?
JS: I would hope that we either have support for listed intents differ for AT, but secondarily having the session invocation be aware of specific intents
JS: and if can't do that it tells you
MK: If that is built into the spec, that theoretically platform independent testing is possible
MK: You're just limited in what you can test
MK: I see a fair amount of value in a web platform test that is supported by W3C and BTT is the best way to go there
MK: If we had the support of the chair and enough other people to make that a reality I hope their minds don't change but I still have that nagging concern
MK: If it doesn't go that way, we would have all the tech we need it just wouldn't be supported by a major platform
jugglinmike: The only other item on the agenda was a review of a design, so I encourage people to review that