Meeting minutes
Update about Windows Speech driver accessibility
Mike: current goal is not just to make Speech driver vocalize, but to make it fully accessible: namely that it remembers the previously enabled voice and reenables the new voice after the fact
Mike: practical reasons for current approach: keep Speech driver minimal, core needs to be written in C++, we also want option to run without vocalization for speed
Michael: from practical sense, both approaches could work. however i have some concerns about what happens while test is executing...
Michael: if there's an exception or failure, how would screen reader recover?
Michael: also, what happens if someone tries to interact with screen reader while tests are running
Weston: what about a virtual machine?
Michael: interesting, this is testing the limits of my familiarity with this tooling. what kind of ergonomics could we provide for developers around this?
Weston: what if you could connect to a remote virtual machine? run remotely execute locally
Seth: how does debuggability work with a virtual machine? what kind of guarantees do you provide?
Weston: very interested in debugging... you could do remote recording, you could read things as it's running etc...
Weston: you could definitely provide that via a CLI or webapp
Michael: the way i envision is you run a single command... behind the scenes it spins up a virtual machine test remotely with mike's voice and then send us the results
Mike: I wonder what the expectation is for previous versions of browsers / ATs
Weston: i've tried to build images with portable versions of different ATs
Michael: I think this is probably a good way forward. i think i still have concerns about AT vocalization that we should verify with other community group members. I agree with others that Weston's cloud service vision is provocative, esp. given that we eventually aspire to run these tests on a daily basis. starting to think about that infrastructure now is really vital
Goals / success criteria for the AT Driver format
Mike: at a high level, my understanding is that we want to build a system that can validate AT/browser behavior automatically. we want to be able to simulate user input and then validate output. i've been thinking about how we format those instructions in a machine-readable way
Mike: i've been thinking we need some sort of Domain Specific Langauge (DSL) to do this. last year, Simon used a JSON-formatted list of instructions, but even since that i think we've pinned it down even further
Mike: for example, we've since gotten feedback that we don't want ambiguous commands like "press until..." we want the commands to be precise and specific about what's going on so we would have "press 3 times" instead
Mike: it seems like a necessary takeaway from the above is the following non-goal: that is that any given test should be shared by multiple ATs.
Mike: also a non-goal: make tests overly concise. right now i think we should focus on getting the right primitives in place... on top of which we could later build abstractions.
Seth: so for Checkbox Test "Navigate to an unchecked checkbox in interaction mode" assertion 1: "Role 'checkbox' is conveyed" the assertion is equivalent for all ATs, but the actual string that is asserted would be different for each AT
Seth: so do individual assertions have meaning in this world or are we just matching the entire output string?
Michael: well, we would lose a lot there because the tests would be more brittle right?
Weston: NVDA at least has an internal object representation before it's rendered to text...
Seth: i think when that's come up in the past, some CG members have felt they couldn't trust an internal object representation
Michael: i think it was more about the fact that it wasn't very interoperable...
Seth: Mike, what do you think about substring matching (per assertion) vs. entire-output string matching?
Mike: I think there's still value in starting with entire-output string matching in order to gather data that might help us design a higher-level abstraction
Mike: obviously there is a downside in the brittleness to the tests, but we can take cues from web development where people use snapshotting... might be a better model for how our CG group should respond to what happens when a vendor totally changes their output
Michael: great discussion. we should get James and Matts' thoughts on this.