<zcorpan> start meeting

<zcorpan> meeting: ARIA-AT one-off discussion on test automation

Automation test format

<zcorpan> scribenick: jesdaigle

<zcorpan> https://github.com/w3c/aria-at/issues/349

<zcorpan> https://docs.google.com/document/d/1nCzxRdufL1l_6WreHJjzSZcdF4lKTGRqMnBwr4noEDk/edit#

Simon Pieters: This week changed custom text format to json based on feedback from NVDA. Changed how output is collect. Two new commands. Press until and press until role that translates ARIA role to specific string.

Simon updated design doc to reflect the changes and documentation in NVDA. Is working. Only one test right now. May be more bugs as we add more tests.

James: Two new commands Press until contains, what is the input? What is it looking for? Simon: If you use contains you can look for anything that is part of the output. Substring match. Most commands are substring matches. Role does a conversion from ARIA role to AT specific string for that role.
... could there be variations? Simon: there is no way to do that. If NVDA changes their output the tests should be changed to match.
... balance between tests that truly fails and false failures. Will it be versioned? Simon: currently the translation lives in NVDA repo. Test runner could look for both variants.
... it would be ideal if ARIA AT could drive different readers and carry out the tests.

Simon: should be possible to automate VoiceOver with Apple Script.

James: How does the system go through the test? Simon: the only new logic is in a loop, press a key and collect the output until the system thinks the output is complete and look for substring. If substring found loop stops. Continues if not.
... How does it execute? Simon: There are two different commands. Two loops. Use one or the other. James: Role, name, state. Simon: Depends on where you navigate.

Simon: existing test for combobox, press x until reach combobox. All accumulated output is in the string.

<michael_fairchild> https://docs.google.com/document/d/1YxHAf6r3E2RG4REaYcmM14DuxfBgP4XoWRdfSk5aAQM/edit#heading=h.n40b82cyf9d8

<s3ththompson> scribenick: s3ththompson

James: can we keep comments in one place (GitHub)?

Simon: having repo root in tests would make it possible to change our minds about where files live later

James, Michael: that makes sense

James: prioritize adjacency

Simon: (summarizing Michael comment) could add argument about how many times `assert_contains` matches
... (summarizing comment) how is `assert_accname` different from `assert_contains`
... it's not now, although it could be if it knew internal state of AT (but we don't have internal support of vendors yet...)

James: in general when we want to do things ourselves, can we run things as addons (rather than having nvda integrate it)

simon: yes, would probably be possible given how the python modules work

James: as an aside, i agree with reef from nvda about reading internal state...

simon: certainly we don't want to *only* read internal state. we could do both

michael: i like the idea of translating a role to a specific AT, but we can't just look at role value... have to consider context too

simon: what about `assert_state_or_property`?
... if consensus is that option role should not be conveyed... that's testable.

james: would this human also understand nuance about when option role should be included?
... e.g. do we consider hint text as important?
... is it possible for each screen reader to use different assertions? we've been under the assumption that the answer is no, but that could change

michael: other example was role mutation based on context: aria-pressed will mutate from button to toggle-button

simon: if it's a one-off we could use assert_contains... or have a special role for toggle button even if not standard per arai
... do we need to enumerate everything for assert_state_or_property? (answer from group: probably not)
... comment from weston thayer: How long does the test runner wait before forwarding the next command?

james: this is a comment question on nvda forum... the answer is usually "you don't." it's not really possible in the wild because speech engines are all different. but maybe in our tests we could pick one engine and know
... (or a custom speak engine...)
... 17 instances of role with modifier... not sure if all of them cause role mutation, but maybe more than we were thinking
... (from core a11y mappings doc)

simon: so is the implication we don't have `assert_role` at all? or do we design it differently?

michael: my take would be leave it as is and follow up after "beta" version
... weston brought up another good point: if we intercept an output and nvda sends multiple chunks after that output, how can we know for sure in the runner that speech output is finished before we move on to the next test
... we really need to think about that more

james: for this and other questions, nvda may have input on how this should be designer... it's all fairly new
... need to continue convo about how test writing is organized... csv files... reusable assertions, etc.

simon: my assumption is definitely that how we write the csv manual tests will need to be updated once we finish designing automated format... but that can come later

<zcorpan> thanks s3ththompson for scribing!

<zcorpan> also thanks jesdaigle for scribing :)

This is scribe.perl Revision of Date 
Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/

Guessing input format: Irssi_ISO8601_Log_Text_Format (score 1.00)

Default Present: zcorpan, jesdaigle, s3ththompson, michael_fairchild
Present: zcorpan jesdaigle s3ththompson michael_fairchild
Found ScribeNick: jesdaigle
Found ScribeNick: s3ththompson
Inferring Scribes: jesdaigle, s3ththompson
Scribes: jesdaigle, s3ththompson
ScribeNicks: jesdaigle, s3ththompson

WARNING: No meeting chair found!
You should specify the meeting chair like this:
<dbooth> Chair: dbooth


WARNING: No date found!  Assuming today.  (Hint: Specify
the W3C IRC log URL, and the date will be determined from that.)
Or specify the date like this:
<dbooth> Date: 12 Sep 2002

People with action items: 

WARNING: Input appears to use implicit continuation lines.
You may need the "-implicitContinuations" option.


WARNING: IRC log location not specified!  (You can ignore this 
warning if you do not want the generated minutes to contain 
a link to the original IRC log.)