Meeting minutes
Review agenda and next meeting dates
Matt_King: Next meeting: Wednesday March 13
Matt_King: CSUN is the week of March 18. Does this effect anyone's ability to join this meeting (it's currently scheduled for Thursday the 21st)
Hadi: I'll bet at the conference
Matt_King: If it's just Hadi, then I think we can still hold the meeting as scheduled
App update status
howard-e: There's now an updated Test Plan page in the application. It matches the page that you can see in the Netlify link
howard-e: There are additional details coming in the update which outline the "problematic behaviors" much more clearly
howard-e: These have been improved internally here at Bocoup and by Matt_King in the staging environment
howard-e: We plan to push this to production on Monday, when we can be available for support if necessary
howard-e: I'll inform PAC and Matt_King when the deployment occurs
Command and toggle button test plans
Matt_King: Done: Role changed to "toggle button" from "button" in role assertions - issue 1031
Matt_King: Any updates on Add 'b' quick nav to VoiceOver commands in button and toggle button test plans issue 1030?
IsaDC: Not yet, but we're targeting Monday
Matt_King: Hopefully we won't need to re-run complete test plans, but we WILL need to run the "b" commands for VoiceOver
Matt_King: By next week's meeting, we should be able to plan these out
Radio test plan
Matt_King: IsaDC and I will "deep dive" into this test plan on Tuesday of next week
Matt_King: We have a pull request that I think we'll be able to merge before that--#1042
https://
Matt_King: It looks good to me (I reviewed it about an hour prior to this meeting), and Alex_Flenniken approved it as well
howard-e: In that case, I can certainly merge it
Matt_King: Maybe on Wednesday of next week, we will also be able to assign people to run the new version of the Radio test plan
Matt_King: I'm really excited about that improvement!
IsaDC: Can we preview it?
Matt_King: I will push a few more changes to your pull request, and then I'll e-mail you. At that point, you can see the changes in the preview attached to your pull request
Assertion priority definitions and alert test plan
Define meaning of assertion priorities: "MUST", "SHOULD", and "MAY"
github: w3c/
Matt_King: We have "required" and "optional" defined in the glossary, but that's now out of date. This issue proposes definitions for "MUST" "SHOULD" and "MAY" that we can add to the glossary
Matt_King: We're able to consider these in terms of feedback we've recently received from Vispero about some specific expectations in the "alert" test plan
Matt_King: Similarly, we've received feedback from Apple about the "not pressed" state for toggle buttons
Matt_King: Last week, we were just talking about "SHOULD" and "MAY"; it didn't seem like there was any feedback on the "MUST"
Matt_King: I do want to consider "MUST" in this meeting, though, because I'd like to arrive on a final decision on the meaning
"The assertion is an absolute requirement. The command or event being tested MUST result in the behavior described by the assertion. Failure to do so could block users of the command from perceiving, understanding, or operating the type of content represented by the test case."
Hadi: I completely support that
Alyssa: Yeah; there isn't much room for interpretation there
Matt_King: I'm trying to take the definition of MUST from RFC2119 and translate it to an assertion in a test (e.g. "the role alert is conveyed")
Matt_King: I'm trying to explain the consequence of any one of our thousands of assertions
Alyssa: It makes sense to me. It differentiates between something like "SHOULD"
Hadi: English is my third language. I think "likely" maybe doesn't go very well with "MUST"
Matt_King: Well, we're not certain that it will block users because we can't be sure that something will block them
Matt_King: We don't know for sure if one single failed assertion would block somebody. It could, but it might take multiple failures
Hadi: I am for equal access. For instance, if you tab to an element and you get information that's different than if you arrive at it some other way. That's unecessary cognitive load
Matt_King: That sounds right to me. We have an assertion for "the role 'toggle button' is conveyed". If an AT conveyed only "button", you could still use that button--it wouldn't block you
Hadi: If I am uncertain at any stage, I call that a failure
Matt_King: Agreed. We're calling this a failure. We're just going on to explain what a "failure" means
IsaDC: I have a similar opinion, though English is also not my native language
Matt_King: The actual definition is just the first sentence. Everything after that is just an explanation
Hadi: Ah, in that case, I'm comfortable with this
Matt_King: Moving on to the definition of "SHOULD"...
"The assertion is a strongly recommended requirement. The command or event being tested SHOULD result in the behavior described by the assertion. Failure to do so is likely to impede users of the command from perceiving, understanding, or operating the type of content represented by the test case. [...]"
"[...] There may exist valid reasons in particular circumstances to ignore this assertion, but the full implications must be understood and carefully weighed before choosing a different course."
Matt_King: Again, modeled after RFC2119
Alyssa: Makes sense to me!
Matt_King: Then, for "MAY"
"The assertion is truly optional. The command or event being tested MAY result in the behavior described by the assertion. Failure to do so is not likely to impede users of the command from perceiving, understanding, or operating the type of content represented by the test case. [...]"
"[...] One assistive technology (AT) vendor may choose to provide the asserted behavior because a particular marketplace requires it or because the vendor feels that it enhances the product while another vendor may omit the same behavior."
Hadi: Can you elaborate a little bit on when vendors might choose?
Matt_King: Some screen reader developers have different ideas about what will better-serve their users
Matt_King: Some screen reader vendors might prioritize a certain market, e.g. people who aren't blind but are using a screen reader for some other purpose
Matt_King: They may make different decisions based on these differing goals
Hadi: Thank you. The default setting is based on which level of verbosity?
Matt_King: Whatever the AT provides immediately after installation. In other words, we use the setting that the vendors themselves implement as default
Hadi: I'm concerned that AT vendors will map the "MUST", "SHOULD", and "MAY" to their verbosity levels
Matt_King: I doubt it. I don't think it's that neat and simple. Especially when it comes to "SHOULD". It's hard to imagine that they would stop doing those things on a medium verbosity
Matt_King: For instance, announcing the position within a radio group (e.g. "one out of three" or "two out of three"). We mark that as a "SHOULD" because you can technically get by without that
howard-e: The word "omit" in the definition of "MAY". To me, in English, that only sometimes represents intentionality
howard-e: Would it be safer to say something like, "while another vendor may not support the same behavior by default" ?
Matt_King: We actually do want that sense of intentionality
Matt_King: Sometimes, users of one AT actively don't want certain behaviors, so the implementers want to leave it out on purpose
<Sam_Shaw> jugglinmike: We are both saying you are able to do what you want within reason, and also you shouldn't do these things. My Questions is, if you do one of these things that we are saying they may do, we will consider it excess verbosity
<Sam_Shaw> Matt_King: With Voiceover or NVDA there is a message it gives, the help message, which we don't test, is wrong. If it is wrong, I think its legitmate to call it out as a problem
<Sam_Shaw> Matt_King: I wouldn't call it excess verbosity, I would call it confusing verbosity
<Sam_Shaw> Matt_King: The other reason for this, is if one SR developer does something and another doesn't, if you are unfamiliar with SR and your testing your component with a SR that does this behavioir, then test with another SR that doesn't, you would think its a bug in your code, and if not then you would think its okay that this behavoir does or doesn'
<Sam_Shaw> Matt_King: doesn't happen
<Sam_Shaw> jugglinmike: I want to guard against confusion when we say you can do anything you want, and you may do this. For me tying into the excessive verbosity reasoning helps clarify that
<Sam_Shaw> Matt_King: Lets push further discussion of Alert role to next week
Priority of alert role assertion:
<Sam_Shaw> no problem jugglinmike
Matt_King: We're almost out of time, so I'd like to push this topic to next week's meeting
Proposal to change terminology used for undesirable behaviors
github: w3c/
Matt_King: I'm having a hard time not thinking more and more about the wording of this and wanting to simplify it further
Matt_King: One of the problems I have here is with the phrase "other behaviors"
Matt_King: That phrase means you have to read all the other assertions. If you read the assertion out of context, it's not clear what it means
Matt_King: I was thinking of calling these other behaviors "negative side effects"
Matt_King: We'd have an assertion for "Severe negative side-effects do not occur" and "Moderate negative side-effects do not occur"
Matt_King: I wonder if other people find that simpler
Hadi: When we talk about "negative side effect", I'm not sure we defined what "negative effect means"
Matt_King: We have a list of them in the test form. things like "excess verbosity", "sluggish behavior", "crashing", and even a text entry field where Testers can explain in some other behavior
Hadi: When we say, "excess verbosity", do we have a kind of guidance where "excess" begins?
Matt_King: I think we're currently relying on peer review for that
Matt_King: It's why we have two people running tests. If one person classified something as "excessive" and another person did not, then that would cause a conflict which we would subsequently discuss in this meeting
Matt_King: I'd love more feedback on this issue--please post it as a reply. I'm trying to improve the clarity for the reports.
Matt_King: We can continue this discussion next week, of course