Accessibility Conformance Testing Teleconference

Rules Format pull request: Expand accessibility requirements section https://github.com/w3c/wcag-act/pull/96

Wilco: I have added some text to the accessibility requirements section
... I can see Moe added something 15 minutes ago

Moe: Yes, it's mostly editorial
... I have cleaned it up a bit

Wilco: The reason why I used the word "fail" there is because it's a fail result

Moe: I wonder if we should have an uppercase F in fail
... I added 2.0 for WCAG

Wilco: I used WCAG 2 on purpose, since it will be true for 2.1 too

Moe: so, you are using 2.0 in the first paragraph because you are referring to a specific success criteria...

<shadi> https://github.com/w3c/wcag-act/pull/102/files

Shadi: I also added editorial comments
... so I just started rewriting the whole thing, I hope I didn't go overboard
... I was looking to use the keywords "pass" and "fail" here

Wilco: so you can pass headings for 1.3.1, but you cannot pass the whole accessibility requirement
... specifically what parts of the text is proposed, what didn't you like?

Shadi: I found it a bit confusing talking in some parts about what the ACT Rule does and doesn't
... first of all, I think the first paragraph can be droppen completely
... does not have to test a whole accessibility requirement, is a double negative
... Next paragraph I found confusing, "a rule must be consistent within a rule"...
... I think we all agree what we want to say here, we just don't know how to say it
... I didn't even understand the part about the mapping. And then I tried to put the note into the text itself
... basically what I tried to do was an editorial change, and maybe editors should look at it
... do we really want to include organizations' accessibility requirements, like we always want to have the logo in the top left? Are these accessibility requirements?

Wilco: examples: only have one H1, never skip heading levels - these are accessibility criteria

Shadi: what is in scope of ACT Rules and accessibility? Do we need to call this "Accessibility Requirements", or could we call it "Testing Requirements". It's often unclear what is accessibility requirements or usability requirements

Wilco: it's not a testing requirement per se

Shadi: another name, so that we don't define the accessibility requirements

Charu: if we have particular rules that go beyond WCAG or is coorperate specific, we can make it clear to everyone that this tests for certain cooperate criteria
... but I think we should stick with accessibility criteria, or else it will become too much, if we try to include every kind of testing

<MoeKraft> Our original intent of this section "Accessibility Requirements Explain the accessibility requirement being tested such as the WCAG 2 Success criterion and / or the technique the rule maps to; For example WCAG 2.0 Technique H67."

Charu: ... make it clear that it maps to WCAG

Moe: +1 to Charu, we should stick to that we want to define an accessibility rules format

Shadi: I hear the concern, and I agree. I am only concerned about the very last sentence: "Often organizations have accessibility requirements in addition to the WCAG success criteria. These too can be tested using ACT Rules."
... I have difficulties with the way it's being phrased, like it's accessibility criteria on the same level as WCAG, when it's in fact techniques being brought up to the same level as Success Criteria

Moe: It's a requirement for that organization, but how can it be mapped to something else...

<MoeKraft> Some organizations may have particular requirements, such as specific implementation techniques to meet WCAG 2 Success Criteria; in this cases these would be accessibility requirements. -

Shadi: it's just a way of implementing, a technique that become mandatory

Charu: two different things, a technique that is required, but still maps to WCAG, but also sometimes requirements that go above and beyond WCAG

<Wilco> This section explains the accessibility requirements to which the rule maps, (for example, WCAG 2.0 success criterion 1.1.1). An ACT Rule MAY be a complete or partial test for any number of accessibility requirements.

<Wilco> Outcomes from an ACT Rule MUST be consistent with the accessibility requirement, e.g. if the rule returns a Failed result, this MUST be a failure for the accessibility requirement. This means that the rule maps to the accessibility requirement, as opposed to it merely being related to the requirement, thematically or otherwise.

<Wilco> The actual definition of specific accessibility requirements is beyond the scope of ACT Rules and of this document. For WCAG 2, Success Criteria are considered to be accessibility requirements. Some organizations may have particular requirements, such as specific implementation techniques to meet WCAG 2 Success Criteria; in this cases these would be accessibility requirements.

Wilco: I have no problems with the first paragraph Shadi put in, but I think the first one loses some of it's nuances...
... I wrote a hybrid (above)

Shadi: "related requirement" what does that mean? And "maps"...
... ... I'm happy to go with it

<Wilco> "Each ACT Rule MUST explains the accessibility requirements "

Wilco: for the first sentence, I meant this (above)

Charu: do you mean more "validate" the accessibility requirements?
... maybe "identify"...

Wilco: I'm fine with that

<Wilco> Each ACT Rule MUST identify the accessibility requirements to which the rule maps, (for example, WCAG 2.0 success criterion 1.1.1). An ACT Rule MAY be a complete or partial test for any number of accessibility requirements.

Wilco: then the section becomes this (above)
... I'll send out an update for this, I want to get it merged in

Moe: "In these cases these would be considered accessibility requirements" - it almost makes it sound like these would be instead of WCAG success criteria. Make it sound non-exclusive

Rules Format pull request: Merge change log with version history https://github.com/w3c/wcag-act/pull/94

Wilco: so I got a request to take out the word "huge", which is fine

Moe: I made some grammatical changes

<MoeKraft> It is important to keep track of changes to the ACT rules so that users of the rules can understand if changes in test results are due to changes in the rules used when performing the tests, rather than changes in the content itself.

I don't like to use the word "them"

Romain: maybe we should require that all versions are linked from the change log. With this requirement it seems that we create a chain of links
... in this way you will have one entry point to all of the versions

Wilco: you are certainly allowed to do that, but I don't think it should be required, it will get messy

Romain: ... ignore what I just said...

Wilco: taking in Moes rewritten paragraph, removing the word "huge"

<Wilco> An example of when a new rule should be created would be when going from a rule that tests the use of a blink element, to a rule that looks for animated style changes.

Tobias: I think Anne had a comment for the next paragraph. There is something wrong with the sentence

Wilco: So the sentence would look like this. Change made

Use of the term "Test case" in the spec

<Wilco> https://www.w3.org/TR/act-rules-format/#test-proc-cases

Wilco: Our current ACT Rule Format includes a section on test cases
... problem is that we have started using the word "test cases" differently. Like a sub test or test step
... where in other places we are using "test case" for a html snippet that should either pass or fail when using the rule on it
... So I think we should change the wording here
... in auto-wcag we call it test steps, in aXe core we call it tests

<rdeltour> I like "steps", it's pretty explicit

aXe core: checks. Steps imply an order

Charu: in IBM we have test steps

Moe: but a rule could have multiple tests, right? But a test could have multiple steps

<shadi> https://en.wikipedia.org/wiki/Test_case

<Wilco> https://auto-wcag.github.io/auto-wcag/rules/SC1-1-1-longdesc.html

Wilco: looking at a random auto-wcag rule...
... you kind of break it up into higher level procedures...
... Shadi dropped in a wikipedia article of test case

Shadi: "test steps"

Wilco: so we have got "test procedure", made up of "steps"
... anyone having a problem with calling it test steps?

Moe: to me it seems that the steps have steps themselves
... I mean to test that it's a valid URL, it will have to go through steps themselves
... each test case is a seperate test inside the rule
... we can use step as long as it's consistent and we explain it... we can call it test execution steps
... I'll take a stab at it

Shadi: We are using "test cases" in the review process too

Moe: I'll open an issue for this as well