Accessibility Conformance Testing Teleconference

23 Nov 2017


Wilco, Anne, Martin, Romain, Kasper, Mads, Shadi


Possibly allow meta-rules https://github.com/w3c/wcag-act/issues/141

<Wilco> https://github.com/w3c/wcag-act/issues/121

Wilco: Would it possible to have rules that rely on other rules? Either super rules or meta rules. Similar to rule groups in the current draft. Next step from simple aggregates where rules directly use the results from other rules.

rdeltour: Is there any difference between this and the current rule groups?

Wilco: Rule groups are more simple. They allow you to pass a rule of groups by simple passing any one rule within that group.
... Meta rules can do more. It gives you a hierarchy of rules as Kathy talked about.

rdeltour: It does make sense, though an example would be nice.

Anne: We don't think it would be a good as it breaks the atomic nature of rules.
... Changes to results in one rule might affect the results of another if we go this way.

Mads: From an implementation standpoint, shared functionality would make sense. Have it be an implementation question rather than a rule question.
... The rule definitions should be atomic, though optimizations at the implementation level can be made.

Wilco: This would lean towards a specific implementation of a rule where we're looking to optimise things.
... Would be too descriptive in a way we don't want.

Anne: Color contrast checks for level AA and AAA would make sense when talking about shared definitions. Same goes for definitions of visibility. These definitions could be shared across rules, but the rules themselves need to be atomic.

Wilco: I like the glossary idea which makes sense for a lot of stuff. These definitions could be a lot more technical, which we don't need for rules.

Mads: What super tests do allow you to do is to allow having a single rule fail which would fail the depending rules.

Anne: Couldn't we build logic into rule groups?

Mads: Not the way it's worded now.

Wilco: Do we have examples of rules where when one of them fails the whole group should fail? Sounds like an implementation detail.
... If you like at image rules, a first rule would be to make sure the image has an alt attribute. If it doesn't, it's always a violation. A second rule in a similar group would be checking if the alt attribute is descriptive.
... If we're looking for more atomic rules, then separating them out into groups has a certain logic to it.
... Would be interesting to see what makes sense for manual tests.

Anne: I think that would be interesting to look into. From the manual perspective, tests are often reversed versions of the automated rules.

Wilco: The question we need to answer is which is preferred for documentation?
... If you want to rate how a tool should do image testing, would it make more sense for it to be more documents or just one?

Anne: What are talking about splitting out?

Wilco: Any violation meaning that a whole group would fail would allow to have less steps in a rule.

Michael: Both examples are atomic rules, so both need to pass. It sounds crazy to have them in different documents in that case. We might also be able to reuse them.

Wilco: Different documents would mean different rules.

Anne: It would make sense to implement only automated rules if needed and having atomic rules would allow this.
... The rule group description would then link to the associated rules.

Wilco: Makes sense.
... If think what we're missing is allowing a rule group to fail if any single rule within them fails. Meta-rules would lean more towards implementation.

<rdeltour> +1

Wilco: What we right now allow is for different rules within a group to have multiple selectors.
... Does that still make sense to us?

Mads: It might be necessary.
... Different input might be required for rules within a rule group.

<rdeltour> IMO it still makes sense to have different selectors

Mads: Two separate pieces of data might need to be passed to a rule.

Wilco: The point where this confuses me is that you've got the different selectors, now an element is only tested by one rule in a rule group. What does that mean for those other rules?
... You've might have a conflict of applicability if only a single rule within a group says that an element passes.

rdeltour: If you're looking at applicability for an element within a rule group, it doesn't matter for the individual rule. Applicability is aggregated for the rule group.
... There shouldn't be a conflict, but the rule should explicitly define the applicability. I don't think it's an issue.

Wilco: What happens if you haven't implemented everything in a rule group?

Michael: Can you do that?

Wilco: I think so.
... We need to list out the different scenarios.
... It simplifies things if you can't claim anything about a group unless you've implemented everything.

rdeltour: Maybe another approach is to assume that we can use different selector and then come up with counter examples. If we don't come up with any then it should work.

Wilco: We went from something straight forward with an inverted aggregation where a single rule passing lead to everything else passing.
... We want to make that we thought through the whole thing. What happens with elements that only apply to one rule within a group?

Mads: Some rules within a group might be automated, some might be manual.

rdeltour: It might be a bit overkill to restrict things by design.
... We have a process in place for approving the rules.

Wilco: I'm worrited about getting inconsistent results.
... Are we getting back to the problem we're trying to solve with rules format in the first place?

Martin: Isn't it just a question of transparency?
... If we're good at showing how we got certain results is it really a problem?

Shadi: Some tools might support more test cases than the others.
... That brings back the question of when to adopt a rule.
... What if tools support different parts of the test cases?
... At what point do we say that a rule is supported sufficiently?

Wilco: Unless you pass every test case, I don't think we should say that you support a rule.

Shadi: What if someone brings in new test cases to a rule?
... Some vendors might then adapt their tools to those cases.

Wilco: That would be a new version of the rule then.

Shadi: That reduces complexity. Where would the differences then be?

Wilco: For rules that require X number of passes, you can then only claim the rule as a violation if you've implemented everything.
... Reverse is true for a rule that require X number of violations.
... That's the thing that is missing.

Shadi: If we do at a later point add a rule to a group, that would be a new version of the group.

Wilco: Groups have to versioned, yes.

Anne: Do groups have test cases?

Wilco: Groups don't have logic, so they don't need to.

Shadi: Could there be interaction between the pieces that require unit testing?

Wilco: I would say not.

Anne: If the test cases indicate how much you support, don't we also need test cases for rule groups?
... Also in relation to the matrix we talked about.

Output data format for rule groups https://github.com/w3c/wcag-act/issues/116

Wilco: You would have to aggregate the data for the success criteria, though we have to define that.
... Anyone who doesn't think that's the case?
... Also means we have two types of rule groups. Do we need definitions for them?

Anne: What are two types?

Wilco: A group that requires certain number of passes vs. a group that requires certain number of failures.
... Anybody have ideas for what to call them?

Martin: Wouldn't it make sense to introduce logic for rule groups?

Wilco: That was the proposal for the meta rules.
... It's overly prescriptive of the implementation.

Mads: What we opposed with meta rules was the dependencies between rules.
... Rule groups seem more like an aggregation tool.
... We could have more conditions for rule groups.

Martin: In the current draft, we're still saying the standard should be technology agnostic. If we want an example for everything, we might limit ourselves.

Wilco: Rules are not technology agnostic. A rule is written for a specific technology.

Martin: We're trying to find examples to back up our decisions, which might be dangerous. The examples we might think of in a web environment might not be enough for the standard to support.

Wilco: Let's talk about next time how we can be more abstract and get away from implementation details.

Summary of Action Items

Summary of Resolutions

[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.152 (CVS log)
$Date: 2017/12/06 15:08:22 $