W3C

Accessibility Conformance Testing Teleconference

22 Feb 2018

Attendees

Present
Wilco, MaryJo, SteinErik, Romain, Anne
Regrets

Chair
Wilco, MaryJo
Scribe
Anne

Contents


Change of meeting day? Same time but maybe a different day - see Doodle poll at: https://doodle.com/poll/dnzani3y7nvfvpqp

Wilco: It's between Monday and Thursday. We will stick with Thursday meetings, or else we will have no one from Daisy. On Thursdays we will still have others from Siteimprove. We will keep it as it is

Issue #174: Review process comments from Detlev Fischer (see Shadi's proposal in issue) https://github.com/w3c/wcag-act/issues/174

Wilco: Shadi put together a response for Detlev's comment.
... for the comment "I was a bit unsure whether the sentence "Implementations for ACT Rules may be added, modified, or removed at any time." applies to implementations of third parties (so saying 'do as you like, use what feels fit') or to some centrally maintained rule base at WAI."
... I am a bit unsure what "former" means in Shadi's response

Stein Erik: Now that I read it again, I'm a bit unsure what the boundary is between the two

Wilco: What I read first was that implementers are in charge of their own manifest documents. And we are not going to do QA on that. So if someone claims they have implemented a rule, we will trust them

Romain: Sounds all right to me

Next thing I want to talk about is Detlev's comment "Does the text need to say anything about implementers' scope for modifying rules in the way they are implemented in their tools or processes, or about any constraints on that, at least regarding claims that they 'use the ACT rule base' or similar?"

scribe: I don't think Shadi's response to this answers the question

Stein Erik: It rather dismisses the question

scribe: If you haven't implemented the Rule, you haven't implemented it. You have done something else

Wilco: All right, I'll leave these two as comments

<Wilco> 1. Implementors can update their manifest at their own pace

<Wilco> 2. Implementers MUST implement the rule as is, they aren't allowed to implement a variation of it

<Skotkjerra> +1

<rdeltour> +1

Wilco: Does this look all right? No comments, so let's go with this

<maryjom> +1

Wilco: Any other comments to Shadi's comments?

Stein Erik: We are committing to a lot of work in the coming time

Issue #116: Output data format for rule groups https://github.com/w3c/wcag-act/issues/116

<Wilco> https://github.com/w3c/wcag-act/issues/165

Wilco: Issue 116 relates to Romains point in Issue 165
... I think the question here is valid. How do you format results that come in a Rule Group. And it relates to how we do it with aggregation.

Stein Erik: 116 has a dependency on 165, so I suggest we start with that one

Romain: Was there any progress on setting up a task force to work on the JSON MD format?

Wilco: Shadi took the lead on that, but some of the people in auto-WCAG were also interested in being part of that, so I think we should consider involving a broader audience than ACT

Romain: I think the output format depends on serialization

Wilco: I am not sure I agree to that. First we need to find out what we do with Rule Groups before we work on the output format

Romain: I think also for a single Rule, not only Rule Groups, the output format was not very clear for me

<rdeltour> https://github.com/w3c/wcag-act/issues/162

Romain: I think that before we start to look at the Rule aggregation, we need to look at the output format for a single Rule. At least it's not crystal clear to me
... basically we say that the outcome of a Rule is a collection of results for individual test targets. So even for one Rule in one document the result is already some kind of collection
... and I don't think it's very clear how this collection is calculated
... and we sometimes talk about the result for single test target and sometimes for a whole Rule
... and I think these are the things we need to be very clear about when we start talking about aggregations

Wilco: We don't want to be too prescriptive
... it's not a spec for implementing Rules, it's a spec for writing Rules
... As a compromise, the spec could tell Rule writers how they can expect the Rule to be used

Romain: I think we need to define the output format, that was the idea of the data format section. We don't define the excact serialized formats that can be expressed in JSON, XML etc.

Wilco: Can you work on a proposal, a high-level bullet list on the things that we need to change, not actually writing them

Romain: I think I started that in 162, but I will work on clarifying it

Wilco: Just keywords on what needs to be added and removed where

Stein Erik: That would be great for getting an overview of the consequences of these changes

Issue #139: Consider a new section on "Evaluating a Rule"? https://github.com/w3c/wcag-act/issues/139

Romain: I haven't made any progress on this one yet

Wilco: Which actions are we expecting to come out of this one?

Romain: At least in earlier versions of the spec, maybe not so clearly any more, we had wording that mixes statements on the Rule format with information on how Rules are processed

Wilco: Would it make sense to create a non-normative section that describes how a Rule can be used?

Romain: I agree that a descriptive section is a good idea
... I could work on writing a section on this

Wilco: Informative sections are also something we could ask Moe to write

Stein Erik: I'm a bit concerned about adding too many informative sections, making things too complex. Does this description have to live inside the spec?

scribe: I think we should try to limit the amount of informative sections
... We could consider supporting documents that are easily updated, doesn't have to go through a review process
... especially considering Detlev's comments, we can write something that is a bit more pragmatic

Wilco: Interesting idea. An "Understanding ACT Rules". Then we won't have to keep it as short, we can just update it along the way

Stein Erik: Something like the "Inapplicable is a pass in WCAG" could also live there

Wilco: I doubt we will get away with that, that is a sensitive area...

Mary Jo: I agree that it's a good idea to split out the explanatory information. When working on the presentation for CSUN it was difficult to find out what is actually part of the spec and what is informative

Romain: For this particular issue 139, we probably need some normative parts too, some algorithms for aggregation

Stein Erik: Maybe we should include a comment on that in the reply to Detlev's comment, Issue 174, on the part about the tone being stern

Wilco: done
... Some more stuff, that might need to go into an Understanding document

Stein Erik: I think Romain has a very good point, because it doesn't make sense for a manual Rule to be compared to an expert evaluation

scribe: this is something that needs to be solved to make the ACT Rules spec useful for semi-automated and manual Rules

Wilco: I am not sure I completely agree

Stein Erik: How would you then evaluate a manual Rule?

Wilco: I think I would go with the assumption that it's non-accessibility experts, e.g. QA people, that are using the ACT Rule

Stein Erik: So you would compare the non-expert results of an ACT Rule to the gut-feeling of an accessibility expert?

Wilco: Yes

Stein Erik: We know about the disagreement among accessibility experts, so it's an uncertain method for validating something that we want to validate the certainty of

scribe: the problem that we are trying to solve is the inconsistent way of evaluating accessibility, so it doesn't make sense to use this inconsistent way of evaluating as a measure for how precise our Rules are
... then we at least need to compare different expert's evaluations

Wilco: What we are trying to do is to find false positives. There could be cases where the Rule doesn't cover an edge case, and that is what we want to find with expert reviews

Stein Erik: But we also want to find false negatives

Stein Erik: And we shouldn't assume that the expert is doing perfect every time

Wilco: No, but every time there is differences, there is something to look into
... it should be done regularly to check that the Rule is doing what it should

Stein Erik: Maybe we need to add something about frequency and different evaluators

Stein Erik: Maybe we should involve some manual testers to give input on the validation methodology

Romain: We can add a note that this validation only apply to automated Rules, and we cannot validate manual Rules

Stein Erik: Are there some way we can collect some feedback from people working with manual testing methodologies

Wilco: I will take this on and work with it to see if we can come up with a solution

Issue #165: Clarify the definition of Rule Aggregation https://github.com/w3c/wcag-act/issues/165

Already covered

Wilco: Update on the publication of next draft
... Working Group only has comments now on getting rid of some typos
... Shadi and Wilco are working on it
... It will be up for review for around a month, closing around the time of CSUN

Summary of Action Items

Summary of Resolutions

[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.152 (CVS log)
$Date: 2018/02/23 14:29:03 $