Wilco: It's between Monday and Thursday. We will stick with Thursday meetings, or else we will have no one from Daisy. On Thursdays we will still have others from Siteimprove. We will keep it as it is
Wilco: Shadi put together a response for
Detlev's comment.
... for the comment "I was a bit unsure whether the sentence
"Implementations for ACT Rules may be added, modified, or removed at any
time." applies to implementations of third parties (so saying 'do as you
like, use what feels fit') or to some centrally maintained rule base at
WAI."
... I am a bit unsure what "former" means in Shadi's response
Stein Erik: Now that I read it again, I'm a bit unsure what the boundary is between the two
Wilco: What I read first was that implementers are in charge of their own manifest documents. And we are not going to do QA on that. So if someone claims they have implemented a rule, we will trust them
Romain: Sounds all right to me
Next thing I want to talk about is Detlev's comment "Does the text need to say anything about implementers' scope for modifying rules in the way they are implemented in their tools or processes, or about any constraints on that, at least regarding claims that they 'use the ACT rule base' or similar?"
scribe: I don't think Shadi's response to this answers the question
Stein Erik: It rather dismisses the question
scribe: If you haven't implemented the Rule, you haven't implemented it. You have done something else
Wilco: All right, I'll leave these two as comments
<Wilco> 1. Implementors can update their manifest at their own pace
<Wilco> 2. Implementers MUST implement the rule as is, they aren't allowed to implement a variation of it
<Skotkjerra> +1
<rdeltour> +1
Wilco: Does this look all right? No comments, so let's go with this
<maryjom> +1
Wilco: Any other comments to Shadi's comments?
Stein Erik: We are committing to a lot of work in the coming time
<Wilco> https://github.com/w3c/wcag-act/issues/165
Wilco: Issue 116 relates to Romains point in
Issue 165
... I think the question here is valid. How do you format results that
come in a Rule Group. And it relates to how we do it with aggregation.
Stein Erik: 116 has a dependency on 165, so I suggest we start with that one
Romain: Was there any progress on setting up a task force to work on the JSON MD format?
Wilco: Shadi took the lead on that, but some of the people in auto-WCAG were also interested in being part of that, so I think we should consider involving a broader audience than ACT
Romain: I think the output format depends on serialization
Wilco: I am not sure I agree to that. First we need to find out what we do with Rule Groups before we work on the output format
Romain: I think also for a single Rule, not only Rule Groups, the output format was not very clear for me
<rdeltour> https://github.com/w3c/wcag-act/issues/162
Romain: I think that before we start to look
at the Rule aggregation, we need to look at the output format for a
single Rule. At least it's not crystal clear to me
... basically we say that the outcome of a Rule is a collection of
results for individual test targets. So even for one Rule in one
document the result is already some kind of collection
... and I don't think it's very clear how this collection is calculated
... and we sometimes talk about the result for single test target and
sometimes for a whole Rule
... and I think these are the things we need to be very clear about when
we start talking about aggregations
Wilco: We don't want to be too prescriptive
... it's not a spec for implementing Rules, it's a spec for writing
Rules
... As a compromise, the spec could tell Rule writers how they can
expect the Rule to be used
Romain: I think we need to define the output format, that was the idea of the data format section. We don't define the excact serialized formats that can be expressed in JSON, XML etc.
Wilco: Can you work on a proposal, a high-level bullet list on the things that we need to change, not actually writing them
Romain: I think I started that in 162, but I will work on clarifying it
Wilco: Just keywords on what needs to be added and removed where
Stein Erik: That would be great for getting an overview of the consequences of these changes
Romain: I haven't made any progress on this one yet
Wilco: Which actions are we expecting to come out of this one?
Romain: At least in earlier versions of the spec, maybe not so clearly any more, we had wording that mixes statements on the Rule format with information on how Rules are processed
Wilco: Would it make sense to create a non-normative section that describes how a Rule can be used?
Romain: I agree that a descriptive section
is a good idea
... I could work on writing a section on this
Wilco: Informative sections are also something we could ask Moe to write
Stein Erik: I'm a bit concerned about adding too many informative sections, making things too complex. Does this description have to live inside the spec?
scribe: I think we should try to limit the
amount of informative sections
... We could consider supporting documents that are easily updated,
doesn't have to go through a review process
... especially considering Detlev's comments, we can write something
that is a bit more pragmatic
Wilco: Interesting idea. An "Understanding ACT Rules". Then we won't have to keep it as short, we can just update it along the way
Stein Erik: Something like the "Inapplicable is a pass in WCAG" could also live there
Wilco: I doubt we will get away with that, that is a sensitive area...
Mary Jo: I agree that it's a good idea to split out the explanatory information. When working on the presentation for CSUN it was difficult to find out what is actually part of the spec and what is informative
Romain: For this particular issue 139, we probably need some normative parts too, some algorithms for aggregation
Stein Erik: Maybe we should include a comment on that in the reply to Detlev's comment, Issue 174, on the part about the tone being stern
Wilco: done
... Some more stuff, that might need to go into an Understanding
document
Stein Erik: I think Romain has a very good point, because it doesn't make sense for a manual Rule to be compared to an expert evaluation
scribe: this is something that needs to be solved to make the ACT Rules spec useful for semi-automated and manual Rules
Wilco: I am not sure I completely agree
Stein Erik: How would you then evaluate a manual Rule?
Wilco: I think I would go with the assumption that it's non-accessibility experts, e.g. QA people, that are using the ACT Rule
Stein Erik: So you would compare the non-expert results of an ACT Rule to the gut-feeling of an accessibility expert?
Wilco: Yes
Stein Erik: We know about the disagreement among accessibility experts, so it's an uncertain method for validating something that we want to validate the certainty of
scribe: the problem that we are trying to
solve is the inconsistent way of evaluating accessibility, so it doesn't
make sense to use this inconsistent way of evaluating as a measure for
how precise our Rules are
... then we at least need to compare different expert's evaluations
Wilco: What we are trying to do is to find false positives. There could be cases where the Rule doesn't cover an edge case, and that is what we want to find with expert reviews
Stein Erik: But we also want to find false negatives
Stein Erik: And we shouldn't assume that the expert is doing perfect every time
Wilco: No, but every time there is
differences, there is something to look into
... it should be done regularly to check that the Rule is doing what it
should
Stein Erik: Maybe we need to add something about frequency and different evaluators
Stein Erik: Maybe we should involve some manual testers to give input on the validation methodology
Romain: We can add a note that this validation only apply to automated Rules, and we cannot validate manual Rules
Stein Erik: Are there some way we can collect some feedback from people working with manual testing methodologies
Wilco: I will take this on and work with it to see if we can come up with a solution
Already covered
Wilco: Update on the publication of next
draft
... Working Group only has comments now on getting rid of some typos
... Shadi and Wilco are working on it
... It will be up for review for around a month, closing around the time
of CSUN