WCAG 2.0 Evaluation Methodology Task Force Teleconference

22 Sep 2011

See also: IRC log


Vivienne, Kathy, Katie, Eric, Shadi, Detlev, Amy, Sarah, Alistair, Mike, Kerstin, Leonie, Vincent, Tim
Denis, Kostas, Liz, Emanuelle
Leonie, Vivienne


Discussing requirements

EV: Replace critical path analysis by processes.
... We'll use the term "complete processes"
... People asked for methodology to be cross tested. It's a good addition.

DF: Is it clear when we say we'll include reference to the test results?

EV: I can make it longer than this, agreed.
... Can we add the reson for the methodology?
... Reasonable to add this.
... Do we want to add both preliminary and full evaluation in section goals?
... My proposal is to work on the full evaluation, and keep the preliminary evaluation out of this more formal way of doing things.

AC: Will the full evaluation cover parts of a website or just complete websites?

<agarrison> Agree with Eric - concentrate on full evaluation

EV: I don't think a preliminary check would be suitable in the full methodology.

VC: We need to be clear what the term preliminary check means.
... If we decide not to include preliminary checks, we need to be clear what it is we're not including.

EV: Good addition.

KHS: For this methodology, I think we should focus on the full evaluation, but Vivienne's point is a good one.

EV: A full evaluation could be applied to a page or a full site.
... Preliminary to me is an evaluation that gives a sense of the accessibility, rather than a full evaluation.

<shadi> http://www.w3.org/WAI/eval/preliminary.html

KHS: I wouldn't call that preliminary, it's a target full evaluation?

KP: It depends on the choice over whether to test 1, or 10 or some other number of pages.

EV: We'll add in a section about co-ordination with other groups.

DF: Shadi has provided a scope that does nicely. Tim suggested checking it against other existing definitions.

EV: In section scope we should add members of the group involved in other sepcifications, for example ATAG. This was agreed.
... We decided to keep unique interpretation, but need to change the description.

AC: I like "unambiguous" better than "unique".

EV: There was a requirement for independent verification and quality control.
... Would this be a good requirement to add?

<vivienne> I'm okay with it

DF: Looking at conformance statements for WCAG, it says a list of URIs should be included to which the claim relates. This means a conformance claim has to be backed up by this list.

<Detlev> http://www.w3.org/TR/WCAG20/

<shadi> http://www.w3.org/TR/WCAG20/#conformance-reqs

AC: Could we change the wording to say "this methodology can be used by a party providing independent verification...".

EV: The idea is to say that this methodology supports independent verification.

AC: Needs to be more clear.

EV: Agreed, we'll look at that.

KHS: Say "may" instead of "can"?

TB: Are these requirements supposed to be testable?

EV: I'm not sure other W3C documents are always testable, so I don't think we have to be formal about it.

SAZ: The more clear we are, the less ambiguity we'll have.
... We need to balance between spending too much time on requirements, and having them too ambiguous to use.
... Ideally yes, requirements should be testable though.

DF: Two things have been mixed up. Independent verification is required by the WCAG conformance claim guidance, whether our requirements can be tested or backed up is a different thing.

rssagent, make minutes

EV: So we had Tim's remark about testability of requirements, and Detlev's remark about conformance claims under WCAG.

<Detlev> ok

LW: The methodology and the requirements are separate things. We need to be mindful of that.

EV: Not sure we can have a requirement for objectivity?

EV: should we add objectivity as a requirement if there is a clear way of representing it?
... maybe we need some more discussion on this
... added J - validity. Could be an extra requirement

SAZ: EARL is a machine-readable way of writing, but doesn't add to the result. It does not provide more validity. EARL would supplement human-readable reports
... re validity issue. 2 types of validity. Methodology is check conformance to WCAG 2. This is slightly different from validity of accessibility itself.

EV: we are focusing on the way that the evaluation results are documented. It is not validity of doing the tests.

SAZ: maybe we're having trouble with the term validity

DF: it is important to have validity in there. It is validity of the conformance to the requirements in WCAG. That would need to be tested and re-tested all the time for the claim you make for a site - is it true? If you re-test or have another definition you may see that the site is not valid because you left something out. It needs to be grounded by use-cases where you can see how that

works for that section of technology. Is our claim valid and can be backed up by user testing?

EV: do you mean that user testing is obligatory?

DF: it has a way of linking the technology in WCAG and those assessments for the techniques.

EV: is that not covered in WCAG 2 itself?

DF: it is in the glossary and conformance, but a methodology may not include that. I think it's important to create a model case base.

LW: having case studies or examples is a good idea. It would support that

KP: validity is important. It should be clear that we measure against the conformance level and not anything else

EV: I will do an update and send it to Shadi after this meeting.

Face to Face meeting

EV: re face to face meeting at CSun or elsewhere? Where to host the meeting? Please send information to Shadi if you can host the meeting at CSUN or elsewhere?

SAZ: face to face meeting are useful to groups working. But it might be too early now, maybe February-April and we should have lots of work by then. If you're interested in hosting such an event let Shadi know. Need a meeting space with coffee etc.

EV: any other business?
... thanks to all who participate in online discussions.Please keep up that work.

Summary of Action Items

[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.136 (CVS log)
$Date: 2011/09/26 07:16:14 $