W3C

WCAG 2.0 Evaluation Methodology Task Force Teleconference

14 Feb 2013

Agenda

See also: IRC log

Attendees

Present
Vivienne, Shadi, Martijn, Detlev, Liz, Mike, Katie, Kathy, Sarah, Eric, Peter, Moe
Regrets
Tim
Chair
Eric
Scribe
Sarah

Contents


<shadi> https://www.w3.org/2002/09/wbs/48225/WCAG-EM20130208/results

<shadi> https://www.w3.org/2002/09/wbs/35422/WCAGevalTaskForce/results

shadi: first internal survey, we don't need to fill in both surveys
... objection from Samuel - does not object to idea of adding a note; can be discussed after public review; editor note will be added to step 5.c
... text from current working draft will be used for the public comments draft

<shadi> [Review Note: Eval TF is particularly looking for feedback on this section.]

+1

<ericvelleman> +1

<MartijnHoutepen> +1

<Liz> +1

<Vivienne> _1

<MoeKraft> +1

<Detlev> plus one

<Kathy> +1

<Vivienne> +1

katie: +1

<Mike_Elledge> +1

peter: +1

<shadi> kathy: +1

<ericvelleman> https://www.w3.org/2002/09/wbs/35422/WCAGevalTaskForce/results

eric: most responses accept as is

<shadi> [[I think there is still sometimes ambiguity in the term WCAG 2.0 Techniques, which sometimes is used to refer to any technique that meets a WCAG success criterion and sometimes to refer to the techniques published in the Techniques document. It is not necessary for this draft, but at some point please make a review and use consistent terminology to distinguish those two meanings. You may need to define those two terms in your document to distinguish them.]]

eric: conformance for particular situations - not sure what we should do with this one; will be discussed in wcag group tonight

shadi: loretta asks for a review, so shadi and eric will look for obvious ambiguities and adjust wording for next draft

eric: proposed solution ok?

+1

<shadi> shadi: propose Eric and Shadi to take a quick pass at catching obvious ambiguity, then do a more thorough review of terminology before the next publication

+1

<ericvelleman> +1

<MartijnHoutepen> +1

<Vivienne> +1

<Detlev> fine

<Mike_Elledge> +1

<Liz> +1

<MoeKraft> +1

<Kathy> +1

<ericvelleman> https://www.w3.org/2002/09/wbs/35422/WCAGevalTaskForce/results

<ericvelleman> https://www.w3.org/2002/09/wbs/48225/WCAG-EM20130208/results

vivienne: looking for samuel's actual comment

<Kathy> I am still working on an updated version

eric: evaluation flow graphic

kathy: using waterfall diagram that Peter came up with, and taking colorful style from previous one. also, trying to show the interaction without having it be overwhelming

shadi: we need to have final version by tuesday for approval process
... deadline for external comments is today

<shadi> [[Samuel's comments resent: http://lists.w3.org/Archives/Public/public-wai-evaltf/2013Feb/0022.html]]

shadi: final version for public review is due tuesday

kathy: will try to have revised diagram by saturday

<Detlev> fine

<MartijnHoutepen> +1

<Mike_Elledge> +1

<Vivienne> that sounds fine to me

eric: proposal to use new diagram if everybody agrees. otherwise we will use the existing diagram if there are issues with the new one

+1

<Kathy> +1

<Liz> +1

<MoeKraft> +1

<ericvelleman> +1

shadi: please fill out the survey - need to have comments if there are concerns

peter: still concerned about sampling template and not being able to test web apps in their entirety
... should there be something other than a total conformance claims for cases where you can't test everything, e.g. web applications and software? a conformance (or another word) claim (or another word), like 'to our knowledge'?
... want this to be helpful for others who haven't worked on the website

shadi: agrees that this a bigger issue than this methodology which is bound to WCAG - we are not chartered or scoped to address this issue. inherent aspect of wcag, that the methodology applies to the entire scope of the website
... we will not be able to address it in this draft, even though this is an important issue

peter: want to avoid the language, 'alternative to a conformance claim'. this is not an alternative. really likes the phrase, 'evaluation findings.'
... is not recommending that this issue hold up the public working draft

<Vivienne> Sorry guys, I can't get back on the call

eric: can you really do a conformance claim, maybe to the 'best of our knowledge' would be good to do in the next version of the working doc.

<Vivienne> zakim keeps tellingme my passcode is not valid!

detlev: all or nothing evaluation methodology is difficult. evaluation findings is usually more the case. we don't place ourselves outside of the practical side of this
... using conformance claim, the risk is that people will downplay the risks, and impact the credibility of the claim

shadi: two aspects - peter - you can't evaluate everything - scoping issue; detlev - tolerance aspect - websites will typically have some oversight or issue; maybe they need to be addressed slightly differently

peter: developers probably can't make conformance claims. they have more knowledge, but all apps have bugs. wacg is structured around conformance claims, but nothing is ever perfect, so a conformance claim approach is problematic.
... this is not only scoping or complexity, but a mixture of both. could we put this as a action item - to explore 'evaluation finding' and challenges of making claims on dynamic, complex apps

<Detlev> thumbs up for that issue!

eric: will make this part of the "issue" list

<korn> Issue: Further explore "evaluation findings" as a concept, and language that describes the challenges of making a "conformance claim" for large sites, dynamic sites, web apps

<trackbot> Created ISSUE-13 - Further explore "evaluation findings" as a concept, and language that describes the challenges of making a "conformance claim" for large sites, dynamic sites, web apps; please complete additional details at <http://www.w3.org/WAI/ER/2011/eval/track/issues/13/edit>.

<ericvelleman> Thanks Peter

mike: conformance claims seems to be more objective than they actually are; however, for all practical purposes, even if the findings are all transparent, someone is going to compare them. they will try to create a score or metric to compare the two tools. so, if we don't establish the metric for conformance claim, someone else will.

peter: we should also be aware of this this will be used. language is important and if we suggest that this is 'the number' or
... or 'yes, it passes' can be risky, since it usually isn't true

shadi: i don't disagree with the discussion, but does this belong in wcag, or in this methodology? needs to be discussed in the wcat working group.

detlev: the methodology should draw a clear line between the 'check' and what it really does, which is show where the problems in the site are. more realistic and more usable.

peter: we are all members of wcag or eval tf, so anything has to be approved by the wcag working group anyway

shadi: it is our responsibility to bring back findings to the working group. so far in our draft, we have that you do a 'conformance check with reasonable confidence.'

<Detlev> I have been going on about that repeatedly but it did not find consinsus of course

<shadi> sometimes consensus building takes its time, Detlev :)

<Detlev> sure :-)

eric: watch for the revised diagram and get comments back to the group quickly

Summary of Action Items

[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.137 (CVS log)
$Date: 2013/02/14 16:08:35 $