WCAG 2.0 Evaluation Methodology Task Force Teleconference

Walkthrough smaller changes in editor draft

<ericvelleman> http://www.w3.org/WAI/ER/conformance/ED-methodology-20130903

Eric working on a new editor's draft, so woud like to skip over topics we discussed earlier.

Vivienne - suggest we say "this methodology describes the min. req. for one approach to evaluating a webiste"

<shadi> peter: keep in mind that every part of this note is advisory

<shadi> ...but vivienne's wording highlights this well

Mike: ~+1 to Peter. EvalTF is a pretty "weighty" document. Anything we can do to help folks understand that this is simply 1 way of doing this. Also keep the document streamlined.

<Mike_Elledge> +

<shadi> peter: could try progressive disclosure approach

<Kathy> +1

<shadi> ...first describe methodology at a higher level

<shadi> ...then drill deeper into specifics later on

<shadi> ...not sure what specific changes are needed for this

Kathy - a verbal +1 as well.

Kathy: adding a summary section at the top might be an approach to this.

<Mike_Elledge> +1

Eric: another approach as this is throughout the docment is to put more into appendicies.

Kathy: does seem overwhelming.

Peter suggested expand/collapse widgets as an option - see WCAG2ICT doc.

<shadi> http://www.w3.org/WAI/ACT/deliverables#eval_guide

Peter: take a look at http://www.w3.org/WAI/GL/2013/WD-wcag2ict-20130905/accordion

Shadi: many folks we expect to use this document won't be the typical readers of a W3C spec.
... W3C / WAI has funding to work on interactive guides for evaluators. Could include WCAG-EM, as well as other educational materials for evaluation

Mike: discussion reminds him of how handy the old WCAG 1.0 "wallet card" was.

<shadi> http://www.w3.org/WAI/WCAG20/glance/

<shadi> called WCAG 2 at a Glance

<ericvelleman> http://www.w3.org/WAI/ER/conformance/ED-methodology-20130903#step3f

(sub-topic) Step 3F

Eric: discuss optional/non-optionality of this step

Peter: rather than the optional/non-optional-ality of this section, we should use this section to CAPTURE the fact that redundant pages were found. E.g. a site with 3 unique pages and 500 all of the same template is a site on which the evaluation will have a high degree of confidence - what is found on one/two of the 500 would likely apply to all.
... so we should retain the knowledge of the reduncancy, not just toss it out.

Kathy: agrees we shouldn't take this out; reasons to have redundancy in the sample. Esp. if the site has different developers, etc.
... so there are reasons to have redundancy in sample.

Sarah: reading the exact language, and what we are saying in 3f - if you happen to pick the same page as you did earlier, you would then select a different one (don't evaluate the precisely same page twice)
... "... thta are identical with other web pages in the sample..."

/thta/that

Eric: this discussion illustrates that we need to work on the text, as people on the call are reading the text differently.
... thinks we all agree if we mean the "same page" we should take it out.

Peter: interpretation of "... that are identical with other web pages in the sample..." as meaning the precisely same page is not consistent with "Filter the sample to eliminate excessive redundancies." (as "excessive" suggests that sometimes looking at the same/identical page multiple times is OK, so long as you aren't doing it "excessively"

Eric: we all agree that taking identical pages out is OK?

Peter: put "uniqueness" of items in the sample into step 3.e. Then 3.f can be a discussion about "appearing to be similar", and about hetero-/homo-genaity

<Sarah_Swierenga> +1

<Vivienne> Yes, I agree with the other callers about this

Eric: thought's on Peter's suggestion?

<ericvelleman> +1

<MartijnHoutepen> +1

(sub-topic) Section 5 introduction

<ericvelleman> http://www.w3.org/WAI/ER/conformance/ED-methodology-20130903#step5

(sub-topic) move now to discuss performance score

<shadi> peter: if we don't have a common way to score, there can't be comparability

<shadi> ...but again don't think should have performance score whatsoever

Kathy: agrees with Peter - so much that can go into a score (not just WCAG 2.0 compliance, but severity of issue, impact on various different PwDs, needs) it get so very difficult.

<Kathy> http://jimthatcher.com/favelets/

Kathy: maybe look at what Knowbility has done for their scoring, also Jim Thatcher's work. So may want to lok at that, lots of different ideas there.

Vivienne: understand's what Peter is saying, his concerns. But it's not just because people want it; in some situations it is required (for when you are comparing things).
... so need some way of recording things in a stastical manner. How to compare the same website over time. So has no problem with saying "scoring isn't part of the scope", but still providing some suggestions on this. Because the question is going to get raised, etc.
... we shouldn't just bury our heads in the sand and say "don't do it". We need to deal with the issue.

Eric: we should find some way to give a score, even if we provide caveats around it.

Shadi: is going to be somewhat contradictory... (is really torn). Agrees with lots of the arguments raised. There are uses cases for scoring, use cases against.
... we previous agreed that we wanted to provide scoring for comparing their own websites over time, not to compare different websites with each other.
... so thinks it might be good that we don't provide a single approach, but rather several they can use. We are far away from comparing different sites effectively.
... concern is that people use this to compare against each other, in which case maybe we should dig our heads into the sand.

<ericvelleman> We do not make it a step but make it a section about Performance scores, and say we will not make it. They can be helpful but are dubious. Then have appendix about resources of performance scores… Not part of WCAG-EM to make performance score.

<MartijnHoutepen> +1

Peter: perhaps a way forward: (a) we don't make this a formal step, (b) we talk about all of the challenges in performance scores - that they are NOT valid for inter-site comparisons but can be helpful for giving a quick look at improvements/regressions in a single site, and (c) have an appendix with resources for folks that want to use them ONLY for site improvement evaluation.

<Vivienne> sounds like it might work - depends on the wording

<Sarah_Swierenga> +1

Eric: will put this onto a survey; thinks this may be the best way forward.

Moe: have no problem with this as a separate section, but want to echo some of the thoughts/ideas discussed. May be that you have a single evaluator looking at multiple websites, using the same scoring methods to compare them (may also be for the same commissioner).
... so comparing multiple websites could be OK. But no problem in a separte section.

Further steps and agenda

Eric: working on a survey for final changes on the doc, & new editor draft. You will first get the survey, then next week we can discuss outcomes from that survey.