WCAG 2.0 Evaluation Methodology Task Force Teleconference

Section 4

EV mentions discussion on sampling, wanted to send newer version of methodology

EV: lots of discussions on scope, will be worked into draft
... sampling: 1 core resources, 2 Complete processes, 3 Random sampling
... Question: what are the minimum requirements for sampling?

<Mike_Elledge> looked good to me

Kathy: liked task-oriented sampling approach, identify core tasks by differnet types of users. Random sampling also OK, mor as validation

EV: Barriers and amoount of pages in sample an issue

Kathy: Regarding processes: if you run automated scan, that indicates pages with more technical issues

DF: raises issue of dropping issues in evaluation when they have been identified once, they may be played down too much

Sarah: asks clients about the different templates used to assist in page sampling; start with core pages then extend
... concern about the random sample is the number of pages ending up in the sample

EV: agree to add templates as important element

Vivienne: Important to go through WCAG checkpoints to select pages to whcih as many checkpoints as possible apply
... also uses random sample, but mainly uses combination of pages manually selected. Complete processes must not be missed

ME: Frequency of issue appearing - raises question of consistency. Templates help to check consistency of coding in website. Problem with limited sample is that developers may claim a structured approach but that may not be true and should be checked, ideally.

MH: In partial evaluations, can we use the same sampling rules?

<houtepen> zakim

EV: Needs discussion. Focus is full evaluations, but there may be claims for partial conformance.

MH: Maybe - full sample but parial conformance claim

EV: confirms focus on full evaluation
... any comment on the issue of trust between tester and site owner?

Vivienne: Does not see this tension as a big problem - parties need to develop trust, tester wants to be helpful for project owner, gets more out of it if communication is open and trustful

EV: Input from developer regarding setting the page sample OK?

Vivienne: owner should not dictate sample, but communicate to assist tester in sampling, explain scenarios of use, identify cirtical pages and processes

RW: When identifying a barrier, evaluator cannot tell site owner exactly how to fix a barrier, the developer must address the issue - a number of occurences might not be needed

Sarah: You cannot dictate trust but llook at pages and paths to check whether design and a11y is used consistently, check several parts from top to bottom
... not so much complete process but user task-based approach, looking for paths and features that tell what WCAG items are relevant
... Scenarios and tasks included

Detlev: difference between developer focus where issues are identified once, and user perspective where quantify of problems also counts

EV: raises question of automatic, semi-auutom. and manual sampling
... can you have a full evaluation with an automatic sample?

<Kathy> manual sample is important

<SarahSwierenga> Manual sample is very important!

<Tim> automatic plus manual

Vivienne: Seems clear that relying on automated tools is not sufficient, issue is choice of methods to select pages that are then checked manually

EV: Raises issue of number of pages to sample

Vivienne: Australian Government has decided for national transition strategy that 10% of the page total should be checked - on large sites this is probably not feasible...

Sarah: Above 20 or 25 pages will not make sense, takes a significant amount of time, below 20 should be OK - customers will not want larger samples, costs get prohibitive

<vivienne> I agree with Sarah. Even 20 pages would take a significant amount of time

Sarah: Discount usability procedure is an example of a weay of getting a pretty good idea of the state, there are diminishing returns with more users / pages

RW: With a site like Amazon you got 1000s of pages but templates mean tzhey are mostly similar pages; on other more diverse sites you need a larger sample. A ruigit perecntage won't work

<SarahSwierenga> Responding to Richard - depends on what you call the 'site.' You seemed to be describing a portal-type design.

DF: is it fair to selct pages with more issues?

<kerstin> sorry, I have to leave

Kathy: If client has tight budget, should areas pages identified that have key a11y issues

EV: will be difficult to make part of the methodology text...

Kathy: Value of taking selectively pages

AG: Run a tool over pages to identify pages that where relevant for each checkpoint, makimng up the sample

EV: Good approach, Alistair?

<vivienne> What kind of tool would you use to identify this Alistair?

AG: (inaudible)

<agarrison> What we used to do was run a tool over the site to identify pages which would be most relevant to each checkpoint - maybe identifying 5 - 10 per checkpoint. All identified pages went into the sample.

<agarrison> This was a good approach for WCAG 1.0 - as pages were in effect delivered to the evaluator - however, WCAG 2.0 is a bigger beast.

TB: Site analytics could suggest most visited pages to enter the sample. There is no one sampling method working across the board; therefore the evaluator should document the sampling method used

MH: Choose some pages with multimedia if there are many in the site

EV: should pages be added manually (q to Aliastair)

AG: good point to select pages that are frequently used - link that to pages to which many checkpoints apply
... Helps finding the few pages that have special elements like Video

<agarrison> http://code.google.com/p/googlesitemapgenerator/ have a look at this for selecting pages which are frequently viewed

EV: Proposes discussing number of pages in mailing list, continue discussion with AG about automatic selection of pages

Publication Planning

EV: Is this version fit to be sent to WCAG WG as draft?

Shadi: no meetings next weeks, use draft and study at the fireplace ;-)
... wonders whether current draft will give sufficient impression of the shape of methodology - may be not yet...
... use time to make it ready for publication

Availability for f2f

<vivienne> It might make a difference how much Christmas cheer we have consumed before that reading!

EV: will work in comments and hopes to publish update hopefully before Xmas

<ericvelleman_> Happily we have Sinterklaas in the Netherlands, so we already had our presents

Shadi: Availability for face-to-face meetings (survey) - some will be at CSUN, other opportunities in Europe

<agarrison> Should try England for meeting - seems to be in the atlantic between Europe and America ;-)

TPAC is a W3C meeting with other groups

<vivienne> Alistair, a little far from Western Australia.

Shadi: face-to-face good for detailed work, but expensive
... more ideas for meetings at conferences are welcome, but may be difficult to get to

EV: Shadi might send list of meetings where he will be

Shadi: will attend most of them (in the survey)

EV: any other issues? No. Closes meeting.

<vivienne> okay, bye for now. A very blessed Chrismtas and New Year to you all.

<Kathy> Thanks! Merry Christmas to you too... bye

<Mike_Elledge> Merry Xmas and Happy Holidays, all!

EV: Merry Xmas to all, read Methodology, meet again on 5. January