WCAG 2.0 Evaluation Methodology Task Force Teleconference

EV: updated Overview document by EOWG http://www.w3.org/WAI/eval/conformance
... updated Working Draft out for review
... now expecting comments on it

http://lists.w3.org/Archives/Public/public-wai-evaltf/2013Feb/0009.html

PK: prior comments from ERT WG and WCAG WG that we said we'll address after publication
... need to put these in our list too

EV: will review previous disposition of comments
... and create a new one

MK: deadline for comments is not very long

EV: can extend if needed

SAZ: can extend if needed, but also want to have new draft before the summer breaks

EV: can keep the deadline for now and extend in the last week

Testing WCAG-EM

EV: proposal for Eval TF participants to check a website using WCAG-EM
... and compare the results we get

PK: like the idea

VC: good test for our process
... but tricky which website we pick
... good to use the same website
... need to consider what features it presents
... should cover as many WCAG 2 requirements as possible

<Sarah_Swierenga> +1

<MartijnHoutepen> +1

<MoeKraft> +1

<Vivienne> +1

EV: people like the idea?

MK: wonder if we could reach out to the WCAG community for someone to volunteer a website

PK: ideally would be testing two sets of things
... the level of how we are all following the same guidance
... but also looking at what the challenge areas are in the guidance
... for example web applications could be a challenge area
... could be different then how effective the guidance is for people to follow equally

EV: may mean that we look at more than one website

SAZ: Kathy had the idea of splitting volunteers into groups
... then we can address coverage of different types of websites
... but also have comparable notes

PK: could also use fictional websites

SAZ: we have the Before and After Demo (BAD) but quite difficult to develop such sites
... might be better to use existing ones

VC: may have some candidate sites in mind

SAZ: please send to me and Eric before contacting them
... need to determine volunteers first

SW: could have someone do the sampling then others evaluate these samples
... otherwise not comparable results

PK: part of what we are testing is the sampling methodology
... need to include the sampling as part of the testing
... if we get diverging results then the methodology is not ready for prime time
... could have one group follow this approach
... or be able to factor out the sampling aspect
... but need to also test how the sampling guidance works

MH: could also evaluate our evaluations after each step
... like compare notes after each step

EV: had similar idea in a project in The Netherlands
... first see what samples people come up with
... then all evaluate the same sample
... but could be extended to different scenarios

SW: agree with Peter, like the idea that everyone has the same core of web pages
... but need to test the sampling procedure as well

[Proposal: ~3 teams of ~3-4 people to evaluate a website per team in two phases; phase would combine steps 1-3 of WCAG-EM (then compare notes); phase 2 would combine steps 4-5 of WCAG-EM (then compare notes)]

PK: do we need teams?
... need to compare notes
... but if they compare notes in between it mixes results

EV: so not comparison between the phases?

PK: comparing notes part way through could muddy the results
... why have people talk if we are trying to test the methodology in its entirety?

EV: should get to the same result with different samples

<MartijnHoutepen> I will volunteer

<Vivienne> I'm happy to and I can get some of my staff to help

<Sarah_Swierenga> +1

<MoeKraft> +1 I would like to be involved and/or have someone from my team assist in this effort.

SAZ: the more data points we have the more we know what we need to improve

<MoeKraft> I will be away however from 3/28-4/11

EV: who would volunteer?

SAZ: need to ask absent people as well

EV: ask for public volunteers?

PK: could do 2-step
... inviting the public at this stage would impact confidentiality

EV: already asking people to test the methodology

VC: better if we don't for this first test
... because we know more about how this methodology was intended
... would be testing too many parameters if we open it up

SAZ: agree with Vivienne, need to do our homework first

<ericvelleman> <https://www.w3.org/2002/09/wbs/48225/evaltfq9/>

Survey 09

EV: your input on what we need to discuss in next steps
... already good suggestions there
... but please add others you think are missing
... or that need additional attention