WCAG 2.0 Evaluation Methodology Task Force Teleconference

Update on WCAG-EM publication

http://www.w3.org/blog/news/archives/3618

http://www.w3.org/TR/WCAG-EM/

SAZ: WAI IG announcement coming up
... feel free to circulate that around
... also to re-tweet the WAI twitter message

MK: link the WAI IG announcement fromn blog post

SAZ: the W3C News item comes first, then IG message
... but IG message will be sent to Eval TF list
... feel free to circulate that when it arrives

TB: next steps?

EV: want to do a test run
... commenting period until 28 February
... can also talk about some of the conceptual issues

Conceptual issues in WCAG-EM

EV: which areas in the current document need more discussion?

SAZ: areas highlighted in the document with "Review Note"
... sample selection, scoring, and documentation

VC: need to refine some of the sections
... read through them in detail

EV: combined with the test run to ready very carefully in practice
... also to compare with people's own current procedures
... which sections work and which don't
... and why not, if certain sections don't work

SAZ: need this to be an action?
... to read it by next week?

EV: can put it as a request on the mailing list

Test Run

EV: goal is to see how people use the methodology
... do people come to the same evaluation result?
... important time to specifically test this
... do an evaluation and compare the outcomes
... need to decide what we want to measure, which website, and how to capture the data
... and also find people willing to do a full review
... probably take a full-day worth of work for this

VC: have been following WCAG-EM with my team
... works very well but not sure what you are asking for
... would need to compare it to something else

TB: would need to compare it to another methodology
... do a comparative study

RW: need to run WCAG-EM and our own methodologie
... and provide feedback on how it worked in practice
... usually needs a couple of runs until you get a hold of it
... need a panel to do that

EV: could we ask people to use WCAG-EM on the same website
... and compare what people did in each case

<Sarah_Swierenga> +1 I agree that we should use the same website.

<Vivienne> We still have our university library's approval to use them as an example for this comparison

EV: like which sample did they pick, how they got the results, etc

<richard> +1 for same website

<Vivienne> +1

<MoeKraft> +1

MK: would be helpful to have a survey to help organize the feedback
... liked previous organization of results

RW: can do that for the new data as well

EV: document has flexibility in it so might not be as comparable
... but what data do we want to collect?
... think we need to pre-decide the website but also the scope

<richard> Accurate scope is essential

EV: that would set the perimeter

<MoeKraft> ------------------------------------------------------ 1. Introduction Please remember that we are testing the methodology, not WCAG or your personal ability to evaluate websites. As a member of the Eval task-force you fit into our profile by having the required expertise (http://www.w3.org/TR/WCAG-EM/#expertise) so please feel free to comment as much as possible. Please follow the text in the Methodology as close as possible. 2. Step 1: Define the Ev[CUT]

MK: previous results reminded that we are testing WCAG-EM not WCAG itself

<Sarah_Swierenga> Sorry, I need to hop off of this call. I have a few work issues I need to deal with right now. Have a good week. Sarah

<Vivienne> sounds good

<MoeKraft> +1

<Vivienne> +1

http://lists.w3.org/Archives/Public/public-wai-evaltf/2013May/0027.html

<richard> +1 we must issue step 1

RW: need to get confirmation that it is understandable and reliable
... testing if the website complies with WCAG 2.0
... looking for fails most of the time
... counting failures better shows the issues
... 54 problems is more powerful than 17 of 34 Success Criteria met

EV: two evaluators could have different interpretation of the Success Criteira

SAZ: true, that happens
... but we'd be discussing WCAG 2.0 rather than WCAG-EM

EV: want to understand if differences are on Success Criteria level versus sampling
... this would give feedback on WCAG-EM

SAZ: think listing in Step 5.a would produce sufficient data

VC: evaluating a full website is probably more than a day
... about 40 hours

EV: already shows some differences in approaches

VC: we use teams as well

<MoeKraft> +1

<richard> Could do one

<EricVelleman> we could also do one

<MoeKraft> possibly with help from my team

<Vivienne> I could but not until after March 31

Face-to-Face Meeting

SAZ: looking more likely but not yet fully confirmed

MK: MaryJo will need to know the dates pretty soon

<EricVelleman> Shadi: We are looking at monday and tuesday

<EricVelleman> Tuesday WCAG is meeting

<EricVelleman> Shadi: We will hear within next week more news