WCAG 2.0 Evaluation Methodology Task Force Teleconference

27 Mar 2014

See also: IRC log


Shadi, Liz, Kathy, Detlev, Sarah, Alistair, Tim
Moe, Eric, Martijn
Sarah Swierenga, Detlev


Debrief from Face-to-Face meeting

<shadi> http://www.w3.org/2014/03/17-eval-minutes

<shadi> Shadi_AZ, MaryJo_M, Eric_V, Katie_HS, Kathy_W, Tim_B, David_M, Michael_C (partially), Mike_E (partially), Judy_B (partially), Shawn_H (partially), Klaus_M (partially)

<shadi> http://www.w3.org/WAI/ER/conformance/comments-20140130

during the face-to-face mtg , we went through all substantial comments received so far

We relayed to the WCAG working group that there is a need for point to point evaluation of individual success criteria.

We also discussed the issue of scoring, but we need to have more discussion about that

Call for Participation: WCAG-EM Test Run

<shadi> http://lists.w3.org/Archives/Public/public-wai-evaltf/2014Mar/0083.html

april 10 is the preferred deadline for the internal website test run

we need to gather experiential feedback on the EM from the test run to finalize the document

<Liz> Liz volunteered

Sarah: UARC is almost done with the review, and will be filling in the survey early next week.

Detlev: no audio
... questions about external eval

Liz: looking at survey that Eric sent. He asked several questions about timing. I am willing to try the methodology, but am questionning the need for very precise timings.

Shadi: does not need to be precise/seconds on timing, if possible, will be helpful.

Liz: I will be adding comments about the EM, and will try to get some timings.

Shadi: Correction: timings do not need to be precise, but general timings are helpful
... we want to have varying level of expertise using the EM
... We want to understanding general timings for various experience levels, to see what we can do to minmize the time involved to do the review.
... We want to get as many internal and external surveys completed as possible. Two surveys are up and running.

<Detlev> http://lists.w3.org/Archives/Public/public-wai-evaltf/2014Mar/0082.html

Detlev: External/open survey - it is a good idea to communicate that we want lots of feedback - even if they don't complete the whole survey. Concerned that we have 0 responses.
... The open survey url doesn't seem to have been distributed widely. Can we do more to get the news out?

Shadi: Eric and I recruited heavily at CSUN.
... We don't typically ask for this level of volunteer work for these kinds of projects. Also wondering if we will get the right people if we broadcast too widely.

Detlev: This is a public doc and a EM that we expect the public to use when it's published.
... We should ask people to try it out, and get their feel for how the EM works. We should send it out broadly to get as much participation as possible.

Shadi: We have published the docuement to get public comment, and have asked people to do a test run (which is more unusual for a W3C working group). Is there a quality vs quantity concern?
... We should certainly reach out to people who currently do evaluations to try out the EM.
... Does the group know of other people to reach out to for the test run?
... time remaining is working against us now, too. We would likely need to extend the timeline.

Detlev: We still need to get the word out. This is a small group, so we need an outside view. Awareness is key, and it is a shame not to get that broader feedback. We need to work harder to get more participation - even a handful would be helpful.
... Recommends around 10, but we need to ask many to get a few to actually conduct the evaluation.

Shadi: If each of us recruits 1-2 people to participate, we could get the feedback we need.

Detlev: is willing to post to WebAim list. is that ok?
... Disappointed that we haven't seen broader requests for participation in the test run.

<Detlev> I won't

Shadi: You can send to the WebAim list, but we really want people who are qualified to do the evaluation, and who can take the time to do the evaluation, so that we can get valuable, targeted feedback.

<Detlev> Ok, fine.

Shadi: we have the public comments and survey out there right now. Shadi and Eric will explore this further.

Removing version number from title

Shadi: Do we need a version number in the document.
... every pub is dated

Sarah: having 1.0 in the title might be confusing since wcag is 2.0.

<shadi> http://www.w3.org/TR/WCAG-EM/

Shadi: every pub in TR space has a date, and revision dates. "this version" and "latest version"

Merging "Review Teams" and "Required Expertise"

+1 for no version number in the title

Shadi: concern about the review teams. the purpose of the review teams is that the evaluator needs a combination of skills and expertise, which one person may or may not have. A team approach with the combined expertise is the key.

Clarifying "Define Evaluation Methods to be Used"

Shadi: recommendation was to have the mulitple person review teams in the Required Expertise section. Think about this.
... we will also need to discuss this section in a survey and weekly call. this section needs to be refined again. Also discussed with the WCAG Working Group that we need to think about recommened techniques, but specific techniques should be clarified in the W3C Working Group.
... Green light that we reference the WG's techniques, as they clarify them more.

Continued discussion on "Scoring"

Art CSUN there was an engaged discussion with the WCAG WG on scoring, varied views of peope reg. scoring: needed for accounting fo rminor issues, afford comparison, etc, but others having valid concerns

<Sarah_Swierenga_> Shadi: We will need to come back to this item.

<Sarah_Swierenga_> Tim: Lots of discussion in his CSUN session - and concerns

<Sarah_Swierenga_> Shadi: Comment that got traction was that since any approach to scoring will have issues associated with it, that W3C should not sanction any specific scoring system. Rather provide do

<Sarah_Swierenga_> Shadi: provide do's and dont's

<Sarah_Swierenga_> Shadi: Providing a variety of scoring methods, but let people know they can use their own. Or, we can recommend what they doing with their particular scoring method.

<Sarah_Swierenga_> Shadi: Please perform the test run and find others to do it as well.

Summary of Action Items

[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.138 (CVS log)
$Date: 2014/04/02 18:58:26 $