WCAG 2.0 Evaluation Methodology Task Force Teleconference

05 Sep 2013

See also: IRC log


Mike, Liz, Shadi, Martijn, Eric, Kathy, Detlev, Peter, Sarah, Tim, Vivienne


Eric: welcome to the call
... tried to incorporate all changes into the recent Editor Draft

new editor draft

<ericvelleman> http://www.w3.org/WAI/ER/methodology/

<ericvelleman> http://www.w3.org/WAI/ER/conformance/comments-20130226

Eric: key topics to discuss, likely on the mailing list in the coming week...
... New conformance section, changes in 1b, how to choose a sample
... in document, noted where the change comes from (e.g. disposition of comments), why the change, etc.

Discussion start about new section Conformance in the context of WCAG-EM

<ericvelleman> http://www.w3.org/WAI/ER/conformance/ED-methodology-20130903#conformanceintro

Shadi: issue with the title of the section... "Conformance in the context of WCAG-EM"
... may need to tweek the wording of the title. Also comment AC81 (from Peter) and the use of the word "statements"
... also should be clear we aren't proposing a new/different conformance model here.

<Detlev> Why not "Step 5.b: Provide an Accessibility Evaluation Statement"?

Fredrick: worried about using the word "conformance" itself here. Could cause confusion.

Eric: uses the term "Accessibility Evaluation Conformance Statement", to try to be distinct.

Fredrick: Prefer to not use "conformcne" here; so "Accessibility Evaluation Statement"

Peter (from earlier in the thread): purpose / goal of this new section is to make clear that WCAG conformance claims cannot be made, by their definition, to our output, as our output is intentionally a sampling.

<Vivienne> Sorry i was late in from a meeting . Can someone fill me in on what section we're discussing?

<shadi> http://www.w3.org/WAI/ER/conformance/ED-methodology-20130903#conformanceintro

Peter (still from earlier): therefore we want to lay out the issues in this section, and set forth the idea that the output of WCAG-EM is a statement addressing the extent to which conformance was found.

Shadi: likes what Tim is saying - avoiding the words "conformance" and "conformance claim" as much as possible.
... Q (to Peter perhaps): is it that we are trying to define the scope of WCAG-EM here?
... thinks that the second paragraph is more important; maybe flip the order. Seems we are more describing the limitations & scope, rather than any conformance requirements. Hence the disconnect of the title from the contents of the section.

Peter: appreciate to try to avoid conformance but important to talk about it when we introduce the evaluation statements or such
... because conforms is used inherently and needs to be addressed straight on
... need to address the many ways people will use this
... agree that title could be improved
... maybe "output from WCAG-EM"

<shadi> ...but critical to have description of these aspects of the document

Eric: suggests we suspend further discussion of this section here, and shift it to the mailing list.

Discussion start about changes to Step 1.b

<MartijnHoutepen> http://www.w3.org/WAI/ER/conformance/ED-methodology-20130903#step1b

<MartijnHoutepen> Define the goal of the evaluation

<shadi> http://www.w3.org/TR/WCAG-EM/#step1b

Eric: changing 1b to be about "additional requirments", beyond what is set forth in WCAG-EM

Peter: might be good to list examples of some of the things that are additional
... not sure if the term "requirements" is precise
... maybe better "aspects" as they are more content than process aspects

Detlev: might this also be a place to define an eval that is more limited (e.g. to explicitly NOT evaluate against some SCs)
... or to evaluation portions of a page (since WCAG applies to entire pages)
... thus you might skip "Page title", etc. since it wouldn't apply to a portion of a page.

+1 to this from Peter

Shadi: paraphrasing Detlev: if you have a bunch of templates / code objects / pieces of content, and you want to evaluate those on their own, and make an eval statement about those, you would want to set that forth here in step 1b.

Eric: this would be the place to put such a thing, though maybe a bit different of an "addition" (being a subtraction) than we had at first thought.

Peter: perhaps "define any modifications to the evaluation process"
... then in examples, we might have the one Detlev set forth

Shadi: need to look back at the section "Using this methodology" and "scope" and "applicability". Those sections also list types of websites.
... we encourage the use of this methodology in different situations, but the primary focus is the website as a whole, as presented to user.
... lots of disucssion about incremental eval, development eval, regression eval. But... adding this (only) in step 1b may break what we said earlier. Perhaps it should (instead) be said earlier?

Eric: pausing this disucssion here, to shift it to mailing list.
... Primary observation from here is from Detlev - focusing eval to a subset of a page

Shadi: in the spirit of "addressing things head on", it might really be good under "scope of applicability" to have a section saying "you can still apply this guidance in other situations" and list them (such as Detlev's "part of a page", also "during development", etc.)

<MartijnHoutepen> +1

Discussion start about changes to Step 3: Select a Representative Sample

Eric: took outcome of earlier discussion (even from last year) and put it here. Not sure if it should be here, or in different form...
... added "factors related to likelihood of a sample being representative"

Peter: likes section, but remains concerned with the lack of statistic rigor guidance here for how to determine the "likelihood of a sample being representative"
... suggests we might again do more website testing under WCAG-EM with a focus on how similar/different the results are, to inform how much further guidance may be needed here for "representative sample" and "confidence"

Eric: concerned that there are so many ways in literature for doing sampling, we don't want to pick one above others
... +1 to doing another test run again

Discussion start about changes to Step 3.d: Include Complete Processes in the Sample

Detlev: evaluators will need some guidance for following processes (not just "put in URL of a page"). Menus, forms, lightboxes, etc.
... only way to include such generated pages is to refer to the "base page" and how to get to the "generated page"
... sets forth doing it the "default" way, and then also suggests looking at some variants (e.g. input error)
... an obvious can of worms for very complex sites - all the potential complexity, and need to draw the line somewhere, make a judgement call as to how far to go.

Eric: this is one place in the methodology where we really take people by the hand. Originally just had a little text here; now we are doing much more leading people through "how you do this". Perhaps we should look at doing that amount of leading in other places.

Detlev: risk is that we are too descriptive - too focused on a particular type of website. Need to think about that too.

Peter: consider "at least" in number 2; make clear that going through the default branch is a minimum.

Eric: discussion about the prescriptive-ness of this... anyone think this is too much? That we should do more of that level elsewhere?

Peter: seems like the right direction but maybe not the right level of prescriptiveness

<shadi> ...a bit more elsewhere may be helpfull too but need to check

Kathy: good to have this level of detail in this section, but have to be careful. If you lead too much by the hand, it may not fit certain circumstances. So agrees - evaluate level of prescriptive-ness on a section-by-section basis

Vivienne: likes level of detail here. Finds with clients that guidance around complete processes is necessary.
... thinks we will find other areas where more point-by-point instructions will be very helpful.

Mike: thinks we keep running into issue of how to define a representative sample. What is sufficient... A little reluctant to make things too detailed, but this seems like a good place for this level of detail.
... maybe have examples here as well.

Eric: (summarizing) not hearing people saying they don't want this level of detail; should discuss on mailnig list whether we want this much, and where else in document we want this level of detail (or at least more detail than is there presently)
... will start discussions of each of these on the mailing list (after Dutch dinner), and hopes we continue discussions there.
... next week, hopes to have a survey so we can close down some of these discussions

Other issues

<Mike_Elledge> Bravo!

Shadi: really great work Eric

Summary of Action Items

[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.138 (CVS log)
$Date: 2013/09/05 15:10:16 $