WCAG 2.0 Evaluation Methodology Task Force Teleconference

06 Oct 2011

See also: IRC log


Samuel, Liz, Kathy, Shadi, Amy, Katie, Richard, Emmanuelle, Tim, Mike, Kostas, Alistair, Leonie (via IRC)
Detlev, Sarah, Vincent, Denis, Kerstin, Roberto



Shadi: top of list are candidates for scribing

<shadi> http://www.w3.org/WAI/ER/2011/eval/eval-tf


Shadi: some progress on list, transition from requirements phase to content development phase

<shadi> http://lists.w3.org/Archives/Public/public-wai-evaltf/2011Oct/0010.html

Shadi: Richard proposed alternative concise requirements

<shadi> http://lists.w3.org/Archives/Public/public-wai-evaltf/2011Oct/0025.html

Shadi: Note that it is not engraved in stone, initial set of requirements for better understanding, good discussions
... From my perspective, req pretty stable. Kristen raised Objective and Validity. Open that up for discussion.

Tim: Intents of requirement, when possible should be the same. Present active tense, testability, partial results. Better to be consistent. Will employ versus "should" get equivalent results. General comments. Better to use declarative present tense to evaluate when see fit, pass/fail.

Richard: Agree about the tense. Word "should" to try and limit it, discussions instead of exact comparison to be less strict. Not identical results. Used the word "should." Don't mind if changed.

Shadi: We want as much correlation as possible btw different evaluators, but how to find the wording and measure. Don't know if we can sort out at this early phase. Try to find wording we are all comfortable with, then get methodlogy where we want it to be.

Mike: Think we should have consistent headings in their structure and formatting.

Shadi: Leave this for discussion for editorial work. Does help to have consistency.

<shadi> [[RQ 01 : Define methods for evaluating WCAG 2.0 conformance

<shadi> The Methodology provides methods to measure conformance with WCAG 2.0 that

<shadi> can be used by the target audience (see section 2 above) for evaluating

<shadi> small and large websites, sections of websites or web-based applications.]]

Shadi: Req 1 - note that we did want to look at the heading for each of the req and tense. Any other comments on Req 1?

Tim: what is def of large website?

Shadi: proper wording, website regardless of size

<shadi> [[RQ 02 � Clear, understandable, and translatable language

<shadi> The methodology is written in clear language, understandable to the target

<shadi> audience and capable of translation to other languages.]]

<ssirois> rq2 is just fine with me

<Kathy> capable is fine

<shadi> [[RQ 04 - Tool and browser independent

<shadi> The use and application of the Methodology is independent of any particular

<shadi> evaluation tools, browsers, and assistive technology. It requires combined

<shadi> use of manual and automated testing approaches to carry out a full

<shadi> evaluation according to the Methodology.]]

<agarrison> should we use the term evaluation or assessment

Mike: Question about Req 2. Objectivity could be addressed in Req 2 to resolve some of the issues brought up.

Shadi: Please re-raise if forget.

<shadi> [[RQ 05 - QA framework specification guidelines

<shadi> The Methodology will conform to the Quality Assurance framework

<shadi> specification guidelines as set in: http://www.w3.org/TR/qaframe-spec/.]]

Shadi: Good question. (Agarrison) Keep for later.
... R5 pretty straightforward.

<shadi> [[RQ 06 - Reporting

<shadi> The Methodology includes recommendations for reporting evaluation findings.

<shadi> It will be based on the [href=http://www.w3.org/WAI/eval/template.html

<shadi> standard template] and supplemented with machine-readable

<shadi> [href=http://www.w3.org/WAI/intro/earl reports using Evaluation and Report

<shadi> Language (EARL)].]]

Katie: Back to Req 4. Whoever recommened use and application language. independent of any browser, tools, AT. Like that wording.

Mike: Req 6 - difficulty understanding the statement. Methodology include info on how to report the findings, or recommendations based on the eval findings?

Shadi: Should include both. What do others think?

Richard: Req 6, the template talks about WAI 1, standard template to look from for report, know what questions we have to answer, such as define website to make it valid and reliable.

Shadi: Tried to separate the two points. We can update the template. Group that maintains that is EOWG. Base intial thoughts from there and update template with methology that we have. Template could be part of methology. Standardized reporting template.
... Methodology includes "how", woudl that make it clearer?

sorry, can you repeat the changes?

<shadi> [[RQ 07 - Use of existing WCAG 2.0 processes

<shadi> Wherever possible the Methodology will employ existing testing procedures in

<shadi> the WCAG 2.0 Techniques documents rather than replicate them.]]

<shadi> The Methodology includes recommendations for howe to report evaluation findings based on ...

<Kathy> RQ 07 is good

<agarrison> is testing procedures the right term?

Shadi: "The methology employes existing testing procedures."

<shadi> Methodology employs existing WCAG 2.0 Techniques rather than replicate them

Allistair: There was a testing procedure at the end of every technique.

<shadi> The Methodology employs existing testing procedures from WCAG 2.0 Techniques rather than replicate them

<Ryladog_> +1 test procedure

Allistair: What about test procedure? nice to have one for each technique to limit it to that technique

Shadi: Diff to expalin lower level testing and the evaluation. interrelationship.

<shadi> The Methodology employs existing testing procedures within WCAG 2.0 Techniques rather than replicate them

Katie: Agree with Test Procedure. Not sure about limiting it to one technique. Test Procedure covers overall intent of the requirement, instead of identifying specific techniques.

Shadi: Tried changing one word. Instead of "from" to "within"

<Kathy> that is better


<agarrison> within is better

<Tim> better

Q from phone: methodology for manual eval. Testing and tech not complete, only for automatic eval?

Shadi: Techniques not only for automatic testing.
... Techniques on WCAG working group.

<Ryladog_> shadi is echoing

Shadi: Techniques not exhaustive. This group not focus on the individual testing procedures, but rather on overall procedure aspect.
... Work being done outside W3C. Vendors, tools, testing techniques. Open initiatives for testing procedures. Some work to compile that. Fit into methodology.

<shadi> [[RQ 08 - Recommendations for scope and sampling

<shadi> It includes recommendations for methods of sampling web pages and how to

<shadi> ensure that complete processes (such as for a shopping site where all the

<shadi> pages that are part of the steps in an ordering process) are included. Such

<shadi> selections would be reflected in any conformance claim.]]

Shadi: Title of Req 7 needs to be updates

<agarrison> "It includes recommendations" might be "it defines"

Kostas: Concerned about req about sampling. General methods, but give examples? How can we cover all of the possibilities? Effective sampling? General recommendations and methods? How can we ensure the complete process?

Shadi: good example, give some numbers, based on website size. Need to develop. One of the big parts. Req 8 and 9 the biggest parts of the methodology. Depending on how well we do 8 and 9, then number 3, replicability will emerge

Kostas: Maybe need to be more specific.

<shadi> Amy: sampling will be a big part but we have lots of experience at Oracle

<shadi> ...but ok with the wording as it stands now

Kathy: Suggest taking out the example. "Such as for a shopping site" Scenarios, concerned with throwing one specific thing in there.

Shadi: Better to define complete process in terminology section and link to it

<Ryladog_> +1

<Mike_Elledge> +1

Allistair: Recommendations, methods have to be consistent. Doesn't mean they have to be using those methods.
... Change to recommendations for scope and sampling

<Mike_Elledge> +1


<Tim> +1

<kostas> +1

<agarrison> should be change from recommendations to just scope and sampling

<Kathy> +1

<richard> +1

<Ryladog_> +1

Sorry, Allistair means, take out recommendations

Tim: WCAG and ATAG discussions on what would be in a conformance claim.

Shadi: could go in reporting section on conformance claim

<Ryladog_> +1

<richard> +1

<Mike_Elledge> +1

<Kathy> +1

<sinarmaya> +1

<shadi> [[RQ 09 - Includes tolerance metrics

<shadi> It includes calculation methods for determining nearness of conformance.

<shadi> Depending on the amount of tolerance, a failure could fall within a certain

<shadi> tolerance level meaning that the page or website might be considered

<shadi> conformant even though there is a failure. Such tolerances would be

<shadi> reflected in any conformance claim.]]

<agarrison> remove includes from title, and change includes to defines

Shadi: Remove the last sentence, but any other thoughts?

<Kathy> I agree

<shadi> [[RQ 10 - Support documentation

<shadi> The document will give a short description of the knowledge necessary for

<shadi> using the Methodology for evaluations.]]

Change Includes Tolerance metrics to Tolerance Metrics.

<shadi> [[RQ 03 � Reliable

<shadi> Different Web accessibility evaluators using the same methods on the same

<shadi> website(s) should get equivalent results. Evaluation process and results are

<shadi> documented to support independent verification.]]

Shadi: From Kriten on mailing list, that this does not go far enough.

<Ryladog_> +q

Shadi: diluted the wording somewhat, will there really be 100%, same results. What would be reliable?
... Tim also raised the concern "should"

sorry, from earlier, nevermind

<agarrison> The wording for R03 sounds nice and scientific - I support current form (should is fine)

Katie: 100% not going to happen, on the 80/20 reviewers. Will probably need something better than that when we get to the actual requirement.

<ssirois> sorry, irc connection trouble, but was on the phone all alone and haven't have something to disagry on that would have urge me to find the way to unmute myself over touchtone! ;)

Mike: "Should get equivalent and equally valid results." and define valid

Shadi: what diff btw equivalent and equally valid?

Mike: Equivalent results, but make sure valid. Maybe don't want to say equally. Maybe valid results.

Kathy: Comment on not 100% possible, if we define equivalent results.

<scribe> ACTION: Equivalent results need to be defined. High correlation. Weave in validity. [recorded in http://www.w3.org/2011/10/06-eval-minutes.html#action01]

<trackbot> Sorry, couldn't find user - Equivalent

Shadi: Mike earlier raised objectiveness and objectivity. Fits in req 2
... Reliable, R3, define equiv results and link to it

<Mike_Elledge> +1

<agarrison> +1

<Kathy> +1

<Ryladog_> +1


<ssirois> +1

<Tim> +1

<sinarmaya> +1

<Mike_Elledge> +1

Shadi: go back to Objective R2, how about Clear language that is unambiguous and understandable to audience

<agarrison> clear = unambiguous

Mike: Yes, seems to get at it to me.

Allistair: How about language that is clear and understandable to the audience.

<Ryladog_> clear is not testable

sorry, repeat please?

Allistair: Unambigous might be harder to understand

Shadi: Replace clear rather than unambigous

<Mike_Elledge> <clapping hands>


Methodology for Assessing Website Conformity

Kostas: Sampling, Inspection, and Technical Evaluation (SITE) has the word "site" not to be confused with websites

(Kostas - was that what you were saying?)

<kostas> What about MEAC? INSTEAD OF MAC?

Shadi: Let the creative juices flow for naming
... take it to the list and talk about it next week.

Summary of Action Items

[NEW] ACTION: Equivalent results need to be defined. High correlation. Weave in validity. [recorded in http://www.w3.org/2011/10/06-eval-minutes.html#action01]
[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.136 (CVS log)
$Date: 2011/10/07 16:05:51 $