[contents]
This document specifies goals and requirements for Web Accessibility Evaluation Methodology for Web Content Accessibility Guidelines (WCAG) 2.0.
This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.
This 28 September 2011 Editors Draft of Website Accessibility Evaluation Methodology for WCAG 2.0 Requirements is an initial contribution to capture requirements for future work. This document is intended to be published and maintained as a W3C Working Group Note after review and refinement.
The WCAG 2.0 Evaluation Methodology Task Force (Eval TF) invites discussion and feedback about this document by developers, evaluators, researchers, and other practitioners who have interest in web accessibility evaluation. In particular, Eval TF is looking for feedback on how complete, precise, and achievable these requirements are.
Please send comments on this Website Accessibility Evaluation Methodology for WCAG 2.0 Requirements document to public-wai-evaltf@w3.org (publicly visible mailing list archive).
Publication as Editor Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.
Editor note:A brief introduction about the need for a harmonized Methodology here and what it will add to WCAG 2.0.
Editor note:Added what was previously in goals to this section to be clearer following the discussions and made target audience a separate section.
The main goal is to define an internationally harmonized methodology for evaluating the conformance of websites to WCAG 2.0. This methodology will support evaluation in different contexts, such as self-assessment and third-party evaluation of small and larger websites. This Methodology will also be independent of any particular evaluation tools and services, and applicable to any type of website. It does not focus on preliminary but on full evaluations.
The Methodology intends to cover recommendations for sampling web pages and for expressing the scope of a conformance claim, complete processes, computer assisted content selection, manual content selection, the evaluation of web pages, integration and aggregation of the evaluation results and conformance statements. The methodology should also address tolerance metrics and include recommendations for harmonized (machine-readable) reporting. The methodology will be cross-tested by the Task Force and will include a reference to the test results.
While the Methodology will provide guidance on evaluation throughout the development process, it is specifically designed to inform on evaluation of existing websites; complementary WAI resources will provide further advice on evaluation during other stages of the development process.
Eval TF has the following dependencies:
- WCAG WG - to ensure accurate interpretation and representation of WCAG requirements
- ERT WG - to facilitate coordination with other W3C and WAI testing efforts
Eval TF will coordinate with EOWG where necessary to improve the writing, presentation, and educational value of the methodology. Members of the Task Force will monitor the work in the authoring tools accessibility guidelines (ATAG) working group: http://www.w3.org/WAI/AU/ and the developing ATAG2.0 specification. The Methodology will be written to be agnostic to the context of conformance evaluation so that in can be used for self-assessment by the manufacturer or supplier, for acceptance-testing by the user or purchaser, or for third-party evaluation by an independent body. This makes the Methodology compatible with quality assurance processes such as that defined by ISO/IEC Guide 7.
There are different possible scenarios for use. The primary target audience of the Methodology are people who want to evaluate the conformance of websites to WCAG 2.0. This includes:
Editor note: Did some remodeling below to capture all groups named in the discussions
- Website developers, suppliers, procurers, and owners wishing to evaluate during the development process
- Web accessibility monitoring and benchmarking activities
- Web content producers wishing to evaluate their content
- Developers of Evaluation and Repair Tools
- Web accessibility consultancies and evaluation services
- Web accessibility researchers and disability advocates
They could use tools and/or do manual evaluations.
Other target audience of the Methodology include:
- People who want to use the Methodology for education and outreach activities.
- Policy makers, project managers, and other decision makers who need a standard
Users of the Methodology are assumed to be knowledgeable of the WCAG 2.0, accessible design, assistive technology, and of how people with disabilities use the Web.
For the purpose of this Methodology, the relevant definitions in WCAG 2.0 and the following definitions apply:
Editor note:Changes from the 24 september version: split the definition of webpage and website and added complete process instead of process
- Webpage
- As defined by WCAG 2.0 in: http://www.w3.org/TR/WCAG20/#webpagedef.
- Website
- A coherent collection of one or more related web pages that together provide common use or functionality. It includes static web pages, dynamically generated web pages, and web applications.
- Complete process
- As defined by WCAG 2.0 in: http://www.w3.org/TR/WCAG20/#cc3
- R01: Define methods for evaluating WCAG 2.0 conformance
- The Methodology provides a method to measure conformance with WCAG 2.0. The level of the Methodology does not stretch into techniques and tests. It builds on the conformance section in WCAG2.0.
- R02: Tool and browser independent
- The use and application of the Methodology is independent from (not limited to) specific tools or browsers.
- Editor note: Does this include assistive technologies?
- R03: Unambiguous interpretation
- The Methodology itself is unambiguous to people who want to use it.
- Editor note: Deleted next part after comment from Detlev that it is not precise: It should make it clear to users what they can do if they choose a certain evaluation approach.
- R04: Replicable
- Different Web accessibility evaluators using the same methods on the website(s) should get the same results.
- R05: Translatable
- It should be translatable into different languages. This also means the use of standards terminology if possible and available.
- Editor note:Proposal for rephrasing: The Methodology will be written using terminology and a style that is as easy to translate as possible; where necessary terms will be defined in a glossary.
- R06: Use of existing WCAG 2.0 techniques
- The Methodology will use existing testing procedures in the WCAG 2.0 Techniques documents rather than to reproduce them.
- R07: Support for both manual and automated evaluation
- It supports people doing automated evaluation and/or manual evaluation. It is not limited to only manual or only automated evaluation. One of the methods would be to provide a way to report on the use of automated and/or manual evaluation.
- R08: Address the needs of the target audience
- The Methodology is usable for the target audiences. The document will give a short description of the knowledge necessary for using the Methodology for evaluations.
- R09: Support for different contexts
- It supports different contexts like self-assessment, third-party evaluation of small or larger websites.
- R10: Recommendations for scope and sampling
- It includes recommendations for sampling web pages and for expressing the scope of a conformance claim.
- R11: Complete processes
- It includes adding complete processes into the sample of an evaluation. I.e. for a shopping site all pages that are part of the steps in an ordering process would be in the evaluation sample (the webpages that will be evaluated).
- R12: computer assisted and manual content selection
- It covers computer assisted (automated) and manual selection of pages or content for the evaluation sample. Computers can spider websites and choose pages with certain characteristics (like images, tables etc.) and/or choose random pages. This can also be done manually. The Methodology will support both manual and computer assisted or automated.
- R13: Includes integration and aggregation of the evaluation results and related conformance statements
- It includes methods for aggregation of evaluation results and proposal for conformance statements based on such aggregation of data.
- R14: Includes tolerance metrics
- Depending on the amount of tolerance, a failure could fall within a certain tolerance level meaning that the page or website would be considered conformant even though there is a failure.
- R15: Machine-readable reporting
- The Methodology includes recommendations for harmonized (machine-readable) reporting. It provides a format for machine-readable reports using EARL.
- R16: Support for independent verification
- The Methodology supports independent verification of the evaluation results
- R17: Support validity
- Evaluation results are documented in a manner that it is later possible to see if what has been measured is valid to support a claim to conformance.
- R18: QA framework specification guidelines
- The Methodology will conform to the Quality Assurance framework specification guidelines as set in: http://www.w3.org/TR/qaframe-spec/.
- WCAG 2.0
- http://www.w3.org/TR/WCAG/. The Web Content Accessibility Guidelines version 2.0.
- ISO/IEC guide 7
- Guide for conformity assessment documents.
- ATAG2.0
- http://www.w3.org/TR/ATAG20/. the Authoring Tool Accessibility Guidelines.
- EIAO Indicator Refinement study
- Indicator refinement document made by EU project EIAO: http://eiao.net/publications/Indicator_refinement_FINAL_v1.31.pdf
- Quality Assurance Framework Specification Guidelines
- http://www.w3.org/TR/qaframe-spec/