Website Accessibility Evaluation Methodology for WCAG 2.0

Status of this document

This clause describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

This 2 November 2011 Editors Draft of Website Accessibility Evaluation Methodology for WCAG 2.0 is an initial outline for future refinement. This document is intended to be published and maintained as a W3C Working Group Note after review and refinement.

The WCAG 2.0 Evaluation Methodology Task Force (Eval TF) invites discussion and feedback about this document by developers, evaluators, researchers, and other practitioners who have interest in web accessibility evaluation. In particular, Eval TF is looking for feedback on the completeness of this outline.

Please send comments on this Website Accessibility Evaluation Methodology for WCAG 2.0 document to public-wai-evaltf@w3.org (publicly visible mailing list archive).

Publication as Editor Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

1. Introduction

Editor note:For the drafting of this document we use ISO/IEC Directives — Part 2: Rules for the structure and drafting of International Standards [1] as far as practicable.

This WCAG 2.0 Evaluation Methodology (WCAG-EM) proposes an internationally harmonized methodology for evaluating the conformance of websites to WCAG 2.0. The Methodology supports evaluation in different contexts, such as self-assessment and third-party evaluation of websites. It is applicable to all websites (including web applications) regardless of size and it is independent of any particular evaluation tools, browsers, and assistive technology.

2. Scope of this document

Editor note:This clause defines without ambiguity the subject of the document and the aspects covered, thereby indicating the limits of applicability of the document or particular parts of it. It does not contain requirements.

The WCAG 2.0 Evaluation Methodology describes a formal and harmonized way to evaluate the conformance of websites to WCAG 2.0. The Methodology defines manual and computer assisted methods for selecting representative samples of web pages from websites that include complete processes. It provides guidance for evaluating the selected web pages to WCAG 2.0, and defines methods for integration and aggregation of the evaluation results into structured reports and conformance claims.

The Methodology is an editors draft that provides guidance on evaluation throughout the development process but it is specifically applicable to conformance evaluation of existing websites. It extends the existing WAI resource Conformance Evaluation of Websites for Accessibility. Complementary WAI resources such as Preliminary Review of Websites for Accessibility and Involving Users in Evaluating Web Accessibility provide further advice on evaluation during other stages of the development process.

3. Target audience

Editor note:Description of the target audience of the Methodology. We did some work for this in the requirements document.

The primary target audience of the Methodology is anyone, including individuals and organizations, wanting to evaluate the conformance of existing websites to WCAG 2.0. This includes but is not limited to:

Users of the Methodology are assumed to be knowledgeable of WCAG 2.0, accessible web design, assistive technology, and of how people with different disabilities use the Web.

4. References

Editor note:A list of the referenced documents cited in this document as far as they are indispensable for the application.

The following documents, in whole or in part, are (normatively) referenced in this document and are indispensable for its application. For dated references, only the edition cited applies. For undated references, the latest edition of the referenced document (including any amendments) applies.

5. Terms and definitions

Editor note:Terminology that is really important for the understanding of the Methodology. General words and terms could be placed in the glossary at the end of the document. We already did some work in the requirements document like for website etc.

6. Expertise for evaluating accessibility

The W3C Evaluation suite extensively describes the expertise necessary for the evaluation of the accessibility of Web content for people with disabilities. Evaluation activities require diverse kinds of expertise and perspectives. Individuals evaluating the accessibility of web content require training and experience across a broad range of disciplines. A collaborative approach can bring better results for individuals evaluating web content. The W3C/WAI evaluation suite writes: "Effective evaluation of Web accessibility requires more than simply running an evaluation tool on a Web site. Comprehensive and effective evaluations require evaluators with an understanding of Web technologies, evaluation tools, barriers that people with disabilities experience, assistive technologies and approaches that people with disabilities use, and accessibility guidelines and techniques." This document describes evaluation methodology. Automatic evaluation can significantly reduce the time and effort needed for an evaluation but please note that many accessibility checks require human judgement and must be evaluated manually. This is described in this document.

6.1 Involving People with Disabilities in the process

Editor note:As discussed in the requirements discussion we wanted to address in the Methodology that involvement of users with disabilities is important. There is text about this in the eval suite on the WAI pages.

7. Procedure to express the Scope of the evaluation (based on:)

Editor note:How can an evaluator express the scope of a website. What is in and what can be left out? Below are some possible subclauses that cover things that look necessary to describe to pinpoint the exact scope of what is in and what can be left outside the scope of a website:

7.1 Technologies used on the webpages

While the WCAG 2.0 Recommendation focus on webpages, the Evaluation Methodology focuses on evaluating the conformance of a website. This means that it is important to define what is part of the website and what is not. To be more precise, to define what is part of the coherent collection of one or more related web pages that together provide common use or functionality and can include static web pages, dynamically generated web pages, and web applications.
Web pages are not limited to html. It includes PDF, Office documents, audio, video and photos. All web pages that are part of the website fall within the scope of the evaluation.

The scope of a website can include subdomains and parts of external domains. The setting of the scope delimites the pages and applications to be included into the sample.

A webpage is part of the scope of an evaluation if one or more of the following apply:

7.2 Base URI of the evaluation

First determine the main URI of the website. For most websites this will consist of a domain name and top level domain (like .com or .eu). The base URI can also be more specific, for example a directory added to the URI or a subdomain.

The base URI does not determine whether a page is inside or outside the scope. It is only the starting point from which the rest of the scope is determined. All pages that use the base URI, both because they have an extensive directory structure or use of sub-(sub-) domains fall within the scope unless there is a clearly described exception.

Some websites have multiple domains linked for the same content and dynamics, in that case the scope is limited to the primary domain. A conformity declaration would in that case also be valid for the other websites.

7.3 Perception and function

7.4 Complete processes

7.5 Webpages behind authorization

7.6 Dividing the scope into multiple evaluations

Editor note:Imagine a website is large and the would like to divide the evaluation over different parts for which different people are responsible. If all parts are in the scope of the website, then the scope could be divided into multiple parts that all have to be evaluated.

8. Sampling of pages

Editor note:An evaluator can manually evaluate all pages, but on website with 9M pages that is a lot of work. How to select a sample of a website is described in this clause. How many pages and how do you choose them?

8.1 Sample selection (random and targeted sampling)

8.2 Size of evaluation samples (related to barriers)

9. Evaluation

Editor note:This is the clause describing step by step how to do the evaluation. The evaluation is depending on many factors, like technologies used, technologies evaluated, Accessibility supported etc. Part of the story is the barrier that are encountered during evaluation. When are they a real problem? Is it possible to have an error margin and how do we describe that?

9.1 Manual and machine evaluation

9.2 Technologies

9.3 Procedure for evaluation

9.4 Barrier recognition

9.5 Error margin

10. Conformity

Editor note:This clause is largely from WCAG 2.0 with additional information.

10.1 Conformity requirements

10.2 Accessibility supported

10.3 Partial conformance

10.4 Conformance claims

10.5 Score function (barrier change)

11. Reporting

Editor note:How to write a report from the evaluation that is human readable and one that is machine readable. And what should be in the report. Templates are included in the appendices.

11.1 Text based report

11.2 Machine readable report using EARL

12. Limitations and underlying assumptions

13. Acknowledgements

14. Glossary

Editor note:Glossary for this document and link to general glossary for WCAG 2.0.

15. References

Annex 1: Template for manual evaluation report

Annex 2: Template for EARL report

Editor note:a complete example template for evaluation reporting using EARL.