[contents]

Abstract

This document, Website Accessibility Conformance Evaluation Methodology (WCAG-EM) 1.0, describes one possible approach for evaluating the conformance of existing websites to the Web Content Accessibility Guidelines (WCAG) 2.0. It provides guidance on defining parameters for the evaluation scope; on exploring and understanding websites including their key features and functionality; on sampling representative web pages from websites where it is not feasible to evaluate all web pages of a website; on evaluating the conformance of these selected web pages to the target level of WCAG 2.0 conformance; and on documenting and reporting the findings from such evaluation. It is part of the guidance on Evaluating Websites for Accessibility and complements existing WCAG 2.0 resources but it does not define additional WCAG 2.0 requirements nor does it replace or supersede it in any way.

This methodology is primarily intended for accessibility evaluation of existing websites, for example to validate conformance claims. It can also be useful for other purposes, such as for the evaluation of websites during their development and for ongoing monitoring. However, this methodology does not replace the need for quality assurance measures that are implemented throughout the design, development, and maintenance of websites to ensure their accessibility conformance. The methodology is applicable to any website including dynamically generated websites, rich web applications, and mobile websites. It is independent of any particular web technology, evaluation tool, web browser, assistive technology, and other software, and it addresses different evaluation contexts, including self-assessment and third-party evaluation. It is primarily intended for use by experienced accessibility evaluators.

Status of this document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

This 8 February 2013 Editors Draft of Website Accessibility Conformance Evaluation Methodology (WCAG-EM) 1.0 addresses the comments received on the previously published Working Draft of 26 February 2013. It addresses all issues raised though some sections may need further refinement.

This document is intended to be published and maintained as an informative W3C Working Group Note after review and refinement. The WCAG 2.0 Evaluation Methodology Task Force (Eval TF) invites discussion and feedback on this document by developers, evaluators, researchers, and others with interest in web accessibility evaluation.

Please send comments on this Editors Draft of Website Accessibility Conformance Evaluation Methodology 1.0 to public-wai-evaltf@w3.org (publicly visible mailing list archive). These comments will be considered in the internal discussions of Eval TF.

Publication as Editors Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

This document has been produced by the WCAG 2.0 Evaluation Methodology Task Force (Eval TF, a joint task force of the Web Content Accessibility Guidelines Working Group (WCAG WG) and Evaluation and Repair Tools Working Group (ERT WG), as part of the Web Accessibility Initiative (WAI) Technical Activity.

This document was produced by two groups operating under the 5 February 2004 W3C Patent Policy. The groups do not expect this document to become a W3C Recommendation. W3C maintains a public list of any patent disclosures for WCAG WG and public list of any patent disclosures for ERT WG made in connection with the deliverables of each group; these pages also include instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy.

Table of Contents

Overview

List of Sections

List of Appendices

Introduction

Evaluation needs to be carried out throughout the web development process to ensure that websites are accessible. For example, designers need to ensure that the colors for text and background are sufficiently distinguishable when the website design is being created. Later on when content is being created, content authors need to ensure that structural elements such as headings and lists are correctly coded. Everyone involved in the (re-)design, (re-)development, and ongoing creation of web content needs to evaluate the accessibility of their contribution to the website to ensure that it is uniformly accessible.

In some situations it is necessary to evaluate the accessibility of websites after they are developed. Such evaluation can be part of quality assurance checks before releasing, acquiring, or redesigning websites. It can also be part of monitoring the accessibility of websites over time or comparing the accessibility of websites to each other. This methodology, Website Accessibility Conformance Evaluation Methodology (WCAG-EM) 1.0, describes one possible approach to evaluate the conformance of existing websites to the Web Content Accessibility Guidelines (WCAG) 2.0. This methodology can also be useful for other purposes such as for ongoing evaluation of websites during their development to continually assess progress towards conformance.

This document is part of the guidance on Evaluating Websites for Accessibility and complements existing WCAG 2.0 resources but it does not define additional WCAG 2.0 requirements nor does it replace or supersede it in any way.

Scope of this Document

The methodology described in this document, WCAG-EM, provides guidance on defining parameters for the evaluation scope; on exploring and understanding websites including their key features and functionality; on sampling representative web pages from websites where it is not feasible to evaluate all web pages of a website; on evaluating the conformance of these selected web pages to the target level of WCAG 2.0 conformance; and on documenting and reporting the findings from such evaluation. The methodology can be applied to any website including dynamically generated websites, rich web applications, mobile websites and other types of websites. It is independent of the size of the website, the tools and formats used to create the website, and of any particular web accessibility evaluation tools, web browsers, assistive technologies, and other software tools.

This methodology does not replace the need for quality assurance measures that are implemented throughout the design, development, and maintenance of websites to ensure their accessibility conformance. Following this methodology may not identify every possible occurrence of support for or violation of WCAG 2.0 conformance. It is one possible approach for evaluating the conformance of existing websites to WCAG 2.0 in different evaluation contexts, including self-assessment and third-party evaluation. It is primarily intended for use by experienced accessibility evaluators.

Purpose of this document

The methodology described in this document can be useful in a variety of situations but it does not specifically provide guidance on web accessibility in situations other than conformance evaluation of existing websites to WCAG 2.0, nor does it provide general guidance on web accessibility evaluation. Some of the use cases for this document include providing a common procedure for:

Complementary resources on Web Content Accessibility Guidelines (WCAG) 2.0, Evaluating Websites for Accessibility, Web Accessibility Evaluation and Testing Activities, and WAI Guidelines and Techniques provide further guidance on web accessibility.

Background Reading

The information below related to WCAG 2.0 and evaluation is important background for using this methodology:

Web Accessibility Essentials

The following documents introduce the essential components of web accessibility and how people with disabilities use the Web, and are critical for understanding the broader context of web accessibility evaluation:

Evaluating Websites for Accessibility

A multi-page resource suite that outlines different approaches for evaluating websites for accessibility. The following resources are particularly important in this context:

Web Content Accessibility Guidelines (WCAG) 2.0

Internationally recognized standard explaining how to make web content more accessible to people with disabilities. The following resources are particularly important in this context:

Terms and Definitions

For the purposes of this document, the following terms and definitions apply:

Complete Processes
From WCAG 2.0 Conformance Requirement for Complete Processes:
When a web page is one of a series of web pages presenting a process (i.e., a sequence of steps that need to be completed in order to accomplish an activity), all web pages in the process conform at the specified level or better. (Conformance is not possible at a particular level if any page in the process does not conform at that level or better.)
Common Functionality
[For review: Proposal to rename Common Functionality to Core Functionality. This would also include to change the term accordingly in later sections where this term is used. This would cover comment 3, comment 5, comment 6 and comment 7]. To be discussed in the 13 June 2013 Telco.
Functionality of a website that, if removed, fundamentally changes the use or purpose of the website for users. This includes information and tasks that users of a website carry out to perform this functionality.
Note: Examples of functionality include "selecting and purchasing a product from the shop area of the website", "filling and submitting the form provided on the website", and "registering for an account on the website".
Note: Other functionality are not excluded from the scope of evaluation. The term "common functionality" is intended to help identify critical pages and include them among others in an evaluation.
Common Web Pages
Web pages that are relevant to the entire website. This includes the homepage, login page, and other entry pages, and, where applicable, the sitemap, contacts page, site help, legal information, and similar web pages that are typically linked from all web pages (usually from the header, footer, or navigation menu of a web page).
Evaluator
The person, team of people, organization, in-house department, or other entity responsible for carrying out the evaluation.
Evaluation Commissioner
The person, team of people, organization, in-house department, or other entity that commissioned the evaluation.
Note: In many cases the evaluation commissioner may be the website owner or website developer, in other cases it may be another entity such as a procurer or an accessibility monitoring survey.
Relied Upon (Technologies)
From WCAG 2.0 definition for "relied upon":
The content would not conform if that technology is turned off or is not supported
Templates
From ATAG 2.0 definition for "templates":
Content patterns that are filled in by authors or the authoring tool to produce content for end users (e.g., document templates, content management templates, presentation themes). Often templates will pre-specify at least some authoring decisions.
[Editor Note: Given that ATAG 2.0 is still a Working Draft, this definition might change in future drafts of this document. This is from the 10 April 2012 Working Draft]
Website
A coherent collection of one or more related web pages that together provide common use or functionality. It includes static web pages, dynamically generated web pages, and web applications.
Note: The focus of this methodology is on full, self-enclosed websites. Websites may be composed of smaller sub-sites, each of which can be considered to be an individual website. For example, a website may include an online shop, an area for each department within the organization, a blog area, and other areas that may each be considered to be a website.
Website Developer
The person, team of people, organization, in-house department, or other entity that is involved in the website development process including but not limited to content authors, designers, programmers, quality assurance testers, and project managers.
Website Owner
The person, team of people, organization, in-house department, or other entity that is responsible for the website.
Web page
From WCAG 2.0 definition for "Web Page":
A non-embedded resource obtained from a single URI using HTTP plus any other resources that are used in the rendering or intended to be rendered together with it by a user agent
Note: Web pages may include multimedia content, interactive components, and complex applications.

Using This Methodology

The methodology defined by this document is used for evaluating the conformance of websites to WCAG 2.0. A more cursory approach for exploring the accessibility of a website is described in Preliminary Review of Websites for Accessibility. In many cases it is advisable to carry out such preliminary reviews before applying this methodology, for example to identify obvious errors and to develop a rough understanding of the overall performance of the website.

Scope of Applicability

The methodology defined by this document applies to full, self-enclosed websites. This includes websites of organizations and entities, persons, events, products, and services. It also includes web applications and mobile websites. Examples include:

A website may include areas with smaller collections of related web pages such as an online shop, an area for each department within the organization, a blog area, and other parts. In some situations such areas can be considered to be a full, self-enclosed website each. This methodology can be applied to such individual sub-sites (a website within another website) and to the main website in its entirety. However, this methodology may not be applied to a website excluding any of its parts. Excluding parts of the website from the scope of evaluation would likely conflict with the WCAG 2.0 conformance requirements full pages and complete processes, or significantly distort the evaluation results.

Example of a Website:
Diagram of a University Website explained in the following paragraph.

In the diagram above, the university website is made of several sub-sites: "Information for Students", "Information for Lecturers", a "Courseware Application", and a "Library Application". The "Courseware Application" includes "Physics Courses", "Medical Courses", and "History Courses" that are aggregated into the application. Also, the university website has individual web pages such as legal notices, sitemap, and others that are not part of any particular sub-site.

In the example above, none of the depicted parts may be excluded from the scope of evaluation in the context of this methodology, if it is to be applied to the university website. This includes any aggregated and embedded content such as online maps for the university campus and forms for credit card transactions, including when such parts originate from third-party sources.

Note: WCAG 2.0 defines "Statement of Partial Conformance" for individual web pages that are known not to conform to WCAG 2.0, for example due to third-party content and/or languages lacking accessibility support. Such web pages may not be excluded from the scope of evaluation when using this methodology. In some cases this may mean that a website as a whole does not fully conform to WCAG 2.0 because it contains only partially conforming web pages. Step 5: Report the Evaluation Findings provides more guidance on reporting evaluation results and on making accessibility statements for entire websites.

Particular Types of Websites

This methodology is applicable to any website including dynamically generated websites, rich web applications, mobile websites and other types of websites. Some particular types of websites include:

Small Websites
For websites with few web pages the sampling procedure defined in Step 3: Select a Representative Sample will likely result in selecting most or all of the web pages from the target website. In cases where all web pages can be evaluated, the sampling procedure can be skipped and the selected sample is considered to be the entire website in the remaining steps.
Web Applications
In some cases only one or few web pages may dynamically generate all the content with the functionality that normally an entire set of web pages would provide. Such web applications are usually not separable into sub-sites and are considered as a single website or part of a website. Each state of a web page and generated web page (individual representation of content and functionality) in which such a web application is considered to be a web page for the purpose of this document should be included as a candidate for sampling.
Note: These individual states of web page and generated web pages of a web application may not have unique URIs and may need to be identified by describing the settings, input, and actions required to generate them. Due to the many possibilities to generate these states of web page and generated web pages it may not be feasible to exhaustively identify every possible instance of web pages as per Step 2.a: Identify Common Web Pages of the Website, functionality as per Step 2.b: Identify Common Functionality of the Website, and variety of web pages as per Step 2.c: Identify the Variety of Web Page Types, so that evaluating such web applications may be more challenging than for other types of websites.
Website with Separable Areas
In some cases websites may have clearly separable areas, such as a password-restricted area of a website (extranet) that is not part of using the public area (log-in is not required to complete a function or process). Such areas can be considered as individual websites rather than sub-sites for the purpose of this document.
Website in Multiple Versions
Some websites are available in versions that are independent of each other. For example, mobile versions of a website and versions of a website in different languages, where using one version does not require or depend on using another version of the website (usually such website versions have a different set of URIs each). Such website versions can be considered as individual websites rather than sub-sites for the purpose of this document.
Note: Websites using responsive design techniques (ie. adapt the presentation according to user hardware, software, and preferences) are not considered to be separable, independent websites.

Required Expertise

Users of the methodology defined by this document are assumed to be knowledgeable of WCAG 2.0, accessible web design, assistive technologies, and of how people with different disabilities use the Web. This includes understanding of the relevant web technologies, barriers that people with disabilities experience, assistive technologies and approaches that people with disabilities use, and evaluation techniques and tools to identify potential barriers for people with disabilities. In particular, it is assumed that users of this methodology are deeply familiar with the resources listed in section Background Reading.

Evaluation Tools (Optional)

The methodology defined by this document is independent of any particular web accessibility evaluation tool, web browser, and other software tool. However, web accessibility evaluation tools can significantly assist an evaluator during the evaluation process and contribute to more effective evaluations. For example, some web accessibility evaluation tools can scan entire websites to help identify relevant pages for manual evaluation, and other tools can assist manual evaluation in a variety of ways. Specific guidance is provided in Selecting Web Accessibility Evaluation Tools.

Note: Web accessibility evaluation tools can only automatically check a limited set of WCAG 2.0 Success Criteria that are automatable. Conformance evaluation requires manual testing and judgment by experienced evaluators.

Review Teams (Optional)

The methodology defined by this document can be carried out by an individual evaluator with the skills described in section Required Expertise. However, using the combined expertise of review teams provides better coverage for the required skills and helps identify accessibility barriers more effectively. While not required, it is recommended to employ review teams for conformance evaluation of websites. Specific guidance is provided in Using Combined Expertise to Evaluate Web Accessibility.

Involving Users (Optional)

The methodology defined by this document can be carried out by an individual evaluator with the skills described in section Required Expertise. However, involving people with disabilities and people with aging-related impairments helps identify additional accessibility barriers that are not easily discovered by the evaluator alone. While not required, it is strongly recommended to involve real people covering a wide range of abilities during the evaluation process. Specific guidance is provided in Involving Users in Web Accessibility Evaluation.

Conformance Evaluation Procedure

This section defines the stages and activities of the evaluation procedure. The stages are not necessarily sequential. Also the exact sequence of the activities carried out during the evaluation process depends on the type of website, the purpose of the evaluation, and the process used by the evaluator. Some of the activities overlap or may be carried out in parallel. The following diagram illustrates the iterations between the stages defined in this section:

Diagram about the iterations between the steps in this methodology. Explanation in the following paragraph.

The diagram depicts each of the five steps defined in this section with arrows back and forth between each of two consecutive steps: 1. Define the Evaluation Scope; 2. Explore the Target Website; 3. Select a Representative Sample; 4. Audit the Selected Sample and 5. Report the Evaluation Findings.

Note: See also section Considerations for Particular Situations that may influence how an evaluation procedure is carried out.

Step 1: Define the Evaluation Scope

Methodology Requirement 1: Define the evaluation scope according to Methodology Requirement 1.a, Methodology Requirement 1.b, Methodology Requirement 1.c, and optionally Methodology Requirement 1.d.

During this step the overall scope of the evaluation is defined. It is a fundamental step that affects the subsequent steps in the evaluation procedure and is ideally carried out together with the evaluation commissioner (who may or may not be the website owner) to ensure common expectations about the scope of the evaluation. Some exploration throughout this stage may be necessary to understand the full scope of the website and the required evaluation (see Explore the Target Website).

Note: Involvement of the website owner and/or website developer (in addition to the evaluation commissioner) is not required but often helps identify use cases, functionality, and other aspects of the implementation that makes the evaluation procedure more efficient and effective. However, the evaluator is responsible for an objective and thorough assessment.

Step 1.a: Define the Scope of the Website

Methodology Requirement 1.a: Define the scope of the website.

During this step the exact scope of the website (the web pages for which the evaluation will apply) is defined and documented. This scope definition may not contradict the terms established in section Scope of Applicability. It is essential to carefully document particular aspects such as any use of content and services developed externally, different languages, and parts of the website that may not be easily identifiable as such (for example an online shop that is not integrated into the website but considered to be part of it).

The outcome of this step is an unambiguous definition for the target website, so that for each web page it can be determined whether it is within the scope of evaluation or not. Where possible, it is recommended to use formalizations such as regular expressions or listings of URIs that define the web pages that are within the scope of evaluation (part of the target website).

Step 1.b: Define the Goal of the Evaluation

Methodology Requirement 1.b: Define the goal of the evaluation.

Defining the goal of the evaluation is particularly relevant to the auditing stage defined in Step 4: Audit the Selected Sample and the reporting stage defined in Step 5: Report the Evaluation Findings. Some of the evaluation goals include:

Basic Report
Only identifies whether a website conforms or not without providing additional information. This type of evaluation is typically carried out when the website is assumed to conform, for example to verify an existing conformance claim, and for large-scale evaluations with less resources to explore the details of individual websites.
Detailed Report
Identifies whether a website conforms or not, and provides further information about the conformance of each evaluated web page. This type of evaluation is particularly useful for instructing web developers and for acquiring statistics for monitoring progress over time.
In-Depth Analysis Report
Identifies whether a website conforms or not, and analyzes any identified issues in detail. This includes descriptions of the errors and suggestions for possible repair options. This type of evaluation is particularly useful for organizations that want to improve the accessibility of their website and need a detailed analysis.

Step 1.c: Define the Conformance Target

Methodology Requirement 1.c: Define the target WCAG 2.0 conformance level as "A", "AA", or "AAA".

Part of initiating the evaluation process is to define the target WCAG 2.0 conformance level ("A", "AA", or "AAA"). In many situations WCAG 2.0 Level AA is the generally accepted level of accessibility.

Note: It is often useful to expand the scope of the evaluation beyond the planned conformance target to get a more complete picture of the accessibility performance of the website. An evaluator could include a selection of success criteria from higher conformance levels.

Step 1.d: Define the Techniques and Failures to be Used (Optional)

Methodology Requirement 1.d: Specify the techniques and failures (that have been documented by W3C and others as meeting the Success Criteria) that will be used to evaluate WCAG 2.0 Success Criteria. (Optional).

Techniques and failures in the context of WCAG 2.0 are informative and not required for determining conformance with WCAG 2.0; WCAG 2.0 Success Criteria are written as testable statements. Techniques provide documented ways of meeting WCAG 2.0 Success Criteria and failures document commonly made mistakes. More information on techniques and failures is provided in WCAG 2.0 Layers of Guidance.

W3C/WAI provides a set of publicly documented Techniques for WCAG 2.0. However, it is not necessary to use these particular techniques. In fact, in some situations, such as in a closed network, it may be necessary to use techniques that are specifically developed to the particular needs of the users of that network. Individuals and organizations developing techniques have to employ methods for establishing the technique's ability to meet the WCAG 2.0 Success Criteria.

It is good practice to specify the sets or sources of techniques and failures (that have been documented by W3C and others as meeting the Success Criteria) that are intended to be used during the evaluation at this stage already to ensure consistent expectation between the evaluator and the evaluation commissioner. This definition is typically refined in later stages of the evaluation process, for example during the website exploration and evaluation stages. Step 4.c: Use WCAG 2.0 Techniques and Failures Where Possible (Optional) provides more guidance on using techniques and failures during evaluation.

Step 2: Explore the Target Website

Methodology Requirement 2: Explore the website to be evaluated according to Methodology Requirement 2.a, Methodology Requirement 2.b, Methodology Requirement 2.c, and Methodology Requirement 2.d.

During this step the evaluator explores the target website to be evaluated, to develop a better understanding of the website use, purpose, and functionality.

Carrying out initial cursory checks during this stage helps identify web pages that are relevant for more detailed evaluation later on. For example, an evaluator may identify web pages that seem to be lacking color contrast, consistent navigation, or document structures (headings, links, etc.) with simple checks while studying the website, and note them down for more detailed evaluation later on.

To carry out this step it is critical that the evaluator has access to all the relevant parts of the website. For example, it may be necessary to create accounts or otherwise provide access to restricted areas of a website that are part of the evaluation.

Note: Involvement of the website owner and/or website developer can be useful to help identify common web pages, functionality, technologies, and other aspects of the implementation that makes the evaluation procedure more efficient and effective. However, the evaluator is responsible for an objective and thorough evaluation.

Step 2.a: Identify Common Web Pages of the Website

Methodology Requirement 2.a: Identify the common web pages of the website.

During this step the common web pages of the website are identified and documented. Common web pages are pages that are relevant to the entire website. They include the homepage, login page, and other entry pages, and, where applicable, the sitemap, contacts page, site help, legal information, and similar web pages that are typically linked from all web pages (usually from the header, footer, or navigation menu of a web page).The outcome of this step is a list of all common web pages. These will be part of the sample to be selected through the steps defined in Step 3: Select a Representative Sample.

This step also helps understand key aspects of the website, such as the navigation and overall structure of the website.

Step 2.b: Identify Common Functionality of the Website

Methodology Requirement 2.b: Identify the common functionality of the website.

During this step the common functionality of the website is identified and documented, to provide a comprehensive basis for the sample to be selected through the steps defined in Step 3: Select a Representative Sample. Common functionality is all functionality of a website that, if removed, fundamentally changes the use or purpose of the website for users. This includes information and tasks that users of a website carry out to perform this functionality.

The outcome of this step is a list of user tasks such as "selecting and purchasing a product from the shop area of the website", "filling and submitting the form provided on the website", and "registering for an account on the website".

Step 2.c: Identify the Variety of Web Page Types

Methodology Requirement 2.c: Identify the variety of web page types.

Web pages with varying styles, layouts, structures, and functionality often have different implementations of accessibility features. They are also often generated by different templates and authored by different people. Web pages may also appear differently, behave differently, and contain different content depending on the user. The outcome of this step is a list of the different types of web pages on the website, to provide a comprehensive basis for the sample to be selected through the steps defined in Step 3: Select a Representative Sample. Different web page types include:

Note: This step is intended to identify groups of web page instances, including those in web applications.

Step 2.d: Identify Web Technologies Relied Upon

Methodology Requirement 2.d: Identify the technologies relied upon to provide the website.

During this step the web technologies relied upon to provide the website are identified and documented, to provide a basis for the sample selection through the steps defined in Step 3: Select a Representative Sample. This includes base web technologies such as HTML and CSS, HTML5, auxiliary web technologies such as Java, JavaScript and WAI-ARIA, as well as specific web technologies such as SMIL and SVG. The outcome of this step is a list of web technologies that are relied upon.

Only the web technologies that are relied upon to provide the website need to be identified for evaluation. This relates closely to the WCAG 2.0 concepts of conforming alternate versions and non-interference.

Note: Where possible, it is often also useful to identify the libraries and components used to create the website, such as Dojo, JQuery, and others. Particularly for web applications, much of the accessibility support is built into these libraries and components, and evaluation can become more effective and efficient when these are identified.

Step 3: Select a Representative Sample

Methodology Requirement 3: Select a representative sample of web pages from the website according to Methodology Requirement 3.a, Methodology Requirement 3.b, Methodology Requirement 3.c, Methodology Requirement 3.d, Methodology Requirement 3.e and optionally Methodology Requirement 3.f.

While ideally every web page of a website is evaluated, usually this is not possible on most websites. In cases where all web pages can be evaluated, this sampling procedure can be skipped and the selected sample is considered to be the entire website in the remaining steps.

Exploration of the target website in Step 2: Explore the Target Website (within the scope set in Step 1: Define the Evaluation Scope) provides sufficient understanding of the website to facilitate selection of a sample of web pages that is representative of the entire target website with reasonable confidence. Depending on the size and complexity of the website, the size of this sample will vary. For example, a website with few types of web pages that are all generated from a confined set of templates, such as a data repository, will likely require a smaller sample than a website with many types of web pages that are authored using different mechanisms. Also a web application could have a limited number of web pages that dynamically generate content with varying types of appearance, behavior, and information that each need to be sampled.

The web pages identified through the exploration will typically relate to more than one design aspect. For example, web pages with particular functionality such as scripting, multimedia, and forms will typically also use technologies such as Flash, JavaScript, PDF, Silverlight, and WAI-ARIA, and in many cases these web pages may have different design to others. Careful selection of web pages can significantly reduce the required sample size while maintaining appropriate representation of the entire website.

Note: In this step the evaluator identifies the comprehensive sample of web pages for evaluation. However, many web pages will have repetitive content, such as the header, navigation, and other common components that may not need to be re-evaluated on each occurrence depending on the goal of the evaluation per Step 1.b: Define the Goal of the Evaluation. Guidance on evaluating the sample identified in this step is provided in Step 4: Audit the Selected Sample.

Step 3.a: Include Common Web Pages of the Website

Methodology Requirement 3.a: Include all common web pages into the selected sample of web pages.

All common web pages, including the common states of these web pages for web applications, are part of the selected sample. These web pages are identified in step Step 2.a: Identify Common Web Pages of the Website.

Step 3.b: Include Exemplar Instances of Web Pages

Methodology Requirement 3.b: Include (where applicable and available) of each (1) common functionality, (2) distinct types of web pages, and (3) web technologies into the selected sample of web pages.

From the variety and types of web pages identified in Step 2: Explore the Target Website (within the scope of the evaluation as defined per Step 1: Define the Evaluation Scope), select at least one distinct web page for all of the following features (where applicable and available):

Note: A selected web page could have any number of these features. For example, a selected web page could be used to represent the use of forms and scripting at the same time. The important aspect is to select at least one distinct web page for each relevant feature identified on the website, though more web pages may be necessary depending on the complexity of the website.

Step 3.c: Include other Relevant Web Pages

Methodology Requirement 3.c: Include other web pages relevant for people with disabilities and accessibility into the selected sample of web pages.

Websites frequently include web pages that are relevant for people with disabilities and accessibility but do not explicitly match the criteria described in the previous sections. These web pages are also part of the selected sample. They typically include:

Step 3.d: Include Complete Processes

Methodology Requirement 3.d: Include all web pages that are part of a complete process.

The selected sample has to include all web pages that belong to a series of web pages presenting a complete process. No web page in the selected sample may be part of a process without all other web pages that are part of that process to be also included into the selected sample.

Step 3.e: Include a Randomly Selected Sample

Methodology Requirement 3.e: Include a randomly selected sample.

[Review Note: This section has been added since the previous draft and feedback on it is particularly welcome. For example, is 5% randomly selected web pages (to complement structured sampling of web pages) sufficient? Is a minimum of 5 web pages sufficient? What alternatives are there and what benefits do they provide?]

A randomly selected portion of the sample, even if it is small, can act as a simple verification indicator of the results found with the structured sample. In that case, a few web pages would then be sufficient to increase confidence in the results of the evaluation. Therefore, from the scope of the website as defined in Step 1: Define the Evaluation Scope, randomly select at least 5 percent of the number of web pages that are in the structured sample with a minimum of 5 randomly selected web pages. How the random sample was selected is reported in Step 5.a: Provide Documentation for Each Step.

Step 3.f: Eliminate Redundancies in the Sample (Optional)

Methodology Requirement 3.f: Filter the sample to eliminate excessive redundancies.

Once a sample has been selected according to Methodology Requirement 3.a, Methodology Requirement 3.b, Methodology Requirement 3.c, Methodology Requirement 3.d, and Methodology Requirement 3.e, evaluators may identify web pages that are identical with other web pages in the sample. Replace these redundant web pages in the sample with other web pages using the same Methodology Requirement as for the removed web pages.

Step 4: Audit the Selected Sample

Methodology Requirement 4: Audit the selected sample of web pages according to Methodology Requirement 4.a, Methodology Requirement 4.b, and optionally, Methodology Requirement 4.c, Methodology Requirement 4.d and Methodology Requirement 4.e.

WCAG 2.0 defines five conformance requirements that need to be met for each web page in the sample selected per Step 3: Select a Representative Sample. This includes checking whether each WCAG 2.0 Success Criterion in the target conformance level (per Step 1.c: Define the Conformance Target) has been met or not met for each of these web pages.

Note: Many web pages will have repetitive content, such as the header, navigation, and other common components that may not need to be re-evaluated on each occurrence. Depending on the Goal of the Evaluation as defined by Step 1.b: Define the Goal of the Evaluation, an evaluator may not need to continue to identify successes and failures in meeting the conformance target for these repetitive elements on every web page. Step 5: Report the Evaluation Findings provides more guidance on reporting.

Note: According to WCAG 2.0, Success Criteria to which there is no matching content are deemed to have been satisfied. An outcome such as 'Not Applicable' may be used to denote the particular situation where Success Criteria were satisfied because no relevant content was applicable.

Step 4.a: Check for the Broadest Variety of Use Cases

Methodology Requirement 4.a: Check that each web page in the selected sample per Step 3: Select a Representative Sample meets each of the WCAG 2.0 conformance requirements, including all WCAG 2.0 Success Criteria relevant to the target conformance level (as per Step 1.c: Define the Conformance Target).

For each web page in the sample selected per Step 3: Select a Representative Sample, check whether each of the WCAG 2.0 conformance requirements have been met, including all WCAG 2.0 Success Criteria relevant to the target conformance level (as per Step 1.c: Define the Conformance Target).

To carry out an evaluation effectively, it is often useful to construct and apply personas, use cases, and scenarios of users with a variety of abilities and using different web browsing techniques, including assistive technology and adaptive strategies. It is critical to consider the broadest possible spectrum of use cases to help identify issues that may occur to different audiences. It is strongly recommended to also involve real users during this process, to help identify issues that may not be easily identified through expert testing alone. See Involving Users in Evaluating Web Accessibility for more guidance.

This assessment also needs to be complemented with focused testing of particular situations including:

Note: Templates are often used to create many web pages, sometimes entire parts of a website. While evaluating templates is optional in this methodology, in some contexts it can be helpful to check templates on their own. Evaluating templates may identify potential issues that may not be easily identified through evaluating individual instances of web pages. However, issues identified in templates alone do not necessarily imply that these issues occur on the website and need to be validated on individual instances of web pages. Also, identifying no issues in templates does not necessarily imply that no issues occur on instances of web pages.

Step 4.b: Assess Accessibility Support for Features

Methodology Requirement 4.b: Check if accessibility features provided on the website are accessibility supported.

To ensure that the accessibility features such as text-alternatives, captions, keyboard access are actually usable in practice, each of these accessibility features has to be accessibility supported. The Level of Assistive Technology Support Needed for "Accessibility Support" defined by WCAG 2.0 needs to be supported throughout the website.

In some situations techniques for meeting WCAG 2.0 and repositories on accessibility support provide insights on the level of support for accessibility features in particular combinations of web technologies, web browsers, and assistive technology. However, note that techniques, including "sufficient technqiues", are not automatically accessibility supported. The evaluator is responsible for the accuracy of the assessment of accessibility support and the resulting evaluation.

Step 4.c: Use Techniques and Failures Where Possible (Optional)

Methodology Requirement 4.c: Where possible, use techniques and failures (that have been documented by W3C and others as meeting the Success Criteria) to help assess successes and failures in meeting the WCAG 2.0 Success Criteria relevant per Step 1.c: Define the Conformance Target (Optional).

Reminder: Techniques and failures in the context of WCAG 2.0 are only informative. They can help assess if WCAG 2.0 Success Criteria are met by providing documented ways of meeting them and commonly occurring failures in meeting them. However, as per the WCAG 2.0 conformance requirements, only the Success Criteria have to be met. And you can use any techniques that meet the Success Criteria, whether they are documented yet as part of WCAG 2.0 or not.

The initial sets or sources of techniques and failures to be used during evaluation may be defined in Step 1.d: Define the Techniques to be Used (Optional). However, during evaluation initial sets may often need to be refined according to the particular situation, such as for evaluating particular web technologies and accessibility features that are identified on the website.

Techniques in the context of WCAG 2.0 are documented ways for meeting or for going beyond what is required by individual WCAG 2.0 Success Criteria. A WCAG 2.0 Success Criterion is met on a web page when:

Conversely, failures in the context of WCAG 2.0 are documented ways of not meeting individual WCAG 2.0 Success Criteria. A WCAG 2.0 Success Criterion is not met on a web page when a failure applies to any instance of web content that is addressed by the WCAG 2.0 Success Criterion.

Techniques are not the only way to meet WCAG 2.0 Success Criteria, and failures are not the only way to fail WCAG 2.0 Success Criteria. Techniques and failures are not exhaustive as they cannot cover every possible situation. Also, the techniques used to meet WCAG 2.0 Success Criteria during the development may not be known to the evaluator. Particularly for newly released web technologies, or when these web technologies are used in particular contexts, there may be no publicly or proprietary documented techniques and failures available to the evaluator. The evaluator has to consider these limitations when using techniques and failures to evaluate conformance to WCAG 2.0.

Step 4.d: Archive Web Pages for Reference (Optional)

Methodology Requirement 4.d: Archive every evaluated web page for reference. (Optional)

Archive web pages for future reference. At the very least, include the title and URI of the web page and the date of the evaluation. To avoid ambiguity, complement this with any of the following as appropriate:

Step 4.e: Record Software Tools and Methods Used (Optional)

Methodology Requirement 4.e: Record the web accessibility evaluation tools, web browsers, assistive technologies, other software, and methods used to evaluate the web pages. (Optional).

For future reference, record software tools and methods used to evaluate the web pages. This includes versions of web accessibility evaluation tools, web browsers and add-ons, assistive technology, and other software used during the evaluation. Depending on the level of detail for reporting defined by Step 1.b: Define the Goal of the Evaluation, this recording may apply globally for the entire evaluation, to individual web pages, or to individual checks carried out within the audited web pages. For detailed reporting a table or grid may be useful to record what was used for the different pages and Success Criteria in the sample.

Step 5: Report the Evaluation Findings

Methodology Requirement 5: Document the evaluation findings according to Methodology Requirement 5.a and optionally Methodology Requirement 5.b, Methodology Requirement 5.c, and Methodology Requirement 5.d.

This section describes how to report the evaluation findings that have been gathered during the previous steps. Reporting is a key element of every evaluation and helps facilitate replicability of the results.

Note: Individual pieces of the reporting are gathered throughout the evaluation process, not necessarily at the end of it.

Step 5.a: Provide Documentation for Each Step

Methodology Requirement 5.a: Document each outcome of the steps defined in Step 1: Define the Evaluation Scope, Step 2: Explore the Target Website, Step 3: Select a Representative Sample, and Step 4: Audit the Selected Sample.

Document the outcomes of each of the previous steps defined in Step 1: Define the Evaluation Scope, Step 2: Explore the Target Website, Step 3: Select a Representative Sample, and Step 4: Audit the Selected Sample, to ensure justifiable, transparent, and repeatable evaluation results. In particular, this includes documenting:

Documentation of the outcome of auditing the representative sample depends on the level of detail for reporting defined by Step 1.b: Define the Goal of the Evaluation. Outcome of the auditing has to include at least the following information depending on the set goal.

Note: Optional example reports are provided in Appendix C: Example Reports.

[Review Note: Feedback on the following types of reports is particularly welcome. For example, how well do they reflect actual situations? What other types of reporting are typically provided in conformance evaluation?]

Basic Report
Only captures the successes and failures in meeting WCAG 2.0 globally for the entire website. For each WCAG 2.0 Success Criterion applicable as per Step 1.c. Define the Conformance Target, the report identifies if it is met or not met in the selected sample of web pages. Where failures in meeting WCAG 2.0 Success Criteria are identified, at least one example web page from the sample in which such a failure has been identified has to be indicated in the report.
Detailed Report
Captures the successes and failures in meeting WCAG 2.0 for each web page. For each WCAG 2.0 Success Criterion applicable as per Step 1.c. Define the Conformance Target, the report identifies if it is met or not met in each web page in the selected sample of web pages. Where failures in meeting WCAG 2.0 Success Criteria on a web page are identified, each identified occurrence of such a failure has to be indicated in the report.
In-Depth Analysis Report
Captures the successes and failures in meeting WCAG 2.0 Success Criterion for each page. For each WCAG 2.0 Success Criterion applicable as per Step 1.c. Define the Conformance Target, the report identifies if it is met or not met in the selected sample of web pages. Where failures in meeting WCAG 2.0 Success Criteria are identified, examples of the identified occurrence of such a failure has to be indicated in the report. The In-Depth Analysis Report includes a summary of the issues identified on the website, examples of frequently occurring issues and an assessment of their impact on the users of the website in completing tasks, and if possible suggestions for improving the overall accessibility of the website and suggesting guidance for the future.

Note: While such a report is required for conformance with this methodology, it is not required for any parts of this report to be made public unless an optional public accessibility statement is provided per Step 5.b: Provide an Accessibility Statement (optional) in which case the minimum evaluation information to be made public is as per the requirements for WCAG 2.0 conformance claims (see Step 5.b: Provide and Accessibility Statement). In other situations, the level of confidentiality for evaluation reports is usually determined by the evaluation commissioner.

Step 5.b: Provide a Conformance Evaluation Statement (Optional)

Methodology Requirement 5.b: Provide an accessibility conformance evaluation statement (Optional).

WCAG 2.0 conformance claims can only be made for the web pages that have been actually evaluated and identified to conform with its conformance requirements. Accessibility conformance evaluation statements for entire websites can be made according to this methodology when:

As per the requirements for WCAG 2.0 conformance claims, also accessibility evaluation statements have to include the following information:

  1. Date of the accessibility evaluation statement
  2. Guidelines title, version and URI: "Web Content Accessibility Guidelines 2.0 at http://www.w3.org/TR/WCAG20/"
  3. Conformance level satisfied: (Level A, AA or AAA)
  4. A concise description of the Web pages, such as a list of URIs for which the claim is made, including whether subdomains are included in the claim.
  5. A list of the web technologies relied upon.

Accessibility evaluation statements can also be made when only partial conformance has been achieved according to the requirements defined in third-party content and languages lacking accessibility support. In such cases the evaluation statements have to also include information to identify the following:

  1. Website areas that do not conform to WCAG 2.0;
  2. Reason for not conforming to WCAG 2.0: "third-party content" or "lack of accessibility support";

Note: It is not possible to make an accessibility evaluation statement for a website that is still in development.

Step 5.c: Provide a Performance Score (Optional)

Methodology Requirement 5.c: Provide a performance score. (Optional).

Performance scores can provide more granular measure for the level of conformance of a website than the WCAG 2.0 conformance levels provide. This can be useful to monitor progress of websites over time. However, scores alone do not provide sufficient context and information to understand the actual accessibility state of a website. In some situation such scores can also be misleading as they can be subjective towards certain kinds of web content.

Performance scores provided in this methodology require that the evaluation is carried out in conformance with this Methodology. They are intended to be supplementary information to evaluation reports and not to be used separately. The type of scoring system used has to be indicated along with the score whenever such a score is provided.

Currently the following scoring approaches are provided by this methodology:

Per Website

This score calculates a ratio over the entire website. It is a simple approach to determine overall conformance but also very sensitive towards conformance failures. For example, any failure to meet a Success Criterion on any web page is directly reflected as failure of the website to meet the respective Success Criterion.

This score is calculated as follows:

  1. List the WCAG 2.0 Success Criteria applicable to the evaluation (as per Step 1.c. Define the Conformance Target).
  2. During the evaluation mark the Success Criteria that are met on all of the web pages within the selected sample (as per Step 3: Select a Representative Sample). For example, if Sucess Criterion 1.1.1 is met on all web pages, then this Success Criterion is marked as "met").
  3. After the evaluation calculate the sum of Success Criteria that are met consistently across the entire sample, and divide this by the sum of all applicable Success Criteria.
Per Web Page

This score calculates an average ratio over each web page. It is less sensitive to conformance failures such as occassional oversight but does not consider the relative frequency of failures. For example, a website that has relatively few videos but consistenly fails to provide captions for these videos may still have a high score even though it is profoundly disadvantaging certain users.

This score is calculated as follows:

  1. List the WCAG 2.0 Success Criteria applicable to the evaluation (as per Step 1.c. Define the Conformance Target).
  2. During the evaluation mark any Success Criteria that are met for each of the web pages within the selected sample (as per Step 3: Select a Representative Sample). For example, if Sucess Criterion 1.1.1 is met on a given web page, then this Success Criterion is marked as "met" for that one web page.
  3. After the evaluation calculate the entire sum of Success Criteria that are met on each web page, and divide this by the sum of all applicable Success Criteria across all pages (ie. multiplied by the number of web pages).
Per Instance

This score calculates an average ratio over all Success Criteria instances. It is least sensitive to conformance failures such as occassional oversight and considers the relative frequency of failures. However, this score is quite demanding to calculate without appropriate tools support. It is also not always easy to determine each possible instance for Success Criteria.

This score is calculated as follows:

  1. List the WCAG 2.0 Success Criteria applicable to the evaluation (as per Step 1.c. Define the Conformance Target).
  2. During the evaluation of each web page within the selected sample (as per Step 3: Select a Representative Sample) calculate the sum instances for which each Success Criterion is applicable for each web page and the number of instances for which they are met (eg. the number of times in which Sucess Criterion 1.1.1 was applicable on a given web page and the number of times in which it was met).
  3. After the evaluation calculate the entire sum of Success Criteria that were met on each web page, and divide this by the sum of all applicable Success Criteria across all pages (ie. multiplied by the number of web pages).

Note: According to WCAG 2.0, Success Criteria that do not apply to the content are deemed to have been met. Also, all Success Criteria are not met for web pages on which any of the WCAG 2.0 conformance requirements are not met.

Step 5.d: Provide Machine-Readable Reports (Optional)

Methodology Requirement 5.d: Provide machine-readable reports. (Optional).

Machine-readable reports facilitate processing the results by authoring and evaluation tools, for example to help monitor accessibility of a website over time. The Evaluation and Report Language (EARL) is a machine-readable format that was specifically designed for this purpose. It is recommended to use EARL for providing machine-readable reports. See also Understanding Metadata to learn more about uses of metadata, including machine-readable reports, such as EARL.

Considerations for Particular Situations

The methodology defined by this document is flexible to allow its implementation in different situations and contexts. This section provides guidance for applying this methodology in some of these situations.

Evaluating a Large Website with Separate Parts

Evaluation of large websites according to this methodology may be carried out by evaluating the individual website areas, such as the online shop, departments within an organization, blogging area, and other sub-sites. In this case, each sub-site has to meet the terms established in section Scope of Applicability and has to be evaluated using this methodology. Evaluation of each sub-site has to be carried out to at least the same conformance target (as per Step 1.c. Define the Conformance Target) as for the main website.

An additional evaluation has to be carried out according to this methodology, whereby the selected sample (as per Step 3: Select a Representative Sample) has to include at least all common web pages of the main website plus two randomly selected web pages from the representative sample selected for each sub-site. The reporting, scoring, and accessibility statements per Step 5: Report the Evaluation Findings applies to the entire set of sampled web pages from the main website and all its sub-sites.

Re-Running a Website Conformance Evaluation

Conformance evaluation according to this methodology may be re-run after a short period, for example when issues are identified and repaired by the website owner or website developer, or periodically to monitor progress. In such cases, for the issues that were identified, include at least:

Large-Scale Evaluation of Multiple Websites

Carrying out full conformance evaluation of multiple websites can involve a lot of effort. This methodology can not be undertaken using just automated evaluation, which can only evaluate a small portion of the accessibility requirements, it also requires manual evaluation by experts. Limiting the goal of the evaluation in Step 1.b: Define the Goal of the Evaluation to "Basic Reporting" and careful selection of the samples in Step 3: Select a Representative Sample can optimize the effort required for each evaluation.

Application of this Methodology

The methodology defined by this document is flexible to allow its implementation in different situations and contexts. It is not required to carry out any of the steps and activities defined by this methodology in any particular sequence. For proper application of this methodology, use all the following Methodology Requirements:

Results, reports, accessibility statements, and performance scores can only be claimed to be in accordance with this methodology when the evaluation carried out meets these Methodology Requirements.

Appendices

Appendix A: Acknowledgements

This publication has been developed with support from the European Commission funded WAI-ACT Project (IST 287725).

Contributors

Past and present active participants of the WCAG 2.0 Evaluation Methodology Task Force (Eval TF) include: Shadi Abou-Zahra; Frederick Boland; Denis Boudreau; Amy Chen; Vivienne Conway; Bim Egan; Michael Elledge; Wilco Fiers; Detlev Fischer; Elizabeth Fong; Vincent François; Alistair Garrison; Emmanuelle Gutiérrez y Restrepo; Katie Haritos-Shea; Martijn Houtepen; Peter Korn; Maureen Kraft; Aurelien Levy; David MacDonald; Kerstin Probiesch; Donald Raikes; Corominas Ramon; Roberto Scano; Samuel Sirois; Sarah J Swierenga; Eric Velleman; Konstantinos Votis; Kathleen Wahlbin; Elle Waters; Richard Warren; Léonie Watson.

Appendix B: References

QA-FRAMEWORK
Dubost K, Rosenthal L, Hazaël-Massieux D, Henderson L (eds) (2005) W3C Quality Assurance Framework: Specification Guidelines. W3C. Available at: http://www.w3.org/TR/qaframe-spec/
RFC3986
Fielding R, Gettys J, Mogul J, Frystik H, Masinter L, Berners-Lee T (eds) (1999). Hypertext Transfer Protocol - HTTP/1.1. Request for Comments: 2616. IETF. Available at: http://www.ietf.org/rfc/rfc3986.txt
RFC2616
Berners-Lee T, Fielding R, Masinter L (eds) (2005). Uniform Resource Identifier (URI): Generic Syntax. Request for Comments: 3986. IETF. Available at: http://www.ietf.org/rfc/rfc2616.txt
UWEM
Velleman E.M, Velasco C.A, Snaprud M (eds) (2007). D-WAB4 Unified Web Evaluation Methodology (UWEM 1.2 Core). Wabcluster. Available at: http://www.wabcluster.org/uwem1_2/
WCAG20
Caldwell B, Cooper M, Guarino Reid L, Vanderheiden G, eds (2008). Web Content Accessibility Guidelines 2.0. W3C. Available at: http://www.w3.org/TR/WCAG20/

Appendix C: Example Reports

[Editor Note: This section will be updated to better align it with the guidance in Step 5: Report the Evaluation Findings, in particular to make it more useful in situations where no conformance claims are being made.]

This Appendix proposes three different levels of reporting following the goals defined in Step 1.b: Define the Goal of the Evaluation and the documentation defined in Step 5.a: Provide Documentation for Each Step:

  1. Basic Report
  2. Detailed Report
  3. In-Depth Analysis Report
All reports should at least have the following basic information:

For Detailed Report add to basic information:

For In-Depth Analysis Report add to basic information:

Reports could also indicate the impact of issues found and specifically identify any high impact issues that are easy to fix. This could include: Any accessibility issues that interfere with the user completing tasks, issues that affect navigation and orientation, issues that occur very frequently and issues that can cause any loss or harm to a user. A lot of the important but technical information could be put in an appendix (such as documentation of each outcome of the steps as per Step 5.a: Provide Documentation for Each Step).

Appendix D: Document Changes

Changes since the Public Working Draft of 26 February 2013 include:

A full disposition of comments of all the comments received on the 26 February Working Draft is available.