[contents]

Abstract

This document provides guidance on evaluating the extent of conformance of websites to the Web Content Accessibility Guidelines (WCAG) 2.0. It describes a procedure, highlights considerations, and promotes good practice to guide accessibility evaluators in creating conformance evaluation reports. It does not provide instructions on a feature-by-feature evaluation of web content; it relies on WCAG 2.0 techniques for this. It is an informative resource that is part of a series of W3C/WAI resources on Evaluating Websites for Accessibility and that complements the existing WCAG 2.0 Documents. It does not define additional WCAG 2.0 requirements nor does it replace or supersede it in any way.

The methodology described in this document is intended for use by people who are experienced in accessibility evaluation using WCAG 2.0 and its supporting resources. It is primarily designed for evaluating existing websites, for example to learn about and monitor their level of accessibility, but it can also be useful during earlier development stages of website creation. It is applicable to static and dynamically generated websites, mobile websites and applications, and any other type of websites. It is independent of particular web technologies, evaluation tools, web browsers, assistive technologies, and other software, and it is suitable for use in different evaluation contexts, including self-assessment and third-party evaluation.

[For Review: Significantly rewritten (see previous Abstract) to reflect group discussions and to address comment 1, comment 2, and comment 20.]

Status of this document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

This 19 November 2013 Editors Draft of Website Accessibility Conformance Evaluation Methodology (WCAG-EM) 1.0 addresses the comments received on the previously published Working Draft of 26 February 2013. It addresses all issues raised though some sections may need further refinement.

This document is intended to be published and maintained as an informative W3C Working Group Note after review and refinement. The WCAG 2.0 Evaluation Methodology Task Force (Eval TF) invites discussion and feedback on this document by developers, evaluators, researchers, and others with interest in web accessibility evaluation.

Please send comments on this Editors Draft of Website Accessibility Conformance Evaluation Methodology 1.0 to public-wai-evaltf@w3.org (publicly visible mailing list archive). These comments will be considered in the internal discussions of Eval TF.

Publication as Editors Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

This document has been produced by the WCAG 2.0 Evaluation Methodology Task Force (Eval TF, a joint task force of the Web Content Accessibility Guidelines Working Group (WCAG WG) and Evaluation and Repair Tools Working Group (ERT WG), as part of the Web Accessibility Initiative (WAI) Technical Activity.

This document was produced by two groups operating under the 5 February 2004 W3C Patent Policy. The groups do not expect this document to become a W3C Recommendation. W3C maintains a public list of any patent disclosures for WCAG WG and public list of any patent disclosures for ERT WG made in connection with the deliverables of each group; these pages also include instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy.

Table of Contents

Overview

List of Sections

List of Appendices

Introduction

Evaluating the extent to which a website conforms to the Web Content Accessibility Guidelines (WCAG) 2.0 is a process involving several steps. The activities carried out within these steps are influenced by many aspects such as the type of website (e.g. static, dynamic, mobile, etc.), its size, complexity, and the technologies used to create it (e.g. HTML, PDF, etc.), how much knowledge the evaluators have of how the website was or is being developed (e.g. white-box, black-box, or gray-box testing), as well as the main purpose for the evaluation (e.g. to issue an accessibility statement, to plan a redesign process, to do research, etc.).

This methodology, the Website Accessibility Conformance Evaluation Methodology (WCAG-EM) 1.0, describes the steps that are common to website conformance evaluation processes and highlights considerations to help evaluators apply these steps in the context of a particular website. Following this methodology helps evaluators to apply good practice, avoid common mistakes, and achieve more comparable results. It supports common approaches and understanding for evaluating the extent of conformance of websites to WCAG 2.0, though in the majority of use cases it does not directly result into WCAG 2.0 conformance claims.

Also, this methodology does not in any way add to or change the requirements defined by the normative WCAG 2.0 standard, nor does it provide instructions on feature-by-feature evaluation of web content. The methodology relies on WCAG 2.0 techniques such as the Techniques for WCAG 2.0 documented by W3C/WAI, but is not limited to this set of techniques.

[For Review: Significantly rewritten (see previous Introduction) to reflect group discussions and to address comment 20.]

Purposes for this Methodology

In many situations it is necessary to evaluate the accessibility of a website, for example before releasing, acquiring, or redesigning the website. Periodic evaluation is also useful for monitoring the accessibility performance of websites over time. WCAG-EM is for anyone who wants to follow a common procedure for conformance evaluation of websites to WCAG 2.0. This includes:

[For Review: Significantly rewritten (see previous "Purpose of this document") and merged with previous "Scope of this document" to reflect group discussions and to address comment 20.]

Relation to WCAG 2.0 Conformance Claims

WCAG 2.0 defines conformance requirements for individual web pages rather than for entire websites. It also defines the optional conformance claims that can be made to cover individual web pages, series of web pages such as a multi-page form, and multiple related web pages such as a website. However, conformance claims in WCAG 2.0 can only be made to cover specific web pages that are known to satisfy each conformance requirement. This includes when all web pages that are in the scope of a conformance claim have been each evaluated or created in a process that ensures that they each satisfy all the conformance requirements.

WCAG 2.0 conformance claims cannot be made for entire websites based upon the evaluation of a selected sub-set of web pages and functionality alone, as it is always possible that there will be unidentified conformance errors on these websites. However, in the majority of use cases of this methodology only a sample of web pages and functionality from a website is selected for evaluation. Thus in the majority of use cases, using WCAG-EM does not directly result into WCAG 2.0 conformance claims for the target websites. Instead, Step 5.b provides guidance on how to make "Accessibility Conformance Evaluation Statements" for websites that have been evaluated according to WCAG-EM.

[For Review: New section that addresses discussion during telco of 25 July, telco of 12 September, telco of 17 October, and comment 81.]

Background Reading

The information below related to accessibility, WCAG 2.0, and evaluation is important background for using this methodology:

Web Accessibility Essentials

The following documents introduce the essential components of web accessibility and how people with disabilities use the Web, and are critical for understanding the broader context of web accessibility evaluation:

Evaluating Websites for Accessibility

A multi-page resource suite that outlines different approaches for evaluating websites for accessibility. The following resources are particularly important in this context:

Web Content Accessibility Guidelines (WCAG) 2.0

Internationally recognized standard explaining how to make web content more accessible to people with disabilities. The following resources are particularly important in this context:

Terms and Definitions

For the purposes of this document, the following terms and definitions apply:

Complete Processes
From WCAG 2.0 Conformance Requirement for Complete Processes:
When a web page is one of a series of web pages presenting a process (i.e., a sequence of steps that need to be completed in order to accomplish an activity), all web pages in the process conform at the specified level or better. (Conformance is not possible at a particular level if any page in the process does not conform at that level or better.)
Conformance
From WCAG 2.0 definition for "conformance":
Satisfying all the requirements of a given standard, guideline or specification
Core Functionality
Functionality of a website that, if removed, fundamentally changes the use or purpose of the website for users. This includes information and tasks that users of a website carry out to perform this functionality.
Note: Examples of functionality include "selecting and purchasing a product from the shop area of the website", "filling and submitting the form provided on the website", and "registering for an account on the website".
Note: Other functionality are not excluded from the scope of evaluation. The term "core functionality" is intended to help identify critical web pages and include them among others in an evaluation.
Common Web Pages
Web pages that are relevant to the entire website. This includes the homepage, login page, and other entry pages, and, where applicable, the sitemap, contacts page, site help, legal information, and similar web pages that are typically linked from all other web pages (usually from the header, footer, or navigation menu of a web page).
Evaluator
The person, team of people, organization, in-house department, or other entity responsible for carrying out the evaluation.
Evaluation Commissioner
The person, team of people, organization, in-house department, or other entity that commissioned the evaluation.
Note: In many cases the evaluation commissioner may be the website owner or website developer, in other cases it may be another entity such as a procurer or an accessibility monitoring survey.
Relied Upon (Technologies)
From WCAG 2.0 definition for "relied upon":
The content would not conform if that technology is turned off or is not supported
Templates
From ATAG 2.0 definition for "templates":
Content patterns that are filled in by authors or the authoring tool to produce web content for end users (e.g., document templates, content management templates, presentation themes). Often templates will pre-specify at least some authoring decisions.
[Editor Note: This definition is from the 7 November 2013 Candidate Recommendation (CR) Working Draft of ATAG 2.0. While this is a stable Working Draft, this definition might change in future drafts depending on how ATAG 2.0 evolves.]
Website
A coherent collection of one or more related web pages that together provide common use or functionality. It includes static web pages, dynamically generated web pages, and web applications.
Note: The focus of this methodology is on full, self-enclosed websites. Websites may be composed of smaller sub-sites, each of which can be considered to be an individual website. For example, a website may include an online shop, an area for each department within the organization, a blog area, and other areas that may each be considered to be a website.
Website Developer
The person, team of people, organization, in-house department, or other entity that is involved in the website development process including but not limited to content authors, designers, programmers, quality assurance testers, and project managers.
Website Owner
The person, team of people, organization, in-house department, or other entity that is responsible for the website.
Web Page
From WCAG 2.0 definition for "web page":
A non-embedded resource obtained from a single URI using HTTP plus any other resources that are used in the rendering or intended to be rendered together with it by a user agent
Note: Web pages may include multimedia content, interactive components, and rich and mobile web applications.
Web Page States
[For review: Added this new definition, also to address Comment 13.]
Web pages with dynamic content can have different states; for example, they might generate different content and provide different presentation or functionality depending on the particular user and on actions initiated by the user. Each such web page state is considered as an individual web page in the context of this methodology.
Note: Examples of web page states are the individual pages of a multi-part online form that are dynamically generated depending on users input. These individual states may not have unique URIs and may need to be identified by describing the settings, input, and actions required to generate them.

Using This Methodology

WCAG-EM is used for thorough evaluation of the extent of conformance of websites to WCAG 2.0. Before applying WCAG-EM on an entire website it is usually good to do a preliminary evaluation of different web pages from the target website to identify obvious conformance errors and to develop an overall understanding of the accessibility performance of the website. Easy Checks - A First Review of Web Accessibility describes an approach for such preliminary evaluation that is complementary to this methodology.

Required Expertise

Users of WCAG-EM are assumed to be knowledgeable of WCAG 2.0, accessible web design, assistive technologies, and of how people with different disabilities use the Web. This includes understanding of the relevant web technologies, barriers that people with disabilities experience, assistive technologies and approaches that people with disabilities use, and evaluation techniques and tools to identify potential barriers for people with disabilities. In particular, it is assumed that users of WCAG-EM are deeply familiar with the resources listed in section Background Reading.

Evaluation Tools (Optional)

WCAG-EM is independent of any particular web accessibility evaluation tool, web browser, and other software tool. However, web accessibility evaluation tools significantly assist evaluators during the evaluation process and contribute to more effective evaluation. For example, some web accessibility evaluation tools can scan entire websites to help identify relevant pages for manual evaluation. Tools can also assist in manually evaluating the many checks that are not automatable. Selecting Web Accessibility Evaluation Tools provides further guidance beyond the scope of this document.

Review Teams (Optional)

WCAG-EM can be carried out by an individual evaluator with the skills described in section Required Expertise. However, using the combined expertise of review teams provides better coverage for the required skills and helps identify accessibility barriers more effectively. While not required for using this methodology, it is recommended to employ review teams for conformance evaluation of websites. Using Combined Expertise to Evaluate Web Accessibility provides further guidance beyond the scope of this document.

Involving Users (Optional)

Involving people with disabilities and people with aging-related impairments helps identify additional accessibility barriers that are not easily discovered by the evaluators alone. While not required for using this methodology, it is strongly recommended to involve real people with a wide range of abilities during the evaluation process. Involving Users in Web Accessibility Evaluation provides further guidance beyond the scope of this document.

Scope of Applicability

WCAG-EM is designed for evaluating full, self-enclosed websites. This includes websites of organizations, entities, persons, events, products, and services. It also includes mobile websites and applications. Specific examples include:

A website can be part of a larger website, such as the online shop in the examples above. It can also be a clearly separable version of the website such as the mobile or Dutch language versions of the website in the examples above. WCAG-EM can be applied to any such determinable website, regardless if it is part of a larger website or the larger website itself. The exact definition of a target website to be evaluated is determined as part of Step 1.a.

Principle of Website Enclosure

When a target website is defined for evaluation, it is essential that all web pages and functionality that are within the scope of this website definition are considered for evaluation. Excluding such aspects of a website from the scope of evaluation would conflict with the WCAG 2.0 conformance requirements for full pages and complete processes, or otherwise distort the evaluation results.

Example 1: Website Enclosure

Diagram of a University Website explained in the following paragraph.

In the diagram above, the university website is made of the distinct areas "Information for Students", "Information for Lecturers", "Courseware Application", and "Library Application". The "Courseware Application" includes "Physics Courses", "Medical Courses", and "History Courses" that are aggregated into the application. Also, the university website has individual web pages such as legal notices, sitemap, and others that are common to all areas.

In the example above, if the university website in its entirety is defined as the target for evaluation then all of the depicted areas are within the scope of the evaluation. This includes any aggregated and embedded content such as maps of the campus, forms for online payments, and discussion boards, including when such parts originate from third-party sources.

Similarly, if only a website area such as the "Courseware Application" is defined as the target for evaluation then all the parts of this area are within the scope of the evaluation. In this case it would include all depicted courses, as well as the individual web pages that are common to all areas of the university (see also: definition for Common Web Pages).

Particular Types of Websites

WCAG-EM is applicable to any website including dynamically generated websites, rich web applications, mobile websites and other types of websites. Some particular types of websites include:

Small Websites
For websites with few web pages the sampling procedure defined in Step 3: Select a Representative Sample will likely result in selecting most or all of the web pages from the target website. In cases where all web pages can be evaluated, the sampling procedure can be skipped and the selected sample is considered to be the entire website in the remaining steps.
Web Applications
Web applications are generally composed of dynamically generated content and functionality (web page states). They tend to be more complex and interactive. Examples of web applications are webmail clients, document editors, and online shops. Web applications are usually part of a larger website but can sometimes also constitute a website of their own for evaluation.
Note: Due to the many possibilities to generate content and functionality in web applications it is sometimes not feasible to exhaustively identify every possible web page, web page state, and functionality. Web applications will typically require more time and effort to evaluate and larger web page samples to reflect the different types of content, functionality, and processes.
Website with Separable Areas
In some cases websites may have clearly separable areas where using one area does not require or depend on using another area of the website. For example, an organization might provide an extranet for employees that is linked from the public website but is otherwise disjoint from it, or it might have sub-sites for the different departments of the organization that are clearly distinct from one another. Such separable areas can be considered as an individual website each for evaluation.
Note: Some websites provide additional or different content and functionality depending on the user (typically after a log-in). This additional content and functionality is generally part of the core purpose and functionality of the website and is thus not considered to be a separable website area. Note the role of common web pages in this context too.
Website in Multiple Versions
Some websites are available in multiple versions that are each independent of one another in usage. For example, a mobile version of a website and versions of a website in different languages, where using one version does not require or depend on using another version of the website. Usually such website versions have a different set of URIs each. Such website versions can be considered as individual websites for evaluation.
Note: Websites using responsive design techniques (ie. adapt the presentation according to user hardware, software, and preferences) as opposed to redirect the user to a different location are not considered to be independent website versions.

Particular Evaluation Contexts

WCAG-EM is flexible to allow its applicability in different situations and contexts. This section provides guidance for applying this methodology in some of these situations.

Self-Assessment of Conformance
In this context evaluators often have easier access to the website developers and maintainers, the development enviornment and authoring tools, and the materials used for development. Particularly use cases, design analysis, technical specifications and documentation, and testing resources can make evaluation more effective and should be leveraged where possible.
Third-Party Assessment of Conformance
Third-party evaluators (ie. pure black-box testing) typically have very little information about how a website was developed and all its areas and functionality. Therefore it is more difficult, often impossible, to make exhaustive conformance claims based on such evaluation alone. However, it can be very effective for validating conformance claims and statements made for a website.
Evaluating During Development
It is critical to evaluate accessibility throughout the design and development stages of a website to manage its conformance. This methodology provides guidance that can be useful during these stages, though some adaptation may be needed for this. However, it is important to beware that evaluations carried out during the design and development stages can be very quickly obsoleted even by minor changes so that they should not be used for making statements about the finalized website.
Evaluating Composite Websites
When evaluating websites that are composed of separable areas (such as online shop, blog area, and other sub-sites), it can be useful to evaluate each website area separately according to this methodology, then do an overall evaluation with samples from each website area and the common web pages. This would ensure more complete coverage of the website in its entirety as well as provide more insights about how each website area performed, which is often significantly different.
Evaluating Aggregated Websites
Websites that are generated using content that is combined from different sources, such as portals with portlets, are usually much more challenging to evaluate because of the many different content instances that can be generated. Generally it is not possible to evaluate the content from their sources separately but rather as displayed to the users when they are combined.
Evaluating Third-Party Content
Third-party content can be that embedded into a website from a different source, or content that is generated by users such as in an online forum. If such content that is outside the control of website authors is regularly monitored and repaired (within two business days), then it can be deemed as conforming. Otherwise it needs to be clearly identified as non-conforming parts of the web pages that it appears in. WCAG 2.0 provides more guidance on providing a Statement of Partial Conformance.
Re-Running a Conformance Evaluation

Conformance evaluation according to this methodology may be re-run after a short period, for example when issues are identified and repaired by the website owner or website developer, or periodically to monitor progress. In such cases, for the issues that were identified, include at least:

Large-Scale Evaluation
Carrying out mass evaluation of many websites, for example for national or international surveying, is usually carried out using primarily automated evaluation tools and relatively few web pages for full manual inspection. Such evaluations do not usually address the neccessary qualitative depth of review per website that this methodology is primarily designed for.

Evaluation Procedure

This section defines the stages and activities of an evaluation procedure. The stages are not necessarily sequential. Also the exact sequence of the activities carried out during the evaluation stages depends on the type of website, the purpose of the evaluation, and the process used by the evaluator. Some of the activities overlap or may be carried out in parallel. The following diagram illustrates the iterations between the stages defined in this section:

Diagram about the iterations between the steps in this methodology. Explanation in the following paragraph.

The workflow diagram depicts each of the five steps defined in this section with an arrow to the next step and arrows back to all of the previous steps. This way evaluators can proceed from one step to the next and return to any preceding step in the process whenever the evaluation reveals a need to return to any one of the previous steps: 1. Define the Evaluation Scope; 2. Explore the Target Website; 3. Select a Representative Sample; 4. Audit the Selected Sample and 5. Report the Evaluation Findings.

Note: See also considerations for Particular Evaluation Contexts that may influence how an evaluation procedure is carried out.

Step 1: Define the Evaluation Scope

Methodology Requirement 1: Define the evaluation scope according to Methodology Requirement 1.a, Methodology Requirement 1.b and optionally Methodology Requirement 1.c and Methodology Requirement 1.d.

During this step the overall scope of the evaluation is defined. It is a fundamental step that affects the subsequent steps in the evaluation procedure and is ideally carried out together with the evaluation commissioner (who may or may not be the website owner) to ensure common expectations about the scope of the evaluation. Some exploration throughout this stage may be necessary to understand the full scope of the website and the required evaluation (see Explore the Target Website).

Step 1.a: Define the Scope of the Website

Methodology Requirement 1.a: Define the target website according to Scope of Applicability, so that for each web page it is unambiguous whether it is within the scope of evaluation or not.

During this step the target website (the web pages and states of web pages that are in scope of the evaluation) is documented. This scope of the website is defined according to the terms established in section Scope of Applicability.

To avoid later mismatches of expectations between the evaluator, evaluation commissioner, and readers of the resulting evaluation report, it is important to define the target website in such a way that for each web page it is unambiguous whether it is within scope or not. Formalizations such as regular expressions and listings of web addresses (URIs) are recommended where possible.

It is also important to document any particular aspects of the target website to support its identification. This includes:

[For Review: This addresses comment 35 (no change).]

Step 1.b: Define the Conformance Target

Methodology Requirement 1.b: Select a target WCAG 2.0 conformance level ("A", "AA", or "AAA") for the evaluation.

Part of initiating the evaluation process is to define the target WCAG 2.0 conformance level ("A", "AA", or "AAA") to evaluate for. WCAG 2.0 Level AA is the generally accepted and recommended target.

Note: It is often useful to evaluate beyond the conformance target of the website to get a more complete picture of its accessibility performance. For example, while a website might not fully meet a particular conformance level, it might meet individual requirements from a higher conformance level. Having this information can help plan future improvements more effectively.

Step 1.c: Define the Techniques and Failures to be Used (Optional)

[For Review: Updated links to new Understanding Techniques for WCAG Success Criteria page — do we still need all the content in this section?.]

Methodology Requirement 1.c: Define any initial sets of techniques and failures that will be used to evaluate conformance to WCAG 2.0 Success Criteria. (Optional).

Techniques and failures in the context of WCAG 2.0 are informative and not required for determining conformance with WCAG 2.0; WCAG 2.0 Success Criteria are written as testable statements. Techniques provide documented ways of satisfying WCAG 2.0 Success Criteria and failures document commonly made mistakes. More information on techniques and failures is provided in WCAG 2.0 Layers of Guidance.

W3C/WAI provides a set of publicly documented Techniques for WCAG 2.0. However, it is not necessary to use these particular techniques. In fact, in some situations, such as in a closed network, it may be necessary to use techniques that are specifically developed to the particular needs of the users of that network. Individuals and organizations developing techniques have to employ methods for establishing the technique's ability to satisfy the WCAG 2.0 Success Criteria.

While optional, it is good practice to specify initial sets (or sources) of techniques and failures to be used during the evaluation, as it helps to ensure consistent expectation between the evaluator and the evaluation commissioner. This definition is typically refined in later stages of the evaluation process, for example during the website exploration and evaluation stages. Step 4.c: Use WCAG 2.0 Techniques and Failures Where Possible (Optional) provides more guidance on using techniques and failures during evaluation.

Step 1.d: Define Additional Evaluation Requirements (Optional)

Methodology Requirement 1.d: Define any additional evaluation requirements agreed on between the evaluator and evaluation commissioner (optional).

[For Review: Proposed change this section as discussed on mailing list and in telco 18 July, and telco 5 september, to be more flexible and open for additional requirements/whishes from the evaluation commissioner(s). This section now presumes that the minimum requirements for reporting are set in Step 5: Report the Evaluation Findings with examples in Appendix C. Because the requirements are additional to the minimum requirements, the step is optional. Please review. Also look at the diff version provided in the agenda of telco for 5 september for small changes in the document as a result of changing the title and presuming that step 5 will contain the minimum requirements. This also covers comment 36 and comment 37.]

An evaluation commissioner may be interested in additional information beyond what is needed to evalaute the extent of conformance of the target website to WCAG 2.0. For example, an evaluation commissioner might be interested in:

Such additional evaluation requirements that are agreed on with the evaluator need to be clarified early on and documented. This also needed to be reflected in the resulting report, for example to clarify how the selection of web pages was carried out.

Step 2: Explore the Target Website

Methodology Requirement 2: Explore the website to be evaluated according to Methodology Requirement 2.a, Methodology Requirement 2.b, Methodology Requirement 2.c, and Methodology Requirement 2.d.

During this step the evaluator explores the target website to be evaluated, to develop an initial understanding of the website and its use, purpose, and functionality. Much of this will not be immediately apparent to evaluators, in particular to those from outside the development team. In some cases it is also not possible to exhaustively identify and list all functionality, types of web pages, and technologies used to realize the website and its applications. The initial exploration carried out in this step is typically refined in the later stages Step 3: Select a Representative Sample and Step 4: Audit the Selected Sample, as the evaluator learns more about the target website. Involvement of website owners and website developers can help evaluators make their explorations more effective.

Note: Carrying out initial cursory checks during this stage helps identify web pages that are relevant for more detailed evaluation later on. For example, an evaluator may identify web pages that seem to be lacking color contrast, document structure, or consistent navigation, and note them down for more detailed evaluation later on.

Note: To carry out this step it is critical that the evaluator has access to all the relevant parts of the website. For example, it may be necessary to create accounts or otherwise provide access to restricted areas of a website that are part of the evaluation.

Step 2.a: Identify Common Web Pages of the Website

Methodology Requirement 2.a: Identify and document the common web pages of the target website.

Explore the target website to identfy and document its common web pages. Typically these are linked directly from the main entry point (home page) of the target website, and often linked from the header, navigation, and footer sections of other web pages. The outcome of this step is a list of all common web pages.

Step 2.b: Identify Core Functionality of the Website

Methodology Requirement 2.b: Identify and document an initial list of core functionality of the target website.

Explore the target website to identify and document its core functionality. While some functionality will be easy to identify, others will need more deliberate discovery. For example, an online shop is expected to have a payment function though it might be less easy to identify that it also has a currency conversion function that is core to the particular context of the online shop. The outcome of this step is a list of functionality that users can perform on the website, for example:

Note: The purpose of this step is not to exhaustively identify all functionality of a website but to determine those that are core to the purpose and goal of the target website. This will inform later selection of web pages and their evaluation. Other functionality will also be included in the evaluation but through other selection mechanisms.

Step 2.c: Identify the Variety of Web Page Types

Methodology Requirement 2.c: Identify and document the types of web pages.

Web pages with varying styles, layouts, structures, and functionality often have different implementations of accessibility features. They are also often generated by different templates and authored by different people. Web pages may also appear differently, behave differently, and contain different content depending on the particular user. The outcome of this step is a list of the different types of web pages (as opposed to instances of web pages), including states of web pages, that appear on the target website.

Examples of web page types include:

Step 2.d: Identify Web Technologies Relied Upon

Methodology Requirement 2.d: Identify and document the technologies relied upon to provide the website.

During this step the web technologies relied upon to provide the website are identified and documented. This includes base web technologies such as HTML and CSS, auxiliary web technologies such as Java, JavaScript and WAI-ARIA, as well as specific web technologies such as SMIL and SVG. The outcome of this step is a list of technologies that are relied upon according to WCAG 2.0.

Only the web technologies that are relied upon to provide the website need to be identified for evaluation. This relates closely to the WCAG 2.0 concepts of conforming alternate versions and non-interference.

Note: Where possible, it is often also useful to identify the libraries and components used to create the website, such as Dojo, JQuery, and others. Particularly for web applications, much of the accessibility support is built into these libraries and components, and evaluation can become more effective and efficient when these are identified.

Step 2.e: Identify Other Relevant Web Pages

Methodology Requirement 3.c: Identify and document other web pages that are relevant to people with disabilities and to accessibility of the website.

Some websites include web pages that are relevant for people with disabilities and accessibility of the website. The outcome of this step is a documentation of such web pages. This includes:

Step 3: Select a Representative Sample

Methodology Requirement 3: Select a representative sample of web pages from the website according to Methodology Requirement 3.a, Methodology Requirement 3.b, Methodology Requirement 3.c, Methodology Requirement 3.d, Methodology Requirement 3.e and optionally Methodology Requirement 3.f.

While ideally every web page of a website is evaluated, usually this is not possible on most websites. In cases where all web pages can be evaluated, this sampling procedure can be skipped and the selected sample is considered to be the entire website in the remaining steps.

[For Review: Proposal for explanation of the factors has been added. Please review.]

Exploration of the target website in Step 2: Explore the Target Website (within the scope set in Step 1: Define the Evaluation Scope) provides sufficient understanding of the website to facilitate selection of a sample of web pages that is within the scope and representative of the entire target website with reasonable confidence. The size of the sample needed for an accessibility evaluation evaluation with reasonable confidence depends on different factors. Below are a few examples of these factors and how they may aditionally influence the size of the sample:

The web pages identified through the exploration will typically relate to more than one design aspect. For example, web pages with particular functionality such as scripting, multimedia, and forms will typically also use technologies such as Flash, JavaScript, PDF, Silverlight, and WAI-ARIA, and in many cases these web pages may have different design to others. Careful selection of web pages can significantly reduce the required sample size while maintaining appropriate representation of the entire website.

Step 3.a: Include Common Web Pages of the Website

Methodology Requirement 3.a: Include all common web pages into the selected sample of web pages.

All common web pages, including the common states of these web pages for web applications, are part of the selected sample. These web pages are identified in step Step 2.a: Identify Common Web Pages of the Website.

Step 3.b: Include Exemplar Instances of Web Pages

Methodology Requirement 3.b: Include (where applicable and available) of each (1) common functionality, (2) distinct types of web pages, and (3) web technologies into the selected sample of web pages.

From the variety and types of web pages identified in Step 2: Explore the Target Website (within the scope of the evaluation as defined per Step 1: Define the Evaluation Scope), select at least one distinct web page for all of the following features (where applicable and available):

Note: Every one of the features may require more pages depending on the distinct pages that have been identified. A selected web page could however also have any number of these features. For example, a selected web page could be used to represent the use of forms and scripting at the same time. The important aspect is to select at least one distinct web page for each relevant feature identified on the website, though more web pages may be necessary depending on the complexity of the website.

Step 3.c: Include Other Relevant Web Pages

Methodology Requirement 3.c: Include other web pages relevant for people with disabilities and accessibility into the selected sample of web pages.

Websites frequently include web pages that are relevant for people with disabilities and accessibility but do not explicitly match the criteria described in the previous sections. These web pages are also part of the selected sample. They typically include:

Step 3.d: Include Complete Processes in the Sample

Methodology Requirement 3.d: Include all web pages that are part of a complete process.

The selected sample has to include all web pages that belong to a series of web pages presenting a complete process. No web page in the selected sample may be part of a process without all other web pages that are part of that process to be also included into the selected sample.

  1. For any complete process identified in Step 2.b: Identify Key Functionalities of the Website, the base page is included in the sample. The base page is the page that acts as launch pad for other states called up when traversing the process.
  2. For complete processes with branches including error handling, record at least the default branch of the complete process. The default branch follows the standard use case, describing the default way through the complete process. It assumes that there are no user input errors and no selection of additional options. For example, in a checkout system, the user would proceed to checkout, confirm the default payment option, provide all required payment details correctly, and complete the purchase, without changing contents of the shopping cart, using a stored user profile, selecting alternative options for payment or shipping address, providing erroneous input, and so forth.
  3. For processes that consequently call up or generate further pages (this may be new URIs), add these further pages to the page sample, directly where possible or by specifying the actions on the base page of the process that led to it (e.g., "fill out form correctly and press "Submit"). It should be clear that there is a relationship between the pages added the sample for complete processes. The record should describe that this further page is part of the same process as the first page.
  4. For each web page of the complete process, the record should at least describe the steps / user interactions that lead to the consequent process states on the same web page. Whenever a process calls up a new URI, consequent states should be taken into the sample and their relation recorded.
  5. The record needs only specify additional states to be evaluated, i.e. all elements that are added, modified or made visible on top of the base page of the complete process.
  6. For complete processes with branches that are commonly used or entered and are critical for the successful completion of the process, add critical alternative branches of the same complete process to the sample. Branches may terminate where they re-enter the default branch of the process, or be covered until their completion. For example, adding a new shipping address will be registered as a critical alternative branch that leads back to the default branch of the process.

Step 3.e: Include a Randomly Selected Sample

Methodology Requirement 3.e: Include a randomly selected sample.

A randomly selected portion of the sample, even if it is small, can act as a simple verification indicator of the results found with the structured sample. In that case, a few web pages would then be sufficient to increase confidence in the results of the evaluation. Therefore, from the scope of the website as defined in Step 1: Define the Evaluation Scope, randomly select at least 10 to 15 percent of the number of web pages in the sample with a minimum of 5 randomly selected web pages (if available on the website). How the random sample was selected is reported in Step 5.a: Provide Documentation for Each Step.

[Editor Note: We plan to add more information about random sampling in an appendix or in a seperate document. Possibly with some examples. See discussion on the list and minutes of telco of 12 september.]

Step 3.f: Eliminate Redundancies in the Sample (Optional)

Methodology Requirement 3.f: Filter the sample to eliminate excessive redundancies.

[For Discussion: Discussed in the telco of 19 september. Peter proposed "to put "uniqueness" of items in the sample into step 3.e. Then 3.f can be a discussion about "appearing to be similar", and about hetero-/homogeneity".]

Once a sample has been selected according to Methodology Requirement 3.a, Methodology Requirement 3.b, Methodology Requirement 3.c, Methodology Requirement 3.d, and Methodology Requirement 3.e, evaluators may identify web pages that are completely identical with other web pages in the sample. Replace these redundant web pages in the sample with other web pages using the same Methodology Requirement as for the removed web pages.

Note: Redundancy can also indicate that a website is homogenous.

Step 4: Audit the Selected Sample

Methodology Requirement 4: Audit the selected sample of web pages according to Methodology Requirement 4.a, Methodology Requirement 4.b, and optionally, Methodology Requirement 4.c, Methodology Requirement 4.d and Methodology Requirement 4.e.

WCAG 2.0 defines five conformance requirements that need to be met for each web page in the sample selected per Step 3: Select a Representative Sample. This includes checking whether each WCAG 2.0 Success Criterion in the target conformance level (per Step 1.b: Define the Conformance Target) has been met or not met for each of these web pages.

Note: Many web pages will have repetitive content, such as the header, navigation, and other common components that may not need to be re-evaluated on each occurrence. Depending on the additional requirements for evaluation (optional) as defined by Step 1.d: Define Additional Requirements for Evaluation (Optional), an evaluator may not need to continue to identify successes and failures in meeting the conformance target for these repetitive elements on every web page. Step 5: Report the Evaluation Findings provides more guidance on reporting.

Note: According to WCAG 2.0, if there is no content related to a success criteria (for example, no video), then the success criteria is "satisfied". In such cases, an evaluator may use text such as "not applicable" as a description associated to the outcome "satisfied", to denote the particular situation where a success criterion was satisfied because no relevant content was applicable.

Proposal: "According to WCAG 2.0, Success Criteria to which there is no matching content are satisfied. ."

[For Review: This addresses comment 58, comment 59, comment 60, comment 61, comment 62, comment 63 and comment 57 (no change).]

Step 4.a: Check web pages for WCAG2.0 conformance requirements

Methodology Requirement 4.a: Check that each web page in the selected sample per Step 3: Select a Representative Sample meets each of the WCAG 2.0 conformance requirements, including all WCAG 2.0 Success Criteria relevant to the target conformance level (as per Step 1.b: Define the Conformance Target).

[For Review: Changed the title from "step 4.a: Check for the Broadest Variety of Use Cases" to "step 4.a: Check web pages fort WCAG2.0 conformance requirements". This addresses comment 65. Added the text "Note that techniques and failures in the context of WCAG 2.0 are only informative" to the text below addressing comment 74.]

For each web page in the sample selected per Step 3: Select a Representative Sample, check whether each of the WCAG 2.0 conformance requirements have been met, including all WCAG 2.0 Success Criteria relevant to the target conformance level (as per Step 1.b: Define the Conformance Target). Note that techniques and failures in the context of WCAG 2.0 are only informative.

[For Review: Take the text below out ("To carry out ... for more guidance."). It is not relevant to this section and it is addressed elsewhere in the document. This is about checking for conformance requirements. Use cases influence the selection of the sample and are addressed earlier. Addresses comment 66, comment 67, comment 68 and comment 64 (no change). ]

To carry out an evaluation effectively, it is often useful to construct and apply personas, use cases, and scenarios of users with a variety of abilities and using different web browsing techniques, including assistive technology and adaptive strategies. It is critical to consider the broadest possible spectrum of use cases to help identify issues that may occur to different audiences. It is strongly recommended to also involve real users during this process, to help identify issues that may not be easily identified through expert testing alone. See Involving Users in Evaluating Web Accessibility for more guidance.

[For Discussion: Can we add more guidance below? See comment 66: Can we get guidance out of WCAG as proposed in comment 66?]

This assessment also needs to be complemented with focused testing of particular situations including:

Note: Templates are often used to create many web pages, sometimes entire parts of a website. While evaluating templates is optional in this methodology, in some contexts it can be helpful to check templates on their own. Evaluating templates may identify potential issues that may not be easily identified through evaluating individual instances of web pages. However, issues identified in templates alone do not necessarily imply that these issues occur on the website and need to be validated on individual instances of web pages. Also, identifying no issues in templates does not necessarily imply that no issues occur on instances of web pages.

Step 4.b: Assess Accessibility Support for Features

Methodology Requirement 4.b: Check if accessibility features provided on the website are accessibility supported.

[For Review: This addresses comment 69, comment 70 and comment 71.]

To ensure that the accessibility features such as text-alternatives, captions, keyboard access are actually usable in practice, each of these accessibility features has to be accessibility supported. The Level of Assistive Technology Support Needed for "Accessibility Support" defined by WCAG 2.0 needs to be supported throughout the website.

In some situations techniques for meeting WCAG 2.0 and repositories on accessibility support provide insights on the level of support for accessibility features in particular combinations of web technologies, web browsers, and assistive technology. However, note that techniques, including "sufficient techniques", are not automatically accessibility supported. The evaluator is responsible for the accuracy of the assessment of accessibility support and the resulting evaluation.

Step 4.c: Use Techniques and Failures Where Possible (Optional)

Methodology Requirement 4.c: Where possible, use techniques and failures (that have been documented by W3C and others as satisfying the Success Criteria) to help assess successes and failures in satisfying the WCAG 2.0 Success Criteria relevant per Step 1.b: Define the Conformance Target (Optional).

Reminder: Techniques and failures in the context of WCAG 2.0 are only informative. They can help assess if WCAG 2.0 Success Criteria are satisfied by providing documented ways of satisfyiing them and commonly occurring failures in meeting them. However, as per the WCAG 2.0 conformance requirements, only the Success Criteria have to be satisfied. And you can use any techniques that satisfy the Success Criteria, whether they are documented yet as part of WCAG 2.0 or not.

The initial sets or sources of techniques and failures to be used during evaluation may be defined in Step 1.c: Define the Techniques and Failures to be Used (Optional). However, during evaluation initial sets may often need to be refined according to the particular situation, such as for evaluating particular web technologies and accessibility features that are identified on the website.

Techniques in the context of WCAG 2.0 are documented ways for satisfying or for going beyond what is required by individual WCAG 2.0 Success Criteria. A WCAG 2.0 Success Criterion is satisfied on a web page when:

Conversely, failures in the context of WCAG 2.0 are documented ways of not satisfying individual WCAG 2.0 Success Criteria. A WCAG 2.0 Success Criterion is not satisfied on a web page when a failure applies to any instance of web content that is addressed by the WCAG 2.0 Success Criterion.

Techniques are not the only way to satisfy WCAG 2.0 Success Criteria, and failures are not the only way to fail WCAG 2.0 Success Criteria. Techniques and failures are not exhaustive as they cannot cover every possible situation. Also, the techniques used to satisfy WCAG 2.0 Success Criteria during the development may not be known to the evaluator. Particularly for newly released web technologies, or when these web technologies are used in particular contexts, there may be no publicly or proprietary documented techniques and failures available to the evaluator. The evaluator has to consider these limitations when using techniques and failures to evaluate conformance to WCAG 2.0.

[For Review: This addresses comment 72, comment 73 and comment 74 (No change). Comment 74 is also addressed in step 4.a.]

Step 4.d: Archive Web Pages for Reference (Optional)

Methodology Requirement 4.d: Archive every evaluated web page for reference. (Optional)

[For Review: Changed first sentence to address comment 75. This should make it clear that optional only relates to the extra things. It now points to step 5 for requirements about how to report the evaluation findings.]

Archive web pages for future reference. Please read Step 5 for requirements about how to report the evaluation findings. These can be complemented, to avoid ambiguity, with any of the following as appropriate:

Step 4.e: Record Software Tools and Methods Used (Optional)

Methodology Requirement 4.e: Record the web accessibility evaluation tools, web browsers, assistive technologies, other software, and methods used to evaluate the web pages. (Optional).

For future reference, record software tools and methods used to evaluate the web pages. This includes versions of web accessibility evaluation tools, web browsers and add-ons, assistive technology, and other software used during the evaluation. Depending on the desired level of detail for reporting (if more than the minimum required) defined by Step 1.d: Define Additional Requirements for Evaluation (Optional), this recording may apply globally for the entire evaluation, to individual web pages, or to individual checks carried out within the audited web pages. For detailed reporting a table or grid may be useful to record what was used for the different pages and Success Criteria in the sample.

[For Review: This addresses comment 76 (no change).]

Step 5: Report the Evaluation Findings

Methodology Requirement 5: Document the evaluation findings according to Methodology Requirement 5.a and optionally Methodology Requirement 5.b, Methodology Requirement 5.c, and Methodology Requirement 5.d.

[For review: This section describes the minimum requirements for reporting. There can be additional requirements added by the evaluation commissioner, but they are not required by WCAG-EM. In this section, we can indicate where this extra information can be reported (todo). Additional requirements have been discussed on mailing list and in telco 18 July and make the methodology more flexible for people who want to do more. Please review.]

This section describes how to report the evaluation findings that have been gathered during the previous steps. Reporting is a key element of every evaluation and helps facilitate replicability of the results.

Note: Individual pieces of the reporting are gathered throughout the evaluation process, not necessarily at the end of it.

Step 5.a: Provide Documentation for Each Step

Methodology Requirement 5.a: Document each outcome of the steps defined in Step 1: Define the Evaluation Scope, Step 2: Explore the Target Website, Step 3: Select a Representative Sample, and Step 4: Audit the Selected Sample.

Document the outcomes of each of the previous steps defined in Step 1: Define the Evaluation Scope, Step 2: Explore the Target Website, Step 3: Select a Representative Sample, and Step 4: Audit the Selected Sample, to ensure justifiable, transparent, and repeatable evaluation results. In particular, this includes documenting:

[Editor Note: Reading this, and then Appendix C, the reference to this information in Appendix C is to minimal. It should be more clearly spelled out in the appendix.]

Documentation of the outcome of auditing the representative sample has to include at least the following information for the WCAG 2.0 Succes Criteria:

Besides the minimum requirements of WCAG-EM, evaluation commissioners may require additional detail as per Step 1.d: Define Additional Requirements for Evaluation (Optional). Additions could be: Capturing the successes and failures for every Success Criterium in meeting WCAG 2.0 for each web page. For instance, if an evaluation commissioner wants to give the report to different people for repair work, this requires more detail than minimally required by WCAG-EM. Additional requirements could also include adding use-cases or asking for possible solutions for the failures that were found during the conformance evaluation. It could also be interesting for the target audience of the report to consider adding a section explaining the consequences of the success criteria for users. An optional example report is provided in Appendix C: Example Report.

Note: While such a report is required for conformance with this methodology, it is not required for any parts of this report to be made public unless an optional public accessibility conformance evaluation statement is provided per Step 5.b: Provide a Conformance Evaluation Statement (optional) in which case the minimum evaluation information to be made public is as per the requirements for WCAG 2.0 conformance claims (see Step 5.b: Provide a Conformance Evaluation Statement). In other situations, the level of confidentiality for evaluation reports is usually determined by the evaluation commissioner.

Step 5.b: Provide a Accessibility Conformance Evaluation Statement (Optional)

Methodology Requirement 5.b: Provide an accessibility conformance evaluation statement (Optional).

WCAG 2.0 conformance claims can only be made for the web pages that have been actually evaluated and identified to conform with its conformance requirements. WCAG-EM Accessibility conformance evaluation statements for entire websites can be made according to this methodology when:

As per the requirements for WCAG 2.0 conformance claims, accessibility conformance evaluation statements also have to include the following information:

  1. Date of the accessibility evaluation statement
  2. Guidelines title, version and URI: "Web Content Accessibility Guidelines 2.0 at http://www.w3.org/TR/WCAG20/"
  3. Conformance level satisfied: (Level A, AA or AAA)
  4. A concise description of the Web pages, such as a list of URIs for which the claim is made, including whether subdomains are included in the claim.
  5. A list of the web technologies relied upon.

Accessibility conformance evaluation statements can also be made when only partial conformance has been achieved according to the requirements defined in third-party content and lack of accessibility support due to language. In such cases the accessibility conformance evaluation statements have to also include information to identify the following:

  1. Website areas that do not conform to WCAG 2.0;
  2. Reason for not conforming to WCAG 2.0: "third-party content" or "lack of accessibility support for languages";

Note: It is not possible to make an accessibility conformance evaluation statement for a website that is still in development. Also, in case of a sample, the WCAG-EM Accessibility Conformance Evaluation Statement only shows the extent to which a website 'likely' conforms because it is always possible that there are errors on pages that are within the scope, but that are not selected in the sample.

[Editor Note: In this section we can also address the notion of "reasonable confidence" that is introduced in the introduction of section 3. And provide the relationship to the likely used here. It should be linked to the section about sampling. Also we want to have a closer look at the word conformance (take the word out). This is discussed in the survey 11]

Step 5.c: Provide a Performance Score (Optional)

Methodology Requirement 5.c: Provide a performance score. (Optional).

[Editor Note: This is still under discussion. In minutes of 25 August and 29 August telco and also in the 19 september telco. Input for calculating a score can be found at http://www.w3.org/TR/accessibility-metrics-report/ and other places. See minutes. Added "(performance scores are not very useful for inter-site comparisons)" and changed "websites" to "website" (singular).]

[For Discussion: There are a number of comments relevant for this section: comment 84, comment 85, comment 86, comment 87, comment 88, comment 89 and comment 90.]

Performance scores can provide more granular measure for the level of conformance of a website than the WCAG 2.0 conformance levels provide. This can be useful to monitor the progress of a website over time (performance scores are not very useful for inter-site comparisons). However, scores alone do not provide sufficient context and information to understand the actual accessibility state of a website. In some situation such scores can also be misleading as they can be subjective towards certain kinds of web content.

Performance scores as provided in this methodology require that the evaluation is carried out in conformance with this Methodology. They are intended to be supplementary information to evaluation reports and not to be used separately. The type of scoring system used has to be indicated along with the score whenever such a score is provided.

Currently the following scoring approaches are provided by this methodology:

Per Website

This score calculates a ratio over the entire website. It is a simple approach to determine overall conformance but also very sensitive towards conformance failures. For example, any failure to satisfy a Success Criterion on any web page is directly reflected as failure of the website to satisfy the respective Success Criterion.

This score is calculated as follows:

  1. List the WCAG 2.0 Success Criteria applicable to the evaluation (as per Step 1.b. Define the Conformance Target).
  2. During the evaluation mark the Success Criteria that are satisfied on all of the web pages within the selected sample (as per Step 3: Select a Representative Sample). For example, if Success Criterion 1.1.1 is satisfied on all web pages, then this Success Criterion is marked as satisfied).
  3. After the evaluation calculate the sum of Success Criteria that are satisfied consistently across the entire sample, and divide this by the sum of all applicable Success Criteria.
Per Web Page

This score calculates an average ratio over each web page. It is less sensitive to conformance failures such as occassional oversight but does not consider the relative frequency of failures. For example, a website that has relatively few videos but consistenly fails to provide captions for these videos may still have a high score even though it is profoundly disadvantaging certain users.

This score is calculated as follows:

  1. List the WCAG 2.0 Success Criteria applicable to the evaluation (as per Step 1.b. Define the Conformance Target).
  2. During the evaluation mark any Success Criteria that are satisfied for each of the web pages within the selected sample (as per Step 3: Select a Representative Sample). For example, if Success Criterion 1.1.1 is satisfied on a given web page, then this Success Criterion is marked as "satisfied" for that one web page.
  3. After the evaluation calculate the entire sum of Success Criteria that are satisfied on each web page, and divide this by the sum of all applicable Success Criteria across all pages (ie. multiplied by the number of web pages).
Per Instance

This score calculates an average ratio over all Success Criteria instances. It is least sensitive to conformance failures such as occassional oversight and considers the relative frequency of failures. However, this score is quite demanding to calculate without appropriate tools support. It is also not always easy to determine each possible instance for Success Criteria.

This score is calculated as follows:

  1. List the WCAG 2.0 Success Criteria applicable to the evaluation (as per Step 1.b. Define the Conformance Target).
  2. During the evaluation of each web page within the selected sample (as per Step 3: Select a Representative Sample) calculate the sum instances for which each Success Criterion is applicable for each web page and the number of instances for which they are satisfied (eg. the number of times in which Success Criterion 1.1.1 was applicable on a given web page and the number of times in which it was satisfied).
  3. After the evaluation calculate the entire sum of Success Criteria that were satisfied on each web page, and divide this by the sum of all applicable Success Criteria across all pages (ie. multiplied by the number of web pages).

Note: According to WCAG 2.0, Success Criteria that do not apply to the content are deemed to have been satisfied. Also, all Success Criteria are not satisfied for web pages on which any of the WCAG 2.0 conformance requirements are not satisfied.

Step 5.d: Provide Machine-Readable Reports (Optional)

Methodology Requirement 5.d: Provide machine-readable reports. (Optional).

Machine-readable reports facilitate processing the results by authoring and evaluation tools, for example to help monitor accessibility of a website over time. The Evaluation and Report Language (EARL) is a machine-readable format that was specifically designed for this purpose. It is recommended to use EARL for providing machine-readable reports. See also Understanding Metadata to learn more about uses of metadata, including machine-readable reports, such as EARL.

Contributors

Past and present active participants of the WCAG 2.0 Evaluation Methodology Task Force (Eval TF) include: Shadi Abou-Zahra; Frederick Boland; Denis Boudreau; Amy Chen; Vivienne Conway; Bim Egan; Michael Elledge; Wilco Fiers; Detlev Fischer; Elizabeth Fong; Vincent François; Alistair Garrison; Emmanuelle Gutiérrez y Restrepo; Katie Haritos-Shea; Martijn Houtepen; Peter Korn; Maureen Kraft; Aurelien Levy; David MacDonald; Kerstin Probiesch; Donald Raikes; Corominas Ramon; Roberto Scano; Samuel Sirois; Sarah J Swierenga; Eric Velleman; Konstantinos Votis; Kathleen Wahlbin; Elle Waters; Richard Warren; Léonie Watson.

Appendix B: References

QA-FRAMEWORK
Dubost K, Rosenthal L, Hazaël-Massieux D, Henderson L (eds) (2005) W3C Quality Assurance Framework: Specification Guidelines. W3C. Available at: http://www.w3.org/TR/qaframe-spec/
RFC3986
Fielding R, Gettys J, Mogul J, Frystik H, Masinter L, Berners-Lee T (eds) (1999). Hypertext Transfer Protocol - HTTP/1.1. Request for Comments: 2616. IETF. Available at: http://www.ietf.org/rfc/rfc3986.txt
RFC2616
Berners-Lee T, Fielding R, Masinter L (eds) (2005). Uniform Resource Identifier (URI): Generic Syntax. Request for Comments: 3986. IETF. Available at: http://www.ietf.org/rfc/rfc2616.txt
UWEM
Velleman E.M, Velasco C.A, Snaprud M (eds) (2007). D-WAB4 Unified Web Evaluation Methodology (UWEM 1.2 Core). Wabcluster. Available at: http://www.wabcluster.org/uwem1_2/
WCAG20
Caldwell B, Cooper M, Guarino Reid L, Vanderheiden G, eds (2008). Web Content Accessibility Guidelines 2.0. W3C. Available at: http://www.w3.org/TR/WCAG20/

Appendix C: Example Report

[For review: Rewrite to be better aligned with the guidance in Step 5: Report the Evaluation Findings. Please review. This rewrite also addresses comment 91.]

This Appendix proposes a minimal level of reporting following the goals defined in Step 1.c: Define the Conformance Target and the documentation defined in Step 5.a: Provide Documentation for Each Step:

The report includes at least the following information (if available):

The website evaluator can also include further documentation as defined per Step 1.d: Define Additional Requirements for Evaluation (Optional).

Reports could also indicate the impact of issues found and specifically identify any high impact issues that are easy to fix. This could include: Any accessibility issues that interfere with the user completing tasks, issues that affect navigation and orientation, issues that occur very frequently and issues that can cause any loss or harm to a user. A lot of the important but technical information could be put in an appendix (such as documentation of each outcome of the steps as per Step 5.a: Provide Documentation for Each Step).

Appendix D: Document Changes

Changes since the Public Working Draft of 26 February 2013 include:

A full disposition of comments of all the comments received on the 26 February Working Draft is available.