[contents]


Abstract

This document (WCAG-EM) describes a methodology for evaluating the conformance of websites to the Web Content Accessibility Guidelines (WCAG) 2.0. It provides guidance on defining the evaluation scope and parameters, exploring the website features and functionality, sampling representative web pages where it is not feasible to evaluate all web pages of a website, applying the WCAG 2.0 success criteria and conformance requirements in this setting, and documenting and reporting the evaluation findings. It complements the existing guidance for WCAG 2.0 but it does not define additional WCAG 2.0 requirements nor does it replace or supersede it in any way.

This methodology only addresses accessibility evaluation of existing websites, for example to validate conformance with reasonable confidence or to better understand the accessibility performance for improvement. It is applicable to any website, including web applications and websites for mobile devices. It is independent of any particular evaluation tool, web browser, and assistive technology, and it addresses different contexts, including self-assessment and third-party evaluation. However, ensuring and maintaining accessibility requires that accessibility is considered throughout the development process. Complementary resources from W3C/WAI on evaluation provide guidance for other situations.

This work is part of complementary W3C/WAI activities on Web Accessibility Evaluation and Testing that address different W3C/WAI guidelines and specifications. It is developed through the collaborative W3C Process with the involvement of international experts and stakeholders.

[Editor note: We decided to take all numbering out of the substeps after finalizing processing the comments. This will help locate where the comments refer to in the current document. Please refer to Comment 166.

Status of this document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

This 22 January 2013 Editors Draft of Website Accessibility Conformance Evaluation Methodology 1.0 is based on the same approach and expanding the previously published W3C First Public Working Draft of 20 September 2012. This Editors Draft addresses the following comments received, to seek approval for publication as an updated Working Draft:

This document is intended to be published and maintained as an informative W3C Working Group Note after review and refinement. The WCAG 2.0 Evaluation Methodology Task Force (Eval TF) invites discussion and feedback on this document by developers, evaluators, researchers, and others with interest in web accessibility evaluation. Eval TF is particularly looking for feedback on the sections about 2.1 Scope of Applicabilityand 3.5.3 Step 5.c: Provide Performance Scores (Optional).

Please send comments on this Editors Draft of Website Accessibility Conformance Evaluation Methodology 1.0 to public-wai-evaltf@w3.org (publicly visible mailing list archive). These comments will be considered in the internal discussions of Eval TF.

Publication as Editors Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

This document has been produced by the WCAG 2.0 Evaluation Methodology Task Force (Eval TF, a joint task force of the Web Content Accessibility Guidelines Working Group (WCAG WG) and Evaluation and Repair Tools Working Group (ERT WG), as part of the Web Accessibility Initiative (WAI) Technical Activity.

This document was produced by two groups operating under the 5 February 2004 W3C Patent Policy. The groups do not expect this document to become a W3C Recommendation. W3C maintains a public list of any patent disclosures for WCAG WG and public list of any patent disclosures for ERT WG made in connection with the deliverables of each group; these pages also include instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy.


Table of Contents

Overview

  1. Introduction
  2. Using this Methodology
  3. Conformance Evaluation Procedure
  4. Considerations for Particular Situations
  5. Application of this Methodology

List of Sections

  1. Introduction
  2. Using this Methodology
  3. Conformance Evaluation Procedure
  4. Considerations for Particular Situations
  5. Application of this Methodology

List of Appendices


1. Introduction

[For review: Changed the first sentence with redundant list of target audience that is covered in section 1.2 Target Audience as discussed in the 17 January 2013 Telco. Made a small adjustment to the proposed text in the Telco, please review (this also covers for comment 1)]

This methodology (WCAG-EM) describes how to evaluate the conformance of websites to the Web Content Accessibility Guidelines (WCAG) 2.0. For example, such evaluation may be carried out as part of releasing or purchasing a website to validate its conformance with reasonable confidence, or it may be to better understand and monitor the level of accessibility provided on a website for improvement. The methodology described in this document provides one possible approach to address this particular aspect of web accessibility evaluation. It does not specifically explain evaluation in other situations nor does it provide general guidance on evaluation with WCAG 2.0. It extends the existing guidance for WCAG 2.0 but it does not define additional WCAG 2.0 requirements nor does it replace or supersede it in any way.

1.1 Scope of this Document

The methodology described in this document (WCAG-EM) provides guidance on defining the evaluation scope and parameters, exploring the website features and functionality, sampling representative web pages where it is not feasible to evaluate all web pages of a website, verifying the application of the WCAG 2.0 success criteria and conformance requirements in this setting, and documenting and reporting the evaluation findings. It is applicable to all websites, including web applications, websites intended to be used with mobile devices, and other types of websites. It is independent of the size of the website, the web technologies used to create the website, and of any particular web accessibility evaluation tools, web browsers, assistive technologies, and other software tools.

[For Review: Rewrite according to discussion in DoC and during Telco of 6 december 2012 and discussion in Telco of 17 January 2013 (main purpose but encourage evaluation throughout development). Also changed sentence in Abstract to existing websites. Please review the text below and check if this covers Comment 4, Comment 5, Comment 7, comments 20, 23, 24, 25, 26, 27 and 28 and Comment 155. Input from EOWG still necessary to replace "@@@". As a result of our discussion in the 6 December Telco 2012, a note was added at the bottom of Step 5.b stating that it is not possible to make an accessibility evaluation statement based on the evaluation of a website in development. Please note that this rewrite also covers Comment 30.]

The main purpose of this Methodology is to evaluate an existing website's conformance to WCAG 2.0. This can also be a website that is still in development. When developing a website, accessibility should be considered early and throughout the design and development process. We specifically want to encourage evaluation throughout development. Related information outside the scope of this document is provided in @@@(New link from EOWG to be provided here) and in Evaluating Websites for Accessibility, W3C WAI Web Accessibility Evaluation and Testing Activities, and WAI Guidelines and Techniques.

Also, WCAG 2.0 conformance requirements apply to individual web pages. However, on most websites it is not feasible to evaluate every web page with reasonable effort. To ensure conformance of a website to WCAG 2.0 it is necessary to ensure that this is considered throughout the development process. Following this methodology helps assess the level of conformance with reasonable confidence where it is not feasible to evaluate every web page.

1.2 Target Audience

[Editor note: Todo: Work on purpose versus target audience.]

The primary target audience for this methodology are individuals and organizations, wanting to evaluate the conformance of existing websites to WCAG 2.0. This includes but is not limited to:

Other audiences for whom this methodology is also relevant include:

Users of the methodology are assumed to have expertise in WCAG 2.0, accessible web design, assistive technologies, and of how people with different disabilities use the Web; read more in section 2.2 Required Expertise.

1.3 Background Reading

It is assumed that the reader of this document is familiar with the following related resources from W3C/WAI:

Evaluating Websites for Accessibility

A multi-page resource suite that outlines different approaches for evaluating websites for accessibility. The following resources are particularly important in this context:

Web Content Accessibility Guidelines (WCAG) 2.0

Internationally recognized standard explaining how to make web content more accessible to people with disabilities. The following resources are particularly important in this context:

Accessible Web Design

The following documents introduce the essential components of web accessibility, inclusive design for people with disabilities and people with aging-related impairments, and overlap with design for mobile devices. The documents help managers, designers, developers, policy makers, researchers, and others optimize their efforts in these overlapping areas:

1.4 Terms and Definitions

For the purposes of this document, the following terms and definitions apply:

Complete Processes
From WCAG 2.0 Conformance Requirement for Complete Processes:
When a web page is one of a series of web pages presenting a process (i.e., a sequence of steps that need to be completed in order to accomplish an activity), all web pages in the process conform at the specified level or better. (Conformance is not possible at a particular level if any page in the process does not conform at that level or better.)
Common Web Pages
Web pages that are relevant to the entire website. This includes the homepage, login page, and other entry pages, and, where applicable, the sitemap, contacts page, site help, legal information, and similar web pages that are typically linked from all web pages (usually from the header, footer, or navigation menu of a web page).
Evaluator
The person, team of people, organization, in-house department, or other entity responsible for carrying out the evaluation.
Evaluation Commissioner
The person, team of people, organization, in-house department, or other entity that commissioned the evaluation.
Note: In many cases the evaluation commissioner may be the website owner or website developer, in other cases it may be another entity such as a procurer or an accessibility monitoring survey.
Common Functionality
Functionality of a website that, if removed, fundamentally changes the use or purpose of the website for users. This includes information and tasks that users of a website carry out to perform this functionality.
Note: Examples of functionality include "selecting and purchasing a product from the shop area of the website", "filling and submitting the form provided on the website", and "registering for an account on the website". Please note that other functionality is not out of scope. The sections in which "common functionality" occurs is intended to help identify critical pages but is not intended to limit evaluation to these pages alone.
Relied Upon (Technologies)
From WCAG 2.0 definition for "relied upon":
The content would not conform if that technology is turned off or is not supported
Template
From ATAG 2.0 definition for "templates":
Content patterns that are filled in by authors or the authoring tool to produce content for end users (e.g., document templates, content management templates, presentation themes). Often templates will pre-specify at least some authoring decisions.
[Editor Note: Given that ATAG 2.0 is still a Working Draft, this definition might change in future drafts of this document. This is from the 10 April 2012 Working Draft]
Website
A coherent collection of one or more related web pages that together provide common use or functionality. It includes static web pages, dynamically generated web pages, and web applications.
Note: The focus of this methodology is on full, self-enclosed websites. Websites may be composed of smaller sub-sites, each of which can be considered to be an individual website. For example, a website may include an online shop, an area for each department within the organization, a blog area, and other areas that may each be considered to be a website.
Website Developer
The person, team of people, organization, in-house department, or other entity that is involved in the website development process including but not limited to content authors, designers, programmers, quality assurance testers, and project managers.
Website Owner
The person, team of people, organization, in-house department, or other entity that is responsible for the website.
Web page
From WCAG 2.0 definition for "Web Page":
A non-embedded resource obtained from a single URI using HTTP plus any other resources that are used in the rendering or intended to be rendered together with it by a user agent
Note: Web pages may include multimedia content, interactive components, and complex applications.

2. Using This Methodology

The methodology defined by this document is used for comprehensively assessing the conformance of websites to WCAG 2.0. A more cursory approach for exploring the accessibility of a website is described in Preliminary Review of Websites for Accessibility. In many cases it is advisable to carry out such preliminary reviews before applying this methodology, for example to identify obvious errors and to develop a rough understanding of the overall performance of the website.

2.1 Scope of Applicability

The methodology defined by this document applies to full, self-enclosed websites. This includes websites of organizations and entities, persons, events, products, and services. It also includes web applications and mobile websites. Examples include:

[Discussion note: Do we need to add more to this list? examples from the comments: "The extranet for the online tax payment system of Public Administration X" - "The tablet version of the collaborative web game..." - "Version 3.1 of the Word processing and File sharing tool...". This relates to Comment 22. Please find the discussion in evaltfq7.]

A website may include areas with smaller collections of related web pages such as an online shop, an area for each department within the organization, a blog area, and other parts. In some situations such areas can be considered to be a full, self-enclosed website each. This methodology can be applied to such individual sub-sites (a website within another website) and to the main website in its entirety. However, this methodology may not be applied to a website excluding any of its parts. Excluding parts of the website from the scope of evaluation would likely conflict with the WCAG 2.0 conformance requirements full pages and complete processes, or significantly distort the evaluation results.

Example of a Website:
Diagram of a University Website explained in the following paragraph.

In the diagram above, the university website is made of several sub-sites: "Information for Students", "Information for Lecturers", a "Courseware Application", and a "Library Application". The "Courseware Application" includes "Physics Courses", "Medicine Courses", and "History Courses" that are aggregated into the application. Also, the university website has individual web pages such as legal notices, sitemap, and others that are not part of any particular sub-site.

In the example above, none of the depicted parts may be excluded from the scope of evaluation in the context of this methodology, if it is to be applied to the university website. This includes any aggregated and embedded content such as online maps for the university campus and forms for credit card transactions, including when such parts originate from third-party sources.

Although the purpose of this document is to evaluate an existing website, we encourage evaluation throughout the development process.

Note: WCAG 2.0 defines "Statement of Partial Conformance" for individual web pages that are known not to conform with WCAG 2.0 due to third-party content and/or languages lacking accessibility support. Such web pages may not be excluded from the scope of evaluation according to this methodology. In some cases this means that the website as a whole does not conform with WCAG 2.0 due to partially conforming web pages. Section 3.5 Step 5: Report the Evaluation Findings provides more guidance on reporting evaluation results and making accessibility statements for entire websites.

2.1.1 Particular Types of Websites

This methodology is applicable to all websites, including web applications, websites intended to be used with mobile devices, and other types of websites. Some particular types of websites include:

[Note For Discussion: Do we want to add more specific types of websites like "Responsive websites..". (Comment 33). Please find the discussion in the evaltfq7]

Small Websites
For websites with few web pages the sampling procedure defined in 3.3 Step 3: Select a Representative Sample will likely result in selecting most or all of the web pages from the target website. In cases where all web pages can be evaluated, the sampling procedure can be skipped and the selected sample is considered to be the entire website in the remaining steps.
Web Applications

[For review: Please review rewrite of the text in this definition (use the diff version for support). The changes are a result of our discussion in the 17 January Telco and of comments in the DoC. Changed the use of "State". As commented in Comment 31 and in the questionnaire6. Changed "state" to "page state or generated page" and made correct sentence of first paragraph. Added "as a candidate for sampling" at the end of the sentence to provide a possibility to avoid extremely large samples as a result of the use of the word "each" (17 January telco) . Please review.

In some cases only one or few web pages may dynamically generate all the content with the functionality that normally an entire set of web pages would provide. Such web applications are usually not separable into sub-sites and are considered as a single website or part of a website. Each page state or generated page (individual representation of content and functionality) in which such a web application is considered to be a web page for the purpose of this document and should be included as a candidate for sampling.
Note: These individual page states or generated pages of a web application may not have unique URIs and may need to be identified by describing the settings, input, and actions required to generate them. Due to the many possibilities to generate these page states or generated pages it may not be feasible to exhaustively identify every possible instance of web pages as per 3.2.1 Step 2.a: Identify Common Web Pages of the Website, functionality as per 3.2.2 Step 2.b: Identify Common Functionality of the Website, and variety of web pages as per 3.2.3 Step 2.c: Identify the Variety of Web Page Types, so that evaluating such web applications may be more challenging than for other types of websites.
Website with Separable Areas
In some cases websites may have clearly separable areas, such as a password-restricted area of a website (extranet) that is not part of using the public area (log-in is not required to complete a function or process). Such areas can be considered as individual websites rather than sub-sites for the purpose of this document.
Website in Multiple Versions
Some websites are available in versions that are independent of each other. For example, mobile versions of a website and versions of a website in different languages, where using one version does not require or depend on using another version of the website (usually such website versions have a different set of URIs each). Such website versions can be considered as individual websites rather than sub-sites for the purpose of this document.

[For Review: This also closes the following comments where 'no changes' where proposed resolution: Comment 35, Comment 36, . Please review.]

2.2 Required Expertise

Users of the methodology defined by this document are assumed to be knowledgeable of WCAG 2.0, accessible web design, assistive technologies, and of how people with different disabilities use the Web. This includes understanding of the relevant web technologies, barriers that people with disabilities experience, assistive technologies and approaches that people with disabilities use, and evaluation techniques and tools to identify potential barriers for people with disabilities. In particular, it is assumed that users of this methodology are deeply familiar with the resources listed in section 1.3 Background Reading.

2.3 Evaluation Tools (Optional)

The methodology defined by this document is independent of any particular web accessibility evaluation tool, web browsers, and other software tools. However, web accessibility evaluation tools can significantly assist an evaluator during the evaluation process and contribute to more effective evaluations. For example, some web accessibility evaluation tools can scan entire websites to help identify relevant pages for manual evaluation, and other tools can assist manual evaluation in a variety of ways. Specific guidance is provided in Selecting Web Accessibility Evaluation Tools.

Note: Web accessibility evaluation tools can only automatically check a limited set of accessibility requirements that are automatable. Conformance evaluation requires manual testing and judgment by experienced evaluators.

2.4 Review Teams (Optional)

The methodology defined by this document can be carried out by an individual evaluator with the skills described in section 2.2 Required Expertise. However, using the combined expertise of review teams provides better coverage for the required skills and helps identify accessibility barriers more effectively. While not required, it is strongly recommended to employ review teams for conformance evaluation of websites. Specific guidance is provided in Using Combined Expertise to Evaluate Web Accessibility.

2.5 Involving Users (Optional)

The methodology defined by this document can be carried out by an individual evaluator with the skills described in section 2.2 Required Expertise. However, involving people with disabilities and people with aging-related impairments helps identify additional accessibility barriers that are not easily discovered by the evaluator alone. While not required, it is strongly recommended to involve real people covering a wide range of abilities during the evaluation process. Specific guidance is provided in Involving Users in Web Accessibility Evaluation.

3. Conformance Evaluation Procedure

This section defines the stages and activities of the evaluation procedure. The stages are not necessarily sequential. Also the exact sequence of the activities carried out during the evaluation process depends on the type of website, the purpose of the evaluation, and the process used by the evaluator. Some of the activities overlap or may be carried out in parallel. The following diagram illustrates the iterations between the stages defined in this section:

Diagram about the iterations between the steps in this methodology. Explanation in the following paragraph.

The diagram depicts each of the five steps defined in this section with arrows back and forth between each of two consecutive steps: 1. Define the Evaluation Scope; 2. Explore the Target Website; 3. Select a Representative Sample; 4. Audit the Selected Sample and 5. Report the Evaluation Findings.

Note: See also section 4. Considerations for Particular Situations that may influence how an evaluation procedure is carried out.

3.1 Step 1: Define the Evaluation Scope

Methodology Requirement 1: Define the evaluation scope according to Methodology Requirement 1.a, Methodology Requirement 1.b, Methodology Requirement 1.c, Methodology Requirement 1.d and Methodology Requirement 1.e , Methodology Requirement 1.d, and optionally Methodology Requirement 1.e.

During this step the overall scope of the evaluation is defined. It is a fundamental step that affects the subsequent steps in the evaluation procedure and is ideally carried out together with the evaluation commissioner (who may or may not be the website owner) to ensure common expectations about the scope of the evaluation. Some exploration throughout this stage may be necessary to understand the full scope of the website and the required evaluation (see 3.2: Explore the Target Website).

Note: Involvement of the website owner and/or website developer (in addition to the evaluation commissioner) is not required but often helps identify use cases, functionality, and other aspects of the implementation that makes the evaluation procedure more efficient and effective. However, the evaluator is responsible for an objective and thorough assessment.

3.1.1 Step 1.a: Define the Scope of the Website

Methodology Requirement 1.a: Define the scope of the website.

During this step the exact scope of the website (the web pages for which the evaluation will apply) is defined and documented. This scope definition may not contradict the terms established in section 2.1 Scope of Applicability. It is essential to carefully document particular aspects such as any use of content and services developed externally, different languages, and parts of the website that may not be easily identifiable as such (for example an online shop that is not integrated into the website but considered to be part of it).

The outcome of this step is an unambiguous definition for the target website, so that for each web page it can be determined whether it is within the scope of evaluation or not. Where possible, it is recommended to use formalizations such as regular expressions or listings of URIs that define the web pages that are within the scope of evaluation (part of the target website).

3.1.2 Step 1.b: Define the Goal of the Evaluation

Methodology Requirement 1.b: Define the goal of the evaluation.

[For review: Changed the Methodology Requirement text from "Define the goal of the evaluation as "Basic Report", "Detailed Report", or "In-Depth Analysis Report" to "Define the goal of the evaluation". This partly covers Comment 47. Please review.]

[For review: Detailed report looks at 'each page' for all SC. In-Depth Analysis Report looks at SC and gives examples of pages where it is not conformant. This is now in line with rest of document. Please review text below as general introduction to the goals that will be described in more detail later in this document. This also covers Comment 46. and is discussed in evaltfq7 survey.]

Defining the goal of the evaluation is particularly relevant to the auditing stage defined in 3.4 Step 4: Audit the Selected Sample and the reporting stage defined in 3.5 Step 5: Report the Evaluation Findings. Some of the evaluation goals include:

Basic Report
Only identifies whether a website conforms or not without providing additional information. This type of evaluation is typically carried out when the website is assumed to conform, for example to verify an existing conformance claim, and for large-scale evaluations with less resources to explore the details of individual websites.
Detailed Report
Identifies whether a website conforms or not, and provides further information about the conformance of each evaluated web page. This type of evaluation is particularly useful for instructing web developers and for acquiring statistics for monitoring progress over time.
In-Depth Analysis Report
Identifies whether a website conforms or not, and analyzes any identified issues in detail. This includes descriptions of the errors and suggestions for possible repair options. This type of evaluation is particularly useful for organizations that want to improve the accessibility of their website and need a detailed analysis.

[For review: The proposed changes to this section also close the following comments that have "no change" as proposed resolution: Comment 42, Comment 43, Comment 44, Comment 45 and Comment 48]

3.1.3 Step 1.c: Define the Conformance Target

Methodology Requirement 1.c: Define the target WCAG 2.0 conformance level as "A", "AA", or "AAA".

Part of initiating the evaluation process is to define the target WCAG 2.0 conformance level ("A", "AA", or "AAA"). In many situations WCAG 2.0 Level AA is the generally accepted level of accessibility.

Note: It is often useful to expand the scope of the evaluation beyond the planned conformance target to get a more complete picture of the accessibility performance of the website. An evaluator could include a selection of success criteria from higher conformance levels.

[For review: Added note at the end of the section to emphasize that it is good to look at more than just the selected conformance target. Comments indicate that we should encourage this. Naturally it is always free for anyone to do more, but it will depend on budget, time etc.. Therefor the text now says it is often usefull, but not that it is required. Text was copied from section 4. Please review.]

3.1.4 Step 1.d: Define the Context of Website Use

[For review: Deleted this step as it is redundant and was leading to more confusion on possible exclusion of groups of people. See the previous text in the September Public Working Draft. The context as described could indicate possible exclusions for target groups: "the website is only for blind people" or "our intranet is only for people using Assistive Technology X in combination with user agent Y". User tasks are more clearly included in the next step 2 in Common Functionality. The use of AT is already in the step on auditing the website. This adresses Comment 51, Comment 52, Comment 53, Comment 55, Comment 56, Comment 57, Comment 58, Comment 59, Comment 60 and Comment 61. Please review.]

Methodology Requirement 1.d: Define the context of website use and the (minimum) set of web browsers and assistive technology for evaluation.

It is often not feasible for websites to support accessibility on every combination of web browser, assistive technology, and operating system that they run on, nor is it possible to test with every such combination of tools. It is necessary to determine the context in which a website is used and the minimum set of web browsers and assistive technology that must be supported by the website.

For example, a website may be public, only available within a closed network (intranet), or limited to authenticated users (extranet). Users of intranet and extranet websites may be known to have a certain level of training on using the website and may be using pre-determined web browsers and assistive technology. This is usually not the case for public websites.

This definition of context and tools needs to meet the terms defined in WCAG 2.0 Level of Assistive Technology Support Needed for "Accessibility Support" and needs to be supported throughout the website. For example, if one part of a website is accessible using one set of tools that is different from a set of tools that is needed to access another part of the same website, then the website is effectively not accessible for some users. Accessibility support needs to be uniform throughout a single website.

Note: It is not possible to single out or exclude individual groups of users, such as "people with visual impairments" in the target audience. All WCAG 2.0 Success Criteria must be considered throughout the evaluation.

3.1.5 Step 1.e: Define the Techniques to be Used (Optional)

[For review: As a result of deleting previous step, this step has to be renamed to 3.1.4 Step 1.d: Define the Techniques to be Used. (Optional).

Methodology Requirement 1.e: Specify the WCAG 2.0 techniques that will be used to evaluate WCAG 2.0 Success Criteria. (Optional).

Techniques in the context of WCAG 2.0 are informative and not required for satisfying the WCAG 2.0 conformance requirements; WCAG 2.0 Success Criteria are written as testable statements. Techniques provide documented ways of meeting WCAG 2.0 Success Criteria and failures in meeting them. More information on techniques is provided in WCAG 2.0 Layers of Guidance.

W3C/WAI provides a set of publicly documented Techniques for WCAG 2.0. However, it is not necessary to use these particular techniques. In fact, in some situations, such as in a closed network, it may be necessary to use techniques that are specifically developed for such situations. Individuals and organizations developing techniques must employ methods for establishing the technique's ability to meet the WCAG 2.0 Success Criteria.

It is good practice to specify the sets or sources of techniques that are intended to be used during the evaluation at this stage already to ensure consistent expectation between the evaluator and the evaluation commissioner. This definition is typically refined in later stages of the evaluation process, for example during the website exploration and evaluation stages. Section 3.4.2 Step 4.b: Use WCAG 2.0 Techniques Where Possible (Optional) provides more guidance on using WCAG 2.0 techniques during evaluation.

3.2 Step 2: Explore the Target Website

Methodology Requirement 2: Explore the website to be evaluated according to Methodology Requirement 2.a, Methodology Requirement 2.b, Methodology Requirement 2.c, and Methodology Requirement 2.d.

During this step the evaluator explores the target website to be evaluated, to develop a better understanding of the website use, purpose, and functionality.

Carrying out initial cursory checks during this stage helps identify web pages that are relevant for more detailed evaluation later on. For example, an evaluator may identify web pages that seem to be lacking color contrast, consistent navigation, or document structures (headings, links, etc.) with simple checks while studying the website, and note them down for more detailed evaluation later on.

To carry out this step it is critical that the evaluator has access to all the relevant parts of the website. For example, it may be necessary to create accounts or otherwise provide access to restricted areas of a website that are part of the evaluation.

Note: Involvement of the website owner and/or website developer can be helpful to help identify common web pages, functionality, technologies, and other aspects of the implementation that makes the evaluation procedure more efficient and effective. However, the evaluator is responsible for an objective and thorough assessment.

[For review: This section includes comments that have "no change" as proposed solution: Comment 67. Please review]

3.2.1 Step 2.a: Identify Common Web Pages of the Website

Methodology Requirement 2.a: Identify the common web pages of the website.

[For review: Added part of the definition in the text below. In the next steps, there is a pointer back to this requirement, but then this requirement points back to the definitions list. This is easier to read. This also covers Comment 68, Comment 69. Please review.]

During this step the common web pages of the website are identified and documented. Common web pages are pages that are relevant to the entire website. They include the homepage, login page, and other entry pages, and, where applicable, the sitemap, contacts page, site help, legal information, and similar web pages that are typically linked from all web pages (usually from the header, footer, or navigation menu of a web page).The outcome of this step is a list of all common web pages. These will be part of the sample to be selected through the steps defined in 3.3 Step 3: Select a Representative Sample.

This step also helps understand key aspects of the website, such as the navigation and overall structure of the website.

3.2.2 Step 2.b: Identify Common Functionality of the Website

Methodology Requirement 2.b: Identify the common functionality of the website.

[For review: Added part of the definition. In the next steps, there is a pointer back to this requirement, but then this requirement points back to the definitions list. This is easier to read. Please review.]

During this step the common functionality of the website are identified and documented, to provide a comprehensive basis for the sample to be selected through the steps defined in 3.3 Step 3: Select a Representative Sample. Common functionality is all functionality of a website that, if removed, fundamentally changes the use or purpose of the website for users. This includes information and tasks that users of a website carry out to perform this functionality.

The outcome of this step is a list of user tasks such as "selecting and purchasing a product from the shop area of the website", "filling and submitting the form provided on the website", and "registering for an account on the website".

3.2.3 Step 2.c: Identify the Variety of Web Page Types

Methodology Requirement 2.c: Identify the variety of web page types.

Web pages with varying styles, layouts, structures, and functionality often have different implementations of accessibility features. They are also often generated by different templates and authored by different people. Web pages may also appear differently, behave differently, and contain different content depending on the user. The outcome of this step is a list of the different types of web pages on the website, to provide a comprehensive basis for the sample to be selected through the steps defined in 3.3 Step 3: Select a Representative Sample. Different web page types include:

Note: This step is intended to identify groups of web page instances, including in web applications.

3.2.4 Step 2.d: Identify Web Technologies Relied Upon

Methodology Requirement 2.d: Identify the technologies relied upon to provide the website.

During this step the web technologies relied upon to provide the website are identified and documented, to provide a basis for the sample selection through the steps defined in 3.3 Step 3: Select a Representative Sample. This includes base web technologies such as HTML and CSS, HTML5, auxiliary web technologies such as Flash, Java, JavaScript, PDF, Silverlight and WAI-ARIA, as well as advanced web technologies such as SMIL and SVG. The outcome of this step is a list of web technologies that are relied upon.

[Editor note: Does this include features like in html5? and do we have to say more about accessibility support here?]

Only the web technologies that are relied upon to provide the website need to be identified for evaluation. This relates closely to the WCAG 2.0 concepts of conforming alternate versions and non-interference.

Note: Where possible, it is often also useful to identify the libraries and components used to create the website, such as Dojo, JQuery, and others. Particularly for web applications, much of the accessibility support is built into these libraries and components, and evaluation can become more effective and efficient when these are identified.

3.3 Step 3: Select a Representative Sample

[For review: Added section on random sampling at the end. Also worked on Comment 78. Please review]

[Editor note: Rewrite or explain "with reasonable confidence". This was introduced during the telco's and agreed by the group. We should then provide explanation of what "reasonable confidence" is and how evaluators can achieve this. See also Comment 83.]

Methodology Requirement 3: Select a representative sample of web pages from the website according to Methodology Requirement 3.a, Methodology Requirement 3.b, Methodology Requirement 3.c, Methodology Requirement 3.d, Methodology Requirement 3.e and Methodology Requirement 3.f.

While ideally every web page of a website is evaluated, usually this is not possible on most websites. In cases where all web pages can be evaluated, this sampling procedure can be skipped and the selected sample is considered to be the entire website in the remaining steps.

Exploration of the target website in 3.2 Step 2: Explore the Target Website (within the scope set in 3.1 Step 1: Define the Evaluation Scope) provides sufficient understanding of the website to facilitate selection of a sample of web pages that is representative with reasonable confidence of the entire target website. Depending on the size and complexity of the website, the size of this sample will vary. For example, a website with few types of web pages that are all generated from a confined set of templates, such as a data repository, will likely require a smaller sample than a website with many types of web pages that are authored using different mechanisms. Also a web application could have a limited number of web pages that dynamically generate content with varying types of appearance, behavior, and information that each need to be sampled.

The web pages identified through the exploration will typically relate to more than one design aspect. For example, web pages with particular functionality such as scripting, multimedia, and forms will typically also use related technologies such as Flash, JavaScript, PDF, Silverlight, and WAI-ARIA, and in many cases these web pages may have different design to others. Careful selection of web pages can significantly reduce the required sample size while maintaining appropriate representation of the entire website.

Note: In this step the evaluator identifies the comprehensive sample of web pages for evaluation. However, many web pages will have repetitive content, such as the header, navigation, and other common components that may not need to be re-evaluated on each occurrence. Guidance on evaluating the sample identified in this step is provided in 3.4 Step 4: Audit the Selected Sample.

3.3.1 Step 3.a: Include Common Web Pages of the Website

Methodology Requirement 3.a: Include all common web pages into the selected sample of web pages.

All common web pages, including the common states of these web pages for web applications, must be part of the selected sample. These web pages are identified in step 3.2.1 Step 2.a: Identify Common Web Pages of the Website.

3.3.2 Step 3.b: Include Exemplar Instances of Web Pages

Methodology Requirement 3.b: Include (where applicable and available) of each (1) common functionality, (2) distinct types of web pages, and (3) web technologies into the selected sample of web pages.

[For review: Changed the Methodology Requirement numbered above with (2) to: "distinct types of web pages" to be in line with the text below and the earlier steps (was: "content, design, and functionality"). Addresses Comment 86 and Comment 88. Please review.
For review: Changed first paragraph from "select at least two distinct web pages" to "select at least one" because the sample would otherwise be to large. By using at least, evaluators are free to sample more. Please review.]

From the variety and types of web pages identified in 3.2 Step 2: Explore the Target Website (within the scope of the evaluation as defined per 3.1 Step 1: Define the Evaluation Scope), select at least one distinct web page for all of the following features (where applicable and available):

Note: A selected web page could have any number of these features. For example, a selected web page could be used to represent the use of forms and scripting at the same time. The important aspect is to select at least one distinct web page for each relevant feature identified on the website, though more web pages may be necessary depending on the complexity of the website.

3.3.3 Step 3.c: Include other Relevant Web Pages

Methodology Requirement 3.c: Include other web pages relevant for people with disabilities and accessibility into the selected sample of web pages.

Websites frequently include web pages that are relevant for people with disabilities and accessibility but do not explicitly match the criteria described in the previous sections. These include:

[For review: This addresses comments with "no change": Comment 90. Please review]

3.3.4 Step 3.d: Include Complete Processes in the Sample

Methodology Requirement 3.d: Include all web pages that are part of a complete process.

The selected sample must include all web pages that belong to a series of web pages presenting a complete process. Also, no web page in the selected sample may be part of a process without all other web pages that are part of that process to be also included into the selected sample.

3.3.5 Step 3.e: Include a Random Sample

Methodology Requirement 3.e: Include a random sample.

[For review: Added requirement for random sample. For the moment the random sample is 5 pages.

From the survey we did last year about sampling:
41,4 percent used structured sampling only. Both structured and random sampling was used by 51,7 percent. 37 percent samples between 6 and 10 pages. Also 37 percent between 11 and 25 pages. 21 percent samples between 26 and 50 pages. Further answers include "we use a progressive scale depending on the site size and complexit", "we add the results of an automatic evaluation for a bigger sample", "we sample between 10 and 25 percent".

If we use these figures and add up the non-random sample to about 15, then adding 5 random pages would cover more than 70 percent. Although some organisations intentionally use smaller samples. This will be further discussed on the list, but it seems best to do a trial using the same website with the minimum number of pages and with a larger sample to see if they generate different outcomes in the larger sample. Please review.]

Include at least 5 random web pages (if available) from the scope of the website into the sample. The scope is defined in section 3.1 Step 1: Define the Evaluation Scope.

[For review: This section in combination with the other sections in Step 3 covers a number of comments: Comment 76, Comment 79, Comment xx and Comment xx. Following the definition of web page, this includes web pages from web applications (covering comment 80). Please review]

3.3.6 Step 3.f: Filter the sample to eliminate redundancy

Methodology Requirement 3.f: Filter the sample to eliminate redundancy.

[For review: Added requirement to filter the sample to eliminate unnecessary redundancy. View Comment 84 and Comment 87. Please review.]

Once a complete sample has been selected according to Methodology Requirement 3.a, Methodology Requirement 3.b, Methodology Requirement 3.c, Methodology Requirement 3.d and Methodology Requirement 3.e, the evaluator may identify web pages that are identical with other web pages in the sample. Replace redundant identical pages in the sample with new pages from the same Methodology Requirement.

[Editor note: We will discuss this addition in the Survey. This section does not cover repetitive elements of pages as they are identified in the introduction of step 4: Audit the Selected Sample.]

3.4 Step 4: Audit the Selected Sample

Methodology Requirement 4: Audit the selected sample of web pages according to Methodology Requirement 4.a, Methodology Requirement 4.b, and optionally, Methodology Requirement 4.c, Methodology Requirement 4.d and Methodology Requirement 4.e.

WCAG 2.0 defines five conformance requirements that need to be met for each web page in the sample selected per 3.3 Step 3: Select a Representative Sample. This includes checking whether each WCAG 2.0 Success Criterion in the target conformance level (per 3.1.3 Step 1.c: Define the Conformance Target) has been met or not met for each of these web pages.

Note: Many web pages will have repetitive content, such as the header, navigation, and other common components that may not need to be re-evaluated on each occurrence. Depending on the level of detail for reporting defined by 3.1.2 Step 1.b: Define the Goal of the Evaluation, an evaluator may not need to continue to identify successes and failures in meeting the conformance target for these repetitive elements on every web page. Section 3.5 Step 5: Report the Evaluation Findings provides more guidance on reporting.

3.4.1 Step 4.a: Check for the Broadest Variety of Use Cases

Methodology Requirement 4.a: Check each web page in the sample selected per 3.3 Step 3: Select a Representative Sample for meeting each of the WCAG 2.0 Success Criteria in the target conformance level (per 3.1.3 Step 1.c: Define the Conformance Target) and each of the WCAG 2.0 conformance requirements.

For each web page in the sample selected per 3.3 Step 3: Select a Representative Sample, check whether each of the WCAG 2.0 Success Criteria in the target conformance level (per 3.1.3 Step 1.c: Define the Conformance Target) and each of the WCAG 2.0 conformance requirements have been met. This includes that all accessibility features, especially common functionality defined per 3.2.2 Step 2.b: Identify Common Functionality of the Website, are usable for the audiences and with the tools defined per 3.1.4 Step 4.d: Define the Context of Website Us.

Note: According to WCAG 2.0, Success Criteria to which there is no matching content are deemed to have been satisfied. An outcome such as 'Not Applicable' may be used to denote the particular situation where Success Criteria were satisfied because no relevant content was applicable.

To carry out an evaluation effectively, it is often useful to construct and apply personas, use cases, and scenarios of users with a variety of abilities and using different web browsing techniques, including assistive technology and adaptive strategies. It is critical to consider the broadest possible spectrum of use cases to help identify issues that may occur to different audiences. It is strongly recommended to also involve real users during this process, to help identify issues that may not be easily identified through expert testing alone. See Involving Users in Evaluating Web Accessibility for more guidance.

This assessment also needs to be complemented with focused testing of particular situations including:

Note: Templates are often used to create many web pages, sometimes entire parts of a website. While evaluating templates is optional in this methodology, in some contexts it can be helpful to check templates on their own. Evaluating templates may identify potential issues that may not be easily identified through evaluating individual instances of web pages. However, issues identified in templates alone do not necessarily imply that these issues occur on the website and need to be validated on individual instances of web pages. Also, identifying no issues in templates does not necessarily imply that no issues occur on instances of web pages.

Note: It is not possible to single out or exclude individual groups of users, such as "people with visual impairments" in the target audience. All WCAG 2.0 Success Criteria must be considered throughout the evaluation.

3.4.2 Step 4.b: Assess Accessibility Support for Features

Methodology Requirement 4.b: Check if accessibility features provided on the website are accessibility supported.

[For review: Changed text of Methodology Requirement from "Check if accessibility features provided on the website are accessibility supported by the tools defined in Step 1.d: Define the Context of Website Use" to "Check if accessibility features provided on the website are accessibility supported" (if Step 1.d is deleted as proposed).]

[Editor note: discussion on: "accessibility support needs to be uniform throughout a single website" covering Comment 54 (was in Step 1.d first, but that Step was deleted), in the evaltfq7 survey.]

To ensure that the accessibility features such as text-alternatives, captions, keyboard access are actually usable in practice, each of these accessibility features must be accessibility supported by the tools defined in Step 1.d: Define the Context of Website Use.The WCAG 2.0 Level of Assistive Technology Support Needed for "Accessibility Support" needs to be supported throughout the website. For example, if one part of a website is accessible using one set of tools that is different from a set of tools that is needed to access another part of the same website, then the website is effectively not accessible for some users. Accessibility support needs to be uniform throughout a single website.

[For review: Added in text below: "Please mark that "sufficient" techniques are not automatically always accessibility supported." This addresses Comment 62.]

In some situations WCAG 2.0 techniques and repositories on accessibility support provide insights on the level of accessibility support for accessibility features in particular combinations of web technologies and tools. Please mark that "sufficient" techniques are not automatically always accessibility supported and that this is a requirement for conformance. The evaluator is responsible for the accuracy of the assessment of accessibility support and the resulting evaluation.

[For review: The rewrite of this section also covers the following comments that have "no change" as proposed resolution. This is Comment 64. Please review.]

3.4.3 Step 4.c: Use WCAG 2.0 Techniques and Failures Where Possible (Optional)

Methodology Requirement 4.c: Where possible, use documented WCAG 2.0 techniques and failures to help assess successes and failures in meeting the WCAG 2.0 Success Criteria relevant per 3.1.3 Step 1.c: Define the Conformance Target (Optional).

Reminder: Techniques and failures in the context of WCAG 2.0 are only informative. They can help assess if WCAG 2.0 Success Criteria are met by providing documented ways of meeting them and commonly occurring failures in meeting them. However, as per the WCAG 2.0 conformance requirements, only the Success Criteria must be met. And you can use any techniques that meet the Success Criteria, whether they are documented yet by the WCAG WG or not.

The initial sets or sources of WCAG 2.0 techniques and failures to be used during evaluation may be defined in section 3.1.5 Step 1.e: Define the Techniques to be Used (Optional). However, during evaluation such initial sets may often need to be refined according to the particular situation, such as for evaluating particular web technologies and accessibility features that are identified on the website being evaluated.

WCAG 2.0 techniques are documented ways for meeting or for going beyond what is required by individual WCAG 2.0 Success Criteria. A WCAG 2.0 Success Criterion is met on a web page when:

Conversely, failures are documented ways of not meeting individual WCAG 2.0 Success Criteria. A WCAG 2.0 Success Criterion is not met on a web page when a failure applies to any instance of web content that is addressed by the WCAG 2.0 Success Criterion.

WCAG 2.0 techniques are not the only way to meet WCAG 2.0 Success Criteria, and WCAG 2.0 failures are not the only way to fail WCAG 2.0 Success Criteria. WCAG 2.0 techniques and failures are not exhaustive as they cannot cover every possible situation. Also, the techniques used to meet WCAG 2.0 Success Criteria during the development may not be known to the evaluator. Particularly for newly released web technologies, or when these web technologies are used in particular contexts, there may be no publicly or proprietary documented techniques and failures available to the evaluator. The evaluator must consider these limitations when using WCAG 2.0 techniques and failures to evaluate conformance with WCAG 2.0.

3.4.4 Step 4.d: Archive Web Pages for Reference (Optional)

Methodology Requirement 4.d: Archive every evaluated web page for reference. (Optional)

Archive web pages for future reference. At the very least, include the title and URI of the web page and the date of the evaluation. To avoid ambiguity, complement this with any of the following:

3.4.5 Step 4.e: Record Evaluation Tools and Methods (Optional)

Methodology Requirement 4.e: Record the tools and methods used to evaluate the web pages. (Optional).

For future reference, record Evaluation tools and methods used to evaluate the web pages. This includes versions of web accessibility evaluation tools, web browsers and add-ons, and other tools used during the evaluation. Depending on the level of detail for reporting defined by 3.1.2 Step 1.b: Define the Goal of the Evaluation, this recording may apply globally for the entire evaluation, to individual web pages, or to individual checks carried out within the audited web pages.

3.5 Step 5: Report the Evaluation Findings

Methodology Requirement 5: Document the evaluation findings according to Methodology Requirement 5.a and optionally Methodology Requirement 5.b, Methodology Requirement 5.c, and Methodology Requirement 5.d.

[For review: Changed "Reporting your findings is a key element.." to "Reporting is a key element.." to take the personal form out. This is better in line with the rest of the document. This also covers Comment 116. Please review.]

This section describes how to report the evaluation findings that have been gathered during the previous steps. Reporting is a key element of every evaluation and helps facilitate replicability of the results.

Note: Individual pieces of the reporting are gathered throughout the evaluation process, not necessarily at the end of it.

3.5.1 Step 5.a: Provide Documentation for Each Step

Methodology Requirement 5.a: Document each outcome of the steps defined in 3.1 Step 1: Define the Evaluation Scope, 3.2 Step 2: Explore the Target Website, 3.3 Step 3: Select a Representative Sample, and 3.4 Step 4: Audit the Selected Sample.

Document the outcomes of each of the previous steps defined in 3.1 Step 1: Define the Evaluation Scope, 3.2 Step 2: Explore the Target Website, 3.3 Step 3: Select a Representative Sample, and 3.4 Step 4: Audit the Selected Sample, to ensure justifiable, transparent, and repeatable evaluation results. In particular, this includes documenting:

Documentation of the outcome of auditing the representative sample depends on the level of detail for reporting defined by 3.1.2 Step 1.b: Define the Goal of the Evaluation. Outcome of the auditing must include at least the following information depending on the set goal (optional example reports are provided in Appendix C: Example Reports):

[For review: Adjusted the different reports below to align them with the rest of the document and with current evaluation practices.Also added "at least" to the introduction text to this list (above this block) so evaluators are always free to add more information. For instance if requested by the evaluation commissioner. Please review.]

Basic Report
Only captures the successes and failures in meeting WCAG 2.0 conformance requirements globally for the entire website. For each WCAG 2.0 Success Criterion applicable as per 3.1.3 Step 1.c. Define the Conformance Target, the report identifies if it is met or not met in the selected sample of web pages. Where failures in meeting WCAG 2.0 Success Criteria are identified, at least one example web page from the sample in which such a failure has been identified must be indicated in the report.
Detailed Report
Captures the successes and failures in meeting WCAG 2.0 conformance requirements for each web page. For each WCAG 2.0 Success Criterion applicable as per 3.1.3 Step 1.c. Define the Conformance Target, the report identifies if it is met or not met in each web page in the selected sample of web pages. Where failures in meeting WCAG 2.0 Success Criteria on a web page are identified, each identified occurrence of such a failure must be indicated in the report.
In-Depth Analysis Report
Captures the successes and failures in meeting WCAG 2.0 conformance requirements for each Success Criterion. For each WCAG 2.0 Success Criterion applicable as per 3.1.3 Step 1.c. Define the Conformance Target, the report identifies if it is met or not met in the selected sample of web pages. Where failures in meeting WCAG 2.0 Success Criteria are identified, examples of the identified occurrence of such a failure must be indicated in the report. The In-Depth Analysis Report includes a summary of the positive and negative aspects identified on the website, examples of frequently occurring issues and an assessment of their impact on the users of the website in completing tasks, and suggestions for improving the overall accessibility of the website and if possible suggesting guidance for the future.

[For review: Added what to do if an accessibility evaluation statement is provided: "in which case the minimum evaluation information to be made public is as per the requirements for WCAG 2.0 conformance claims (see next section Step 5.b: Provide and Accessibility Statement). In other situations...". Please review.]

Note: While such a report is required for conformance with this methodology, it is not required for any parts of this report to be made public unless an optional public accessibility statement is provided per Step 5.b: Provide an Accessibility Statement (optional) in which case the minimum evaluation information to be made public is as per the requirements for WCAG 2.0 conformance claims (see next section Step 5.b: Provide and Accessibility Statement). In other situations, the level of confidentiality for evaluation reports is usually determined by the evaluation commissioner.

3.5.2 Step 5.b: Provide an Accessibility Evaluation Statement (optional)

[Editor note: Needs refinement.]

Methodology Requirement 5.b: Provide an Accessibility Evaluation Statement. (Optional).

WCAG 2.0 conformance claims can only be made for the web pages that have been actually evaluated and identified to conform with its conformance requirements. Accessibility evaluation statements for entire websites can be made according to this methodology when:

As per the requirements for WCAG 2.0 conformance claims, also accessibility evaluation statements must include the following information:

  1. Date of the claim
  2. Guidelines title, version and URI: "Web Content Accessibility Guidelines 2.0 at http://www.w3.org/TR/WCAG20/"
  3. Conformance level satisfied: (Level A, AA or AAA)
  4. A concise description of the Web pages, such as a list of URIs for which the claim is made, including whether subdomains are included in the claim.
  5. A list of the web technologies relied upon.

Accessibility evaluation statements can also be made when only partial conformance has been achieved according to the requirements defined in third-party content and languages lacking accessibility support. In such cases the accessibility evaluation statements must also include information to identify the following:

  1. website areas that do not conform to WCAG 2.0;
  2. Reason for not conforming to WCAG 2.0: "third-party content" or "lack of accessibility support;

[For review: Added note following discussion in Telco of 6 December 2012 that it is possible to use WCAG-EM during development, but that in that case it is not possible to do a conformance claim. Please review.]

Note: It is not possible to make an accessibility evaluation statement for a website that is still in development.

3.5.3 Step 5.c: Provide a Performance Score (optional)

[Editor Note: Needs refinement.]

Methodology Requirement 5.c: Provide a performance score. (Optional).

Performance scores can provide more granular measures on the level of conformance of a website than the conformance levels defined by WCAG 2.0 provide. However, scores alone do not provide sufficient context and information to understand the actual level of accessibility of a website. Performance scores according to this methodology may only be provided when the evaluation carried out conforms with this methodology as per 5. Conformance with this Methodology.

Depending on the requested level of detail, the performance score is calculated through one of the following approaches:

Per Website
For each WCAG 2.0 Success Criterion applicable as per 3.1.3 Step 1.c. Define the Conformance Target, calculate the number of Success Criteria that were met on all web pages within the selected sample as per 3.3 Step 3: Select a Representative Sample. The Success Criteria are not met for web pages on which any of the WCAG 2.0 conformance requirements are not met. The score is the ratio of the Success Criteria met over all applicable Success Criteria.
Per Web Page
For each web page within the selected sample as per 3.3 Step 3: Select a Representative Sample, calculate the number of applicable Success Criteria as per 3.1.3 Step 1.c. Define the Conformance Target that were met on each web page. All Success Criteria are not met for web pages on which any of the WCAG 2.0 conformance requirements are not met. The score is the average ratio of the Success Criteria met over all applicable Success Criteria for all web pages in the selected sample.
Per Instance
For each web page within the selected sample as per 3.3 Step 3: Select a Representative Sample, calculate the number of success and failure instances in meeting the applicable Success Criteria as per 3.1.3 Step 1.c. Define the Conformance Target that were met on each web page. All instances where Success Criteria are applicable are not met for web pages on which any of the WCAG 2.0 conformance requirements are not met. The score is the ratio of all success and failure instances in meeting the applicable Success Criteria for all web pages in the selected sample.

Note: According to WCAG 2.0, Success Criteria that do not apply to the content are deemed to have been met. Web accessibility evaluation tools may be needed during the evaluation process to help count and calculate the scores.

The approach used to calculate the score must be indicated together with the numeric ratio whenever the score is provided. For example, "X/Y Per Website", "X/Y Per Web Page", and "X/Y Per Instance" are valid statements to express the performance score.

3.5.4 Step 5.d: Provide Machine-Readable Reports (optional)

Methodology Requirement 5.d: Provide machine-readable reports. (Optional).

Machine-readable reports facilitate processing the results by tools, for example to help monitor accessibility of a website over time. The Evaluation and Report Language (EARL) is a machine-readable format that was specifically designed for this purpose. It is recommended to use EARL for providing machine-readable reports. See also Understanding Metadata to learn more about uses of metadata, including machine-readable reports, such as EARL.

4. Considerations for Particular Situations

The methodology defined by this document is flexible to allow its implementation in different situations and contexts. This section provides guidance for applying this methodology in some of these situations.

[For review: Taken out section 4.1. It is not necessary because this is covered earlier in the document and does not add new information. Renumbering of next sections is still to be done. Please review.]

4.1 Initial Conformance Assessment of a Website

When getting started with improving website accessibility it is useful to do a conformance evaluation to assess the current state of the website. In such situations it is often useful to expand the scope of the evaluation beyond the planned conformance target to get a more complete picture of the accessibility performance on the website. Expanding the scope could include setting the goal of the evaluation in 3.1.2 Step 1.b: Define the Goal of the Evaluation to "In-Depth Analysis" and the conformance target in 3.1.3 Step 1.c. Define the Conformance Target to "WCAG 2.0 Level AAA".

Note: It is recommended to carry out preliminary reviews before carrying out a conformance evaluation to identify obvious errors and to develop a rough understanding of the overall performance of the website.

4.2 Evaluating a Large Website with Separate Parts

[Editor note: Needs refinement.]

Evaluation of large websites according to this methodology may be broken down into the evaluation of individual website areas, such as the online shop, departments within an organization, blogging area, and other sub-sites. In this case, each sub-site must meet the terms established in section 2.1 Scope of Applicability and must be evaluated using this methodology. Evaluation of each sub-site must be carried out to at least the same conformance target (as per 3.1.3 Step 1.c. Define the Conformance Target) as for the main website.

An additional evaluation must be carried out according to this methodology, whereby the selected sample (as per 3.3 Step 3: Select a Representative Sample) must include at least all common web pages of the main website plus two randomly selected web pages from the representative sample selected for each sub-site. The reporting, scoring, and accessibility statements per 3.5 Step 5: Report the Evaluation Findings applies to the entire set of sampled web pages from the main website and all its sub-sites.

Sub-sites may be broken down into further sub-site where applicable. In this case this same clause of the methodology is applied.

4.3 Re-Running a Website Conformance Evaluation

Conformance evaluation according to this methodology may be re-run after a short period, for example when issues are identified and repaired by the website owner or website developer, or periodically to monitor progress. In such cases the selected sample (as per 3.3 Step 3: Select a Representative Sample) must include a different set of exemplar web page instances (as per 3.3.2 Step 3.b: Include Exemplar Instances of Web Pages), where possible. This allows a portion of the sample to remain unchanged (common web pages and common functionality), which facilitates comparability between the results.

4.4 Large-Scale Evaluation of Multiple Websites

Carrying out full conformance evaluation of multiple websites can involve a lot of effort. This methodology can not be undertaken using just automated evaluation, which can only evaluate a small portion of the accessibility requirements, it also requires manual evaluation by experts. Limiting the goal of the evaluation in 3.1.2 Step 1.b: Define the Goal of the Evaluation to "Basic Reporting" and careful selection of the samples in 3.3 Step 3: Select a Representative Sample can optimize the effort required for each evaluation.

5. Application of this Methodology

The methodology defined by this document is flexible to allow its implementation in different situations and contexts. It is not required to carry out any of the steps and activities defined by this methodology in any particular sequence. For proper application of this methodology, use all the following Methodology Requirements:

Results, reports, accessibility statements, and performance scores can only be claimed to be in accordance with this methodology when the evaluation carried out meets these Methodology Requirements.

Appendices

Appendix A: Acknowledgements

This publication has been developed with support from the European Commission funded WAI-ACT Project (IST 287725).

Contributors

Past and present active participants of the WCAG 2.0 Evaluation Methodology Task Force (Eval TF) include: Shadi Abou-Zahra; Frederick Boland; Denis Boudreau; Amy Chen; Vivienne Conway; Bim Egan; Michael Elledge; Wilco Fiers; Detlev Fischer; Elizabeth Fong; Vincent François; Alistair Garrison; Emmanuelle Gutiérrez y Restrepo; Katie Haritos-Shea; Martijn Houtepen; Peter Korn; Maureen Kraft; Aurelien Levy; Kerstin Probiesch; Donald Raikes; Corominas Ramon; Roberto Scano; Samuel Sirois; Sarah J Swierenga; Eric Velleman; Konstantinos Votis; Kathleen Wahlbin; Elle Waters; Richard Warren; Léonie Watson.

Appendix B: References

QA-FRAMEWORK
Dubost K, Rosenthal L, Hazaël-Massieux D, Henderson L (eds) (2005) W3C Quality Assurance Framework: Specification Guidelines. W3C. Available at: http://www.w3.org/TR/qaframe-spec/
RFC3986
Fielding R, Gettys J, Mogul J, Frystik H, Masinter L, Berners-Lee T (eds) (1999). Hypertext Transfer Protocol - HTTP/1.1. Request for Comments: 2616. IETF. Available at: http://www.ietf.org/rfc/rfc3986.txt
RFC2616
Berners-Lee T, Fielding R, Masinter L (eds) (2005). Uniform Resource Identifier (URI): Generic Syntax. Request for Comments: 3986. IETF. Available at: http://www.ietf.org/rfc/rfc2616.txt
UWEM
Velleman E.M, Velasco C.A, Snaprud M (eds) (2007). D-WAB4 Unified Web Evaluation Methodology (UWEM 1.2 Core). Wabcluster. Available at: http://www.wabcluster.org/uwem1_2/
WCAG20
Caldwell B, Cooper M, Guarino Reid L, Vanderheiden G, eds (2007). Web Content Accessibility Guidelines 2.0. W3C. Available at: http://www.w3.org/TR/WCAG20/

Appendix C: Example Reports

[For review: Complete rewrite of this Appendix C: Example Reports. Please review.]

This Appendix proposes three different levels of reporting following the goals defined in section 3.1.2 Step 1.b: Define the Goal of the Evaluation and the documentation defined in 3.5.1 Step 5.a: Provide Documentation for Each Step:

  1. Basic Report
  2. Detailed Report
  3. In-Depth Analysis Report
All reports should at least have the following basic information:

For Detailed Report add to basic information:

For In-Depth Analysis Report add to basic information:

Appendix D: Document Changes

Changes since the Public Working Draft of 20 September 2012 include:

A full disposition of comments of all the comments received on the 20 September Working Draft is available.