[ contents ]

Abstract

This document specifies an internationally harmonized describes a methodology for evaluating the accessibility conformance of websites to the Web Content Accessibility Guidelines ( WCAG ) 2.0 . It defines an approach for conformance provides guidance on defining the evaluation scope and parameters, exploring the website features and functionality, sampling representative web pages where it is not feasible to evaluate all web pages of a website, applying the WCAG 2.0 success criteria and conformance requirements in different contexts, including self-assessment this setting, and third-party evaluation. documenting and reporting the evaluation findings. It complements the existing guidance for WCAG 2.0 but it does not define additional WCAG 2.0 requirements nor does it replace or supersede it in any way.

This methodology only addresses accessibility evaluation of websites after their development, for example to validate conformance with reasonable confidence or to better understand the accessibility performance for improvement. It is applicable to any website, including web applications and websites for mobile devices, and devices. It is independent of any particular evaluation tools, tool, web browsers, browser, and assistive technologies. This document is a supporting resource for WCAG 2.0 technology, and does not replace or supersede it in any way. It addresses different contexts, including self-assessment and third-party evaluation. However, ensuring and maintaining accessibility requires that accessibility is considered throughout the need for a standardized approach for evaluating entire websites after their development as opposed to evaluating individual collections of web pages process. Complementary resources from W3C / WAI on evaluation during their development, which is equally important to ensure website accessibility. provide guidance for other situations.

This work is part of complementary W3C / WAI activities on Web Accessibility Evaluation and Testing that address different W3C / WAI guidelines and specifications . It is developed through the collaborative W3C Process with the involvement of international experts and stakeholders.

Status of this document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

This 30 July 10 September 2012 Editors Draft of Website Accessibility Conformance Evaluation Methodology 1.0 is based on the recently same approach and expanding the previously published W3C First Public Working Draft located at http://www.w3.org/TR/2012/WD-WCAG-EM-20120327/ of 27 March 2012 . The This Editors Draft addresses the following comments received, to seek approval for publication as an updated Working Draft:

This document is intended to be published and maintained as a an informative W3C Working Group Note after review and refinement. It includes changes following the resolutions from the comments we received on the Public Working Draft. [@@@] The WCAG 2.0 Evaluation Methodology Task Force ( Eval TF ) invites discussion and feedback on this document by developers, evaluators, researchers, and others with interest in web accessibility evaluation. Eval TF is particularly looking for feedback on these sections:

Please send comments on this Editor Editors Draft of the Website Accessibility Conformance Evaluation Methodology for WCAG 2.0 1.0 to public-wai-evaltf@w3.org (publicly visible mailing list archive ). These comments will be considered in the internal discussions of Eval TF .

Publication as Editor Editors Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

This document has been produced by the WCAG 2.0 Evaluation Methodology Task Force ( Eval TF , a joint task force of the Web Content Accessibility Guidelines Working Group (WCAG WG) and Evaluation and Repair Tools Working Group (ERT WG) , as part of the Web Accessibility Initiative (WAI) Technical Activity .

This document was produced by two groups operating under the 5 February 2004 W3C Patent Policy . The groups do not expect this document to become a W3C Recommendation. W3C maintains a public list of any patent disclosures for WCAG WG and public list of any patent disclosures for ERT WG made in connection with the deliverables of each group; these pages also include instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy .

Table of Contents

Overview

  1. Introduction
  2. Using this Methodology
  3. Conformance Evaluation Procedure
  4. Considerations for Particular Situations
  5. Conformance with Application of this Methodology

List of Sections

  1. Introduction
  2. Using this Methodology
  3. Conformance Evaluation Procedure
  4. Considerations for Particular Situations
  5. Conformance with Application of this Methodology

List of Appendices

1. Introduction

[ Editor Note: This section has been edited and needs to be reviewed by Eval TF.]

Website owners, procurers, suppliers, developers, and others are frequently tasked with assessing often need to evaluate the conformance of existing websites to the Web Content Accessibility Guidelines ( WCAG ) 2.0 . This document provides For example, such evaluation may be carried out as part of releasing or purchasing a standardized methodology for evaluating the website to validate its conformance with reasonable confidence, or it may be to better understand and monitor the level of accessibility provided on a websites website for improvement. The methodology described in this document provides one possible approach to address this particular aspect of web accessibility evaluation. It does not specifically explain evaluation in other situations nor does it provide general guidance on evaluation with WCAG 2.0. It is developed through the collaborative W3C Process with extends the involvement of international stakeholders. It is a supporting document existing guidance for WCAG 2.0 and but it does not define additional WCAG 2.0 requirements nor does it replace or supersede it in any way.

1.1 Scope of this Document

[ Editor Note: This section has been edited and needs to be reviewed by Eval TF.] To ensure accessibility, it needs to be considered and addressed throughout the development process. Accessibility testing and evaluating conformance to WCAG 2.0 needs to be carried out during all stages of the process, including the design, implementation, and maintenance stages of a website .

The guidance methodology described in this document focuses provides guidance on only one aspect of evaluation within the development process, which is defining the accessibility evaluation of an already existing scope and parameters, exploring the website . This features and functionality, sampling representative web pages where it is useful for determining the level of accessibility not feasible to evaluate all web pages of a website at a given point in time, such as at , applying the time of releasing or purchasing a product. It is also useful for initial conformance assessments when getting started with accessibility WCAG 2.0 success criteria and for continual monitoring of website accessibility. The methodology described in this document supports conformance evaluation requirements in different contexts including self-assessment this setting, and third-party assessment of websites . documenting and reporting the evaluation findings. It is applicable to all websites , including web applications, websites intended to be used with mobile devices, and other types of websites . It is independent of the size of the website , the web technologies used to create the website , and of any particular web accessibility evaluation tools, web browsers, assistive technologies, and other software tools.

However, this methodology only addresses evaluation of website conformance to WCAG 2.0 after its development. Complementary resources from W3C / WAI on evaluation provide guidance on evaluation in other situations, and complementary W3C / WAI activities on Web Accessibility Evaluation and Testing address different W3C / WAI guidelines and specifications .

Also, WCAG 2.0 conformance requirements apply to individual web pages . However, on most websites it is not feasible to evaluate every web page with reasonable effort. To ensure conformance of a website to WCAG 2.0 it is necessary to ensure that this is considered throughout the development process. Following this methodology helps assess the level of conformance with reasonable confidence where it is not feasible to evaluate every web page .

1.2 Target Audience

The primary target audience for this methodology are individuals and organizations, wanting to evaluate the conformance of existing websites to WCAG 2.0. This includes but is not limited to:

Other audiences for whom this methodology is also relevant include:

Users of the methodology are assumed to have expertise in WCAG 2.0, accessible web design, assistive technologies, and of how people with different disabilities use the Web; read more in section 2.2 Required Expertise .

1.3 Background Reading

It is assumed that the reader of this document is familiar with the following related resources from W3C resources: / WAI :

Evaluating Websites for Accessibility

A multi-page resource suite that outlines different approaches for evaluating websites for accessibility. The following resources are particularly important in this context:

Web Content Accessibility Guidelines ( WCAG ) 2.0

Internationally recognized standard explaining how to make web content more accessible to people with disabilities. The following resources are particularly important in this context:

Accessible Web Design

The following documents introduce the essential components of web accessibility, inclusive design for people with disabilities and people with aging-related impairments, and overlap with design for mobile devices. The documents help managers, designers, developers, policy makers, researchers, and others optimize their efforts in these overlapping areas:

1.4 Terms and Definitions

For the purposes of this document, the following terms and definitions apply:

Complete Processes
From WCAG 2.0 Conformance Requirement for Complete Processes :
When a web page is one of a series of web pages presenting a process (i.e., a sequence of steps that need to be completed in order to accomplish an activity), all web pages in the process conform at the specified level or better. (Conformance is not possible at a particular level if any page in the process does not conform at that level or better.)
Common Web Pages
Web pages that are relevant to the entire website . This includes the homepage, login page, and other entry pages, and, where applicable, the sitemap, contacts, site help, legal information, and similar web pages that are typically linked from all web pages (usually from the header, footer, or navigation menu of a web page ).
Evaluator
The person, team of people, organization, in-house department, or other entity responsible for carrying out the evaluation.
Evaluation Commissioner
The person, team of people, organization, in-house department, or other entity that commissioned the evaluation.
Note: In many cases the evaluation commissioner may be the website owner or website developer , in other cases it may be another entity such as a procurer or an accessibility monitoring survey. [ Editor Note: Term "web developer" has been added and needs approval.]
Key Common Functionality
Primary functionalities

[ Review Note: We are searching for input on this definition. We previously used the terms key functionality and primary functionality in this definition, but are searching for a better term.]

[...] functionality of a website including tasks that users of a website carry out to perform these functionalities. this functionality.
Note: Examples of functionality include "selecting and purchasing a product from the shop area of the website", "filling and submitting the form provided on the website", and "registering for an account on the website".
Relied Upon (Technologies)
From WCAG 2.0 definition for "relied upon" :
The content would not conform if that technology is turned off or is not supported
Template
From ATAG 2.0 definition for "templates" :
Content patterns that are filled in by authors or the authoring tool to produce content for end users (e.g., document templates, content management templates, presentation themes). Often templates will pre-specify at least some authoring decisions.
[ Editor Review Note: This definition has been added and needs approval.] A specific page Given that provides ATAG 2.0 is still a more or less fixed framework for content. Templates are mostly used to provide the basic components Working Draft, this definition might change in future drafts of a web page (logo, menu, layout, skip links etc.). Content can be viewed when entered into the template. this document.]
Website
A coherent collection of one or more related web pages that together provide common use or functionality. It includes static web pages , dynamically generated web pages , and web applications.
Note: Websites may be composed of smaller sub-sites, each of which can be considered to be an individual website . For example, a website may include an online shop, an area for each department within the organization, a blog area, and other areas that may each be considered to be a website . [ Editor Note: This note has been added and needs approval.]
Website Developer
Anyone involved in the website development process including but not limited to content authors, designers, programmers, quality assurance testers, and project managers.
Website Owner
The person, team of people, organization, in-house department, or other entity that is responsible for the website .
Web page
From WCAG 2.0 definition for "Web Page" :
A non-embedded resource obtained from a single URI using HTTP plus any other resources that are used in the rendering or intended to be rendered together with it by a user agent
Note: Web pages may include multimedia content, interactive components, and complex applications. [ Editor Note: This note has been added and needs approval.] Website Part [ Editor Note: This definition is no longer needed and should be removed.] Web pages that serve the purpose and functionality of a website . This includes web pages that are part of the navigation, design, and complete processes of a website .

2. Using This Methodology

The methodology defined by this document is used for comprehensively assessing the conformance of websites to WCAG 2.0. A more cursory approach for exploring the accessibility of a website is described in Preliminary Review of Websites for Accessibility . In many cases it is advisable to carry out such preliminary reviews before applying this methodology, for example to identify obvious errors and to develop a rough understanding of the overall performance of the website .

2.1 Scope of Applicability

[ Editor Note: This section has been rewritten and needs to be reviewed by Eval TF.] [ Review Note: Eval TF is particularly looking for feedback on this section.]

The methodology defined by this document applies to full, self-enclosed websites . This includes websites of organizations and entities, persons, events, products, and services. It also includes web applications and mobile websites . Examples include:

A website may include several areas with smaller sub-sites, collections of related web pages such as an online shop, an area for each department within the organization, a blog area, and other parts. In some situations such areas that may each can be considered to be a full, self-enclosed website . each. This methodology can be applied to make conformance claims for such individual sub-sites, in which case these sub-sites would be considered to be a (a website each, within another website ) and to the main website in its entirety. However, this methodology may not be applied to a website excluding any of its parts. Excluding parts of the website from the scope of evaluation would likely conflict with the WCAG 2.0 conformance requirements full pages and complete processes , or significantly distort the evaluation results.

Example of a Website:
Diagram of a University Website explained in the following paragraph.

In the diagram above, the university website is made of several sub-sites: "Information for Students", "Information for Lecturers", a "Courseware Application", and a "Library Application". The "Courseware Application" includes "Physics Courses", "Medicine Courses", and "History Courses" that are aggregated into the application. Also, the university website has individual web pages such as legal notices, sitemap, and others that are not part of any particular sub-site.

In the example above, none of the depicted parts may be excluded from the scope of evaluation in the context of this methodology, if a conformance claim it is to be made for applied to the university website according this methodology. . This includes any aggregated and embedded content from third-party sources such as online maps for the university campus and forms for credit card transactions. Excluding transactions, including when such parts of the website originate from the scope of evaluation would likely conflict with the WCAG 2.0 conformance requirements full pages and complete processes . third-party sources.

Note: According to WCAG 2.0 it is possible to make a defines " Statement of Partial Conformance " for individual web pages that are known not to conform with WCAG 2.0 due to third-party content and and/or languages lacking accessibility support . However, these parts must still Such web pages may not be included in excluded from the scope of evaluation. More information evaluation according to this methodology. In some cases this means that the website as a whole does not conform with WCAG 2.0 due to partially conforming web pages . Section 3.5 Step 5: Report the Evaluation Findings provides more guidance on reporting evaluation results and making conformance claims is provided in section 3.5.2 Step 5.b: Provide an Accessibility Statement (Optional) accessibility statements for entire websites .

2.1.1 Particular Types of Websites

[ Editor Note: This section has been added and needs to be reviewed by Eval TF.]

This methodology is applicable to all websites , including web applications, websites intended to be used with mobile devices, and other types of websites . Some particular types of websites include:

Small Websites
For websites with few web pages the sampling procedure defined in 3.3 Step 3: Select a Representative Sample will likely result in selecting most or all of the web pages from the target website . In cases where all web pages can be evaluated, the sampling procedure can be skipped and the selected sample is considered to be the entire website in the remaining steps.
Web Applications
In some cases only one or few web pages may dynamically generate all the content with the functionality that normally an entire set of web pages would provide. Such web applications are usually not separable into sub-sites and are considered as a single website or part of a website . Each state (individual representation of content and functionality) in which such a web application can be considered to be a web page for the purpose of this document.
Note: The individual states of a web application may not have unique URIs and may need to be identified by describing the settings, input, and actions required to generate such states ( web page for the purpose of this document). Due to the many possibilities to generate such states it may not be feasible to exhaustively identify every possible instance of key web pages as per 3.2.1 Step 2.a: Identify Key Common Web Pages of the Website , functionality as per 3.2.2 Step 2.b: Identify Key Functionalities Common Functionality of the Website , and variety of web pages as per 3.2.3 Step 2.c: Identify the Variety of Web Page Types , so that evaluating such web applications may be more challenging than for other types of websites .
Website with Separable Areas
In some cases websites may have clearly separable areas, such as a password-restricted area of a website (extranet) that is not part of using the public area (log-in is not required to complete a function or process). Such areas can be considered as individual websites rather than sub-sites for the purpose of this document.
Website in Multiple Versions
Some websites are available in versions that are independent of each other. For example, mobile versions of a website and versions of a website in different languages, where using one versions version does not require or depend on using another versions version of the website (usually such website versions have a different set of URIs each). Such website versions can be considered as individual websites rather than sub-sites for the purpose of this document.

2.2 Required Expertise

Users of the methodology defined by this document are assumed to be knowledgeable of WCAG 2.0, accessible web design, assistive technologies, and of how people with different disabilities use the Web. This includes understanding of the relevant web technologies, barriers that people with disabilities experience, assistive technologies and approaches that people with disabilities use, and evaluation techniques and tools to identify potential barriers for people with disabilities. In particular, it is assumed that users of this methodology are deeply familiar with the resources listed in section 1.3 Background Reading .

2.3 Evaluation Tools (Optional)

The methodology defined by this document is independent of any particular web accessibility evaluation tool, web browsers, and other software tools. However, web accessibility evaluation tools can significantly assist an evaluator during the evaluation process and contribute to more effective evaluations. For example, some web accessibility evaluation tools can scan entire websites to help identify relevant pages for manual evaluation, and other tools can assist manual evaluation in a variety of ways. Specific guidance is provided in Selecting Web Accessibility Evaluation Tools .

Note: Web accessibility evaluation tools can only automatically check a limited set of accessibility requirements that are automatable. Conformance evaluation requires manual testing and judgment by experienced evaluators.

2.4 Review Teams (Optional)

The methodology defined by this document can be carried out by an individual evaluator with the skills described in section 2.2 Required Expertise . However, using the combined expertise of review teams provides better coverage for the required skills and helps identify accessibility barriers more effectively. While not required, it is strongly recommended to employ review teams for conformance evaluation of websites . Specific guidance is provided in Using Combined Expertise to Evaluate Web Accessibility .

2.5 Involving Users (Optional)

The methodology defined by this document can be carried out by an individual evaluator with the skills described in section 2.2 Required Expertise . However, involving people with disabilities and people with aging-related impairments helps identify additional accessibility barriers that are not easily discovered by the evaluator alone. While not required, it is strongly recommended to involve real people covering a wide range of abilities during the evaluation process. Specific guidance is provided in Involving Users in Web Accessibility Evaluation .

3. Conformance Evaluation Procedure

[ Editor Note: This section has been edited and needs to be reviewed by Eval TF.]

This section defines the stages and activities of the evaluation procedure. The stages are not necessarily sequential but rather iterative. Also the exact sequence of the activities carried out during the evaluation process depends on the type of website , the purpose of the evaluation, and the process used by the evaluator . Some of the activities overlap or may be carried out in parallel. The following diagram illustrates the iterations between the stages defined in this section:

Diagram about the iterations between the steps in this methodology. Explanation in the following paragraph.

The diagram depicts each of the five steps defined in this section with arrows back and forth between each of two consecutive steps: 1. Define the Evaluation Scope; 2. Explore the Target Website; 3. Select a Representative Sample; 4. Audit the Selected Sample and 5. Report the Evaluation Findings.

Note: See also section 4. Considerations for Particular Situations that may influence how an evaluation procedure is carried out.

3.1 Step 1: Define the Evaluation Scope

[ Editor Note: This section has been edited and needs to be reviewed by Eval TF.]

Methodology Requirement 1: The Define the evaluation scope shall be defined according to Methodology Requirement 1.a , Methodology Requirement 1.b , Methodology Requirement 1.c , Methodology Requirement 1.d , and optionally Methodology Requirement 1.e .

During this step the overall scope of the evaluation is defined. It is a fundamental step that affects the subsequent steps in the evaluation procedure and is ideally carried out together with the evaluation commissioner (who may or may not be the website owner ) to ensure common expectations about the scope of the evaluation. Some exploration throughout this stage may be necessary to understand the full scope of the website and the required evaluation (see 3.2: Explore the Target Website ).

Note: Involvement of the website owner and/or website developer (in addition to the evaluation commissioner ) is not required but often helps identify use cases, functionality, and other aspects of the implementation that makes the evaluation procedure more efficient and effective. However, the evaluator is responsible for an objective and thorough assessment.

3.1.1 Step 1.a: Define the Scope of the Website

[ Editor Note: This section has been rewritten and needs to be reviewed by Eval TF.]

Methodology Requirement 1.a: The Define the scope of the website shall be defined, with the definition not contradicting the constraints expressed in section 2.1 Scope of Applicability .

During this step the exact scope of the website (the web pages for which the evaluation will apply) is defined and documented. This scope definition may not contradict the terms established in section 2.1 Scope of Applicability . It is essential to carefully document particular aspects such as any use of third-party content and services, different languages, and parts of the website that may not be easily identifiable as such (for example an online shop that is not integrated into the website but considered to be part of it).

The outcome of this step is an unambiguous definition for the target website , so that for each web page it can be determined whether it is within the scope of evaluation or not. Where possible, it is recommended to use formalizations such as regular expressions or listings of URIs that define the web pages that are within the scope of evaluation (part of the target website ).

3.1.2 Step 1.b: Define the Goal of the Evaluation

[ Editor Note: This section has been edited and needs to be reviewed by Eval TF.]

Methodology Requirement 1.b: The Define the goal of the evaluation shall be defined as " Basic Report ", " Detailed Report ", or " In-Depth Analysis ".

Defining the goal of the evaluation is particularly relevant to the auditing stage defined in 3.4 Step 4: Audit the Selected Sample and the reporting stage defined in 3.5 Step 5: Report the Evaluation Findings . Some of the evaluation goals include:

Basic Report
Only identifies whether a website conforms or not without providing additional information. This type of evaluation is typically carried out when the website is assumed to conform, for example to verify an existing conformance claim, and for large-scale evaluations with less resources to explore the details of individual websites .
Detailed Report
Identifies whether a website conforms or not, and provides further information about the conformance of each evaluated web page . This type of evaluation is particularly useful for instructing web developers and for acquiring statistics for monitoring progress over time.
In-Depth Analysis
Identifies whether a website conforms or not, and analyzes any identified issues in detail. This includes descriptions of the errors and suggestions for possible repair options. This type of evaluation is particularly useful for organizations that want to improve the accessibility of their website and need a detailed analysis

3.1.3 Step 1.c: Define the Conformance Target

[ Editor Note: This section has been edited and needs to be reviewed by Eval TF.]

Methodology Requirement 1.c: The Define the target WCAG 2.0 conformance level ("A", as "A", "AA", or "AAA") to evaluate for shall be defined. "AAA".

Part of initiating the evaluation process is to define the target WCAG 2.0 conformance level ("A", "AA", or "AAA"). In many situations WCAG 2.0 Level AA is the generally accepted level of accessibility.

3.1.4 Step 1.d: Define the Context of Website Use

[ Editor Review Note: This section has been rewritten and needs Eval TF is particularly looking for feedback on requiring uniform accessibility support throughout a single website to be reviewed by Eval TF.] in accordance with this methodology.]

Methodology Requirement 1.d: The Define the target users of the website and the minimum set of web browsers and assistive technology to evaluate for shall be defined, noting that the definition of software support shall not conflict with the WCAG 2.0 guidance on the Level of Assistive Technology Support Needed for "Accessibility Support" . evaluation.

It is often not feasible for websites to support accessibility on every combination of web browser, assistive technology, and operating system that they run on, nor is it possible to test with every such combination of tools. It is necessary to determine the primary context in which a website is used and the minimum set of web browsers and assistive technology that must be supported by the website .

For example, a website may be public, only available within a closed network (intranet), or limited to authenticated users (extranet). Users of intranet and extranet websites may be known to have a certain level of training on using the website and may be using pre-determined web browsers and assistive technology. This is usually not the case for public websites .

This definition of target users and tools must needs to meet the terms defined in WCAG 2.0 Level of Assistive Technology Support Needed for "Accessibility Support" and must needs to be used supported throughout the evaluation. website. For example, it if one part of a website is not possible to evaluate some pages with accessible using one set of tools and other pages with that is different from a set of tools that is needed to access another set. part of the same website, then the website is effectively not accessible for some users. Accessibility support must needs to be uniform throughout a single website . website.

Note: It is also not possible to single out or exclude individual groups of users, such as "people with visual impairments", in the target audience. All WCAG 2.0 Success Criteria must be considered throughout the evaluation.

3.1.5 Step 1.e: Define the Techniques to be Used (Optional)

[ Editor Review Note: This section has been edited and needs to be reviewed by Eval TF.] TF is particularly looking for feedback on this section. We would like to know your opinion on the level of inclusion of the techniques. Specifically in relation to section 3.4.2 Step 4.b: Use WCAG 2.0 Techniques Where Possible (Optional) .]

Methodology Requirement 1.e: The Specify the WCAG 2.0 techniques that will be used to evaluate the conformance to WCAG 2.0 should be specified. Success Criteria. (Optional).

Techniques are an effective way in the context of demonstrating whether particular WCAG 2.0 Success Criteria are met or informative and not met. However, required for satisfying the WCAG 2.0 conformance to requirements ; WCAG 2.0 Success Criteria are written as testable statements. Techniques provide documented ways of meeting WCAG 2.0 applies to its Success Criteria and conformance requirements , and failures in meeting them. More information on techniques is provided in WCAG 2.0 Layers of Guidance .

W3C / WAI provides a set of publicly documented Techniques for WCAG 2.0 . However, it is not necessary to the use these particular techniques. In fact, in some situations, such as in a closed network, it may be necessary to use techniques that are specifically developed for such situations. Individuals and organizations developing techniques must employ methods for establishing the technique's ability to meet the WCAG 2.0 Success Criteria.

It is good practice to specify the sets or sources of techniques that are intended to be used during the evaluation at this stage already, already to ensure consistent expectation between the evaluator and the evaluation commissioner . This definition is typically refined in later stages of the evaluation process, for example to evaluate web content using particular web technologies during the website exploration and accessibility features. Note: W3C / WAI provides a set of publicly documented Techniques for evaluation stages. Section 3.4.2 Step 4.b: Use WCAG 2.0 Techniques Where Possible (Optional) but other provides more guidance on using WCAG 2.0 techniques may be used too. during evaluation.

3.2 Step 2: Explore the Target Website

[ Editor Note: This section has been edited and needs to be reviewed by Eval TF.]

Methodology Requirement 2: The Explore the website to be evaluated shall be explored according to Methodology Requirement 2.a , Methodology Requirement 2.b , Methodology Requirement 2.c , and Methodology Requirement 2.d .

During this step the evaluator explores the target website to be evaluated, to develop a better understanding of the website use use, purpose, and functionality.

Carrying out initial cursory checks during this stage already helps identify web pages that are relevant for more detailed evaluation later on. For example, an evaluator may identify web pages that seem to be lacking color contrast, consistent navigation, or document structures (headings, links, etc.) with simple checks while studying the website , and note them down as candidates for selection in the sampling step defined in 3.3 Step 3: Select a Representative Sample . more detailed evaluation later on.

To carry out this step it is critical that the evaluator has access to all the relevant parts of the website . For example, it may be necessary to create accounts or otherwise provide access to restricted areas of a website that are part of the evaluation.

Note: Involvement of the website owner and/or website developer can be helpful to help identify key common web pages , functionality, technologies, and other aspects of the implementation that makes the evaluation procedure more efficient and effective. However, the evaluator is responsible for an objective and thorough assessment. [ Editor Note: This paragraph has been added and needs approval.]

3.2.1 Step 2.a: Identify Key Common Web Pages of the Website

[ Editor Note: This section has been edited and needs to be reviewed by Eval TF.]

Methodology Requirement 2.a: The Identify the common web pages of the website and templates available to the evaluator shall be identified. .

During this step the common web pages of the website and the templates that are available to the evaluator and used to create the website are identified and documented. The outcome of this step is a list of all common web pages and templates available to , including the evaluator common states of these web pages and used to create the website for web applications . These will be part of the sample to be selected through the steps defined in 3.3 Step 3: Select a Representative Sample .

This step also helps understand key aspects of the website , such as the navigation and overall structure of the website .

3.2.2 Step 2.b: Identify Key Functionalities Common Functionality of the Website

Methodology Requirement 2.b: The Identify the key functionalities common functionality of the website shall be identified. .

During this step the key functionalities common functionality of the website are identified and documented, to provide a comprehensive basis for the sample to be selected through the steps defined in 3.3 Step 3: Select a Representative Sample . The outcome of this step is a list of user tasks such as "selecting and purchasing a product from the shop area of the website", "filling and submitting the form provided on the website", and "registering for an account on the website".

3.2.3 Step 2.c: Identify the Variety of Web Page Types

[ Editor Note: This section has been rewritten and needs to be reviewed by Eval TF.]

Methodology Requirement 2.c: The Identify the variety of web page types shall be identified. types.

Web pages with varying styles, layouts, structures, and functionality often have different implementations of accessibility features. They are also often generated by different templates and authored by different people. Web pages may also appear differently, behave differently, and contain different content depending on the user. The outcome of this step is a list of the different types of web pages on the website , to provide a comprehensive basis for the sample to be selected through the steps defined in 3.3 Step 3: Select a Representative Sample . Different web page types include:

Note: This step is intended to identify instances groups of web pages , page instances, including in web applications , rather than templates. Templates are identified as part of 3.2.1 Step 1.a: Identify Key Web Pages of the Website .

3.2.4 Step 2.d: Identify Web Technologies Relied Upon

[ Editor Note: This section has been edited and needs to be reviewed by Eval TF.]

Methodology Requirement 2.d: The Identify the technologies relied upon to provide the website shall be identified. .

During this step the web technologies relied upon to provide the website are identified and documented, to provide a basis for the sample selection through the steps defined in 3.3 Step 3: Select a Representative Sample . This includes base web technologies such as HTML and CSS , auxiliary web technologies such as Flash, Java, JavaScript, PDF , Silverlight and WAI-ARIA , as well as advanced web technologies such as SMIL and SVG . The outcome of this step is a list of web technologies that are relied upon .

Only the web technologies that are relied upon to provide the website need to be identified for evaluation. This relates closely to the WCAG 2.0 concepts of conforming alternate versions and non-interference .

Note: Where possible, it is often also useful to identify the libraries and components used to create the website , such as Dojo, JQuery, and others. Particularly for web applications, much of the accessibility support is built into these libraries and components, and evaluation can become more effective and efficient when these are identified. [ Editor Note: This paragraph has been added and needs approval.] Note: Only the technologies that are relied upon to provide the website need to be identified for evaluation. This relates closely to the WCAG 2.0 concepts of conforming alternate versions and non-interference .

3.3 Step 3: Select a Representative Sample

[ Editor Review Note: This EvalTF is specifically looking for review of section has been edited 3.3 Step 3 on sampling. We temporarily use the term "with reasonable confidence" in relation to sampling of representative web pages and needs would like your input on how to be reviewed by Eval TF.] better achieve that in this section of the methodology. We plan to add a random sampling approach in a next version of the methodology.]

Methodology Requirement 3: A Select a representative sample of web pages shall be selected from the website according to Methodology Requirement 3.a , Methodology Requirement 3.b , Methodology Requirement 3.c , and Methodology Requirement 3.d .

While ideally every web page of a website is evaluated, usually this is not possible on most websites . In cases where all web pages can be evaluated, this sampling procedure can be skipped and the selected sample is considered to be the entire website in the remaining steps.

Exploration of the target website in 3.2 Step 2: Explore the Target Website (within the scope set in 3.1 Step 1: Define the Evaluation Scope ) provides sufficient understanding of the website to facilitate selection of a sample of web pages that is representative with reasonable confidence of the entire target website . Depending on the size and complexity of the website , the size of this sample will vary. For example, a website with few types of web pages that are all generated from a confined set of templates, templates , such as a data repository, will likely require a smaller sample than a website with many types of web pages that are authored using different mechanisms. Also a web application could have a limited number of web pages that dynamically generate content with varying types of appearance, behavior, and information that each need to be sampled.

The web pages identified through the exploration will typically relate to more than one design aspect. For example, web pages with particular functionality such as scripting, multimedia, and forms will typically also use related technologies such as Flash, JavaScript, PDF , Silverlight, and WAI-ARIA , and in many cases these web pages may have different design to others. Careful selection of web pages can significantly reduce the required sample size while maintaining appropriate representation of the entire website .

Note: In this step the evaluator identifies the comprehensive sample of web pages for evaluation. However, many web pages will have repetitive content, such as the header, navigation, and other common components that may not need to be re-evaluated on each occurrence. Guidance on evaluating the sample identified in this step is provided in 3.4 Step 4: Audit the Selected Sample .

3.3.1 Step 3.a: Include Key Common Web Pages of the Website

[ Editor Note: This section has been edited and needs to be reviewed by Eval TF.]

Methodology Requirement 3.a: All Include all common web pages and templates available to the evaluator shall be part of into the selected sample of web pages .

All common web pages and templates that are available to , including the evaluator common states of these web pages and used for creating the website web applications , must be part of the selected sample. These web pages are identified in step 3.2.1 Step 2.a: Identify Key Common Web Pages of the Website .

3.3.2 Step 3.b: Include Exemplar Instances of Web Pages

[ Editor Note: This section has been edited and needs to be reviewed by Eval TF.]

Methodology Requirement 3.b: At Include at least two distinct web pages (where applicable) applicable and available) of each (1) key common functionality , (2) content, design, and functionality, and (3) web technologies shall be part of into the selected sample of web pages .

From the variety and types of web pages identified in 3.2 Step 2: Explore the Target Website (within the scope of the evaluation as defined per 3.1 Step 1: Define the Evaluation Scope ), select at least two distinct web pages with the following features each (where applicable and available):

Note: A selected web page could have any number of these features. For example, a selected web page could be used to represent the use of forms and scripting at the same time. The important aspect is to select at least two distinct web pages for each relevant feature identified on the website , though more web pages may be necessary depending on the complexity of the website .

3.3.3 Step 3.c: Include Other Relevant Web Pages

Methodology Requirement 3.c: Other Include other web pages relevant for people with disabilities and accessibility shall be part of into the selected sample of web pages .

Websites frequently include web pages that are relevant for people with disabilities and accessibility but do not explicitly match the criteria described in the previous sections. These include:

3.3.4 Step 3.d: Include Complete Processes in the Sample

Methodology Requirement 3.d: All Include all web pages that are part of a complete process shall be included. .

The selected sample must include all web pages that belong to a series of web pages presenting a complete process . Also, no web page in the selected sample may be part of a process without all other web pages that are part of that process to be also included into the selected sample.

3.4 Step 4: Audit the Selected Sample

[ Editor Note: This section has been completed and needs to be approved by Eval TF.]

Methodology Requirement 4: The Audit the selected sample of web pages shall be audited according to Methodology Requirement 4.a , Methodology Requirement 4.b , Methodology Requirement 4.c , and optionally, Methodology Requirement 4.d and Methodology Requirement 4.e .

WCAG 2.0 defines five conformance requirements that need to be met for each web page in the sample selected per 3.3 Step 3: Select a Representative Sample . This includes checking whether each WCAG 2.0 Success Criterion in the target conformance level (per 3.1.3 Step 1.c: Define the Conformance Target ) has been met or not met for each of these web pages .

Note: Many web pages will have repetitive content, such as the header, navigation, and other common components that may not need to be re-evaluated on each occurrence. Depending on the level of detail for reporting defined by 3.1.2 Step 1.b: Define the Goal of the Evaluation , an evaluator may not need to continue to identify successes and failures in meeting the conformance target for each these repetitive elements on every web page . Section 3.5 Step 5: Report the Evaluation Findings provides more guidance on reporting.

3.4.1 Step 4.a: Check for the Broadest Variety of Use Cases

[ Editor Note: This section has been completed and needs to be approved by Eval TF.]

Methodology Requirement 4.a: Each Check each web page in the sample selected per 3.3 Step 3: Select a Representative Sample shall be checked for meeting each of the WCAG 2.0 Success Criteria in the target conformance level (per 3.1.3 Step 1.c: Define the Conformance Target ) and each of the WCAG 2.0 conformance requirements .

For each web page in the sample selected per 3.3 Step 3: Select a Representative Sample , check whether each of the WCAG 2.0 Success Criteria in the target conformance level (per 3.1.3 Step 1.c: Define the Conformance Target ) and each of the WCAG 2.0 conformance requirements have been met. This includes that all accessibility features, especially key common functionality defined per 3.2.2 Step 2.b: Identify Key Functionalities Common Functionality of the Website , are usable for the audiences and with the tools defined per 3.1.4 Step 4.d: Define the Context of Website Use .

Note: According to WCAG 2.0, Success Criteria that do not apply to the which there is no matching content are deemed to have been met. satisfied . An outcome such as 'Not Applicable' may be used to denote the particular situation where Success Criteria were satisfied because no relevant content was applicable.

To carry out an evaluation effectively, it is often useful to construct and apply personas, use cases, and scenarios of users with a variety of abilities and using different web browsing techniques, including assistive technology and adaptive strategies. It is critical to consider the broadest possible spectrum of use cases to help identify issues that may occur to different audiences. It is strongly recommended to also involve real users during this process, to help identify issues that may not be easily identified through expert testing alone. See Involving Users in Evaluating Web Accessibility for more guidance.

This assessment also needs to be complemented with focused testing of particular situations including:

Note: Templates are usually often used to create many web pages , sometimes entire parts of a website . While evaluating templates is optional in this methodology, in some contexts it can be helpful to check templates on their own. Evaluating the templates that are identified per 3.3.1 Step 3.a: Include Key Web Pages of the Website may identify potential issues that may not be easily identified through evaluating individual instances of web pages . However, issues identified in templates alone do not necessarily imply that these issues occur on the website and need to be validated on individual instances of web pages . Also, identifying no issues in templates does not necessarily imply that no issues occur on on individual instances of web pages .

3.4.2 Step 4.b: Use WCAG 2.0 Techniques Where Possible (Optional)

[ Editor Note: This section has been completed is currently under discussion with WCAG WG and needs to be approved by Eval TF.] TF, and we expect to refine it in the next Editor Draft.]

Methodology Requirement 4.b: Where possible, use applicable WCAG 2.0 techniques shall be used to demonstrate successes and failures in meeting the WCAG 2.0 Success Criteria relevant per 3.1.3 Step 1.c: Define the Conformance Target . (Optional).

While The initial sets or sources of WCAG 2.0 techniques are only informative to be used during evaluation may be defined in section 3.1.5 Step 1.e: Define the context of Techniques to be Used (Optional) . However, during evaluation such an initial set may often need to be refined according to the particular situation, such as for evaluating particular web technologies and accessibility features.

WCAG 2.0, they provide an effective way 2.0 defines three types of demonstrating whether techniques:

For each web page are met, it can be assumed that a WCAG 2.0 Success Criterion is:

In some situations, for example WCAG 2.0 techniques are not exhaustive as they cannot cover every possible situation. Also, the techniques used to meet WCAG 2.0 Success Criteria during the development may not be known to the evaluator. Particularly for newly released web technologies or when these web technologies are used in particular contexts, contexts there may be no public publicly or proprietary documented techniques available to the evaluator . In this case the The evaluator must determine whether the WCAG 2.0 Success Criteria are met without the use be considerate of techniques. Otherwise it is good practice (for efficacy and justifiability) to use existing techniques to demonstrate successes and failures in meeting these limitations when using WCAG 2.0 Success Criteria. techniques

Note: The initial sets or sources of Advisory techniques are determined per 3.1.5 Step 1.e: Define the Techniques to be Used . However, these may need to not be extended depending on fully supported by assistive technology. If they are used, make sure that these work with the findings made per 3.2 web browsers and assistive technology defined in 3.1.4 Step 2: Explore 1.d: Define the Target Context of Website Use .

3.4.3 Step 4.c: Assess Accessibility Support for Technologies Features

[ Editor Note: This section has been completed and needs to be approved by Eval TF.]

Methodology Requirement 4.c: Each use of the web technologies used to create Check if accessibility features provided on the website shall be checked to be are accessibility supported by the tools defined in Step 1.d: Define the Context of Website Use .

To ensure that the accessibility features provided on a website such as text-alternatives, captions, keyboard access are actually usable in practice, each use of the web technologies used to create the website these accessibility features must be accessibility supported by the tools defined in Step 1.d: Define the Context of Website Use .

In some situations WCAG 2.0 techniques and repositories on accessibility support provide insights on the level of accessibility support for accessibility features in particular combinations of web technologies and tools. However, the evaluator is responsible for the accuracy of the assessment of accessibility support and the resulting evaluation.

3.4.4 Step 4.d: Archive Web Pages for Reference (Optional)

[ Editor Note: This section has been completed and needs to be approved by Eval TF.]

Methodology Requirement 4.d: Every Archive every evaluated web page evaluated shall be archived for reference. (Optional)

The Archive web pages evaluated should be archived for future reference. At the very least, this should include the title and URI of the web page and the date of the evaluation. To avoid ambiguity, complement this should be complemented with any of the following:

3.4.5 Step 4.e: Record Evaluation Tools and Methods (Optional)

[ Editor Note: This section has been completed and needs to be approved by Eval TF.]

Methodology Requirement 4.e: The Record the tools and methods used to evaluate the web pages shall be recorded. . (Optional).

The For future reference, record Evaluation tools and methods used to evaluate the web pages should be recorded for future reference. . This includes versions of web accessibility evaluation tools, web browsers and add-ons, and other tools used during the evaluation. Depending on the level of detail for reporting defined by 3.1.2 Step 1.b: Define the Goal of the Evaluation , this recording may apply globally for the entire evaluation, to individual web pages , or to individual checks carried out within the audited web pages .

3.5 Step 5: Report the Evaluation Findings

[ Editor Note: This section has been completed and needs to be approved by Eval TF.]

Methodology Requirement 5: The Document the evaluation findings shall be documented according to Methodology Requirement 5.a and optionally Methodology Requirement 5.b , Methodology Requirement 5.c , Methodology Requirement 5.d , Methodology Requirement 5.e , and Methodology Requirement 5.f .

This section describes how to report the evaluation findings that have been gathered during the previous steps. Reporting your findings is a key element of every evaluation and helps facilitate replicability of the results.

Note: Individual pieces of the reporting are gathered throughout the evaluation process, not necessarily at the end of it.

3.5.1 Step 5.a: Provide Documentation for Each Step

[ Editor Note: This section has been completed and needs to be approved by Eval TF.]

Methodology Requirement 5.a: Each Document each outcome of the steps defined in 3.1 Step 1: Define the Evaluation Scope , 3.2 Step 2: Explore the Target Website , 3.3 Step 3: Select a Representative Sample , and 3.4 Step 4: Audit the Selected Sample shall be documented. .

Document the outcomes of each of the previous steps defined in 3.1 Step 1: Define the Evaluation Scope , 3.2 Step 2: Explore the Target Website , 3.3 Step 3: Select a Representative Sample , and 3.4 Step 4: Audit the Selected Sample , to ensure justifiable, transparent, and repeatable evaluation results. In particular, this includes documenting:

Documentation of the outcome of auditing the representative sample depends on the level of detail for reporting defined by 3.1.2 Step 1.b: Define the Goal of the Evaluation . Outcome of the auditing must include the following information depending on the set goal (optional templates for reporting example reports are provided in Appendix C: Reporting Templates Example Reports ):

Basic Report
Only captures the successes and failures in meeting WCAG 2.0 conformance requirements globally for the entire website . For each WCAG 2.0 Success Criterion applicable as per 3.1.3 Step 1.c. Define the Conformance Target , the report identifies if it is met or not met in the selected sample of web pages . Where failures in meeting WCAG 2.0 Success Criteria are identified, each web page in which such a failure has been identified must be indicated in the report.
Detailed Report
Captures the successes and failures in meeting WCAG 2.0 conformance requirements for each web page . For each WCAG 2.0 Success Criterion applicable as per 3.1.3 Step 1.c. Define the Conformance Target , the report identifies if it is met or not met in each web page in the selected sample of web pages . Where failures in meeting WCAG 2.0 Success Criteria on a web page are identified, each identified occurrence of such a failure must be indicated in the report.
In-Depth Analysis
Provides the same information as a detailed report with additional analysis of the successes and failures identified for the website . This includes a summary of the positive and negative aspects identified on the website , examples of frequently occurring issues and an assessment of their impact on the users of the website , and suggestions for improving the overall accessibility of the website .

Note: While such a report is required for conformance with this methodology, it is not required for any parts of this report to be made public unless an optional public accessibility statement is provided per Step 5.b: Provide an Accessibility Statement (optional) . The level of confidentiality for evaluation reports is usually determined by the evaluation commissioner .

3.5.2 Step 5.b: Provide an Accessibility Evaluation Statement (optional)

[ Editor Note: This section has been completed and needs to be approved by Eval TF.] [ Review Note: Eval TF is particularly looking for feedback on this section.]

Methodology Requirement 5.b: An accessibility statement shall be provided. Provide an Accessibility Evaluation Statement. (Optional).

WCAG 2.0 conformance claims can only be made for the web pages that have been actually evaluated and identified to conform with its conformance requirements . Accessibility evaluation statements for entire websites can be made according to this methodology when:

As per the requirements for WCAG 2.0 conformance claims , also accessibility evaluation statements according to this methodology must include the following information:

  1. Date of the claim
  2. Guidelines title, version and URI : "Web Content Accessibility Guidelines 2.0 at http://www.w3.org/TR/WCAG20/ "
  3. Conformance level satisfied: (Level A, AA or AAA)
  4. A concise description of the Web pages , such as a list of URIs for which the claim is made, including whether subdomains are included in the claim.
  5. A list of the web technologies relied upon .

Accessibility evaluation statements according to this methodology can also be made when only partial conformance has been achieved according to the requirements defined in third-party content and languages lacking accessibility support . In such cases the accessibility evaluation statements must also include information to identify the following:

  1. Website areas that do not conform to WCAG 2.0;
  2. Reason for not conforming to WCAG 2.0: "third-party content" or "lack of accessibility support;

3.5.3 Step 5.c: Provide a Performance Score (optional)

[ Editor Note: This section has been completed and needs to be approved by Eval TF.] [ Review Note: Eval TF is particularly looking for feedback on this section.]

Methodology Requirement 5.c: A Provide a performance score shall be provided. score. (Optional).

Performance scores can provide more granular measures on the level of conformance of a website than the conformance levels defined by WCAG 2.0 provide. However, scores alone do not provide sufficient context and information to understand the actual level of accessibility of a website . Performance scores according to this methodology may only be provided when the evaluation carried out conforms with this methodology as per 5. Conformance with this Methodology .

Depending on the requested level of detail, the performance score is calculated through one of the following approaches:

Per Website
For each WCAG 2.0 Success Criterion applicable as per 3.1.3 Step 1.c. Define the Conformance Target , calculate the number of Success Criteria that were met on all web pages within the selected sample as per 3.3 Step 3: Select a Representative Sample . The Success Criteria are not met for web pages on which any of the WCAG 2.0 conformance requirements are not met. The score is the ratio of the Success Criteria met over all applicable Success Criteria.
Per Web Page
For each web page within the selected sample as per 3.3 Step 3: Select a Representative Sample , calculate the number of applicable Success Criteria as per 3.1.3 Step 1.c. Define the Conformance Target that were met on each web page . All Success Criteria are not met for web pages on which any of the WCAG 2.0 conformance requirements are not met. The score is the average ratio of the Success Criteria met over all applicable Success Criteria for all web pages in the selected sample.
Per Instance
For each web page within the selected sample as per 3.3 Step 3: Select a Representative Sample , calculate the number of success and failure instances in meeting the applicable Success Criteria as per 3.1.3 Step 1.c. Define the Conformance Target that were met on each web page . All instances where Success Criteria are applicable are not met for web pages on which any of the WCAG 2.0 conformance requirements are not met. The score is the ratio of all success and failure instances in meeting the applicable Success Criteria for all web pages in the selected sample.

Note: According to WCAG 2.0, Success Criteria that do not apply to the content are deemed to have been met. Web accessibility evaluation tools may be needed during the evaluation process to help count and calculate the scores.

The approach used to calculate the score must be indicated together with the numeric ratio whenever the score is provided. For example, "X/Y Per Website", "X/Y Per Web Page", and "X/Y Per Instance" are valid statements to express the performance score.

3.5.4 Step 5.d: Provide Machine-Readable Reports (optional)

[ Editor Note: This section has been completed and needs to be approved by Eval TF.]

Methodology Requirement 5.f: Machine-readable reports shall be provided. Provide machine-readable reports. (Optional).

Machine-readable reports facilitate processing the results by tools, for example to help monitor accessibility of a website over time. The Evaluation and Report Language ( EARL ) is a machine-readable format that was specifically designed for this purpose. It is recommended to use EARL for providing machine-readable reports. See also Understanding Metadata to learn more about uses of metadata, including machine-readable reports, such as EARL .

4. Considerations for Particular Situations

[ Editor Note: This section has been added and needs to be approved by Eval TF.]

The methodology defined by this document is flexible to allow its implementation in different situations and contexts. This section provides guidance for applying this methodology in some of these situations.

4.1 Initial Conformance Assessment of a Website

[ Editor Note: This section has been added and needs to be approved by Eval TF.]

When getting started with improving website accessibility it is useful to do a conformance evaluation to assess the current state of the website . In such situations it is often useful to expand the scope of the evaluation beyond the planned conformance target to get a more complete picture of the accessibility performance on the website . Expanding the scope could include setting the goal of the evaluation in 3.1.2 Step 1.b: Define the Goal of the Evaluation to " In-Depth Analysis " and the conformance target in 3.1.3 Step 1.c. Define the Conformance Target to " WCAG 2.0 Level AAA".

Note: It is recommended to carry out preliminary reviews before carrying out a conformance evaluation to identify obvious errors and to develop a rough understanding of the overall performance of the website .

4.2 Evaluating a Large Website with Separate Parts

[ Editor Note: This section has been added and needs to be approved by Eval TF.]

Evaluation of large websites according to this methodology may be broken down into the evaluation of individual website areas, such as the online shop, departments within an organization, blogging area, and other sub-sites. In this case, each sub-site must meet the terms established in section 2.1 Scope of Applicability and must be evaluated using this methodology. Evaluation of each sub-site must be carried out to at least the same conformance target (as per 3.1.3 Step 1.c. Define the Conformance Target ) as for the main website .

An additional evaluation must be carried out according to this methodology, whereby the selected sample (as per 3.3 Step 3: Select a Representative Sample ) must include at least all common web pages of the main website plus two randomly selected web pages from the representative sample selected for each sub-site. The reporting, scoring, and accessibility statements per 3.5 Step 5: Report the Evaluation Findings applies to the entire set of sampled web pages from the main website and all its sub-sites.

Sub-sites may be broken down into further sub-site where applicable. In this case this same clause of the methodology is applied.

4.3 Re-Running a Website Conformance Evaluation

[ Editor Note: This section has been added and needs to be approved by Eval TF.]

Conformance evaluation according to this methodology may be re-run after a short period, for example when issues are identified and repaired by the website owner or website developer , or periodically to monitor progress. In such cases the selected sample (as per 3.3 Step 3: Select a Representative Sample ) must include a different set of exemplar web page instances (as per 3.3.2 Step 3.b: Include Exemplar Instances of Web Pages ), where possible. This allows a portion of the sample to remain unchanged ( common web pages , key web pages and functionality, etc.), common functionality ), which facilitates comparability between the results.

4.4 Large-Scale Evaluation of Multiple Websites

[ Editor Note: This section has been added and needs to be approved by Eval TF.]

Carrying out full conformance evaluation of multiple websites can involve a lot effort. This methodology is can not limited to be undertaken using just automated evaluation, which can only evaluate a small portion of the accessibility requirements, and it also requires manual evaluation by experts. Limiting the goal of the evaluation in 3.1.2 Step 1.b: Define the Goal of the Evaluation to " Basic Reporting " and careful selection of the samples in 3.3 Step 3: Select a Representative Sample can optimize the effort required for each evaluation.

5. Conformance with Application of this Methodology

[ Editor Note: This section has been completed and needs to be approved by Eval TF.]

The methodology defined by this document is flexible to allow its implementation in different situations and contexts. It is not required to carry out any of the steps and activities defined by this methodology in any particular sequence. However, it is required that For proper application of this methodology, use all the following requirements defined by this methodology are met: Methodology Requirements:

Results, reports, accessibility statements, and performance scores can only be claimed to be in accordance with this methodology when the evaluation carried out meets these requirements. Methodology Requirements.

Appendices

Appendix A: Acknowledgements

This publication has been developed with support from the European Commission funded WAI-ACT Project ( IST 287725) .

Contributors

Past and present active participants of the WCAG 2.0 Evaluation Methodology Task Force (Eval TF) include: Shadi Abou-Zahra; Frederick Boland; Denis Boudreau; Amy Chen; Vivienne Conway; Bim Egan; Michael Elledge; Wilco Fiers; Detlev Fischer; Elizabeth Fong; Vincent François; Alistair Garrison; Emmanuelle Gutiérrez y Restrepo; Katie Haritos-Shea; Martijn Houtepen; Peter Korn; Maureen Kraft; Aurelien Levy; Kerstin Probiesch; Donald Raikes; Corominas Ramon; Roberto Scano; Samuel Sirois; Sarah J Swierenga; Eric Velleman; Konstantinos Votis; Kathleen Wahlbin; Elle Waters; Richard Warren; Léonie Watson.

Appendix B: References

[ Editor note: List needs correct formatting and updating.]
QA-FRAMEWORK
Dubost K, Rosenthal L, Hazaël-Massieux D, Henderson L (eds) (2005) W3C Quality Assurance Framework: Specification Guidelines. W3C. Available at: http://www.w3.org/TR/qaframe-spec/
RFC3986
Fielding R, Gettys J, Mogul J, Frystik H, Masinter L, Berners-Lee T (eds) (1999). Hypertext Transfer Protocol - HTTP/1.1. Request for Comments: 2616. IETF. Available at: http://www.ietf.org/rfc/rfc2616.txt http://www.ietf.org/rfc/rfc3986.txt
RFC2616
Berners-Lee T, Fielding R, Masinter L (eds) (2005). Uniform Resource Identifier (URI): Generic Syntax. Request for Comments: 3986. IETF. Available at: http://www.ietf.org/rfc/rfc3986.txt http://www.ietf.org/rfc/rfc2616.txt
UWEM
Velleman E.M, Velasco C.A, Snaprud M (eds) (2007). D-WAB4 Unified Web Evaluation Methodology (UWEM 1.2 Core). Wabcluster. Available at: http://www.wabcluster.org/uwem1_2/
WCAG20
Caldwell B, Cooper M, Guarino Reid L, Vanderheiden G, eds (2007). Web Content Accessibility Guidelines 2.0. W3C. Available at: http://www.w3.org/TR/WCAG20/

Appendix C: Reporting Templates Example Reports

Editor note: Review Note: A complete example template for evaluation reporting in human understandable language. Discussion proposal below is built upon the idea to have three different levels of reporting. The template will have to be ok for official evaluations so some change is still necessary. The difference between option 1 and 2 is currently very small (open for discussion).This section needs appending to the requirements Methodology Requirements in section 4. this section may change in later drafts depending on how the other sections evolve.

This Appendix proposes three different levels of reporting following the discussion in clause 5 on levels of evaluation. The three examples are:

The examples should at least have the following information:

Example 1

Results

Guideline X (heading)

Checkpoint: SC XX (subheading)

Result: pass/fail

Character: global/regional (or another wording) if regional: a list of pages where the problem exists

General information about this checkpoint: link to how to meet and to understanding this success criteria.

List of user agents, including assistive technologies that where used during the evaluation of the website and their versions.

List of pages in the sample

Example 2

Results

Guideline X (heading)

Checkpoint: SC XX (subheading)

Result: pass/fail

Character: global/regional (or another wording) if regional: a list of pages where the problem exists

Description of existing problems and barriers for users or link to descriptions on W3C/WAI website.

General information about this checkpoint: link to how to meet and to understanding this success criteria.

List of user agents, including assistive technologies that where used during the evaluation of the website and their versions.

List of pages in the sample

Example 3

Results

Guideline X (heading)

Checkpoint: SC XX (subheading)

Result: pass/fail

Character: global/regional (or another wording) if regional: a list of pages where the problem exists

Description of existing problems and barriers for users or link to descriptions on W3C/WAI website..

General information about this checkpoint: link to how to meet and to understanding this success criteria.

Action: Description of techniques for meeting the SC (could be techniques which are already in the techniques document or new techniques which are not in the document, but with which the SC can be met).

List of user agents, including assistive technologies that where used during the evaluation of the website and their versions.

List of pages in the sample

Appendix D: Document Changes

Changes to the Public Working Draft of 27 March include:

A full disposition of comments of all the comments received on the 27 March Working Draft are available.