[contents]

Abstract

This document specifies an internationally harmonized methodology for evaluating the accessibility conformance of websites to the Web Content Accessibility Guidelines (WCAG) 2.0. deleted content: {This methodology supports} added content: [It defines an approach for] conformance evaluation in different contexts, including self-assessment and third-party evaluation. It is applicable to any website, including web applications and websites for mobile devices, and is independent of any particular evaluation tools, web browsers, and assistive technologies.

added content: [This document is a supporting resource for WCAG 2.0 and does not replace or supersede it in any way. It addresses the need for a standardized approach for evaluating entire websites after their development as opposed to evaluating individual collections of web pages during their development, which is equally important to ensure website accessibility. This work is part of W3C/WAI activities on Web Accessibility Evaluation and Testing that address different W3C/WAI guidelines and specifications.]

Status of this document

This clause describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

This 20 March 2012 Editors Draft (@@ First Public Working Draft @@) of Website Accessibility Conformance Evaluation Methodology 1.0 is intended to be published and maintained as a W3C Working Group Note after review and refinement. deleted content: {This version reflects current discussion and input from the WCAG 2.0 Evaluation Methodology Task Force (Eval TF). It provides a new direction and approach taken by Eval TF to provide guidance for evaluators who want to evaluate accessibility conformance to WCAG 2.0.} added content: [It provides an initial outline for the evaluation methodology to invite feedback on the direction and approach, in particular on its scope of applicability. Some latter sections of this document are currently empty and will be completed in future drafts, also based on the feedback received on this draft.]

The WCAG 2.0 Evaluation Methodology Task Force (Eval TF) invites discussion and feedback deleted content: {about} added content: [on] this document by developers, evaluators, researchers, and other added content: [s with] deleted content: {practitioners who have} interest in web accessibility evaluation. In particular, Eval TF is looking for feedback on added content: [:] deleted content: {section added content: [s 1, 2 and] 3 and the changes to the other sections compared to the previous version.}

Please send comments on this Website Accessibility Evaluation Methodology for WCAG 2.0 document to public-wai-evaltf@w3.org (publicly visible mailing list archive).

Publication as Editor Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

Table of Contents

  1. Introduction
  2. Using this Methodology
  3. Evaluation Procedure
  4. Conformance with this Methodology
  5. Limitations of this Methodology

Appendices

1. Introduction

Website owners, procurers, suppliers, developers, and others are frequently tasked with assessing the conformance of an existing website to the Web Content Accessibility Guidelines (WCAG) 2.0. Because it is generally not feasible nor desired to evaluate every single web page in such a context, it is essential to employ a reliable method for determining the overall conformance of a website.

This document constitutes a standardized methodology for evaluating the conformance of websites, including web applications and websites for mobile devices, to WCAG 2.0. It is developed through the open, collaborative W3C Process with the involvement of international stakeholders. It is a supporting document for WCAG 2.0 and does not replace or supersede it in any way.

1.1 Scope of this Document

The methodology described in this document supports conformance evaluation in different contexts including self-assessment and third-party assessment of websites. It is applicable to all websites, including web applications, websites for mobile devices, and other types of websites. It is independent of the size of the website, the web technologies used to create the website, and of any particular web accessibility evaluation tools, web browsers, assistive technologies, and other software tools.

While this document provides useful guidance for evaluation throughout the website development process it focuses primarily on the last stage of the process, in which confirmation is sought on whether conformance targets have been achieved (commonly referred to as acceptance testing). The document presents concrete steps to define the scope of the evaluation, select a representative sample, audit the selected sample and aggregate and report the results of the evaluation findings in a structured and uniform way.

1.2 Target Audience

The primary target audience for this methodology are individuals and organizations, wanting to evaluate the conformance of existing websites to WCAG 2.0. This includes but is not limited to:

Other audiences for whom this methodology is also relevant include:

Users of the methodology are assumed to have expertise in WCAG 2.0, accessible web design, assistive technologies, and of how people with different disabilities use the Web; read more in section 2. Using this Methodology.

1.3 Background Reading

It is assumed that the reader of this document is familiar with the following added content: [W3C] resources:

Evaluating Websites for Accessibility

A multi-page resource suite that outlines different approaches for evaluating websites for accessibility. The following resources are particularly important in this context:

Web Content Accessibility Guidelines (WCAG) 2.0

Internationally recognized standard explaining how to make web content more accessible to people with disabilities. The following resources are particularly important in this context:

Accessible Web Design

The following documents introduce the essential components of web accessibility, inclusive design for people with disabilities and deleted content: {older people} added content: [people with aging-related impairements], and overlap with design for mobile devices. The documents help managers, designers, developers, policy makers, researchers, and others optimize their efforts in these overlapping areas:

1.4 Terms and Definitions

For the purposes of this document, the following terms and definitions apply:

added content: [Complete Processes]
added content: [When a Web page is one of a series of Web pages presenting a process (i.e., a sequence of steps that need to be completed in order to accomplish an activity), all web pages in the process conform at the specified level or better. (Conformance is not possible at a particular level if any page in the process does not conform at that level or better.), from WCAG 2.0 Conformance Requirement for Complete Processes.]
deleted content: {Elemental} added content: [Common] Web Pages
deleted content: {Fundamental web} added content: [Web] pages that are relevant to the entire website. This includes the homepage, login page, and other entry pages, and, where applicable, the sitemap, contacts, site help, legal information, and similar web pages that are typically linked from all web pages (usually from the header, footer, or navigation menu of a web page).
Evaluator
The person, team of people, organization, in-house department, or other entity responsible for carrying out the evaluation.
Evaluation Commissioner
The person, team of people, organization, in-house department, or other entity that commissioned the evaluation.
Note: In many cases the evaluation commissioner may be the website owner, in other cases it may be another entity such as a procurer or an accessibility monitoring survey.
Key Functionality
Primary functionality of a website including tasks that users of a website carry out to perform functionality.
Note: Examples of functionality include "selecting and purchasing a product from the shop area of the website", "filling and submitting the form provided on the website", deleted content: {or "contributing comments in the forum area of the website"} added content: [and "registering for an account on the website"].
added content: [Relied Upon (Technologies)]
added content: [The content would not conform if that technology is turned off or is not supported, from WCAG 2.0 definition for "relied upon".]
Website developer
Anyone involved in the website development process including but not limited to content authors, designers, programmers, quality assurance testers, and project managers.
Website Owner
The person, team of people, organization, in-house department, or other entity that is responsible for the website.
Web page
A non-embedded resource obtained from a single URI using HTTP plus any other resources that are used in the rendering or intended to be rendered together with it by a user agent, from WCAG 2.0 definition for "Web Page".
Website
A coherent collection of one or more related web pages that together provide common use or functionality. It includes static web pages, dynamically generated web pages, and web applications.
added content: [Website part]
added content: [Web pages that serve the purpose and functionality of a website. This includes web pages that are part of the navigation, design, and complete processes of a website.]

2. Using This Methodology

The methodology defined by this document is used for comprehensively assessing the conformance of websites to WCAG 2.0. A more cursory approach for exploring the accessibility of a website is described in Preliminary Review of Websites for Accessibility. In many cases it is advisable to carry out such preliminary reviews before applying this methodology, for example to identify obvious errors and to develop a rough understanding of the overall performance of the website.

2.1 Scope of Applicability

The methodology defined by this document applies to full, self-enclosed websites. Examples of such websites include:

While a website in this context can be part of a larger website, the methodology always applies to a full website without exclusions or omissions of added content: [website parts]. For example, the methodology could be applied to the website of a university or the library of that university, but the methodology may not be applied to the website of the university excluding the website of the library added content: [, if the library website is considered to be an inherent website part of the university website].

It is also not possible to exclude individual web pages or functions such as an embedded mapping application or a linked credit card authorization form as this would conflict with the added content: [complete processes] requirement deleted content: {of WCAG 2.0} (the WCAG 2.0 concept for conforming alternate versions added content: [and non-interfence] may be useful in some deleted content: {of these} situations). Similarly, for websites that are parts of other websites it is also not possible to exclude shared web pages that are part of both websites, such as a common contacts page or site navigation.

Exception: The methodology can be applied to clearly separable areas of a single website, such as to the public and restricted area of a website deleted content: {or the front-end and back-end of a web-based tool}, provided that this scope matches the evaluation goals and the context of website use deleted content: {; read more in section 3 Step 1: Define the Evaluation Scope of the evaluation procedure}.

added content: [Determining the scope of a particular website is further explained in Step 1: Define the Evaluation Scope of section 3.]

added content: [Editor Note: Eval TF is aware that this scope definition may limit some websites from claiming full conformance according to this methodology, because they may have website parts (such as third-party content) that are known to be inaccessible. Step 5: Report the Evaluation Findings of this methodology will provide guidance on how to report what aspects of a website conform rather than to exclude these parts from the scope. Eval TF welcomes feedback from the public on this aspect.]

2.2 Required Expertise

Users of the methodology defined by this document are assumed to be knowledgeable of WCAG 2.0, accessible web design, assistive technologies, and of how people with different disabilities use the Web. This includes understanding of the relevant web technologies, barriers that people with disabilities experience, assistive technologies and approaches that people with disabilities use, and evaluation techniques and tools to identify potential barriers for people with disaibilities. In particular, it is assumed that users of this methodology are deeply familiar with the resources listed in section 1.3 Background Reading.

2.3 Evaluation Tools (Optional)

The methodology defined by this document is independent of any particular web accessibility evaluation tool, web browsers, and other software tools. However, web accessibility evaluation tools can significantly assist an evaluator during the evaluation process and contribute to more effective evaluations. For example, some web accessibility evaluation tools can scan entire websites to help identify relevant pages for manual evaluation, and other tools can assist manual evaluation in a variety of ways. Specific guidance is provided in Selecting Web Accessibility Evaluation Tools.

Note: Web accessibility evaluation tools can only automatically check a limited set of accessibility requirements that are automatable. Conformance evaluation requires manual testing and judgment by experienced evaluators.

2.4 Review Teams (Optional)

The methodology defined by this document can be carried out by an individual evaluator with the skills described in section 2.2 Required Expertise. However, using the combined expertise of review teams provides better coverage for the required skills and helps identify accessibility barriers more effectively. While not required, it is strongly recommended to employ review teams for conformance evaluation of websites. Specific guidance is provided in Using Combined Expertise to Evaluate Web Accessibility.

2.5 Involving Users (Optional)

The methodology defined by this document can be carried out by an individual evaluator with the skills described in section 2.2 Required Expertise. However, involving people with disabilities and deleted content: {older people} added content: [people with aging-related impairements] helps identify additional accessibility barriers that are not easily discovered by the evaluator alone. While not required, it is strongly recommended to involve real people added content: [covering a wide range of abilities] during the evaluation process. Specific guidance is provided in Involving Users in Web Accessibility Evaluation.

3. Evaluation Procedure

This section defines the steps of the evaluation procedure. The specifics of these steps may shift slightly depending on the type of website, the purpose of the evaluation, and the process used by the evaluator. Some of the steps may also overlap or be carried out in parallel but it is important that all steps are considered to ensure a reliable assessment. It is also important to clearly document each step to ensure verifiable results and justifiable conformance claims.

Step 1: Define the Evaluation Scope

Requirement 1: added content: {he} added content: [The] evaluation scope shall be defined according to Requirement 1.a, Requirement 1.b, Requirement 1.c, Requirement 1.d, and optionally Requirement 1.e.

During this step the overall scope of the evaluation is defined. It is a fundamental step that affects the subsequent steps in the evaluation procedure and is ideally carried out together with the evaluation commissioner (who may be the website owner) to ensure common expectations deleted content: {of what the evaluation should cover} added content: [about the scope of the evaluation].

Step 1.a: Define the Scope of the Website

Requirement 1.a: The scope of the website shall be defined, with the definition not contradicting the constraints expressed in section 2.1 Scope of Applicability.

During this step the exact scope of the website is defined and documented. The scope definition of the website may not contradict the scope of applicability of this methodology, as defined in section 2.1 Scope of Applicability. Examples of scope definitions for websites include:

added content: [Determining the scope of a website may require some knowledge about the ownership and development process for some of the website parts. For example, the web shop of "Example Inc." may have a different design than the main website of the organization or it may be hosted at a completely different web address even though it is essential to the purpose of the organizations' website. In this case the web shop would qualify as a website part and would need to be included in the scope definition.]

Note: This step may be carried out partially with the explorations explained in Step 2: Explore the Target Website, for example to validate that the scope definition does not contradict the scope of applicability of this methodology added content: [, and to select the complete processes that are part of the website that is included into the scope].

Step 1.b: Define the Goal of the Evaluation

Requirement 1.b: The goal of the evaluation shall be defined as "Basic Conformance", "Detailed Review", "In-Depth Analysis", or "Other". In the case of "Other" the exact goal must be clearly defined.

Defining the goal of the evaluation is particularly relevant to the auditing stage defined in Step 4: Audit the Selected Sample and the reporting stage defined in Step 5: Report the Evaluation Findings. Some of the evaluation goals include:

Basic Conformance
Only identifies whether a website conforms or not, but does not provide any additional information. This type of evaluation is typically carried out when the website is assumed to conform, for example to verify a conformance claim, and for large-scale evaluations with less resources to explore the details of individual websites.
Detailed Review
Identifies whether a website conforms or not, and provides further information about any identified errors. deleted content: {This} added content: [It] includes count deleted content: {s} added content: [ing] deleted content: {of} the number of errors and their locations within the web pages. This type of evaluation is particularly useful for instructing web developers and for acquiring deleted content: {statics} added content: [statistics] for monitoring progress over time.
In-Depth Analysis
Identifies whether a website conforms or not, and analyzes any identified errors in detail. This includes descriptions of the errors and suggestions for possible repair options. This type of evaluation is particularly useful for organizations that want to improve the accessibility of their website and need a detailed analysis

Step 1.c: Define the Conformance Target

Requirement 1.c: The target WCAG 2.0 conformance level ("A", "AA", or "AAA") to evaluate for shall be defined.

Part of initiating the evaluation process is to define the target WCAG 2.0 conformance level ("A", "AA", or "AAA") to evaluate for. In many situations it is useful to evaluate to WCAG 2.0 Level AAA even if the added content: [intended] conformance target for added content: [a] website is below that deleted content: {, because a} added content: [. A] full evaluation provides a more complete picture of the website performance added content: [and helps plan imporovements]. For example, it can identify accessibility requirements beyond the deleted content: {website} added content: [intended] conformance target deleted content: {that the website conforms to and helps develop more effective plans for improving entire aspects of a website} added content: [to help improve related aspects] (such as improving how tables, forms, widgets, videos, and other components are generated or developed) rather than fixing individual errors one by one.

Step 1.d: Define the Context of Website Use

Requirement 1.d: The target users of the website and minimum set of web browsers and assistive technology to evaluate for shall be defined, noting that the definition of software support shall not conflict with the WCAG 2.0 guidance on the Level of Assistive Technology Support Needed for "Accessibility Support".

Define who uses the website and how they use it. It is particularly important to define if a website is public or restricted to specific users, such as an intranet or a website that requires authentication for use. In some situations there may be more than one mode of use, for example a website with a public area and a restricted area for selected users and websites that dynamically adapt to the particular user profile. Each of those modalities must be identified and clearly documented.

Following the definition for the mode of use it is important to also define the minimum set of web browsers and assistive technology to evaluate for. For example, if it is an intranet with known software that will access the website then this definition will be significantly different from a public website with unknown software. This definition is also influenced by language, geographical location, and the target audience of the website. This definition of software support may not conflict with the WCAG 2.0 guidance on the Level of Assistive Technology Support Needed for "Accessibility Support".

Step 1.e: Define the Techniques to be Used (Optional)

Requirement 1.e: The deleted content: {WCAG 2.0} techniques deleted content: {to be used during the evaluation} added content: [that will be used to evaluate the conformance to WCAG 2.0] should be specified.

added content: [While optional, techniques are an effective way of demonstrating conformance to WCAG 2.0. It is good practice to specify the sets or sources of techniques that are intended to be used during the evaluation at this stage already, to ensure consistent expectation between the evaluator and the evaluation commissioner. This definition is typically refined in later stages of the evaluation process, for example to evaluate web content using particular web technologies and accessibility features.]

deleted content: {While optional, it is usually quite important to define the WCAG 2.0 Techniques that will be used to carry out the evaluation, to ensure consistent expectation between the evaluation commissioner and the evaluator.} W3C/WAI provides a set of publicly documented Techniques for WCAG 2.0 but other techniques may be used too.

Note: This step may overlap with the exploration of the website defined in Step 2: Explore the Target Website. added content: [The exact techniques used to evaluate conformance to WCAG 2.0, if any, must be documented in Step 5: Report the Evaluation Findings.]

Step 2: Explore the Target Website

Requirement 2: The deleted content: {target} website to be evaluated shall be explored according to Requirement 2.a, Requirement 2.b, Requirement 2.c, and Requirement 2.d.

During this step the evaluator explores the target website to be evaluated, to develop a better understanding of the website use and functionality. Carrying out initial cursory checks during this stage already helps identify web pages that are relevant for more detailed evaluation later on. For example, an evaluator may identify web pages that seem to be lacking color contrast, consistent navigation, or document structures (headings, links, etc.) with simple checks while studying the website, and note them down as candidates for selection in the sampling step defined in Step 3: Select a Representative Sample.

deleted content: {For carrying} added content: [To carry] out this step it is critical that the evaluator has access to all the relevant deleted content: {parts of the website} added content: [website parts] to be evaluated. For example, it may be necessary to create accounts or otherwise provide access for evaluators to restricted areas of a website that are part of the evaluation.

Step 2.a: Identify deleted content: {elemental} added content: [Key] Web Pages of the Website

Requirement 2.a: The deleted content: {elemental} added content: [common] web pages of the website added content: [and templates available to the evaluator] shall be identified.

During this step the deleted content: {elemental} added content: [common] web pages of the website added content: [and templates available to the evaluator] are identified and documented. The outcome of this step is a list of all deleted content: {elemental} added content: [common] web pages added content: [and templates available to the evaluator]. These will be part of the sample to be selected through the steps defined in Step 3: Select a Representative Sample. This step also helps understand key aspects of the website, such as the navigation and overall structure of the website.

Step 2.b: Identify Key Functionalit deleted content: {y} added content: [ies] of the Website

Requirement 2.b: The key functionalit deleted content: {y} added content: [ies] of the website shall be identified.

During this step the key functionalit deleted content: {y} added content: [ies] of the website deleted content: {is} added content: [are] identified and documented, to provide a comprehensive basis for the sample to be selected through the steps defined in Step 3: Select a Representative Sample. The outcome of this step is a list of user tasks such as "selecting and purchasing a product from the shop area of the website", "filling and submitting the form provided on the website", deleted content: {or "contributing comments in the forum area of the website"} added content: [and "registering for an account on the website"].

Step 2.c: Identify the Variety of Web Page Types

Requirement 2.c: The variety of web page types shall be identified.

Web pages with varying styles, layouts, structures, and functionality often have different implementations of accessibility features. They are also often generated by different templates and authored by different people. The outcome of this step is a list of the different types of web pages on the website to provide a comprehensive basis for the sample to be selected through the steps defined in Step 3: Select a Representative Sample. Different web page types include:

added content: [Note: This step is intended to identify instance of web pages rather than templates that are identified as part of Step 2.a: Identify Key Web Pages of the Website.]

Step 2.d: Identify added content: [Web] Technologies deleted content: {Used in the Website} added content: [Relied Upon]

Requirement 2.d: The technologies deleted content: {used} added content: [relied upon] to provide the website shall be identified.

During this step the web technologies deleted content: {used} added content: [relied upon] to provide the website are identified and documented, to provide a basis for the sample to be selected through the steps defined in Step 3: Select a Representative Sample. This includes base web technologies such as HTML and CSS, auxillary web technologies such as Flash, JavaScript, PDF, Silverlight and WAI-ARIA, as well as advanced web technologies such as SMIL and SVG. The outcome of this step is a list of web technologies added content: [relied upon] to provide the website.

added content: [Note: Only the technologies that are relied upon to provide the website need to be identified for evaluation. This relates closely to the WCAG 2.0 concepts of conforming alternate versions and non-interfence, and may need to be determined together with the evaluation commissioner.]

Step 3: Select a Representative Sample

Requirement 3: A representative sample of web pages shall be selected from the website according to Requirement 3.a, Requirement 3.b, Requirement 3.c, and Requirement 3.d.

From exploration of the website deleted content: {defined} in Step 2: Explore the Target Website added content: [(within the scope as set in Step 1: Define the Evaluation Scope)] the evaluator should have an understanding of the website, to facilitate the selection of a sample of web pages that is representative of the entire website. Depending on the size and complexity of the website, the size of this sample will vary. For example, a website with few types of web pages that are all generated from confined templates, such as a data repository, will likely require a much smaller sample to evaluate than a website with many types of web pages that are authored using different mechanisms.

The web pages identified through the exploration will typically relate to more than one design aspect. For example, web pages with particular functionality such as scripting, multimedia, and forms will typically also use related technologies such as Flash, JavaScript, PDF, Silverlight, and WAI-ARIA, and in many cases these web pages may have different designs to other pages. A carefull selection of web pages can significantly reduce the required sample size while maintaining appropriate representation of the entire website.

added content: [Note: This step identifies a comprehensive sample of web pages that need to be evaluated. However, many web pages will have repetitive content, such as the header, navigation, and other common components that may not need to be re-evaluated on each occurrence. Guidance on evaluating the sample identified in this step is provided in Step 4: Audit the Selected Sample.]

Step 3.a: Include deleted content: {Elemental} added content: [Key] Web Pages of the Website

Requirement 3.a: All deleted content: {elemental} added content: [common] web pages added content: [and templates available to the evaluator] shall be part of the selected sample of web pages.

The deleted content: {elemental} added content: [common] web pages added content: [and templates available to the evaluator] identified through the step defined in Step 2.a: Identify deleted content: {Elemental} added content: [Key] Web Pages of the Website usually provide key information and functionality for website users and are to be included as part of the selected sample. deleted content: {Additional key web pages such as particularly popular web pages (identified through the server logs, search engine rankings, etc.), web pages that are particularly relevant for people with disabilities, and web pages that explain the functionality of the website should also be included where applicable.}

Step 3.b: Include Exemplar Instances of Web Pages

Requirement 3.b: At least two distinct web pages (where applicable) of each (1) key functionality, (2) content, design, and functionality, and (3) web technologies shall be part of the selected sample of web pages.

From the variety deleted content: {of web pages} and types of web pages identified through the steps defined in Step 2: Explore the Target Website added content: [(within the scope of the evaluation as defined in Step 1: Define the Evaluation Scope)], select at least two distinct web pages with the following features each (where applicable):

deleted content: {Reminder} added content: [Note]: A selected web page could have any number of these features. For example, a selected web page could be used to represent the use of forms and scripting at the same time. The important aspect is to select at least two distinct web pages for each deleted content: {type of form and scripting} added content: [relevant feature] identified on the website added content: [, though more web pages may be necessary depending on the complexity of the website].

Note: When a website is being re-evaluated, it is sometimes useful for comparison to include a portion of web pages from the sample selected for the previous evaluation. However, deleted content: {the majority of} web pages should be freshly selected to provide an accurate representation of the entire website.

Step 3.c: Include Other Relevant Web Pages

Requirement 3.c: Other web pages relevant for people with disabilities and accessibility shall be part of the selected sample of web pages.

Websites frequently include web pages that are relevant for people with disabilities and accessibility but do not explicitly match the criteria described in the previous sections. These include:

Step 3.d: Include Complete Processes in the Sample

Requirement 3.d: All web pages that are part of a added content: [complete] process added content: [shall be included.] deleted content: {for each web page added content: [that is] selected through Requirement 3.a, Requirement 3.b, and Requirement 3.c shall be part of the selected sample of web pages, as per the deleted content: {WCAG 2.0 conformance} requirement for added content: [complete processes].}

deleted content: {The WCAG 2.0 conformance requirement for complete processes reads as follows:}

deleted content: {"When a Web page is one of a series of Web pages presenting a process (i.e., a sequence of steps that need to be completed in order to accomplish an activity), all Web pages in the process conform at the specified level or better. (Conformance is not possible at a particular level if any page in the process does not conform at that level or better.)"}

deleted content: {For this reason,} The selected sample must include all web pages that belong to a series of web pages presenting a added content: [complete] process deleted content: {for each web page selected}. added content: [Also] deleted content: {In other words}, no web page in the selected sample may be part of a process without all other web pages that are part of that process to be also included in the selection.

Step 4: Audit the Selected Sample

[Editor deleted content: {e} Note: This section will provide guidance on applying WCAG 2.0 to the deleted content: {selected} sample added content: [selected in Step 3: Select a Representative Sample. An introduction to this section will be inserted here. added content: [The introduction will include advice on how to avoid evaluation of repetitive content that would not yield new findings; focus emphasis on Success Criteria rather than on techniques; and reference relevant WCAG 2.0 concepts such as accessibility support, conforming alternatives, and non-intereference.]]

Step 4.a: Check for the Broadest Variety of Use Cases

[Editor deleted content: {e} Note: This section will instruct evaluators to apply all WCAG 2.0 Success Criteria for each web page in the selected sample, and in doing so to consider the broadest possible set of users, software tools, and use cases as scoped out in Step 1.d: Define the Context of Website Use.]

Step 4.b: deleted content: {Apply Documented WCAG 2.0 Techniques} added content: [Use WCAG 2.0 Techniques Where Possible]

[Editor deleted content: {e} Note: This section will instruct evaluators to use deleted content: {documented} WCAG 2.0 techniques added content: [where possible] for auditing added content: [the conformance of] the selected web pages added content: [to WCAG 2.0 Success Criteria], ideally those initially defined in Step 1.e: Define the Techniques to be Used (Optional).]

Step 4.c: Assess Accessibility Support for Techniques

[Editor deleted content: {e} Note: This section will instruct evaluators to assess accessibility support for WCAG 2.0 Techniques used and accessibility features identified in the audited web pages, ideally those initially defined in Step 1.d: Define the Context of Website Use.]

Step 4.d: Archive Web Pages for Reference (Optional)

[Editor deleted content: {e} Note: This section will advise evaluators to archive the audited web pages in any form (screenshots, saving copies of the web page, etc.) for future references and in case of disputes. This step is optional.]

Step 4.e: Record Evaluation Tools and Methods (Optional)

[Editor deleted content: {e} Note: This section will advise evaluators to record the tools and methods used for auditing the web pages for future references and in case of disputes. This step is optional.]

Step 5: Report the Evaluation Findings

[Editor deleted content: {e} Note: This section will provide guidance on reporting the evaluation findings. An introduction to this section will be inserted here.]

Step 5.a: Provide Documentation for Each Step

[Editor deleted content: {e} Note: This section will provide guidance on documenting each step and will link to Appendix C: Template Reports for different types of sample reports. The Methodology will not require such documentation to be public, as it is up to the website owner and/or evaluation commissioner to choose disclosing it.]

Step 5.b: Provide an Accessibility Statement (optional)

[Editor deleted content: {e} Note: This section will provide guidance on providing public accessibility statements referencing the outcomes of an evaluation according to this Methodology. This step is optional. However, in this case the website owners and/or evaluation commissioners choose to make a public conformance statement according to this Methodology so that certain restrictions can be applicable. For instance, it may be important to at least include or reference information about the scope defined in Step 1: Define the Evaluation Scope in such a public statement.]

Step 5.c: Provide a Performance Score (optional)

[Editor deleted content: {e} Note: This section will provide guidance on calculating an overall performance score. This step is optional. There is currently no particular alogrithm for this calculation but it is likely that this step may only be available to evaluations of type Detailed Review and above, to ensure sufficient data for aggregating a useful score.]

Step 5.d: Provide Information on the Findings (optional)

[Editor deleted content: {e} Note: This section will provide guidance on providing additional information on any identified errors, using existing WCAG 2.0 materials where possible. This step is optional.]

Step 5.e: Provide Suggestions for Repairs (optional)

[Editor deleted content: {e} Note: This section will provide guidance on providing suggestions for repairing any identified errors, using existing WCAG 2.0 materials where possible. This step is optional.]

Step 5.f: Provide a Machine-Readable Report (optional)

[Editor deleted content: {e} Note: This section will provide guidance on providing machine-readable reports in addition to a human-readable form. It will include guidance on using the Evaluation and Report Language (EARL) for this purpose. This step is optional.]

4. Conformance with this Methodology

[Editor Note: This section defines requirements, in addition to the requirements already identified in section Evaluation Procedure, for conforming with this methodology. It allows evaluators to use other evaluation procedures while ensuring that these conform with this methodology.]

5. Limitations of this Methodology

[Editor Note: This section will discuss the limitiations of this methodology, in particular that such evaluations are always an approximation to the true conformance (which would require evaluating every single page on a website). It will also highlight aspects of applicability, such as large-scale evaluations that may not be able to include extensive samples so that they would possibly meet the requirements of this methodology. Suggestions for other considerations for this section are requested.]

Appendices

Appendix A: Acknowledgements

This publication has been developed with support from the European Commission funded WAI-ACT Project (IST 287725).

Contributors

Past and present active participants of the WCAG 2.0 Evaluation Methodology Task Force (Eval TF) include: deleted content: {Abou-Zahra, Shadi; Boland, Frederick; Boudreau, Denis; Chen, Amy; Conway, Vivienne; Elledge, Michael; Fiers, Wilco; Fischer, Detlev; Fong, Elizabeth; François, Vincent; Garrison, Alistair; Gutiérrez y Restrepo, Emmanuelle; Haritos-Shea, Katie; Houtepen, Martijn; Probiesch, Kerstin; Raikes, Donald; Sirois, Samuel; Swierenga, Sarah J; Velleman, Eric; Votis, Konstantinos; Waters, Elle; Wahlbin, Kathleen; Warren, Richard; Watson, Léonie} added content: [Shadi Abou-Zahra; Frederick Boland; Denis Boudreau; Amy Chen; Vivienne Conway; Michael Elledge; Wilco Fiers; Detlev Fischer; Elizabeth Fong; Vincent François; Alistair Garrison; Emmanuelle Gutiérrez y Restrepo; Katie Haritos-Shea; Martijn Houtepen; Kerstin Probiesch; Donald Raikes; Samuel Sirois; Sarah J Swierenga; Eric Velleman; Konstantinos Votis; Elle Waters; Kathleen Wahlbin; Richard Warren; Léonie Watson].

Appendix B: References

[Editor note: List needs correct formatting and updating.]

QA-FRAMEWORK
Dubost K, Rosenthal L, Hazaël-Massieux D, Henderson L (eds) (2005) W3C Quality Assurance Framework: Specification Guidelines. W3C. Available at: http://www.w3.org/TR/qaframe-spec/
RFC3986
Fielding R, Gettys J, Mogul J, Frystik H, Masinter L, Berners-Lee T (eds) (1999). Hypertext Transfer Protocol - HTTP/1.1. Request for Comments: 2616. IETF. Available at: http://www.ietf.org/rfc/rfc2616.txt
RFC2616
Berners-Lee T, Fielding R, Masinter L (eds) (2005). Uniform Resource Identifier (URI): Generic Syntax. Request for Comments: 3986. IETF. Available at: http://www.ietf.org/rfc/rfc3986.txt
UWEM
Velleman E.M, Velasco C.A, Snaprud M (eds) (2007). D-WAB4 Unified Web Evaluation Methodology (UWEM 1.2 Core). Wabcluster. Available at: http://www.wabcluster.org/uwem1_2/

Appendix C: Template Reports

Editor note: A complete example template for evaluation reporting in human understandable language. Discussion proposal below built upon the idea to have three different levels of reporting. The template will have to be ok for official evaluations so some change is still necessary. The difference between option 1 and 2 is currently very small (open for discussion).This section needs appending to the requirements in section 4.

This Appendix proposes three different levels of reporting following the discussion in clause 5 on levels of evaluation. The three deleted content: {options} added content: [Examples] are:

The deleted content: {options} added content: [Examples] should at least have the following information:

deleted content: {options} added content: [Example] 1

Results

Guideline X (heading)

Checkpoint: SC XX (subheading)

Result: pass/fail

Character: global/regional (or another wording) if regional: a list of pages where the problem exists

General information about this checkpoint: link to how to meet and to understanding this success criteria.

List of user agents, including assistive technologies that where used during the evaluation of the website and their versions.

List of pages in the sample

deleted content: {options} added content: [Example] 2

Results

Guideline X (heading)

Checkpoint: SC XX (subheading)

Result: pass/fail

Character: global/regional (or another wording) if regional: a list of pages where the problem exists

Description of existing problems and barriers for users or link to descriptions on W3C/WAI website.

General information about this checkpoint: link to how to meet and to understanding this success criteria.

List of user agents, including assistive technologies that where used during the evaluation of the website and their versions.

List of pages in the sample

deleted content: {options} added content: [Example] 3

Results

Guideline X (heading)

Checkpoint: SC XX (subheading)

Result: pass/fail

Character: global/regional (or another wording) if regional: a list of pages where the problem exists

Description of existing problems and barriers for users or link to descriptions on W3C/WAI website..

General information about this checkpoint: link to how to meet and to understanding this success criteria.

Action: Description of techniques for meeting the SC (could be techniques which are already in the techniques document or new techniques which are not in the document, but with which the SC can be met).

List of user agents, including assistive technologies that where used during the evaluation of the website and their versions.

List of pages in the sample

Appendix D: Document Changes

added content: [Changes to the previously announced Draft of 6 March 2012 are documented in the disposition of comments received. A diff-marked version is also available.]

deleted content: {Some of the main changes to the previously announced Draft of 22 February 2012 include:}