[contents]

Abstract

This document specifies an internationally harmonized methodology for evaluating the accessibility conformance of websites to the Web Content Accessibility Guidelines (WCAG) 2.0. It defines an approach for conformance evaluation in different contexts, including self-assessment and third-party evaluation. It is applicable to any website, including web applications and websites for mobile devices, and is independent of any particular evaluation tools, web browsers, and assistive technologies.

This document is a supporting resource for WCAG 2.0 and does not replace or supersede it in any way. It addresses the need for a standardized approach for evaluating entire websites after their development as opposed to evaluating individual collections of web pages during their development, which is equally important to ensure website accessibility. This work is part of W3C/WAI activities on Web Accessibility Evaluation and Testing that address different W3C/WAI guidelines and specifications.

Status of this document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

This 23 May 2012 Editors Draft of Website Accessibility Conformance Evaluation Methodology 1.0 is based on the recently published First Public Working Draft located at http://www.w3.org/TR/2012/WD-WCAG-EM-20120327/. The document is intended to be published and maintained as a W3C Working Group Note after review and refinement. It provides an initial outline for the evaluation methodology to invite feedback on the direction and approach, in particular on section 3 steps 4 and 5 and section 4. These and other sections of this document will be completed in future drafts, also based on the feedback received.

The WCAG 2.0 Evaluation Methodology Task Force (Eval TF) invites discussion and feedback on the current First Public Working Draft located at http://www.w3.org/TR/2012/WD-WCAG-EM-20120327/ by developers, evaluators, researchers, and others with interest in web accessibility evaluation.

Please send comments on this Editor Draft of the Website Accessibility Evaluation Methodology for WCAG 2.0 document to public-wai-evaltf@w3.org (publicly visible mailing list archive). These comments will be considered in the internal discussions of Eval TF.

Publication as Editor Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

This document has been produced by the WCAG 2.0 Evaluation Methodology Task Force (Eval TF, a joint task force of the Web Content Accessibility Guidelines Working Group (WCAG WG) and Evaluation and Repair Tools Working Group (ERT WG), as part of the Web Accessibility Initiative (WAI) Technical Activity.

This document was produced by two groups operating under the 5 February 2004 W3C Patent Policy. The groups do not expect this document to become a W3C Recommendation. W3C maintains a public list of any patent disclosures for WCAG WG and public list of any patent disclosures for ERT WG made in connection with the deliverables of each group; these pages also include instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy.

Table of Contents

  1. Introduction
  2. Using this Methodology
  3. Evaluation Procedure
  4. Conformance with this Methodology
  5. Limitations of this Methodology

Appendices

1. Introduction

Website owners, procurers, suppliers, developers, and others are frequently tasked with assessing the conformance of an existing website to the Web Content Accessibility Guidelines (WCAG) 2.0. Because it is generally not feasible nor desired to evaluate every single web page in such a context, it is essential to employ a reliable method for determining the overall conformance of a website.

This document constitutes a standardized methodology for evaluating the conformance of websites, including web applications and websites for mobile devices, to WCAG 2.0. It is developed through the open, collaborative W3C Process with the involvement of international stakeholders. It is a supporting document for WCAG 2.0 and does not replace or supersede it in any way.

1.1 Scope of this Document

The methodology described in this document supports conformance evaluation in different contexts including self-assessment and third-party assessment of websites. It is applicable to all websites, including web applications, websites for mobile devices, and other types of websites. It is independent of the size of the website, the web technologies used to create the website, and of any particular web accessibility evaluation tools, web browsers, assistive technologies, and other software tools.

While this document provides useful guidance for evaluation throughout the website development process it focuses primarily on the last stage of the process, in which confirmation is sought on whether conformance targets have been achieved (commonly referred to as acceptance testing). The document presents concrete steps to define the scope of the evaluation, select a representative sample, audit the selected sample and aggregate and report the results of the evaluation findings in a structured and uniform way.

1.2 Target Audience

The primary target audience for this methodology are individuals and organizations, wanting to evaluate the conformance of existing websites to WCAG 2.0. This includes but is not limited to:

Other audiences for whom this methodology is also relevant include:

Users of the methodology are assumed to have expertise in WCAG 2.0, accessible web design, assistive technologies, and of how people with different disabilities use the Web; read more in section 2. Using this Methodology.

1.3 Background Reading

It is assumed that the reader of this document is familiar with the following W3C resources:

Evaluating Websites for Accessibility

A multi-page resource suite that outlines different approaches for evaluating websites for accessibility. The following resources are particularly important in this context:

Web Content Accessibility Guidelines (WCAG) 2.0

Internationally recognized standard explaining how to make web content more accessible to people with disabilities. The following resources are particularly important in this context:

Accessible Web Design

The following documents introduce the essential components of web accessibility, inclusive design for people with disabilities and people with aging-related impairements, and overlap with design for mobile devices. The documents help managers, designers, developers, policy makers, researchers, and others optimize their efforts in these overlapping areas:

1.4 Terms and Definitions

For the purposes of this document, the following terms and definitions apply:

Complete Processes
When a Web page is one of a series of Web pages presenting a process (i.e., a sequence of steps that need to be completed in order to accomplish an activity), all web pages in the process conform at the specified level or better. (Conformance is not possible at a particular level if any page in the process does not conform at that level or better.), from WCAG 2.0 Conformance Requirement for Complete Processes.
Common Web Pages
Web pages that are relevant to the entire website. This includes the homepage, login page, and other entry pages, and, where applicable, the sitemap, contacts, site help, legal information, and similar web pages that are typically linked from all web pages (usually from the header, footer, or navigation menu of a web page).
Evaluator
The person, team of people, organization, in-house department, or other entity responsible for carrying out the evaluation.
Evaluation Commissioner
The person, team of people, organization, in-house department, or other entity that commissioned the evaluation.
Note: In many cases the evaluation commissioner may be the website owner, in other cases it may be another entity such as a procurer or an accessibility monitoring survey.
Key Functionality
Primary functionality of a website including tasks that users of a website carry out to perform functionality.
Note: Examples of functionality include "selecting and purchasing a product from the shop area of the website", "filling and submitting the form provided on the website", and "registering for an account on the website".
Relied Upon (Technologies)
The content would not conform if that technology is turned off or is not supported, from WCAG 2.0 definition for "relied upon".
Template
A specific page that provides a more or less fixed framework for content. Templates are mostly used to provide the basic components of a web page (logo, menu, layout, skip links etc.). Content can be viewed when entered into the template.
Website developer
Anyone involved in the website development process including but not limited to content authors, designers, programmers, quality assurance testers, and project managers.
Website Owner
The person, team of people, organization, in-house department, or other entity that is responsible for the website.
Web page
A non-embedded resource obtained from a single URI using HTTP plus any other resources that are used in the rendering or intended to be rendered together with it by a user agent, from WCAG 2.0 definition for "Web Page".
Website
A coherent collection of one or more related web pages that together provide common use or functionality. It includes static web pages, dynamically generated web pages, and web applications.
Website part
Web pages that serve the purpose and functionality of a website. This includes web pages that are part of the navigation, design, and complete processes of a website.

2. Using This Methodology

The methodology defined by this document is used for comprehensively assessing the conformance of websites to WCAG 2.0. A more cursory approach for exploring the accessibility of a website is described in Preliminary Review of Websites for Accessibility. In many cases it is advisable to carry out such preliminary reviews before applying this methodology, for example to identify obvious errors and to develop a rough understanding of the overall performance of the website.

2.1 Scope of Applicability

The methodology defined by this document applies to full, self-enclosed websites. Examples of such websites include:

While a website in this context can be part of a larger website, the methodology always applies to a full website without exclusions or omissions of website parts. For example, the methodology could be applied to the website of a university or the library of that university, but the methodology may not be applied to the website of the university excluding the website of the library, if the library website is considered to be an inherent website part of the university website.

It is also not possible to exclude individual web pages or functions such as an embedded mapping application or a linked credit card authorization form as this would conflict with the complete processes requirement (the WCAG 2.0 concept for conforming alternate versions and non-interfence may be useful in some situations). Similarly, for websites that are parts of other websites it is also not possible to exclude shared web pages that are part of both websites, such as a common contacts page or site navigation.

Exception: The methodology can be applied to clearly separable areas of a single website, such as to the public and restricted area of a website, provided that this scope matches the evaluation goals and the context of website use.

Determining the scope of a particular website is further explained in Step 1: Define the Evaluation Scope of section 3.

Editor Note: Eval TF is aware that this scope definition may limit some websites from claiming full conformance according to this methodology, because they may have website parts (such as third-party content) that are known to be inaccessible. Step 5: Report the Evaluation Findings of this methodology will provide guidance on how to report what aspects of a website conform rather than to exclude these parts from the scope. Eval TF welcomes feedback from the public on this aspect.

2.2 Required Expertise

Users of the methodology defined by this document are assumed to be knowledgeable of WCAG 2.0, accessible web design, assistive technologies, and of how people with different disabilities use the Web. This includes understanding of the relevant web technologies, barriers that people with disabilities experience, assistive technologies and approaches that people with disabilities use, and evaluation techniques and tools to identify potential barriers for people with disaibilities. In particular, it is assumed that users of this methodology are deeply familiar with the resources listed in section 1.3 Background Reading.

2.3 Evaluation Tools (Optional)

The methodology defined by this document is independent of any particular web accessibility evaluation tool, web browsers, and other software tools. However, web accessibility evaluation tools can significantly assist an evaluator during the evaluation process and contribute to more effective evaluations. For example, some web accessibility evaluation tools can scan entire websites to help identify relevant pages for manual evaluation, and other tools can assist manual evaluation in a variety of ways. Specific guidance is provided in Selecting Web Accessibility Evaluation Tools.

Note: Web accessibility evaluation tools can only automatically check a limited set of accessibility requirements that are automatable. Conformance evaluation requires manual testing and judgment by experienced evaluators.

2.4 Review Teams (Optional)

The methodology defined by this document can be carried out by an individual evaluator with the skills described in section 2.2 Required Expertise. However, using the combined expertise of review teams provides better coverage for the required skills and helps identify accessibility barriers more effectively. While not required, it is strongly recommended to employ review teams for conformance evaluation of websites. Specific guidance is provided in Using Combined Expertise to Evaluate Web Accessibility.

2.5 Involving Users (Optional)

The methodology defined by this document can be carried out by an individual evaluator with the skills described in section 2.2 Required Expertise. However, involving people with disabilities and people with aging-related impairements helps identify additional accessibility barriers that are not easily discovered by the evaluator alone. While not required, it is strongly recommended to involve real people covering a wide range of abilities during the evaluation process. Specific guidance is provided in Involving Users in Web Accessibility Evaluation.

3. Evaluation Procedure

This section defines the steps of the evaluation procedure. The specifics of these steps may shift slightly depending on the type of website, the purpose of the evaluation, and the process used by the evaluator. Some of the steps may also overlap or be carried out in parallel but it is important that all steps are considered to ensure a reliable assessment. It is also important to clearly document each step to ensure verifiable results and justifiable conformance claims.

Step 1: Define the Evaluation Scope

Requirement 1: The evaluation scope shall be defined according to Requirement 1.a, Requirement 1.b, Requirement 1.c, Requirement 1.d, and optionally Requirement 1.e.

During this step the overall scope of the evaluation is defined. It is a fundamental step that affects the subsequent steps in the evaluation procedure and is ideally carried out together with the evaluation commissioner (who may be the website owner) to ensure common expectations about the scope of the evaluation. The Evaluation Commissioner can optionally also be of help to the Evaluator. By providing help in defining the use cases, complete processes, templates of the website and the technologies that are used on the website, much time can be saved.

Step 1.a: Define the Scope of the Website

Requirement 1.a: The scope of the website shall be defined, with the definition not contradicting the constraints expressed in section 2.1 Scope of Applicability.

During this step the exact scope of the website is defined and documented. The scope definition of the website may not contradict the scope of applicability of this methodology, as defined in section 2.1 Scope of Applicability. Examples of scope definitions for websites include:

Determining the scope of a website may require some knowledge about the ownership and development process for some of the website parts. For example, the web shop of "Example Inc." may have a different design than the main website of the organization or it may be hosted at a completely different web address even though it is essential to the purpose of the organizations' website. In this case the web shop would qualify as a website part and would need to be included in the scope definition.

Note: This step may be carried out interactively with the explorations explained in Step 2: Explore the Target Website, for example to validate that the scope definition does not contradict the scope of applicability of this methodology, and to select the complete processes and templates that are part of the website that is included into the scope.

Step 1.b: Define the Goal of the Evaluation

Requirement 1.b: The goal of the evaluation shall be defined as "Basic Conformance", "Detailed Review", "In-Depth Analysis", or "Other". In the case of "Other" the exact goal must be clearly defined.

Defining the goal of the evaluation is particularly relevant to the auditing stage defined in Step 4: Audit the Selected Sample and the reporting stage defined in Step 5: Report the Evaluation Findings. Some of the evaluation goals include:

Basic Conformance
Only identifies whether a website conforms or not, but does not provide any additional information. This type of evaluation is typically carried out when the website is assumed to conform, for example to verify a conformance claim, and for large-scale evaluations with less resources to explore the details of individual websites.
Detailed Review
Identifies whether a website conforms or not, and provides further information about any identified errors. It includes counting the number of errors and their locations within the web pages. This type of evaluation is particularly useful for instructing web developers and for acquiring statistics for monitoring progress over time.
In-Depth Analysis
Identifies whether a website conforms or not, and analyzes any identified errors in detail. This includes descriptions of the errors and suggestions for possible repair options. This type of evaluation is particularly useful for organizations that want to improve the accessibility of their website and need a detailed analysis

Step 1.c: Define the Conformance Target

Requirement 1.c: The target WCAG 2.0 conformance level ("A", "AA", or "AAA") to evaluate for shall be defined.

Part of initiating the evaluation process is to define the target WCAG 2.0 conformance level ("A", "AA", or "AAA") to evaluate for. In many situations it is useful to evaluate to WCAG 2.0 Level AAA even if the intended conformance target for a website is below that. A full evaluation provides a more complete picture of the website performance and helps plan imporovements. For example, it can identify accessibility requirements beyond the intended conformance target to help improve related aspects (such as improving how tables, forms, widgets, videos, and other components are generated or developed) rather than fixing individual errors one by one.

Step 1.d: Define the Context of Website Use

Requirement 1.d: The target users of the website and minimum set of web browsers and assistive technology to evaluate for shall be defined, noting that the definition of software support shall not conflict with the WCAG 2.0 guidance on the Level of Assistive Technology Support Needed for "Accessibility Support".

Define who uses the website and how they use it. It is particularly important to define if a website is public or restricted to specific users, such as an intranet or a website that requires authentication for use. In some situations there may be more than one mode of use, for example a website with a public area and a restricted area for selected users and websites that dynamically adapt to the particular user profile. Each of those modalities must be identified and clearly documented.

Following the definition for the mode of use it is important to also define the minimum set of web browsers and assistive technology to evaluate for. For example, if it is an intranet with known software that will access the website then this definition will be significantly different from a public website with unknown software. This definition is also influenced by language, geographical location, and the target audience of the website. This definition of software support may not conflict with the WCAG 2.0 guidance on the Level of Assistive Technology Support Needed for "Accessibility Support".

Step 1.e: Define the Techniques to be Used (Optional)

Requirement 1.e: The techniques that will be used to evaluate the conformance to WCAG 2.0 should be specified.

While optional, techniques are an effective way of demonstrating conformance to WCAG 2.0. It is good practice to specify the sets or sources of techniques that are intended to be used during the evaluation at this stage already, to ensure consistent expectation between the evaluator and the evaluation commissioner. This definition is typically refined in later stages of the evaluation process, for example to evaluate web content using particular web technologies and accessibility features.

W3C/WAI provides a set of publicly documented Techniques for WCAG 2.0 but other techniques may be used too.

Note: This step may overlap with the exploration of the website defined in Step 2: Explore the Target Website. The exact techniques used to evaluate conformance to WCAG 2.0, if any, must be documented in Step 5: Report the Evaluation Findings.

Step 2: Explore the Target Website

Requirement 2: The website to be evaluated shall be explored according to Requirement 2.a, Requirement 2.b, Requirement 2.c, and Requirement 2.d.

During this step the evaluator explores the target website to be evaluated, to develop a better understanding of the website use and functionality. Carrying out initial cursory checks during this stage already helps identify web pages that are relevant for more detailed evaluation later on. For example, an evaluator may identify web pages that seem to be lacking color contrast, consistent navigation, or document structures (headings, links, etc.) with simple checks while studying the website, and note them down as candidates for selection in the sampling step defined in Step 3: Select a Representative Sample.

To carry out this step it is critical that the evaluator has access to all the relevant website parts to be evaluated. For example, it may be necessary to create accounts or otherwise provide access for evaluators to restricted areas of a website that are part of the evaluation.

Step 2.a: Identify Key Web Pages of the Website

Requirement 2.a: The common web pages of the website and templates available to the evaluator shall be identified.

During this step the common web pages of the website and templates available to the evaluator are identified and documented. The outcome of this step is a list of all common web pages and templates available to the evaluator. These will be part of the sample to be selected through the steps defined in Step 3: Select a Representative Sample. This step also helps understand key aspects of the website, such as the navigation and overall structure of the website.

Step 2.b: Identify Key Functionalities of the Website

Requirement 2.b: The key functionalities of the website shall be identified.

During this step the key functionalities of the website are identified and documented, to provide a comprehensive basis for the sample to be selected through the steps defined in Step 3: Select a Representative Sample. The outcome of this step is a list of user tasks such as "selecting and purchasing a product from the shop area of the website", "filling and submitting the form provided on the website", and "registering for an account on the website".

Step 2.c: Identify the Variety of Web Page Types

Requirement 2.c: The variety of web page types shall be identified.

Web pages with varying styles, layouts, structures, and functionality often have different implementations of accessibility features. They are also often generated by different templates and authored by different people. The outcome of this step is a list of the different types of web pages on the website to provide a comprehensive basis for the sample to be selected through the steps defined in Step 3: Select a Representative Sample. Different web page types include:

Note: This step is intended to identify instance of web pages rather than templates that are identified as part of Step 2.a: Identify Key Web Pages of the Website.

Step 2.d: Identify Web Technologies Relied Upon

Requirement 2.d: The technologies relied upon to provide the website shall be identified.

During this step the web technologies relied upon to provide the website are identified and documented, to provide a basis for the sample to be selected through the steps defined in Step 3: Select a Representative Sample. This includes base web technologies such as HTML and CSS, auxillary web technologies such as Flash, JavaScript, PDF, Silverlight and WAI-ARIA, as well as advanced web technologies such as SMIL and SVG. The outcome of this step is a list of web technologies relied upon to provide the website.

Note: Only the technologies that are relied upon to provide the website need to be identified for evaluation. This relates closely to the WCAG 2.0 concepts of conforming alternate versions and non-interfence, and may need to be determined together with the evaluation commissioner.

Step 3: Select a Representative Sample

Requirement 3: A representative sample of web pages shall be selected from the website according to Requirement 3.a, Requirement 3.b, Requirement 3.c, and Requirement 3.d.

From exploration of the website in Step 2: Explore the Target Website (within the scope as set in Step 1: Define the Evaluation Scope) the evaluator should have an understanding of the website, to facilitate the selection of a sample of web pages that is representative of the entire website. Depending on the size and complexity of the website, the size of this sample will vary. For example, a website with few types of web pages that are all generated from confined templates, such as a data repository, will likely require a much smaller sample to evaluate than a website with many types of web pages that are authored using different mechanisms. Also a website with an application in it, could have a limited number of pages or be a single page website with significant content and dynamics inside. In that case, a sample of the screens inside that application is to be added.

[Editor Note: How to cover single web pages with applications in them. These could be covered in the reporting sections in Step 5. We should add how to add 'pages' in an application to the sample]

The web pages identified through the exploration will typically relate to more than one design aspect. For example, web pages with particular functionality such as scripting, multimedia, and forms will typically also use related technologies such as Flash, JavaScript, PDF, Silverlight, and WAI-ARIA, and in many cases these web pages may have different designs to other pages. A carefull selection of web pages can significantly reduce the required sample size while maintaining appropriate representation of the entire website.

Note: This step identifies a comprehensive sample of web pages that need to be evaluated. However, many web pages will have repetitive content, such as the header, navigation, and other common components that may not need to be re-evaluated on each occurrence. Guidance on evaluating the sample identified in this step is provided in Step 4: Audit the Selected Sample.

Step 3.a: Include Key Web Pages of the Website

Requirement 3.a: All common web pages and templates available to the evaluator shall be part of the selected sample of web pages.

The common web pages and templates available to the evaluator identified through the step defined in Step 2.a: Identify Key Web Pages of the Website usually provide key information and functionality for website users and are to be included as part of the selected sample.

Step 3.b: Include Exemplar Instances of Web Pages

Requirement 3.b: At least two distinct web pages (where applicable) of each (1) key functionality, (2) content, design, and functionality, and (3) web technologies shall be part of the selected sample of web pages.

From the variety and types of web pages identified through the steps defined in Step 2: Explore the Target Website (within the scope of the evaluation as defined in Step 1: Define the Evaluation Scope), select at least two distinct web pages with the following features each (where applicable and available):

Note: A selected web page could have any number of these features. For example, a selected web page could be used to represent the use of forms and scripting at the same time. The important aspect is to select at least two distinct web pages for each relevant feature identified on the website, though more web pages may be necessary depending on the complexity of the website.

Note: When a website is being re-evaluated, it is sometimes useful for comparison to include a portion of web pages from the sample selected for the previous evaluation. However, web pages should be freshly selected to provide an accurate representation of the entire website.

Step 3.c: Include Other Relevant Web Pages

Requirement 3.c: Other web pages relevant for people with disabilities and accessibility shall be part of the selected sample of web pages.

Websites frequently include web pages that are relevant for people with disabilities and accessibility but do not explicitly match the criteria described in the previous sections. These include:

Step 3.d: Include Complete Processes in the Sample

Requirement 3.d: All web pages that are part of a complete process shall be included.

The selected sample must include all web pages that belong to a series of web pages presenting a complete process. Also, no web page in the selected sample may be part of a process without all other web pages that are part of that process to be also included into the selection.

Step 4: Audit the Selected Sample

[Editor Note: This section will provide guidance on applying WCAG 2.0 to the sample selected in Step 3: Select a Representative Sample. An introduction to this section will be inserted here. The introduction will include advice on how to avoid evaluation of repetitive content that would not yield new findings; focus emphasis on Success Criteria rather than on techniques; and reference relevant WCAG 2.0 concepts such as accessibility support, conforming alternatives, and non-intereference.]

Requirement 4: Evaluate all the pages in the sample for conformance with the WCAG 2.0 Success Criteria. Apply Requirement 4.a, Requirement 4.b, Requirement 4.c, and optionally, Requirement 4.d and Requirement 4.e to the evaluation.

The main activity of the website accessibility conformance evaluation is to check the pages in the Sample for conformance with the WCAG 2.0 Success Criteria.

In this Methodology we consider a Success Criterium a pass if within the total Sample, it is not evaluated as a fail. In case of repetitive content (like a home logo without proper description) where the same Success Criterium fails in the same way on every page and there is no need to repetitively report this for the evaluation commissioner, the evaluator can stop checking for that specific failure of the Success Criterium after he has found and reported it on 3 distinct pages or places (if available).

Step 4.a: Check for the Broadest Variety of Use Cases

Requirement 4.a: Check pages from the sample within the context of the use of the website.

Apply the WCAG 2.0 Success Criteria to the Use Case web pages in the sample using the minimum set of software tool(s) and (if necessary) assistive technology within the context.

The sample as described in Step 3: Select a Representative Sample already includes web pages and screens (from inside applications) that are representative of Use Cases as identified in Step 1.d: Define the Context of Website Use and Step 2.b: Identify Key Functionalities of the Website.

When auditing the sample, it is important to look at the defined context(s) of the use of the website. This means that the selected Use Cases are tested with the target users of the website in mind using the minimum set of web browsers and assistive technology to evaluate for (as defined in step 1.d). Note that for a user accessing an intranet with known software the context of website use will be very different than for a public website where the software tools are unknown. The context can also be influenced by language, location and target audience. Also check for duplication errors when empty templates are filled with content (like duplicated ID's). Also check beyond the documented use cases for:

We strongly encourage the inclusion of people with disabilities for auditing the use cases.

Step 4.b: Use WCAG 2.0 Techniques Where Possible

Requirement 4.b: Check pages from the sample using the specified sets or sources of techniques

Specified sets or sources of techniques have been set in Step 1.e: Define the Techniques to be Used. It is important to focus on these sets and sources of WCAG 2.0 techniques during the evaluation. The specified sets and sources limit the field of view of the evaluator. If the web page is not accessible or accessibility could be covered by a non selected technology, then the page is considered to be non-conformant. Make sure alternatives experience no interference (non-interference).

Assess the pages in the sample minimally using the sufficient techniques and the common failures. For extensive evaluations, advistory techniques can be assessed.

Step 4.c: Assess Accessibility Support for Techniques

Requirement 4.c: Assess accessibility support for techniques

[Editor Note: This section will instruct evaluators to assess accessibility support for WCAG 2.0 Techniques used and accessibility features identified in the audited web pages, ideally those initially defined in Step 1.d: Define the Context of Website Use.]

In Step 1.d: Define the Context of Website Use the target users of the website and the minimum set of web browsers and assistive technology to evaluate for have been defined. It is necessary to assess the accessibility support for the WCAG 2.0 Techniques used and for the accessibility features that have been identified in the audited web pages. If WCAG 2.0 techniques are used, make sure the techniques are accessibility supported. If there are no WCAG 2.0 techniques available or there is doubt about their accessibility support, then testing with assistive technology and selected software tools is necessary.

Step 4.d: Archive Web Pages for Reference (Optional)

Requirement 4.d: Archive web pages for reference (Optional)

[Editor Note: This section will advise evaluators to archive the audited web pages in any form (screenshots, saving copies of the web page, etc.) for future references and in case of disputes. This step is optional.]

Archive the audited web pages in any form. This could be screenshots, copies of the web pages, files, scripts etc. This backup can be of use as a reference point or to check at any moment in the future on what basis your evaluation has been done. Note that reporting the URL's in the sample is not optional as described in Step 5.a. Provide Documentation for Each Step.

Step 4.e: Record Evaluation Tools and Methods (Optional)

Requirement 4.e: Record evaluation tools and methods (optional)

[Editor Note: This section will advise evaluators to record the tools and methods used for auditing the web pages for future references and in case of disputes. This step is optional.]

Keep record of all the evaluation tools and methods used to evaluate the sample. This may include browser (version), plug-ins, specific web page evaluation tools, validation services and other tools.

Step 5: Report the Evaluation Findings

[Editor Note: This section will provide guidance on reporting the evaluation findings. An introduction to this section will be inserted here.]

This section describes how to report the evaluation findings that have been gathered during Steps 1, 2, 3 and 4. Reporting your findings is a key element of every evaluation and can help facilitate replicability of results. The following information should be added into the Evaluation Report:

Step 5.a: Provide Documentation for Each Step

Requirement 5.a: Provide documentation for each step

[Editor Note: Still discussion needed about how to cover both web pages and applications that may just be one web page]

Document every step of the evalutation process following one of the reporting templates in Appendix C. This documentation need not necessarily be public, as disclosure is up to the owner and/or evaluation commissioner. Be sure to document all choices, selections and identifications made in steps 1, 2, 3 and 4. Also include any specific sequence of steps followed during the evaluation of the Sample, specifically the use cases and the complete processes. This may facilitate the findability of fail and pass results of the evaluation by other evaluators, the website owner, website commissioner or other relevant persons.

As documentation for each Step, describe:

Step 5.b: Provide an Accessibility Statement (optional)

Requirement 5.b: Provide an accessibility statement (optional)

[Editor Note: This section will provide guidance on providing public accessibility statements referencing the outcomes of an evaluation according to this Methodology. This step is optional. However, in this case the website owners and/or evaluation commissioners choose to make a public conformance statement according to this Methodology so that certain restrictions can be applicable. For instance, it may be important to at least include or reference information about the scope defined in Step 1: Define the Evaluation Scope in such a public statement.]

When all evaluated pages of the sample conform to a certain WCAG 2.0 conformance level, a Public Accessibility Statement can be provided for the website. The Accessibility Statement references the outcome of the evaluation according to this Methodology. This step is optional.

If the website owner and/or evaluation commissioner chooses to make a public conformance statement based on evaluation using this Methodology it should at least include or reference information about:

It should also:

If applicable, the Accessibility Statement contains reference to normative documents used for inspection and evaluation quality control like ISO 17020 or others.

Step 5.c: Provide a Performance Score (optional)

Requirement 5.c: Provide a performance score (optional)

[Editor Note: This section will provide guidance on calculating an overall performance score. This step is optional. There is currently no particular alogrithm for this calculation but it is likely that this step may only be available to evaluations of type Detailed Review and above, to ensure sufficient data for aggregating a useful score. More discussion needed]

There are many possibilities for providing a performance score. For this Methodology there are three levels from which can be chosen:

The scores take different forms depending on the purpose and wish of the Evaluation Commissioner:

Additional detail can be added to all the levels of performance scores by providing information about:

Step 5.d: Provide Information on the Findings (optional)

Requirement 5.d: Provide information on the findings (optional)

[Editor Note: This section will provide guidance on providing additional information on any identified errors, using existing WCAG 2.0 materials where possible. This step is optional.]

The level of detail of the findings is depending on the goal of the evaluation as set in Step 1.b: Define the Goal of the Evaluation. For Basic Conformance evaluation, no additional information has to be provided. For detailed Review and in-depth Analysis it is important to provide information on any identified errors, using existing WCAG 2.0 materials where possible.

This step is optional. When evaluation of a success criterion results into a fail statement, explain why this is considered a failure by providing at least 3 examples of the error (if available) using URL's (if available) or else a description of where they where encountered. In case of a complete process in an application (where it is not possible to log the URL's of the web pages, this would mean that the screens are described. This can be done by describing the path followed from the start page of the complete process. Background information on the nature of failures can help formulate solutions and understand success criteria.

Step 5.e: Provide Suggestions for Repairs (optional)

Requirement 5.e: Provide Suggestions for Repairs (optional)

[Editor Note: This section will provide guidance on providing suggestions for repairing any identified errors, using existing WCAG 2.0 materials where possible. This step is optional.]

This step is depending on the goal of the evaluation as defined in Step 1.b: Define the Goal of the Evaluation. For Basic Conformance evaluation, no additional information has to be provided. For the other goals provide suggestions for repair of any identified errors, using existing WCAG 2.0 materials where possible.

Make sure the repairs do not cause another success criterion to evaluate to fail. Suggest the use of sufficient and suggested techniques where possible.

Step 5.f: Provide a Machine-Readable Report (optional)

Requirement 5.f: Provide a Machine Readable Report (optional)

[Editor Note: This section will provide guidance on providing machine-readable reports in addition to a human-readable form. It will include guidance on using the Evaluation and Report Language (EARL) for this purpose. This step is optional.]

Provide a machine-readable report of the output of Step 5a - Step 5e. using the Evaluation and Report Language (EARL). In Appendix C you will find an example of the used EARL-code. A machine readable report makes the exchange and comparison of evaluation results easier.

A machine readable report can be useful for Evaluation Commissioners who want to compare the evaluation results of the same website over time, or compare the evaluations results of different websites.

4. Conformance with this Methodology

For comformance with this Evaluation Methodology, the Evaluator(s) has at least followed and reported all non-optional steps and their requirements as described in section 3 Evaluation Procedure:

5. Limitations of this Methodology

[Editor Note: This section will discuss the limitiations of this methodology, in particular that such evaluations are always an approximation to the true conformance (which would require evaluating every single page on a website). It will also highlight aspects of applicability, such as large-scale evaluations that may not be able to include extensive samples so that they would possibly meet the requirements of this methodology. Suggestions for other considerations for this section are requested.]

Appendices

Appendix A: Acknowledgements

This publication has been developed with support from the European Commission funded WAI-ACT Project (IST 287725).

Contributors

Past and present active participants of the WCAG 2.0 Evaluation Methodology Task Force (Eval TF) include: Shadi Abou-Zahra; Frederick Boland; Denis Boudreau; Amy Chen; Vivienne Conway; Bim Egan; Michael Elledge; Wilco Fiers; Detlev Fischer; Elizabeth Fong; Vincent François; Alistair Garrison; Emmanuelle Gutiérrez y Restrepo; Katie Haritos-Shea; Martijn Houtepen; Peter Korn; Maureen Kraft; Aurelien Levy; Kerstin Probiesch; Donald Raikes; Corominas Ramon; Roberto Scano; Samuel Sirois; Sarah J Swierenga; Eric Velleman; Konstantinos Votis; Kathleen Wahlbin; Elle Waters; Richard Warren; Léonie Watson.

Appendix B: References

[Editor note: List needs correct formatting and updating.]

QA-FRAMEWORK
Dubost K, Rosenthal L, Hazaël-Massieux D, Henderson L (eds) (2005) W3C Quality Assurance Framework: Specification Guidelines. W3C. Available at: http://www.w3.org/TR/qaframe-spec/
RFC3986
Fielding R, Gettys J, Mogul J, Frystik H, Masinter L, Berners-Lee T (eds) (1999). Hypertext Transfer Protocol - HTTP/1.1. Request for Comments: 2616. IETF. Available at: http://www.ietf.org/rfc/rfc2616.txt
RFC2616
Berners-Lee T, Fielding R, Masinter L (eds) (2005). Uniform Resource Identifier (URI): Generic Syntax. Request for Comments: 3986. IETF. Available at: http://www.ietf.org/rfc/rfc3986.txt
UWEM
Velleman E.M, Velasco C.A, Snaprud M (eds) (2007). D-WAB4 Unified Web Evaluation Methodology (UWEM 1.2 Core). Wabcluster. Available at: http://www.wabcluster.org/uwem1_2/

Appendix C: Reporting Templates

Editor note: A complete example template for evaluation reporting in human understandable language. Discussion proposal below built upon the idea to have three different levels of reporting. The template will have to be ok for official evaluations so some change is still necessary. The difference between option 1 and 2 is currently very small (open for discussion).This section needs appending to the requirements in section 4.

This Appendix proposes three different levels of reporting following the discussion in clause 5 on levels of evaluation. The three examples are:

The examples should at least have the following information:

Example 1

Results

Guideline X (heading)

Checkpoint: SC XX (subheading)

Result: pass/fail

Character: global/regional (or another wording) if regional: a list of pages where the problem exists

General information about this checkpoint: link to how to meet and to understanding this success criteria.

List of user agents, including assistive technologies that where used during the evaluation of the website and their versions.

List of pages in the sample

Example 2

Results

Guideline X (heading)

Checkpoint: SC XX (subheading)

Result: pass/fail

Character: global/regional (or another wording) if regional: a list of pages where the problem exists

Description of existing problems and barriers for users or link to descriptions on W3C/WAI website.

General information about this checkpoint: link to how to meet and to understanding this success criteria.

List of user agents, including assistive technologies that where used during the evaluation of the website and their versions.

List of pages in the sample

Example 3

Results

Guideline X (heading)

Checkpoint: SC XX (subheading)

Result: pass/fail

Character: global/regional (or another wording) if regional: a list of pages where the problem exists

Description of existing problems and barriers for users or link to descriptions on W3C/WAI website..

General information about this checkpoint: link to how to meet and to understanding this success criteria.

Action: Description of techniques for meeting the SC (could be techniques which are already in the techniques document or new techniques which are not in the document, but with which the SC can be met).

List of user agents, including assistive technologies that where used during the evaluation of the website and their versions.

List of pages in the sample

Appendix D: Document Changes

Changes to the Draft of 6 March 2012 are documented in the disposition of comments received. A diff-marked version is also available. You are now reading the editor draft. Changes to the Public Working Draft of 27 March are documented in the disposition of comments of the 27 March document.