[contents]
This document specifies the Web site Accessibility Evaluation Methodology (WCAG-EM) for Web Content Accessibility Guidelines (WCAG) 2.0. It is an internationally harmonized methodology for evaluating the conformance of websites to WCAG 2.0. The Methodology supports evaluation in different contexts, such as self-assessment and third-party evaluation of websites. It is applicable to all websites (including web applications and mobile applications) regardless of size and it is independent of any particular evaluation tools, browsers, and assistive technology. It answers a much heard demand for harmonisation of accessibility evaluations for complete websites. The Methodology guides individual evaluators or evaluation teams and supports the rendering of interchangeable results.
This clause describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.
This 4 January 2012 Editors Draft of the Web site Accessibility Evaluation Methodology for WCAG 2.0 is an initial outline for future refinement. This document is intended to be published and maintained as a W3C Working Group Note after review and refinement. This version provides a framework for the direction and focus.
The WCAG 2.0 Evaluation Methodology Task Force (Eval TF) invites discussion and feedback about this document by developers, evaluators, researchers, and other practitioners who have interest in web accessibility evaluation. In particular, Eval TF is looking for feedback on the completeness of this outline. We are currently not looking for details on the different sections of the document but would welcome any input on considerations we may have missed with regards to the evaluation process.
Please send comments on this Web site Accessibility Evaluation Methodology for WCAG 2.0 document to public-wai-evaltf@w3.org (publicly visible mailing list archive).
Publication as Editor Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.
The Web site Accessibility Evaluation Methodology for WCAG2.0 (WCAG-EM) is an internationally harmonized methodology for evaluating the conformance of websites to WCAG 2.0. The Methodology supports evaluation in different contexts, such as self-assessment and third-party evaluation of websites. It is applicable to all websites (including web applications) regardless of size and it is independent of any particular evaluation tools, browsers, and assistive technology.
The Web site Accessibility Evaluation Methodology for WCAG 2.0 (WCAG-EM) is a standardized way to evaluate the conformance of websites to WCAG 2.0. The Methodology defines manual and semi-automated methods for selecting representative samples of web pages from websites that include complete processes. It provides guidance for evaluating the selected web pages to WCAG 2.0, and defines methods for integration and aggregation of the evaluation results into structured reports and conformance claims.
The Methodology provides guidance on evaluation throughout the development process but it is specifically applicable to conformance evaluation of existing websites. It extends the existing WAI resources on evaluating websites for accessibility. It includes complementary resources on Preliminary Review of Web sites for Accessibility and Involving Users in Evaluating Web Accessibility. It also provides further advice on specific contexts, selecting tools and evaluation during other stages of the development process.
1.2 Target Audience
The primary target audience of the Methodology is anyone, including individuals and organizations, wanting to evaluate the conformance of existing websites to WCAG 2.0. This includes but is not limited to:
- Website developers, suppliers, procurers, and owners
- Web accessibility consultants and evaluation services
- Web accessibility monitoring and benchmarking activities
- Web accessibility researchers and disability advocates
Other audiences who might benefit from the Methodology include:
- Developers of web accessibility evaluation and repair tools, and authoring tools
- Website developers who want to do evaluation during the development process
- People who want to use the Methodology for education and outreach activities
- Policy makers, project managers, and other decision makers who need a standard
Users of the Methodology are assumed to be knowledgeable of WCAG 2.0, accessible web design, assistive technology, and of how people with different disabilities use the Web.
1.3 Important Background Reading
The following documents, in whole or in part, are important reading for this document and are indispensable for its application. For dated references, only the edition cited applies. For undated references, the latest edition of the referenced document (including any amendments) applies. More references can be found in clause 15
- Evaluation and Report Language (EARL) 1.0
- EARL can be used to aggregate test results obtained from different testing tools including web accessibitity evaluation tools, validators, and manual evaluation: http://www.w3.org/WAI/intro/earl.php
- WAI resources on evaluating websites for accessibility
- The WAI resources suite outlines different approaches for evaluating websites for accessibility. It is not focused on full website evaluation but provides supplemental information for evaluators: http://www.w3.org/WAI/eval/
- Web Content Accessibility Guidelines (WCAG) 2.0
- The Web Content Accessibility Guidelines (WCAG) provide recommendations for accessibility of Web content. Following these guidelines will make content accessible to a wider range of people with disabilities. Following these guidelines will also often make your Web content more usable to users in general: http://www.w3.org/TR/WCAG20/
- WCAG 2.0 Techniques published by W3C/WAI
- Techniques for WCAG 2.0 offer information to Web content developers who wish to satisfy the success criteria of Web Content Accessibility Guidelines 2.0. Use of the techniques makes it easier to demonstrate conformance to WCAG 2.0 success criteria: http://www.w3.org/TR/WCAG20-TECHS/
For the purposes of this document, the following terms and definitions apply.
- Complete process
- As defined by WCAG 2.0 in: http://www.w3.org/TR/WCAG20/#cc3
- Equivalent results
- High degree of correlation between the evaluation findings so that the final evaluation result is consistent, even when different web pages are selected as part of the representative sample or when different tools are used for evaluation.
- Web page
- As defined by WCAG 2.0 in: http://www.w3.org/TR/WCAG20/#webpagedef.
- Website
- A coherent collection of one or more related web pages that together provide common use or functionality. It includes static web pages, dynamically generated web pages, and web applications.
- Website developer
- Anyone involved in the website development process including but not limited to content authors, designers, programmers, quality assurance testers, and project managers.
- Barrier
- Definition still open and depending on clause about barrier recognition and probability.
- Resources (related to the sample)
- An RDF sequence of resources (of any type)
- More...
- And more important terms and definitions if applicable.
The Methodology can be used by individuals who want to evaluate the conformance of websites to WCAG 2.0. Users of this methodology are assumed to be knowledgeable of WCAG 2.0 accessible webdesign, assistive technology and of how people with different disabilities use the Web. Evaluation according to this methodology can be carried out by individuals but using review teams with combined expertise involving people with disabilities and older people in the evaluation process is strongly recommended.
The expertise necessary for evaluating a website for WCAG2.0 is more explicitely described in the W3C/WAI Evaluation Resource Suite. Evaluation activities require diverse kinds of expertise and perspectives. Individuals evaluating the accessibility of web content require training and experience across a broad range of disciplines. If you do not have all that expertise yet and you want to start evaluating a website, the Evaluation Suite contains a less formal preliminary review to get you started.
The W3C/WAI Evaluation Resource Suite provides complementary guidance on using combined expertise to evaluate Web accessibility. The Methodology can be used by individuals, but review teams can provide better coverage of the expertise required in the understanding of web technologies, evaluation tools, barriers that people with disabilities experience, assistive technologies and approaches that people with disabilities use, and accessibility guidelines and techniques.
The W3C/WAI evaluation resource suite provides complementary guidance on Involving Users in Evaluating Web Accessibility. Evaluating with users with disabilities and with older users can identify additional issues that are not easily discovered by expert evaluation alone. It can make the evaluation process more effective and more efficient, especially when users are involved throughout the development process.
Note: Although testing with users is not a requirement under this Methodology, we strongly recommend that people with disabilities and older people are involved in the evaluation process.
Description: This clause describes the procedure to express the scope of an evaluation. This procedure will include options to express the scope of different kinds of websites including web and mobile applications regardless of size. The scope will also address issues like evaluation tools, browsers and assistive technology. The procedure also includes how to handle web resources where the state and content of the web page change based on user interaction with the page. The same URI could then be accessible in some states and not be accessible in other states based on the user or external events.
In this methodology, a website is defined as a coherent collection of one or more related web pages that together provide common use or functionality. It includes static web pages, dynamically generated web pages, and web applications identified according to the procedures as described in the next subclauses.
One of the purposes of this methodology is to support replicability of results. "For this reason, the unambiguous identification of evaluated resources is essential for the aggregation of results. Blanket statements such as "http://example.org/ conforms to WCAG-EM Level A" are not acceptable. Such a statement would imply that a set of “seed” resources have been crawled to the end following certain pre-determined limits or constraints. However, bearing in mind the wide variety of existing crawlers, the different technologies that they use, the existence of dynamic content etc, it is not possible to verify the reliability of those statements. Furthermore, the different RFCs related to Domain Names leave open room for interpretation in regard to the concept of subdomain and its resolution. Therefore, the scope of a Web site MUST be expressed in the form of a list of resources" [UWEM] for the total evaluation (see section on sampling). The source of the resources must be from the website elements that are described in the next subclauses.
While the WCAG 2.0 Recommendation focuses on webpages, the Evaluation Methodology focuses on evaluating the conformance of a website. This means that it is important to define what is part of the website and what is not. To be more precise, to define what is part of the coherent collection of one or more related web pages that together provide common use or functionality and can include static web pages, dynamically generated web pages, and web applications.
Web pages are not limited to html. They can include PDF, Office documents, audio, video, photos and other technologies. In principle, all web pages that are part of the website or fall withing complete processes fall within the scope of the evaluation but this statement needs refinement. It is important that the scope of an evaluation is properly set because the scope delimites the webpages and applications to be included into the sample.
The following definitions of indicators for setting the scope of an accessibility evaluation apply to the subclauses:
Editor note: It is still te be discussed if the below factors are covering the necessary indicators for setting the scope. We will also indicate the relation with the other subclauses here. The definition of resource needs review as to include more than just webpages.
- Resource as base URI of the evaluation
- A data object identified by a URI [RFC3986]. This definition is adapted from the definition of resource in [RFC2616]. The concept is included for non-HTTP resources like, e.g., those accessed via the FTP protocol. This type of resource must be expressed via an instance of the earl:Content Class.
- Presentation and layout
- Presentation elements include characteristics like graphical design, color, form, layout, logo and other style elements of the website including typography, navigation, buttonstyle and order of the elements of the webpage.
- Complete Process
- As defined by WCAG 2.0 in: http://www.w3.org/TR/WCAG20/#cc3.
- Key functionality
- Information or functionality that is of main interest to the visitors of the website and can be considered one of the core reasons for visiting the website.
- Alternative page
- A page that contains an alternative for inaccessible content as described in WCAG 2.0 and is therefore placed within the scope.
Possibility to exclude webpages from the scope:
- Authorisation
- You need to go through an authorization process to enter part of a website.
- Technologies used on webpages
- Definition needed
- Devided scope
- Part of the website is part of another evaluation. An example could be that part of a complete process is covered by another evaluation. Like with an external creditcard page on a shopping website.
- Special content
- Definition needed: Non-essential content that cannot be made accessible.
First determine the main URI of the website. For most websites this will consist of a domain name and top level domain (like .com or .eu). The base URI can also be more specific, for example a directory added to the URI or a subdomain.
The base URI does not determine whether a page is inside or outside the scope. It is only the starting point from which the rest of the scope is determined. All pages that use the base URI, both because they have an extensive directory structure or use of sub-(sub-) domains fall within the scope unless there is a clearly described exception.
Some websites have multiple domains linked for the same content and dynamics, in that case the scope is limited to the primary domain. A conformity declaration would in that case also be valid for the other websites.
Editor note:This subclause will also address the procedure to handle web resources where the state and or content of the web page change based on user interaction with the page. The same URI could then have one perception and function in one state and another in other states based on the user or external events.
Presentation and layout focuses on 3 criteria, namely Graphical User Interface, general design and navigation. This includes pictograms used for certain functionalities on the website, interaction elements for users to navigate the website and other interaction functionalities on the website.
Presentation elements include characteristics like graphical design, color, form, layout, logo and other style elements of the website including typography, navigation, buttonstyle and order of the elements of the webpage.
All pages of the complete process are included into the scope unless a complete process is excluded from the scope. Complete processes can be excluded from the scope in specific circumstances that will be described in this subclause. In this methodology complete processes can include parts of a website into the scope of an evaluation. In case of a series of steps that need to be completed in order to accomplish an activity, all Web pages in the process are part of the scope of the evaluation and should conform at the specified level or better.
Section description: This subclause will describe functionalities on the website that can be generally regarded belonging to the website as important functionality for users or visitors and should therefore be included into the scope, even if the URI or the perception and function are different. This will be further elaborated.
If a website includes information or functionality that is of main interest to the visitors of the website, then this information or functionality should be part of the scope.
A page that contains an alternative for inaccessible content as described in WCAG 2.0 and is therefore placed within the scope. We will also look at the possibility to exclude or include alternatives. I.e. an alternative that is outside of the scope could be placed inside the scope under certain conditions if it is an accessible alternative for inaccessible content or techniques.
Editor note: Parts of a website can be in or outside the scope of the evaluation if they cannot be viewed without the user having proper authorization or authentication by the organization that manages the website.
You need to go through an authorization process to enter part of the webpage. Examples are login to enter an intranet or extranet. If this is the case, then the part of the website behind the authorization process can be excluded from the scope in the following cases: To be discussed.
This subclause will describe the role that technologies play in setting the scope.
Editor note: This subclause will describe the possibility to divide the evaluation of a website withing a certain scope into different parts i.e. when different people are responsible for those parts. If all parts are in the scope of the website, then the scope could be divided into multiple parts that all have to be evaluated. This subclause will also describe the results for the conformance claim that can be made based on partial evaluations within a total scope.
Editor note: This will describe content for which it is possible to exclude it from the scope. Example could be historical scans of old Italian manuscripts from the 14th century. This exception also includes non-essential content that cannot be made accessible (Example: colorblind simulator will be difficult to make accessible for the blind).
The Methodology describes the roles of manual and semi-automated evaluation. In case of manual evaluation, an evaluator can manually evaluate all pages, but on a website with millions of webpages that may not be feasible or desirable. This clause describes how to select a representative sample of a website. This can include static web pages, dynamically generated web pages, and web applications. This clause describes the contents of a representative sample of a website for evaluation. Also the clause will look at barriers and their impact and at the relation between the sample size and the barrier probability when sampling more pages.
Editor note: This subclause describes how to select a sample. Primarily focusing on the contents of the sample, the techniques and other relevant items to be included. We have to check if we sufficiently include dynamic pages and the different states of a dynamic page like a web application.
For the sample we destinguish three sets of sampled resources to be included into the total evaluation sample of a website:
- Core Resources (non random sample)
- Random Resources
- Task Oriented Resources
Editor note: This subclause is currently taken from [UWEM] for discussion.
Core Resource List (non random sample): The Core Resource List is a set of generic resources, which are likely to be present in most websites, and which are core to the use and accessibility evaluation of a website. The Core Resource List therefore represents a set of resources to be included in any expert accessibility evaluation of the site (if they are available). The Core Resource List cannot, in general, be automatically identified, but will probably requires human judgement to be selected. The Core Resource List should consist of as many of the following resources as are applicable:
- The "Home" resource. This is defined as the resource identified by a URL consisting only of a HTTP protocol component followed by the host name. The mapping from this URL to the Home resource may rely on server side redirection. This resource is required to be present on the site (though it may be differently named).
- The "Contact Information" resource (if any).
- The generic "Help" resource (if any).
- The "Site Map" resource (if any).
- The resources comprising the "Primary Site Search" service (if any). This shall include at least one resource where a search can be initiated and at least one resource showing the results from a sample search.
- Resources describing accessibility features and / or the accessibility policy of the site (if any).
- Examples of resources representative of the primary intended uses of the site (if identifiable).
- Resources representative of each category of resources having a substantially distinct "look and feel" (typically representative of distinct underlying site "templates") (if identifiable).
- Resources representative of each of the following distinct web technologies (where they are in use):
- forms
- data tables
- applets, plug ins, multimedia, etc.
- more...
If the evaluation sample is created manually, it must contain the core resources. If the core resources contains fewer pages than the required sample size, additional resources from the following categories must be added.
Of course, any single resource may belong to more than one of the categories above: the requirement is simply that the Core Resources as a whole should, as far as possible, collectively address all the applicable sampling objectives. Any given resource should appear only once in the total evaluation sample.
Task Orientated Resources are the pages necessary for completing the complete processes on the website. They provide examples of real use cases for the website. This might include tasks such as to source certain information, place an order or participate in a discussion.
Editor note: This subclause is currently taken from [UWEM] for discussion.
To generate a representative and unbiased total evaluation sample, a uniform random selection from the set of all resources belonging to the website is required. [Addition to UWEM: This can be done in many ways. The method used to select a random sample should be described in the Evaluation Report. Random sampling better supports comparability of results, e.g. monitoring and synchronous or asynchronous comparisons. Note that it is very difficult to generate a fully random sample without the use of a tool.]
There are different ways to determine the list of all resources belonging to a website to select a sample from. In some cases the list of resources is known beforehand, e.g. because it is provided by a site owner commissioning the evaluation. If the list of resources is unknown, the website has to be explored prior to the evaluation. This will typically be done by a web crawler automatically exploring the website by following links. The crawl starts out from one or more "seed resources" , e.g. the home page.
If a website is very large, it may not be feasible to identify a complete list of resources belonging to a website. In this case the evaluator may choose to stop the crawling process when a sufficiently large number of resources has been identified.
The evaluation sample is chosen from the list of resources belonging to the website using uniform random sampling without replacement.
Note that both the crawling and sampling algorithm used, and any further restrictions limiting or biasing the result, including, but not limited to the set of restrictions below, should be explicitly disclosed in the Evaluation Report:
- Restriction to pages without form interaction
- Restriction by content type (e.g., not JavaScript/Flash/PDF)
- Restriction by linkage depth from any seed resource
- Restriction by the robots.txt protocol
- Restriction on the specific seed resources used
- Restriction on the total number of resources in the list of resources
Editor note: This subclause describes the ideal size of the sample of a website. The size of a sample will be depending on many different factors including but not limited to the number of webpages per technology related to the overall size of the website.
Editor note: This is the clause describing step by step how to do the evaluation. The evaluation is depending on many factors, like technologies used, technologies evaluated, Accessibility supported etc. Important factor in the evaluation is formed by the accessibility barriers that are encountered and their impact. This clause will show how they are covered and describe a possible margin for errors on a website in relation to the barrierimpact. Note: The methodology will not provide WCAG 2.0 techniques but rather explain how to use them effectively for evaluation.
"The detailed evaluation of a full website has to be carried out by an accessibility expert. Many tests require human judgement and therefore have to be carried out manually. However, not the whole evaluation process has to be done by hand. The expert may choose to use tools to support sampling or perform fully automatable tests. To report the results the expert chooses between the text-based template report or the machine readable report in EARL format" [UWEM].
Editor note: This subclause describes how to relate manual and semi-automated evaluation. It describes the both evaluation types and their relation to accessibility conformance evaluation.
Editor note: This subclause is currently taken from [UWEM] for discussion.
Accessibility testing may be done via semi-automated, manual and user testing. The different types of evaluation methods have a number of strengths and weaknesses. Only semi-automated and manual evaluation are covered in this methodology. Semi-automated evaluation can only test for conformance to a subset of the success criteria. Only a limited subset can be identified reliably by using semi-automated tools. Therefore, coverage of semi-automated evaluation as an overall indicator of accessibility is low, however it can identify many barriers reliably. It may also be applied efficiently to test very large numbers of web resources within a website.
Some tools can also act as support systems for the manual evaluation process. The tools can provide reliable results for a subset of tests and can not only speed up the process by performing some tasks automatically, but also, by providing hints about barrier locations, indicate areas the evaluators should focus on.
User testing is able to identify barriers that are not caught by other testing means, and is also capable of estimating the accessibility for tested web pages. It is not included in this methodology.
The main advantages of semi-automated testing are:
- The method is replicable (although the dynamic nature of some web technologies mean complete replication is impossible).
- It may be applied to very large number of resources on a website and to multiple websites.
Editor note: This subclause describes the role of technologies in the scope and sample. It also describes the dependancy and covering of accessibility supported technologies and the availability of accessible and conformant alternatives.
This subclause provides a step by step description of the evaluation of the website (sample). This does not include going into the guidelines, success criteria etc from WCAG 2.0 or related techniques. It could be possible to propose different ways to evaluate the guidelines: one by one, per theme, per technology, etc. The Methodology will not prescribe one of those ways as necessary.
Editor note: This subclause describes how to recognize the accessibility barriers and ascertain the impact in relation to the sample of webpages. This subclause will provide guidance to setting a barrier level. Also the barrier recognition can help limit the size of a sample of a website. The methodology could provide input for a barrier probability index. This index could provide a breakdown of the need to extend the sample under specific circumstances. We also want to include quantification of the frequency in the status of a barrier (from minutes 20111215).
Editor note: This subclause will define an error margin. This margin will be linked strongly to the barriers described in an earlier subclause. Example: If one out of 1 million images on a website fails the alt-attribute this could mean that the complete websites scores a fail even if the barrier impact would be very low. The W3C/WAI Eval Task Force will define an error margin to cover these non-structural errors that have a low impact. This is already partly covered inside WCAG 2.0.
The Taskforce will also look into the effects of possible extending this error margin to cover techniques, complete processes, dynamic parts of the site of parts of the scope.
Editor note: We will not make changes to WCAG 2.0 sections on conformance. In this section we will link the conformance inside WCAG 2.0 to the extended possibilities and options inside the methodology and to the fact that conformance is for full websites. This include provisions for conformity in relation to scope, error margin etc.
Editor note: We will provide a list of conformity requirements that relate to the scope and sample clauses.
Editor note: This subclause will describe the role of accessibility supported in the methodology and the impact on the website and the relation to the barriers and complete processes. This wil also include what to do if you find technologies that are not accessibility supported. Also we will provide a way to describe the set of AT and User Agents available to the user and the impact on the conformance statement as far as not covered by WCAG 2.0.
Description: Partial conformance seems to be adequately described in and outside of WCAG 2.0 but mostly related to individual webpages. This subclause shows how to claim partial conformance in relation to clauses Scope, Sample and Evaluation for a full website if this should be necessary from the related subclauses.
Editor note: Mostly already inside WCAG 2.0. Additions related to scope, sample, accessibility supported, techniques etc.
Editor note: This section explains how to score both with manual and with machine evaluation of websites. Also this subclause shows how the results can be aggregated and made ready for reporting. The chosen path will be also depending on the way that the evaluation is done (one by one, thematic, etc.).
Description: This clause covers how to write a report from the evaluation that is both human readable and or one that is machine readable. It describes the minimal contents of the report to support optimal interchange of information and harmonization and comparability of data on an international scale. This would make it possible for organisations in different countries, using different evaluators to compare or benchmark the data. Templates will be included in the appendices.
Editor note: Explanation of the manual report in human understandable language. What should minimally be in the report. The template for the text based report is the to be found in the appendix.
Editor note: Explanation of the machine readable report following EARL. The template is then to be found in the appendix.
Desscription: This clause will describe known limitations of the methodology. It will be filled during the further work on the methodology. We will place here the limitations and underlying assumptions that are important for readers to understand the methodology and possible situations where the methodology provides risk i.e. for repeatability. The issues will be limited as possible by the Taskforce.
Appendices
This publication has been funded in part by the European Commission under FP7-ICT-2011-7. The content of this publication does not necessarily reflect the views or policies of the European Commission, nor does mention of trade names, commercial products, or organizations imply any endorsement. Additional information about participation in the Web Content Accessibility Guidelines Working Group (WCAG WG) or the Evaluation and Repair Tools Working Group (ERT) can be found on the Working Group home pages.
Some of the clauses in this document, in particular section 3, 4.1.1, 4.1.3, and 5.1, are derived from the Unified Web Evaluation Methodology [UWEM].
Editor note: Below list needs correct formatting.
- QA-FRAMEWORK
- Quality Assurance Framework: Specification Guidelines: http://www.w3.org/TR/qaframe-spec/
- RFC3986
- Fielding R, Gettys J, Mogul J, Frystik H, Masinter L, Berners-Lee T (eds) (1999). Hypertext Transfer Protocol – HTTP/1.1. Request for Comments: 2616. IETF. Available at: http://www.ietf.org/rfc/rfc2616.txt
- RFC2616
- Berners-Lee T, Fielding R, Masinter L (eds) (2005). Uniform Resource Identifier (URI): Generic Syntax. Request for Comments: 3986. IETF. Available at: http://www.ietf.org/rfc/rfc3986.txt
- UWEM
- Velleman E.M, Velasco C.A, Snaprud M (eds) (2007). D-WAB4 Unified Web Evaluation Methodology (UWEM 1.2 Core). Wabcluster. Available at: http://www.wabcluster.org/uwem1_2/
Editor note: A complete example template for evaluation reporting in human understandable language.
Editor note: A complete example template for evaluation reporting using EARL.