Guidance on the development of web accessibility evaluation tools

Editors' Draft 18 July 2013

This version:: http://www.w3.org/WAI/ER/WD-AERT/ED-AERT20130718
Latest published version:: http://www.w3.org/TR/AERT
Latest internal version:: http://www.w3.org/WAI/ER/WD-AERT/ED-AERT
Previous published version:: http://www.w3.org/TR/2000/WD-AERT-20000426
Previous internal version:: http://www.w3.org/WAI/ER/WD-AERT/ED-AERT20130625
Editors:: Carlos A Velasco, Fraunhofer Institute for Applied Information Technology FIT; Philip Ackermann, Fraunhofer Institute for Applied Information Technology FIT; Evangelos Vlachogiannis, Fraunhofer Institute for Applied Information Technology FIT

Abstract

The purpose of this document is to support developers of web accessibility evaluation tools by identifying typical features of those tools and how to classify them according to different combinations of those features.

Status of this document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

Introduction
Typical features of an accessibility evaluation tool
Example profiles of evaluation tools
References
Acknowledgements
Appendix A: Customising results to different audiences
Appendix B: Integrating the evaluation procedure into the development testing workflows

1. Introduction

There is a wide variety of accessibility evaluation tools to be found on the web. This document intends to support evaluation tool developers to identify their key characteristics. To achieve that, this document:

describes to developers of web accessibility evaluation tools the typical features of these tools and briefly presents different perspectives on these features;
describes typical profiles for web accessibility evaluation tools according to different combinations of the aforementioned features;
supports developers of web accessibility evaluation tools to understand the different types of web accessibility tests: automatic, semiautomatic and manual; and
supports developers of web accessibility evaluation tools to understand how to use WCAG 2.0 success criteria, sufficient techniques, advisory techniques, and common failures for web accessibility testing.

1.1. Audience of this Document

This document is targeted mainly to development managers and developers of web accessibility evaluation tools.

A secondary audience of this document are users of accessibility evaluation tools like accessibility experts or web developers.

Examples of tools that are within the scope of the document include:

Commercial or open source accessibility evaluation tools.
Accessibility evaluation tools that test complete web sites or web applications.
Accessibility evaluation tools that test single web pages or resources.
Focused accessibility evaluation tools, which test a concrete aspect of accessibility, for instance, contrast of images, accessibility of forms, WAI-ARIA implementation, etc.
Accessibility evaluation tools that support research with users or developers of specific aspects of accessibility.

1.2. Document Conventions

The keywords must, required, recommended, should, may, and optional in this document are used in accordance with RFC 2119 [RFC2119].

1.3. Complementary resources

This document must be seen in the context of several others. It is recommended that the reader reviews the following documents as well:

Web Content Accessibility Guidelines 2.0 [WCAG20]
Techniques and Failures for Web Content Accessibility Guidelines 2.0 [WCAG20-TECHS]

In the document you will find additional pointers to other resources like standards, recommendations, technical specifications, etc which are relevant to any developer interested in implementing an accessibility evaluation tool.

2. Typical features of an accessibility evaluation tool

In this section, we will describe typical features and functionalities of accessibility evaluation tools. We have tried to be as complete as possible, but it may be that some features of existing or future evaluation tools are omitted.

It is very important that you analyse and describe for your own development process and for your customers which of those features are supported in your tool and declare any limitations of your tool.

2.1. Types of document formats analyzed

Although the vast majority of web documents are HTML documents, there are many other types of resources that need to be considered when analysing web accessibility. Of particular importance are resources like CSS stylesheets or Javascript scripts, which allow the modification of markup documents in the user agent when they are loaded or via user interaction, as many accessibility tests are the result of the interpretation of those resources.

In general, we can distinguish these types of formats:

Markup documents. These are normally HTML [HTML4, HTML5] or XML documents. Processing these documents requires the ability to build a Document Object Model (DOM) according to the different specifications, which can then parsed and analysed.
Style resources. These are presentation modifiers conformant to the different CSS specifications [CSS2, CSS3], which are processed by the user agents. The interpretation of stylesheets is critical for many accessibility techniques.
Script resources. These are components that modify at a defined point documents and styles, be it because of user interaction or because of other functional requirements of an application (e.g., because of dynamic updates or modifications). Nowadays these resources play a critical role in mobile web and web 2.0 applications. Ignoring these resources may lead to missing accessibility problems or to report non-existing errors. The scripting language commonly used on the web is Javascript, based upon ECMAScript. ECMAScript [ECMAScript] is standardized by the Ecma International standards organization
Multimedia resources. These are images, movies or audio tracks that are inserted into the application. From the accessibility standpoint, the interpretation of those resources is very important, especially for issues like colour contrast, colour blindness, or media alternatives, for instance.
PDF documents. Many reports and administrative information is available in this format, especially in government websites. The format was initially developed by Adobe [PDF] and was later on standardised by ISO.
Resources with other proprietary or open source formats. Although not very frequent on the web, you may encounter other formats like office documents, flash movies and applications, etc. Depending on your customers, you may need to be able to parse and evaluate this type of resources.

Most of the accessibility evaluation tools concentrate on the markup validation, but the most advanced are able to process many of the types described above.

2.2. Crawling of sites

A typical difference between commercial and open source tools is the capability to crawl web resources. That means that the tool incorporates a web crawler [WEBCRAWLER] able to extract hyperlinks out of web resources. It must be kept in mind that, as seen in the previous section, there are many types of resources on the web that contain hyperlinks. The misconception that only HTML documents contain links may lead to wrong assumptions in the evaluation process.

A web crawler defines an starting point and a set of options. The critical features of a web crawler are related to its configuration capabilities. Among them, we can highlight:

Types of documents extracted.
Capability to define inclusion and exclusion filters. Sometimes customers require analysis of concrete parts of the website, or do not want to include others.
Multithreaded crawling. For a large site, it may be important to optimise performance by having a tool able to crawl in parallel threads.
Avoidance of resource duplicates. It is typical from web resources to link many times to the same results. If the crawler is not able to identify such issues, it may lead to a great performance loss.
Session persistence. Some sites include the session ID in the URL. If the site is big, it may occur that the session ID changes over time, leading to identify resources as different when they are in reality the same.
Authentication support. Many sites require some kind of authentication (e.g., HTTP authentication, OpenID, etc.). A crawler should be in the position to handle such use cases.

2.3. Support for different accessibility compliance environments and localisation

Although there is an international effort to harmonisation of legislation in regard to web accessibility, there are still minor differences in the accessibility policy in different countries. It is important that you clearly define in your tool which of those policy environments you support. Most of the tools are focused on the implementation of their Web Content Accessibility Guidelines 2.0 [WCAG20], because it is the most common reference for those policies worldwide.

Additionally it is important that you know your target market. Internationalisation and localisation are important issue to address worldwide markets. There may be cases where your customers are not able to speak English, and you need to generate information in other languages. To that end, you can start by looking into the authorised translations of the Web Content Accessibility Guidelines.

2.4. Ability to test applications, complete resources or fragments

Nowadays, it is typical that it is necessary to test fragments of HTML documents, coming for instance from a web editor in a Content Management System. For those cases, the tool must be able to generate a document fragment to be tested. Furthermore, the tool needs to filter the accessibility tests according to their relevance to the document fragment.

Within this section we include testing of web applications. For them it is typical to emulate different user actions (e.g. activating interface components or filling and sending forms) that modify the status of the current page or load new resources. The user of such an application would need to define these intermediate steps that can be later on interpreted by the tool.

2.5.

3. Example profiles of evaluation tools

4. References

The following are references cited in the document.

CSS2: Cascading Style Sheets Level 2 Revision 1 (CSS 2.1) Specification. W3C Recommendation 07 June 2011. Bert Bos, Tantek Çelik, Ian Hickson, Håkon Wium Lie (editors). Available at: http://www.w3.org/TR/CSS2/
CSS3: CSS Current Status is available at: http://www.w3.org/standards/techs/css#w3c_all
ECMAScript: ECMAScript® Language Specification. Standard ECMA-262 5.1 Edition / June 2011. Available at: http://www.ecma-international.org/ecma-262/5.1/
HTML4: HTML 4.01 Specification. W3C Recommendation 24 December 1999. Dave Raggett, Arnaud Le Hors, Ian Jacobs (editors). Available at: http://www.w3.org/TR/html4/
HTML5: HTML5. A vocabulary and associated APIs for HTML and XHTML. W3C Candidate Recommendation 17 December 2012. Robin Berjon, Travis Leithead, Erika Doyle Navara, Edward O'Connor, Silvia Pfeiffer (editors). Available at: http://www.w3.org/TR/html5/
PDF: PDF Reference, sixth edition. Adobe® Portable Document Format, Version 1.7, November 2006. Adobe Systems Incorporated. Available at: http://www.adobe.com/devnet/pdf/pdf_reference_archive.html
RFC2119: Key words for use in RFCs to Indicate Requirement Levels. IETF RFC, March 1997. Available at: http://www.ietf.org/rfc/rfc2119.txt
WAI-ARIA: Accessible Rich Internet Applications (WAI-ARIA) 1.0. W3C Candidate Recommendation 18 January 2011. James Craig, Michael Cooper (editors). Available at: http://www.w3.org/TR/wai-aria/
WCAG20: Web Content Accessibility Guidelines (WCAG) 2.0. W3C Recommendation 11 December 2008. Ben Caldwell, Michael Cooper, Loretta Guarino Reid, Gregg Vanderheiden (editors). Available at: http://www.w3.org/TR/WCAG20/
WCAG20-TECHS: Techniques for WCAG 2.0. Techniques and Failures for Web Content Accessibility Guidelines 2.0. W3C Working Group Note 3 January 2012. Michael Cooper, Loretta Guarino Reid, Gregg Vanderheiden (editors). Available at: http://www.w3.org/TR/WCAG20-TECHS/
WEBCRAWLER: Wikipedia. http://en.wikipedia.org/wiki/Web_crawler