The purpose of this document is to support developers of web accessibility evaluation tools by identifying typical features of those tools and how to classify them according to different combinations of those features.

Status of this document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

Table of contents

  1. Introduction
    1. Audience of this document
    2. Complementary resources
  2. Typical features of an accessibility evaluation tool
    1. Types of document formats analyzed
    2. Cookies
    3. Authentication support
    4. Session tracking
    5. Crawling of sites
    6. Reporting
    7. Localization
    8. Support for different policy environments
    9. Evaluating document fragments
    10. Evaluating web applications
    11. Support for web testing APIs
    12. Support for semiautomatic and manual tests
    13. Customization to different audiences
    14. Integration in the web development workflow
    15. Support for repair
    16. Persistence of results and monitoring over time
  3. Example profiles of evaluation tools
  4. References
  5. Acknowledgements
  6. Appendix A: Customising results to different audiences
  7. Appendix B: Integrating the evaluation procedure into the development testing workflows

1 Introduction

There is a wide variety of web accessibility evaluation tools. This document intends to support evaluation tool developers to identify their key characteristics. To achieve that, this document:

1.1 Audience of this document

This document is targeted mainly to development managers and developers of web accessibility evaluation tools.

A secondary audience of this document are users of accessibility evaluation tools like accessibility experts or web developers.

Examples of tools that are within the scope of the document include:

1.2 Background documents

This document must be seen in the context of several others. It is recommended that the reader reviews the following documents:

In the document you will find additional pointers to other resources like standards, recommendations, technical specifications, etc., which are relevant to any developer interested in implementing an accessibility evaluation tool.

2 Typical features of an accessibility evaluation tool

In this section, we will describe typical features and functionalities of accessibility evaluation tools. The features of an accessibility evaluation tool can be presented from different perspectives: the subject being tested, the target audiences of the tool, the reporting and presentation of the results, its configurability, etc. We have tried to be as complete as possible, but it may be that some features of existing or future evaluation tools are omitted. The following list of characteristics does not follow any particular order.

It is very important that you analyse and describe for your own development process and for your customers which of those features are supported in your tool and declare any limitations of your tool.

2.1 Types of document formats analyzed

Although the vast majority of web documents are HTML documents, there are many other types of resources that need to be considered when analyzing web accessibility. Of particular importance are resources like CSS stylesheets or Javascript scripts, which allow the modification of markup documents in the user agent when they are loaded or via user interaction, as many accessibility tests are the result of the interpretation of those resources.

In general, we can distinguish these types of formats:

Most of the accessibility evaluation tools concentrate on the markup evaluation, but the most advanced are able to process many of the types described above.

2.2 Cookies

A cookie is a name-value pair that it is stored in the browser of the user. Cookies contain information relevant to the website that is being rendered and often include authentication and session information. This information can be critical for other use cases, like for instance a crawling tool.

2.3 Authentication support

Many sites require some kind of authentication (e.g., HTTP authentication, OpenID, etc.). An accessibility testing tool shall be able to support the typical authentication scenarios. It is important to do so because many sites present different content to authenticated users.

2.4 Session tracking

For security reasons, some sites include the session ID in the URL or in a cookie. With the support of the session information, websites can implement security mechanisms like for instance login out a user after a long inactivity period or track typical interaction paths of the users.

2.5 Crawling of sites

A typical difference between proprietary and open source tools is the capability to crawl web resources. That means that the tool incorporates a web crawler [WEBCRAWLER] able to extract hyperlinks out of web resources. It must be kept in mind that, as seen in the previous section, there are many types of resources on the web that contain hyperlinks. The misconception that only HTML documents contain links may lead to wrong assumptions in the evaluation process.

A web crawler defines an starting point and a set of options. The critical features of a web crawler are related to its configuration capabilities. Among them, we can highlight:

2.6 Reporting

Especially when dealing with the evaluation of several web resources, it is important that the tool is able to generate reports with the results of the evaluation. Among the characteristics of these reports, we can highlight:

2.7 Localization

Localization is important to address worldwide markets. There may be cases where your customers are not able to speak English and you need to present your user interface and your reports in other languages. To that end, you can start by looking into the authorized translations of the Web Content Accessibility Guidelines.

2.8 Support for different policy environments

Although there is an international effort to harmonisation of legislation in regard to web accessibility, there are still minor differences in the accessibility policy in different countries. It is important that you clearly define in your tool which of those policy environments you support. Most of the tools are focused on the implementation of their Web Content Accessibility Guidelines 2.0 [WCAG20], because it is the most common reference for those policies worldwide.

2.9 Evaluating document fragments

Nowadays, it is typical that it is necessary to test fragments of HTML documents, coming for instance from a web editor in a Content Management System. For those cases, the tool must be able to generate a document fragment to be tested. Furthermore, the tool needs to filter the accessibility tests according to their relevance to the document fragment.

2.10 Evaluating web applications

Web and cloud applications are becoming very frequent on the web. These applications present similar interaction patterns as those of the desktop applications. Tools that evaluate such applications must emulate different user actions (e.g., activating interface components or filling and sending forms) that modify the status of the current page or load new resources. The user of such an application would need to define these intermediate steps that can be later on interpreted by the tool (see the following section).

2.11 Support for web testing APIs

When evaluating accessibility of web sites and applications it is sometimes desirable to have own scripts that emulate some kind of user interaction. With the growing complexity of web applications, there has been an effort to standardize such interfaces. One of them is, for instance, the WebDriver API [WebDriver]. With such tools, it is possible to write tests that automate the browser behaviour.

2.12 Support for semiautomatic and manual tests

According to the Evaluation and Report Language specification [EARL10], there are three types of modes to perform accessibility tests:

Most of the tools concentrate on the testing of accessibility requirements which can be performed automatically, although there are some that support accessibility experts by performing the other two types of tests.

Sometimes, the tools do not declare openly that they only perform automatic testing. Since it is a known fact that automatic tests only cover a small set of accessibility issues, accessibility conformance can only be ensured by supporting developers and accessibility experts while testing in the manual and semiautomatic mode.

2.13 Customization to different audiences

Typically, evaluation tools are targeted to accessibility experts. However, there are also tools that allow the customization of the evaluation results to other audiences like for instance:

2.14 Integration in the web development workflow

There are tools that evaluate accessibility as an stand-alone tool, but there are other tools which integrate more naturally into the development process of web applications as plug-ins or extensions in web browsers or in Integrated Development Environments (IDEs) and Content Management Systems (CMS).

The latest tools allow an optimal workflow process because integrates accessibility into the overall quality assurance process of a website or a web application development.

2.15 Support for repair

2.16 Persistence of results and monitoring over time

3 Example profiles of evaluation tools

4 References

The following are references cited in the document.

Cascading Style Sheets Level 2 Revision 1 (CSS 2.1) Specification. W3C Recommendation 07 June 2011. Bert Bos, Tantek Çelik, Ian Hickson, Håkon Wium Lie (editors). Available at: http://www.w3.org/TR/CSS2/
CSS Current Status is available at: http://www.w3.org/standards/techs/css#w3c_all
Evaluation and Report Language (EARL) 1.0 Schema. W3C Working Draft 10 May 2011. Shadi Abou-Zahra (editor). Available at: http://www.w3.org/TR/EARL10-Schema/
ECMAScript® Language Specification. Standard ECMA-262 5.1 Edition / June 2011. Available at: http://www.ecma-international.org/ecma-262/5.1/
HTML 4.01 Specification. W3C Recommendation 24 December 1999. Dave Raggett, Arnaud Le Hors, Ian Jacobs (editors). Available at: http://www.w3.org/TR/html4/
HTML5. A vocabulary and associated APIs for HTML and XHTML. W3C Candidate Recommendation 17 December 2012. Robin Berjon, Travis Leithead, Erika Doyle Navara, Edward O'Connor, Silvia Pfeiffer (editors). Available at: http://www.w3.org/TR/html5/
PDF Reference, sixth edition. Adobe® Portable Document Format, Version 1.7, November 2006. Adobe Systems Incorporated. Available at: http://www.adobe.com/devnet/pdf/pdf_reference_archive.html
Key words for use in RFCs to Indicate Requirement Levels. IETF RFC, March 1997. Available at: http://www.ietf.org/rfc/rfc2119.txt
Accessible Rich Internet Applications (WAI-ARIA) 1.0. W3C Candidate Recommendation 18 January 2011. James Craig, Michael Cooper (editors). Available at: http://www.w3.org/TR/wai-aria/
Web Content Accessibility Guidelines (WCAG) 2.0. W3C Recommendation 11 December 2008. Ben Caldwell, Michael Cooper, Loretta Guarino Reid, Gregg Vanderheiden (editors). Available at: http://www.w3.org/TR/WCAG20/
Techniques for WCAG 2.0. Techniques and Failures for Web Content Accessibility Guidelines 2.0. W3C Working Group Note 3 January 2012. Michael Cooper, Loretta Guarino Reid, Gregg Vanderheiden (editors). Available at: http://www.w3.org/TR/WCAG20-TECHS/
Wikipedia. http://en.wikipedia.org/wiki/Web_crawler
WebDriver. W3C Working Draft 12 March 2013. Simon Stewart, David Burns (editors). Available at: http://www.w3.org/TR/webdriver/


Appendix A: Customising results to different audiences

Appendix B: Integrating the evaluation procedure into the development testing workflows