[Draft] Selecting Web Accessibility Evaluation Tools

Note: This document is a draft [see change log in progress] and should not be referenced or quoted under any circumstances. This document is under development by the Education and Outreach Working Group (EOWG), and will be offered to other W3C groups and the public for review.

1. Introduction

Web accessibility evaluation tools are software programs or online services that help determine if a Web site is accessible. Read the section Conformance Evaluation to the Web Content Accessibility Guidelines 1.0 of the document Evaluating Web Sites for Accessibility to find out how to carry out a comprehensive conformance evaluation which combines semi-automatic, manual, and user testing of accessibility features. This document explains different features of evaluation tools which can be used during a conformance evaluation to the Checkpoints of the Web Content Accessibility Guidelines 1.0, and helps you to determine which types of tools and features would best meet your specific needs.

How Evaluation Tools Can Help You

Web accessibility evaluation tools can significantly reduce the time and effort required to carry out comprehensive conformance evaluations by executing automated accessibility checks and by assisting Web developers in evaluating checkpoints which need to be evaluated manually. Specifically, evaluation tools should perform the following functions:

• Execute automated accessibility checks to determine the largest possible number of checkpoints
  • to which the Web content conforms to,
  • to which the Web content does not conform to,
  • which need to be manually evaluated,
  • which do not apply to the Web content.
• Effectively assist Web developers in evaluating checkpoints which need to be evaluated manually.

How Evaluation Tools Can Not Help You

Many accessibility checks require human judgement and must be manually evaluated by knowledgeable Web developers. The inaccuracy of evaluation tools, especially for non-automatable checkpoints, can be very misleading when Web developers lack fundamental knowledge about key accessibility requirements and concepts. Web accessibility evaluation tools can not make your Web site accessible, they can only assist you in doing so.

Disclaimer

WAI encourages the development and evolution of Web accessibility evaluation tools, and maintains an extensive list of evaluation, repair and transformation tools. WAI does not endorse or promote any single tool or vendor.

2. User Interfaces of Web Accessibility Evaluation Tools

Web accessibility evaluation tools address the requirements of different user roles (such as Web developers, content authors, or project managers) by employing a variety of user interface paradigms; sometimes evaluation tools can provide more than one type of operation mode. Depending on your organization, sometimes it may make sense to utilize a combination of more than one evaluation tool in order to cover all the aspects which are important to your development process.

Results Reports

Generally, report evaluation tools attempt to evaluate Web pages with little or no user interaction, and generate reports with the results of the accessibility checks that they carried out. Report evaluation tools are very useful in quickly determining the conformance of Web content to the checkpoints which can be automatically evaluated, or for identifying which of the remaining checkpoints need to be manually evaluated.

Wizard Interfaces

Wizard-based evaluation tools guide Web developers through a series of checks in a defined sequence in order to determine the conformance of the Web content to accessibility guidelines. Sometimes these tools are able to automatically execute some of the accessibility checks, leaving the remainder of the checks to be manually carried out by the Web developers. For example, an evaluation tool with a wizard interface may display the images of a Web page and their corresponding descriptions one at a time so that Web developers can evaluate how appropriate each of these descriptions are.

In-Page Feedback

In-page feedback evaluation tools display the results of automated accessibility checks on the respective locations of the Web pages by inserting icons and markup into the code of the pages. By marking the location and type of errors onto the Web pages, developers can recognize the context and impact of these errors thus increase their awareness and expertise in Web accessibility.

Page Transformations

Transformation tools assist Web developers in evaluating checkpoints which need to be manually evaluated by modifying the appearence of the Web pages. For example, by displaying the content in text only, by presenting the Web pages without color, or by reading the content aloud, experienced Web developers can effectively evaluate and improve the accessibility features of their Web sites.

3. Evaluating Web Accessibility Evaluation Tools

According to your specific organization, different aspects of Web accessibility evaluation tools may be more relevant. Below are some questions which can be used to compare evaluation tools in order to determine which are most suitable for your specific needs:

Accessibility: How accessible is the evaluation tool for people with disabilities?

It is equally important to ensure that people with disabilities can effectively contribute to the Web, as it is for them to be able to effectively use the Web. Evaluation tool vendors can provide accessibility by following the Authoring Tool Accessibility Guidelines 1.0.

Automation: How many checkpoints is the evaluation tool able to automatically address?

The degree of automation significantly impacts the efficiency of the Web developers. Both the detection and the repair of accessibility barriers should be automatable to the highest degree possible.

Checkpoint Coverage: How many checkpoints is the evaluation tool able to adequately address?

The more checkpoints an evaluation tool can address, the more effective it will be. Checkpoints can be addressed by determining if they are met, not met, do not apply, or need to be manually evaluated.

Collaboration: How well does the evaluation tool support the collaboration of Web developers?

Evaluation tools can support the collaboration of developers in evaluating and repairing Web content by providing shared repositories of the results, or by making the results available in open data formats such as XML or EARL.

Configuration: How well does the evaluation tool adapt to the requirements of the user?

Evaluation tools should be able to adapt to the individual needs of the Web developers. For example by allowing customization of the user interface, the accessibility checks, the generated results, or the repair options.

Documentation: How informative are the educational resources supplied by the evaluation tool?

To better assist developers in learning about the Web accessibility and how to correctly implement it, evaluation tools could provide additional resources to help educate the developers.

Integration: How well does the evaluation tool integrate into the environment of the Web developer?

Some evaluation tools provide plug-in interfaces for editors, content management systems, or browsers; or otherwise integrate into the development environment of the Web developers.

Platform Coverage: How well does the evaluation tool support the operating system of the Web developer?

Even though some evaluation tools may be available on more than one operating system, they may sometimes not support the same features or perform equally on all platforms.

Precision: How accurate are the results delivered by the evaluation tool?

Inaccurate results (such as not detecting accessibility barriers, or detecting wrong ones) decrease the reliability of the evaluation tool thus decrease the efficiency of the evaluation.

Repair: How well does the evaluation tool assist developers in retrofitting Web content?

Evaluation tools can assist developers in retrofitting accessibility barriers. For example by providing repair suggestions, or by automating routine repair measures.

Reporting: How well does the evaluation tool support reporting for different user roles?

According to the roles of the users (for example Web developer, project manager, etc), different types of reports may be better provide targeted information to these user groups.

Site Coverage: How well does the evaluation tool cover the relevant sections of a Web site?

Often Web developers will need to evaluate entire Web sites rather than single Web pages. In such cases, evaluation tools need to support the evaluation of whole groups of related pages.

Technology Coverage: How well does the evaluation tool support the technologies of a Web site?

Not all evaluation tools support all currently available Web technologies (such as HTML, XHTML, CSS, SVG, SMIL, etc.), and they do not support them all equally well.

User Interface: How suitable and user friendly is the interface of the evaluation tool?

Currently evaluation tools employ several user interface paradigms such as report-based or wizard-based interfaces. Regardless which type of interface the evaluation tool employs, the tool should be intuitive and easy to understand.

Last modified: $Date: 2005/03/01 18:48:39 $ by $Author: shadi $

Last updated 24 February 2005 by Shadi Abou-Zahra. Editors: Shadi Abou-Zahra and Judy Brewer, with assistance from participants of the EOWG.

Copyright © 2005 W3C (MIT, ERCIM, Keio ), All Rights Reserved. W3C liability, trademark, document useand software licensingrules apply. Your interactions with this site are in accordance with our public and Member privacy statements.