W3C logoWeb Accessibility Initiative (WAI) logo

WAI: Strategies, guidelines, resources to make the Web accessible to people with disabilities

[Draft] Selecting Web Accessibility Evaluation Tools

Page Contents

Introduction

Web accessibility evaluation tools are software programs or online services that help determine if a Web site is accessible. This document highlights different features of evaluation tools which can assist during evaluation reviews such as the methodologies described in the WAI resource Evaluating Web Sites for Accessibility.

What evaluation tools can do

Web accessibility evaluation tools can significantly reduce the time and effort required to carry out evaluations. When used carefully throughout the design, implementation, and maintenance phases of Web development, these tools can assist their users in preventing accessibility barriers, repairing encountered barriers, and improving the overall quality of Web sites. The following are ways in which tools can assist users in evaluating Web sites for accessibility; some tools can perform both:

What evaluation tools can not do

Many accessibility checks require human judgement and must be evaluated manually using different techniques. Also, in some cases evaluation tools are prone to producing false or misleading results such as not identifying or signal incorrect code. The results from evaluation tools should not be used to determine conformance levels unless they are operated by experienced evaluators who understand the capabilities and limitations of the tools in order to achieve accurate results. Web accessibility evaluation tools can not determine the accessibility of Web sites, they can only assist in doing so.

Disclaimer

WAI encourages the development and evolution of Web accessibility evaluation tools, and maintains an extensive list of evaluation, repair and transformation tools. WAI does not endorse or promote any single tool or vendor.

Considerations for selecting evaluation tools

Web accessibility evaluation tools can be used throughout all stages of Web site development. For example at the early design stage, Web designers may be interested in using tools that help them understand how the site structure, navigation, or look-and-feel perform with respect to accessibility requirements. Later at the implementation stage, developers may be more interested in tools that help them assess the accessibility of the underlying code which is generated by the Web authoring tools (such as editors or content management systems). Web content authors, project managers, and other types of Web site developers have further requirements for evaluation tools that help them fulfill their respective tasks.

According to the specific organization and Web site for which evaluation tools will be used, different characteristics and features of Web accessibility evaluation tools may be more or less important to the tool users. For example, an organization may choose to use fully automated evaluation tools which can examine the whole Web site, and additionally evaluate samples of pages using other types of tools in order to compensate the limitations of fully automated checking. The following are some of the factors which may be considered during the selection of Web accessibility evaluation tools:

Organizational structure and development process

For larger organizations or when several types of Web developers (such as designers, programmers, content authors, quality assurance reviewers, or others) participate in the development of the site, it may be beneficial to use a combination of evaluation tools in order to balance the capabilities of the tools, and to address the needs of the different user roles throughout the development stages.

Complexity and size of the Web site

Examples of complex Web sites are sites that make heavy use of scripting to generate Web pages or to provide functionality in them; employ multimedia content such as audio or video files; incorporate advanced technologies such as SMIL, SVG, or MathML; or are very large and difficult to maintain. In such cases, more specialized evaluation tools might be more useful, even though they may have other limitations.

Skills and knowledge of the Web developers

Some evaluation tools require users to have more knowledge of accessibility requirements or markup code (such as HTML, CSS, ...) than others. Also, some evaluation tools can support Web developers in learning such skills differently than others. It is important to identify the intended tool users and their requirements when selecting appropriate evaluation tools for a specific organization.

Pre-existing Web development environment

It is often beneficial to deploy evaluation tools that work well with the existing operating systems and other development infrastructure. Also, sometimes evaluation tools are plug-in extensions for Web authoring tools (such as editors, content management system, or save-as utilities) or browsers; or they can export evaluation reports in different formats (for example, export results to a database).

Usages of evaluation tools

Web accessibility evaluation tools can be used for different purposes depending on the expertise of the users and what checkpoints they want to evaluate. The following are some of the common characteristics of evaluation tools to support users fulfil different tasks during an evaluation process; some tools provide more than one mode of operation:

Generating reports

Report generating evaluation tools are usually designed to evaluate multiple pages or whole Web sites with little or no user interaction. The results of the accessibility checks that the tools execute are summarized in reports which can often be customized according to the needs of the users. Report generating tools are very useful in quickly determining the conformance of Web sites to checkpoints which can be evaluated automatically, and for identifying remaining checkpoints that need to be evaluated manually.

Step-by-step evaluations

Wizard-based evaluation tools guide users through sequences of checks step by step. Sometimes these tools are able to execute some of the accessibility checks automatically and prompt the users to manually evaluate the remaining checks. For example, an evaluation tool with a wizard interface may be able to automatically check if the images on a Web site have text descriptions, then display the images with their corresponding descriptions to the users to evaluate how appropriate these descriptions are.

In-page feedback

In-page feedback evaluation tools insert (temporary) icons and markup into the code of the Web pages to display the results of automated accessibility checks and their corresponding location within the pages. Sometimes, other types of icons are also inserted into the Web pages to assist the manual evaluation of checkpoints. For example, some tools may insert icons to indicate the hierarchy of the page headings or lists, or the reading sequence of table cells that are be perceived by some Web site users.

Page transformations

Transformation tools modify the appearence of the Web sites to help identify conceptual design issues with regard to Web accessibility. For example, transformation tools may present the content of Web sites in text only, without color, or read the content aloud. These types of evaluation tools are usually especially useful to compensate the limitations of automated accessibility checking and to support the users in evaluating checkpoints that need to be evaluated manually.

Features of evaluation tools

The following is a (non-exhaustive) collection of features which may help users to compare and assess Web accessibility evaluation tools for their specific needs. Some evaluation tools provide some of these features with varying adequacy so that carefull analysis needs to be made when assessing them; sometimes the tool vendors can provide additional information about how their tools support these or other features.

Accessibility: How accessible is the evaluation tool for people with disabilities?
It is equally important to ensure that people with disabilities can effectively contribute to the Web, as it is for them to be able to effectively use the Web. Evaluation tool developers and vendors can provide accessibility in all parts of a tool (user interface, documentation, or generated reports) by following the Authoring Tool Accessibility Guidelines 1.0.
Checkpoint coverage: Which checkpoints is the evaluation tool able to adequately address?
While most Web accessibility evaluation tools support a broad variety of accessibility checks, some tools focus on specific checkpoints that are usually not automatable or require more sophisticated methods to evaluate. Also, evaluation tools may have varying degrees of accuracy or support the evaluation of the same checkpoints in different ways (such as automatically or manually for example).
Configuration: How well does the evaluation tool adapt to the requirements of the users?
Accessibility checks
For specific Web sites, evaluation tool users may want to customize the built-in accessibility checks in order to achieve better performance. For example, tool users may want to suppress certain automated checks, or to modify the parameters which trigger dialogs and prompts that assist users with manual checks.
Reporting
Evaluation tools that generate reports sometimes provide capabilities to customize the format of these reports to different degrees. Customizing reports according to the roles of the developers (for example content author, programmer, project manager, ...) is especially useful for larger development teams.
Web site coverage
In some cases, evaluation tools can be configured to examine entire groups of related pages (such as department sub-sites, pages required to fulfil a specific task on a Web site, ...) rather than single Web pages. This feature may also be useful for Web site monitoring purposes.
Integration: How well does the evaluation tool integrate into the Web development environment of the users?
Platform support
Even though some evaluation tools may be available on more than one platform (hardware, operating system, and system configuration), they may sometimes not support the same features or perform equally on all platforms. It is important to ensure that the required tool features are supported on the platform where it will be deployed.
Software extension
Some evaluation tools integrate into existing development environments by providing plug-in interfaces for Web browsers or authoring tools (such as editors, content management systems, or save-as utilities). This feature may be important to some tool users even though such evaluation tools may sometimes be constrained by the application they are plugged into.
Data export
Some evaluation tools can export evaluation results to databases or other types of data processing tools such as analysis or reporting tools. Some of the commonly supported data formats which facilitate such data exchange are XML or Evaluation and Report Language (EARL).
Policy requirements: Which guidelines and policy requirements does the evaluation tool support?
Some evaluation tools provide support for several accessibility guidelines and national policy requirements. For organizations that are obliged to adhere to one or more national policy requirements, it is important that the selected evaluation tools adequately support these.
Reliability: How reliable are the results delivered by the evaluation tool?
Inaccurate results such as not detecting accessibility barriers or reporting non-existent barriers (that is, false positives or false negatives) decrease the reliability of the evaluation tool and hence decrease the efficiency of the evaluation. Currently there is no widely accepted single method to measure reliability so that careful assessments of the performance of evaluation tools with respect to the specific type of Web site need to be made.
Repair: How well does the evaluation tool assist developers in repairing inaccessible Web sites?
Even though repair is not part of the evaluation process, it is often the next logical step. Evaluation tools can assist developers in repairing accessibility barriers and raising the overall quality of Web sites by providing in-line repair options, or by providing additional information for possible repair measures.
Web technology support: How well does the evaluation tool support the relevant Web technologies?
There several types of Web technologies (such as HTML, XHTML, CSS, ...) and often several versions of each. Even though some of the more advanced Web technologies such as SMIL, SVG, or MathML are currently not widely supported by evaluation tools, it is important to select evaluation tools that best address the specific implementation of Web sites.

Related pages

This document is part of a multi-page Evaluating Web Accessibility resource suite that outlines different approaches for evaluating Web accessibility.