Note: This document is a draft [see change log in progress] and should not be referenced or quoted under any circumstances. This document is under development by the Education and Outreach Working Group (EOWG), and will be offered to other W3C groups and the public for review.
Web accessibility evaluation tools are software programs or online services that help determine if a Web site is accessible. This document highlights different features of evaluation tools which can assist during evaluations such as the methodologies described in the WAI resource Evaluating Web Sites for Accessibility.
Web accessibility evaluation tools can significantly reduce the time and effort required to carry out evaluations. When used carefully throughout the design, implementation, and maintenance phases of Web development, these tools can assist their users in preventing accessibility barriers, repairing encountered barriers, or improving the overall quality of Web sites. The following are ways in which tools can assist users in evaluating Web sites for accessibility; some tools can perform both:
- determine the conformance of Web sites to accessibility checks which can be executed automatically;
- effectively assist tool users in evaluating accessibility checks which need to be evaluated manually.
Many accessibility checks require human judgement and must be evaluated manually using different techniques. Also, in some cases evaluation tools are prone to producing false or misleading results such as not identifying and signal incorrect code. Evaluation tools require operation by users who understand the capabilities and limitations of the tools in order to achieve precise results. Web accessibility evaluation tools can not make Web sites accessible, they can only assist in doing so.
WAI encourages the development and evolution of Web accessibility evaluation tools, and maintains an extensive list of evaluation, repair and transformation tools. WAI does not endorse or promote any single tool or vendor.
Web accessibility evaluation tools can be used throughout all stages of Web site development. For example at the early design stage, Web designers may be interested in using tools that help them understand how the site structure, navigation, or look-and-feel perform with respect to accessibility requirements. Later at the implementation stage, developers may be more interested in tools that help them assess the accessibility of the underlying code which is generated by the Web authoring tools (such as editors or content management systems). Web content authors, project managers, and other types of Web site developers have further requirements for evaluation tools that help them fulfill their respective tasks.
According to the specific organization and Web site for which evaluation tools will be used, different characteristics and features of Web accessibility evaluation tools may be more or less important to the tool users. For example, an organization may choose to use fully automated evaluation tools which can examine the whole Web site, and additionally evaluate samples of pages using other types of tools in order to compensate the limitations of fully automated checking. The following are some of the factors which may be considered during the selection of Web accessibility evaluation tools:
- Organizational Structure and Development Process
- For larger organizations or when several types of Web developers (such as designers, programmers, content authors, quality assurance reviewers, or others) participate in the development of the site, it may be beneficial to use a combination of evaluation tools in order to balance the capabilities of the tools, and to address the needs of the different user roles throughout the development stages.
- Complexity and Size of the Web Site
- Examples of complex Web sites are sites that make heavy use of scripting to generate Web pages or to provide functionality in them; employ multimedia content such as audio or video files; incorporate advanced technologies such as SMIL, SVG, or MathML; or are very large and difficult to maintain. In such cases, more specialized evaluation tools might be more useful, even though they may have other limitations.
- Skills and Knowledge of the Web Developers
- Some evaluation tools require more knowledge of accessibility requirements or markup code (such as HTML, CSS, ...) than others. Also, some evaluation tools can support Web developers in learning such skills differently than others. It is important to identify the intended tool users and their requirements when selecting appropriate evaluation tools for a specific organization.
- Pre-Existing Web Development Environment
- It is often beneficial to deploy evaluation tools that work well with the existing operating systems and other development infrastructure. Also, sometimes evaluation tools are plug-in extensions for Web authoring tools (such as editors, content management system, or save-as utilities) or browsers; or they can export evaluation reports in different formats (for example, export results to a database).
Characteristics of Web accessibility evaluation tools are overall functionalities and behaviour of the tools that characterize their usage, and how they fit into the development process; some evaluation tools provide more than one mode of operation. The following are some of the commonly employed characteristics of evaluation tools:
- Report Generating Tools
- Report generating evaluation tools are usually designed to evaluate Web sites with little or no user interaction. The results of the accessibility checks that the tools execute are summarized in reports which can often be customized according to the needs of the users. Report generating tools are very useful in quickly determining the conformance of Web sites to checkpoints which can be evaluated automatically, and for identifying remaining checkpoints that need to be evaluated manually.
- Wizard Interface Tools
- Wizard-based evaluation tools guide users through sequences of checks step by step. Sometimes these tools are able to execute some of the accessibility checks automatically and prompt the users to manually evaluate the remaining checks. For example, an evaluation tool with a wizard interface may be able to automatically check if the images on a Web site have text descriptions, then display the images with their corresponding descriptions to the users to evaluate how appropriate these descriptions are.
- In-Page Feedback Tools
- In-page feedback evaluation tools insert (temporary) icons and markup into the code of the Web pages to display the results of automated accessibility checks and their corresponding location within the pages. Sometimes, other types of icons are also inserted into the Web pages to assist the manual evaluation of checkpoints. For example, some tools may insert icons to indicate the hierarchy of the page headings or lists, or the sequence of table cells that may be perceived by some Web site users.
- Transformations Tools
- Transformation tools modify the appearence of the Web sites to help identify conceptual design issues with regard to Web accessibility. For example, a transformation tool may present the content of Web sites in text only, without color, or read the content aloud. These types of evaluation tools are usually especially useful to mediate the limitations of automated accessibility checks and to support the users in evaluating checkpoints that need to be evaluated manually.
The following is a (non-exhaustive) collection of features which may help users to compare and assess Web accessibility evaluation tools for their specific needs. Some evaluation tools provide some of these features with varying adequacy so that carefull analysis needs to be made when assessing them; sometimes the tool vendors can provide additional information about how their tools support these or other features.
- Accessibility: How accessible is the evaluation tool for people with disabilities?
- It is equally important to ensure that people with disabilities can effectively contribute to the Web, as it is for them to be able to effectively use the Web. Evaluation tool vendors can provide accessibility in all parts of a tool (user interface, documentation, or generated reports) by following the Authoring Tool Accessibility Guidelines 1.0.
- Checkpoint Coverage: Which checkpoints is the evaluation tool able to adequately address?
-
- Automatic
- The degree of automation in evaluation tools significantly impacts the efficiency of the evaluation process; some vendors have developed specialized accessibility checks for different checkpoints to increase the automation within their tools.
- Manual
- Many accessibility checks require human judgement and must be evaluated manually; some tools assist the users in making the correct evaluation judgements through different approaches, for example by providing additional educational resources.
- Configuration: How well does the evaluation tool adapt to the requirements of the users?
-
- Accessibility Checks
- For specific Web sites, evaluation tool users may want to customize the built-in accessibility checks in order to achieve better performance. For example, tool users may want to suppress certain automated checks, or to modify the parameters which trigger dialogs and prompts that assist users with manual checks.
- Reporting
- Evaluation tools that generate reports sometimes provide capabilities to customize the format of these reports to different degrees. Customizing reports according to the roles of the developers (for example content author, programmer, project manager, ...) is especially useful for larger development teams.
- Web Site Coverage
- In some cases, evaluation tools can be configured to examine entire groups of related pages (such as department sub-sites, pages required to fulfil a specific task on a Web site, ...) rather than single Web pages. This feature may also be useful for Web site monitoring purposes.
- Integration: How well does the evaluation tool integrate into the Web development environment of the users?
-
- Platform Support
- Even though some evaluation tools may be available on more than one platform (hardware, operating system, and system configuration), they may sometimes not support the same features or perform equally on all platforms. It is important to ensure that the required tool features are supported on the platform where it will be deployed.
- Software Extension
- Some evaluation tools integrate into existing development environments by providing plug-in interfaces for Web browsers or authoring tools (such as editors, content management systems, or save-as utilities). This feature may be important to some tool users even though such evaluation tools may sometimes be constrained by the application they are plugged into.
- Data Export
- Some evaluation tools can export evaluation results to databases or other types of data processing tools such as analysis or reporting tools. Some of the commonly supported data formats which facilitate such data exchange are XML or Evaluation and Report Language (EARL).
- Policy Requirements: Which guidelines and policy requirements does the evaluation tool support?
- Some evaluation tools provide support for several accessibility guidelines and national policy requirements. For organizations that are obliged to adhere to one or more national policy requirements, it is important that the selected evaluation tools adequately support these.
- Reliability: How reliable are the results delivered by the evaluation tool?
- Inaccurate results such as false negatives (not detecting accessibility barriers), or false positives (detecting wrong ones) decrease the reliability of the evaluation tool and hence decrease the efficiency of the evaluation. Currently there is no widely accepted single method to measure reliability so that careful assessments of the performance of evaluation tools with respect to the specific type of Web site need to be made.
- Repair: How well does the evaluation tool assist developers in repairing inaccessible Web sites?
- Even though repair is not part of the evaluation process, it is often the next logical step. Evaluation tools can assist developers in repairing accessibility barriers and raising the overall quality of Web sites by providing in-line repair options, or by providing additional information for possible repair measures.
- Web Technology Support: How well does the evaluation tool support the relevant Web technologies?
- There several types of Web technologies (such as HTML, XHTML, CSS, ...) and often several versions of each. Even though some of the more advanced Web technologies such as SMIL, SVG, or MathML are currently not widely supported by evaluation tools, it is important to select evaluation tools that best address the specific implementation of Web sites.
Last modified: $Date: 2005/03/11 15:16:38 $ by $Author: shadi $
Last updated 24 February 2005 by Shadi Abou-Zahra. Editors: Shadi Abou-Zahra and Judy Brewer, with assistance from participants of the EOWG.
Copyright © 2005 W3C (MIT, ERCIM, Keio ), All Rights Reserved. W3C liability, trademark, document useand software licensingrules apply. Your interactions with this site are in accordance with our public and Member privacy statements.