This is an old draft. The published version of this document is at www.w3.org/WAI/bcase/.

Evaluating Web Sites for Accessibility

This is an old draft. The published version of this document is at www.w3.org/WAI/bcase/.

Note: Note: This draft WAI Resource is under development by W3C/WAI's Education and Outreach Working Group (EOWG). We invite review and discussion. Please address your feedback to wai-eo-editors@w3.org, a mailing list with a public archive. Change log available.

Introduction
Preliminary review
Conformance Evaluation to WCAG 1.0
Considerations for specific contexts:

1. Introduction

This document outlines approaches for preliminary review Web site accessibility, and for evaluation of conformance to the Web Content Accessibility Guidelines 1.0. It includes tips for evaluation during development of Web sites, and for monitoring of established Web sites. The measures described here are intended to supplement an organization's existing procedures for content management and quality assurance on their Web sites.

There are a variety of tools and approaches for evaluating Web site accessibility. No single evaluation tool yet provides comprehensive information or captures all problems with regard to the accessibility of a site; therefore evaluation involves a combination of approaches. Goals for evaluating Web sites vary, and require different approaches to meet those goals:

Preliminary review can:
- identify general kinds of barriers on a Web site.
Conformance evaluation can:
- catch major problems during development phase of a new site;
- determine the WCAG 1.0 conformance level for an existing Web site;
- demonstrate that a Web site meets a given WCAG 1.0 conformance level.
Conformance evaluation, plus review of procedures for ongoing monitoring, can:
- help ensure that a site will maintain a given conformance level in the future.

2. Preliminary Review

A preliminary review may help to quickly identify the scope of problems on a Web site. However, the preliminary review will not catch all of the problems on a site and should not be used to determine conformance level. A preliminary review does not include perspectives from a variety of users with disabilities nor does it touch or test every aspect of a site.

A preliminary review combines some manual checking of representative pages on a Web site, along with the use of several semi-automatic accessibility checkers. Reviewers do not need to know Web mark-up languages, but should be able to download software and familiarize themselves with some online tools, and change certain settings on their browser.

To conduct a preliminary review, complete all five steps below.

Select a representative sampling of different kinds of pages from the Web site to be reviewed; must include entry page(s) ("welcome page" etc.)
Use a graphical user interface (GUI) browser (such as Internet Explorer, Netscape Navigator, or Opera) and examine the selection of pages while adjusting the browser settings as follows (NOTE: For reviewers who have disabilities, certain of the following steps may need to be done with another person who does not have the same disability.)
1. turn off images, and make sure that the information is presented in an appropriate sequence relative to the visual presentation on the GUI site.
2. turn off the sound, and make sure audio content is still available through text equivalents.
3. change the font size (larger and smaller) in the browser, and observe whether the page is still readable.
4. set screen resolution to 640 x 480 and observe whether or not this forces the page into horizontal scrolling
5. change the display color to black and white (or print out page on black and white printer) and observe whether color contrast is adequate.
6. put away the mouse and tab through the links and form controls on a page, making sure that you can access all links and form controls, and that the links clearly indicate what they lead to.
Use a voice browser (such as Home Page Reader) or a text browser (such as Lynx) and examine the Web site while answering these questions (NOTE: experienced users of screen readers may substitute a screen reader for a voice or text browser, but if blind, may need a sighted partner to compare information available visually; if sighted, listen to it with eyes closed, then open eyes and confirm whether the information is equivalent)
1. is equivalent information available through the voice or text browser as is available through the GUI browser?
2. is the information presented in a similar logical order as when viewed through the GUI browser?
Use two accessibility evaluation tools and note any problems indicated by the tools, for example:
- WAVE (an online accessibility assessment tool that flags any items on a Web page which should be examined for potential accessibility problems, and provides a description of what the problem might be)
- Bobby (an online or downloadable accessibility checker which provides a semi-automated assessment of accessibility problems on a Web page or group of Web pages; it can identify many problems on sites, and lists problems which it is not able to evaluate automatically and which require manual review)
- A-Prompt (a tool which identifies potential accessibility problems and provides guided editing to correct the problems)
Summarize results
1. summarize the types of problems encountered, as well as best practices that should be continued or expanded on the site
2. indicate the method by which problems were identified, and clearly state that this was not a full conformance evaluation
3. recommend follow-up steps, including full conformance evaluation which includes validation of markup and other tests, and ways to address any problems identified.

3. Conformance Evaluation to WCAG 1.0

A comprehensive evaluation combines semi-automatic, manual, and usability testing. Comprehensive evaluations require familiarity with Web mark-up languages; initial downloading and/or training on a variety of evaluation tools and approaches; configuration of browser settings; and coordination with reviewers with a variety of disabilities. Evaluation with users is important as it helps to identify problems in how the technical solutions are being applied.

A properly conducted comprehensive evaluation can identify potentially major problems during the development phase for a new site; determine what level of accessibility a Web site meets; and/or provide assurance that a Web site meets a required level of accessibility.

A comprehensive evaluation includes all of the steps below except those that are explicitly identified as alternatives or optional:

Identify scope of site to be evaluated, and the targeted conformance level for the evaluation:
1. identify a "page selection" which includes at least one of each different type of page on the site, and all top pages or entry pages to the site.
2. identify and clearly disclose either:
  - the "entire Web site" including all pages at a base URL;
  - or an "expanded page selection," to be clearly explained and disclosed on the Web site. Suggestions for inclusions in this expanded page selection: pages from different sections of the Web site; pages representing different "look & feel"; pages representing different development tools and processes including those generated from databases; pages produced under different guidelines; "contact us" pages; pages critical to your business; etc. If any area of a site is excluded from evaluation, be sure to disclose this information.
3. identify the target conformance level of WCAG 1.0
Semi-automatic and automatic evaluation
1. Validate markup including syntax and style sheets, using all applicable validators, on page selection. Run at least one validation tool across entire Web site or expanded page selection.
2. Use at least two accessibility evaluation tools, on page selection, and run at least one tool across entire Web site.
  - select from examples listed under #4 in preliminary review above
Manual evaluation
1. Examine page selection using relevant level checkpoints from the Checklist of Checkpoints for Web Content Accessibility Guidelines 1.0.
2. Examine page selection with graphical user interface (GUI) browsers: select at least three different configurations from among the following variables: different graphical user interface browsers (Internet Explorer, Netscape, Opera), in different versions (latest, older), running on different platforms (Windows, Linux, Mac) and making the following adjustments:
  1. use all approaches listed under #2 in preliminary review above, plus:
  2. also examine page with scripts, style sheets, and applets not loaded
3. Examine page selection with one text browser (such as Lynx) AND one voice browser (such as Home Page Reader), and answer the following questions. (NOTE: For settings where there is limited choice of assistive technologies, also perform a manual evaluation of the Web site with those assistive technologies; for instance, JAWS is the only screen reader translated into Danish, and therefore in Denmark, a trained evaluator should evaluate the Web site using JAWS.)
  1. use both questions listed under #3 in preliminary review above, plus:
  2. is the text clear and simple to the extent appropriate for the purpose of the Web site? (For English sites, consider using Clear and Appropriate Language and Design (CLAD) test.)
Usability evaluation
- Have people with different disabilities, different levels of technical expertise, and different levels of familiarity with the site, using a variety of assistive technologies and adaptive strategies, review page selection and explore freely across entire Web site or expanded page selection. [NOTE: Sometimes this is done in the context of a testing laboratory, but it is also valuable for people to evaluate the site in their typical Web environment. Different levels of user technical experience and familiarity with the site are important, and note that these will change over time, as people participate in reviewing the site.]
- Ask testers to try to find answers to the most common questions for which people visit the Web site. Note areas where it is difficult or impossible the use the Web site.
Summarize and follow-up
1. Summarize any problems and best practices identified for each page type and a representative URL, and method by which they were identified
2. Recommend follow-up steps, potentially including:
  - repair of accessibility barriers identified through conformance evaluation process. NOTE: for evaluations using "expanded page selection" instead of "entire Web site," apply what you've learned to other pages
  - expanding best practices on site
  - ongoing maintenance and monitoring of site

4. Considerations for Specific Contexts

Evaluation during the development process

Evaluation during the development process is essential. It can sometimes be difficult, as both in-house and subcontracted Web developers sometimes prefer to establish the site design and demonstrate their progress before getting feedback. However, accessibility issues identified early are easier to correct and avoid. Effective evaluation during the design period can include:

establishing clear requirements for the expected accessibility conformance level
involvement in initial planning meetings for the site
agreeing on a review schedule during the development process
providing information on evaluation approaches so that the developers can at least do preliminary reviews on their own

Ongoing monitoring

To maximize likelihood that a Web site will maintain a given conformance level in the future, the following provisions should be in place:

clear statement of expected conformance level and scope of Web site it applies to
clearly identified individuals responsible for monitoring the site, and follow-up procedures they can use to rapidly bring non-conformant pages into conformance
clear expectations with regard to frequency, method, and scope of evaluations
processes for validating and evaluating all new types of pages before they are added to the site
software to facilitate evaluation
incorporation in Web site of address for feedback on accessibility of site

Evaluation of legacy sites

Occasionally Web sites that are "frozen" (legacy; no longer actively maintained ) are found to have substantial accessibility problems. It can be difficult to determine how to address these. It is helpful to:

identify who the current owner is, and whether they have any obligation or interest in making the site accessible;
after evaluating the site, outline the changes that would be required to retrofit the site for accessibility;
identify and propose resources and a timeline for retrofitting the site;
disclose accessibility problems on the site.

Last updated 5 October 2001 by Judy Brewer (jbrewer@w3.org) with assistance from members of EOWG.

Copyright © 2001 W3C (MIT, INRIA, Keio ), All Rights Reserved. W3C liability, trademark, document use and software licensing rules apply. Your interactions with this site are in accordance with our public and Member privacy statements.