W3C logoWeb Accessibility Initiative (WAI) logo

WAI: Strategies, guidelines, resources to make the Web accessible to people with disabilities

[Draft] Conformance Evaluation of Web Sites for Accessibility

Page Contents

Note: This document is a draft [see change log in progress] and should not be referenced or quoted under any circumstances. This document is under development by the Education and Outreach Working Group (EOWG), and will be offered to other W3C groups and the public for review.

Introduction

A conformance evaluation combines automatic, semi-automatic, and manual testing of accessibility features. It is a technical assessment of the conformance to the Web Content Accessibility Guidelines (WCAG) 1.0 and does not include evaluation with users which is described in a separate document. Conformance evaluations require familiarity with Web mark-up languages; initial downloading and/or training on a variety of evaluation tools and approaches; configuration of browser settings; and experience in Web accessibility.

Scope of Conformance Evaluations

A properly conducted conformance evaluation can identify potentially major problems during the development phase for a new site; determine what level of accessibility a Web site meets; and/or provide assurance that a Web site meets a required level of accessibility. However, it does not include evaluation with users which is important to identify problems in how the technical solutions are being applied. Comprehensive evaluations of Web sites combine technical assessments and usability testing of accessibility features.

Steps To Conduct a Conformance Evaluation

A conformance evaluation includes all of the steps below except those that are explicitly identified as alternatives or optional. Note: while determining the scope of the evaluation is a key first step, and summarizing and reporting the results of evaluation is the logical conclusion, the order of the intervening steps is not crucial.

1. Determine and Disclose the Scope of the Conformance Evaluation

Determine and disclose scope of site to be evaluated and the targeted conformance level for the evaluation. Note: disclosure should be internal to the organization during evaluation; if conformance is claimed publicly, disclose externally (e.g. on the Web site).

  1. determine and disclose the target conformance level of WCAG 1.0.
  2. select a representative sampling of pages that match the following criteria:
    • include all pages on which people are more likely to enter your site ("welcome page" etc.)
    • include a variety of pages with different layouts and functionality, for example:
      • Web pages with tables, forms, or dynamically generated results
      • Web pages with informative images such as diagrams or graphs
      • Web pages with scripts or applications that perform functionality
    Note: there are special considerations for web sites with database driven dynamically generated web content.
  3. identify and disclose the entire Web site including all pages at a base URL for automatic and semi-automatic evaluation. Note: special considerations if testing of the entire site is not feasible (e.g. because of its unusual size or dynamic nature).

2. Use Automated or Semi-Automated Web Accessibility Evaluation Tools

  1. Validate markup including syntax and style sheets, using all applicable validators, on the selected sample of pages. Run at least one validation tool across entire Web site:
  2. Use at least two Web accessibility evaluation tools on the selected sample of pages and run at least one tool across entire Web site. Note any problems indicated by the tools. Note: Using at least two evaluation tools helps catch potential misidentification of accessibility problems that might result from using a single evaluation tool.

3. Carry Out Manual Evaluation of the Selected Web Page Sample

  1. Examine the selected sample of pages using checkpoints from the Checklist of Checkpoints for Web Content Accessibility Guidelines 1.0 that are applicable to your site. Note: Applicable can mean checkpoints that cannot be evaluated by automatic or semi-automatic tools; checkpoints that actually apply to the site (e.g. if site contains no audio content, skip those); and, as a minimum, those checkpoints that apply to the level of conformance you are evaluating.
  2. Examine the selected sample of pages with graphical user interface (GUI) browsers: select at least three different configurations from among the following variables: different graphical user interface browsers (such as Internet Explorer, Mozilla FireFox, Netscape Navigator, Opera, Safari, or others), in different versions (latest, older), running on different platforms (Windows, Linux, Mac) and making the following adjustments:
    1. turn off images, and check whether appropriate alternative text is available.
    2. turn off the sound, and make sure audio content is still available through text equivalents.
    3. use browser controls to vary font-size: verify that the font size changes on the screen accordingly; and that the page is still usable at larger font sizes.
    4. test with different screen resolution, and/or by resizing the application window to less than maximum, to verify that horizontal scrolling is not required (caution: test with different browsers, or examine code for absolute sizing, to ensure that it is a content problem not a browser problem)
    5. change the display color to gray scale (or print out page in gray scale or black and white) and observe whether the color contrast is adequate.
    6. without using the mouse tab through the links and form controls on a page, making sure that you can access all links and form controls, and that the links clearly indicate what they lead to.
    7. also examine page with scripts, style sheets, and applets not loaded

    Note: Browser extensions and other plug-in evaluation tools (such as AIS Toolbar for Internet Explorer, WAVE Toolbar for FireFox, Internet Explorer, and Netscape Navigator, or Web Developer Extension for FireFox) provide functionality to help perform many of these manual checks.

    Note: For reviewers who have disabilities, certain of the following steps may need to be done with another person who does not have the same disability.

  3. Examine the selected sample of pages with one text browser (such as Lynx) and one voice browser (such as Home Page Reader), and answer the following questions:
    1. With text browser:
      • is equivalent information and function (e.g. links and scripted events) available through the text browser as is available through the GUI browser?
      • is the information presented in a meaningful order when read serially?
    2. With voice browser:
      • is equivalent information available through the voice browser as is available through the GUI browser?
      • is the information presented in a meaningful order when spoken serially?
      Note: for settings where there is limited choice of assistive technologies, also perform a manual evaluation of the Web site with those assistive technologies; for instance, JAWS is the only screen reader translated into Danish, and therefore in Denmark a trained evaluator should evaluate the Web site using JAWS.
  4. Read over the selected sample of pages: is the text clear and simple to the extent appropriate for the purpose of the Web site? (For English sites, consider using Clear and Appropriate Language and Design (CLAD) test.)

4. Summarize and Reporte Evaluation Findings

Summarize any problems and best practices identified for each page type and a representative URL, and method by which they were identified. Recommend follow-up steps, potentially including:

Note: for evaluations using an expanded page selection instead of the entire Web site, apply what you've learned to other pages.