Editors Draft: $Date: 2009/11/12 15:22:17 $ [changelog]
Status: This document is a draft revision and should not be referenced or quoted under any circumstances. The published version this document is at: www.w3.org/WAI/eval/users.html. Please send comments to wai-eo-editors@w3.org (a publicly archived list).

[Draft Revision] Involving Users in Evaluating Web Accessibility

Introduction

Web accessibility evaluation often focuses on evaluating conformance to accessibility standards, such as the Web Content Accessibility Guidelines (WCAG). Broadening evaluation to involve users helps better understand accessibility issues and implement more effective accessibility solutions. Evaluating with users with disabilities and older users identifies usability issues that are not discovered by conformance evaluation alone.

This page is part of a multi-page Evaluating Web Accessibility resource suite that outlines different approaches for evaluating web accessibility. It is the second of two pages on including real users in accessibility.

Read first: Including Users for Better, Easier Accessibility to learn the benefits of including users early in the development process to help you more efficiently develop accessible websites that work well for people with accessibility needs, thus maximizing your return on any investment (ROI) in accessibility.

Note: Involving users with disabilities in evaluation has many benefits; however, it alone cannot determine if a website is accessible. Combine user involvement with evaluating conformance to WCAG to ensure that accessibility is provided to users with a range of disabilities and situations.

Initial Review

A first step in evaluating web accessibility is conducting a preliminary review of the website to check for any obvious accessibility problems.

Even web developers with little accessibility knowledge can find some accessibility issues through a preliminary review. An accessibility expert with first-hand experience of how people with different disabilities interact with the web can:

evaluate accessibility issues for a broad range of users, which might not be found by individual users;
help fix any known barriers before bringing in users; and
focus the evaluation with users on potential areas of concern.

The initial review identifies any significant accessibility barriers to fix before evaluating with real users. It also helps define what to focus on for evaluation with users.

Range of User Evaluation

Users with disabilities and older users can be included in a wide range of evaluation activities, from brief consultations to large-scale usability studies. There are many options in between these extremes:

Informal evaluation of a specific accessibility issue can be as simple as asking someone you know who uses a screen reader, someone with other disabilities, or even your grandmother, to find some data in an early draft of a data table that you are developing, observing her interaction, and discussing issues.
Formal usability testing of a website follows established protocols to gather quantitative and qualitative data from representative users performing specific tasks. Formal usability tests can be optimized to focus on accessibility issues.

What type of evaluation you do depends on factors such as the stage in your project, for example, initial investigation of design ideas, testing specific areas of prototypes, or reviewing final designs.

Conducting informal evaluations throughout development is more effective than only formal usability testing at the end of a project.

Basics

In most cases, including users in evaluation involves:

getting a few people with disabilities - and depending on your target audience, older users,
including them throughout the development process to complete sample tasks on prototypes so you can see how the different aspects of the design and coding could be improved, and
discussing accessibility issues with them.

See Including Users for Better, Easier Accessibility for guidance on getting a range of users, users' experience interacting with the web, and analyzing accessibility issues.

Just as with any evaluation with users, whether you include novice, average, or advanced users depends on your target users. For example, if you are developing a web application to be used by accountants inside a company, you probably want advanced assistive technology users; for a public website to apply for disability benefits, you want novice assistive technology users.

Caution: Carefully consider all feedback and avoid assuming that feedback from one person with a disability applies to all people with disabilities. A person with a disability does not necessarily know how other people with the same disability interact with the web, nor know enough about other disabilities to provide valid guidance on other accessibility issues.

Note: In addition to finding accessibility problems, evaluating with users with disabilities usually reveals general usability problems that impact all users, including those without disabilities.

Analyzing Accessibility Issues

Web accessibility depends on several components of web development and interaction working together, including web browsers, assistive technologies (AT), and web content. For any accessibility problems you identify, determine which components are responsible. For example, if a user who cannot use a mouse has trouble with keyboard access, it could be because:

the developer did not markup/code the web page properly, or
the browser or media player isn't handling the markup properly, or
the user's AT isn't handling the markup properly, or
the user doesn't know how to use the browser, media player, or AT's keyboard access features, or
the page is poorly designed and it is a general usability problem for all users, including those without disabilities.

Drawing Conclusions and Reporting

Be careful drawing conclusions from limited evaluations or studies. See the Caution above.

Reports should include the scope of the study and the evaluation parameters, such as the testing methods and the user characteristics. For example, if a study included only usability testing with participants who are blind, its report should clarify that it did not evaluate conformance to accessibility guidelines and that it does not apply to all people with disabilities. Thus the report can help readers draw appropriate conclusions.

While small studies often provide useful information, they are not robust enough to provide statistical significance.

Note for usability professionals

When defining usability tests specifically to find accessibility issues, the protocol will be different from a typical general usability test; for example:

you would likely use a think-aloud technique with high facilitator interaction;
data collection would focus on understanding errors related to accessibility issues, rather than on time-on-task or user satisfaction; and
tasks would concentrate on specific areas of concern for potential accessibility problems, rather than general site usage.

The For More Information section below provides additional guidance specifically for usability professionals.

For More Information

This document briefly addresses a few points of a very complex topic. Many resources on other aspects of involving users in evaluation are available on the Web.

Involving Users in Web Development for Better, Easier Accessibility @@
Relationship between Web Accessibility and Usability@@
@@ Consider which (or all) of the existing references should be included
@@ Consider if any of the external references should be included

Terminology and Notes

adaptive strategies: Adaptive strategies are techniques that people with disabilities use to improve interaction with the web, such as increasing the font size in a common browser. Adaptive strategies include techniques with mainstream browsers or with assistive technologies.
assistive technologies: Assistive technologies are software or equipment that people with disabilities use to improve interaction with the web, such as screen readers that read aloud web pages for people who cannot read text, screen magnifiers for people with some types of low vision, and voice recognition software and selection switches for people who cannot use a keyboard or mouse.
user characteristics: User characteristics typically include things like age, job responsibilities, software, hardware, environment (for example, home, shared office, private office, shared public terminal), computer experience, and web experience.User characteristics can also include type of disability, adaptive strategies used, and experience with specific assistive technologies.
web content: Web "content" generally refers to the information in a web page or web application, including text, images, forms, sounds, and such. More specific definitions are available in the WCAG documents, which are linked from the Web Content Accessibility Guidelines (WCAG) Overview.

Site Navigation