W3C logoWeb Accessibility initiative

WAI: Strategies, guidelines, resources to make the Web accessible to people with disabilities

Involving Users in Evaluating Web Accessibility

Page Contents

Introduction

Web accessibility evaluation often focuses on conformance to accessibility standards such as WCAG. While conformance is important, there are many benefits to evaluating with real people to learn how your website or web tool really works for users and to better understand accessibility issues. Evaluating with users with disabilities and with older users identifies usability issues that are not discovered by conformance evaluation alone.

This page is part of a multi-page Evaluating Web Accessibility resource suite that outlines different aspects of evaluating web accessibility. It is the second of two pages on including users in web projects; please read Including Users in Web Projects for Better, Easier Accessibility, which covers broader issues of including users early in website design, tool development, standards, and other web projects.

Initial Review

A first step in evaluating web accessibility is reviewing the website to check for any obvious accessibility problems. The Preliminary Review and Conformance Evaluation pages provide guidance.

Even web developers with little accessibility knowledge can find some accessibility issues through a preliminary review. An accessibility expert with first-hand experience of how people with different disabilities interact with the web can:

The initial review identifies any significant accessibility barriers to fix before evaluating with users. It also helps define what to focus on for evaluation with users.

Range of User Evaluation

Users with disabilities and older users can be included in a wide range of evaluation activities, from brief consultations to large-scale usability studies. There are many options in between these extremes:

What type of evaluation you do depends on factors such as the stage in your project, for example, initial investigation of design ideas, testing specific areas of prototypes, or reviewing final designs.

Conducting informal evaluations throughout development is more effective than only formal usability testing at the end of a project.

Basics

In most cases, including users in evaluation involves:

See Including Users in Web Projects for Better, Easier Accessibility for guidance on getting a range of users and users' experience interacting with the web

Just as with any evaluation with users, whether you include novice, average, or advanced users depends on your target users. For example, if you are developing a web application to be used by accountants inside a company, you probably want advanced assistive technology users; for a public website to apply for disability benefits, you want novice assistive technology users.

Caution: Carefully consider all feedback and avoid assuming that feedback from one person with a disability applies to all people with disabilities. A person with a disability does not necessarily know how other people with the same disability interact with the web, nor know enough about other disabilities to provide valid guidance on other accessibility issues.

Note: In addition to finding accessibility problems, evaluating with users with disabilities usually reveals general usability problems that impact all users, including those without disabilities.

Analyzing Accessibility Issues

Web accessibility depends on several components of web development and interaction working together, including web browsers, assistive technologies (AT), and web content. Accessibility problems can be caused by one or more different components. For example, if a user who cannot use a mouse has trouble with keyboard access, it could be because:

Combine User Evaluation with Standards

Involving users with disabilities in evaluation has many benefits; however, it alone cannot determine if a website is accessible. Combine user involvement with evaluating conformance to WCAG to ensure that accessibility is provided to users with a range of disabilities and situations.

Drawing Conclusions and Reporting

Be careful drawing conclusions from limited evaluations or studies. Results from only a couple of people with disabilities cannot be generalized to apply to all people with similar disabilities or people with other disabilities. See the Caution above.

Reports should include the scope of the study and the evaluation parameters, such as the testing methods and the user characteristics. For example, if a study included only usability testing with participants who are blind, its report should clarify that it did not evaluate conformance to accessibility guidelines and that it does not apply to all people with disabilities. Thus the report can help readers draw appropriate conclusions.

While small studies often provide useful information, they are not robust enough to provide statistical significance.

Note for usability professionals

When specifically exploring accessibility barriers, the protocol is usually different from a general usability test; for example:

Note that is is also important to evaluate general usability, user satisfaction, and other such criteria for users with disabilities.

The More Information section below includes additional guidance specifically for usability professionals.

More Information and Guidance

This document briefly addresses a few points of a very complex topic. Many resources on other aspects of involving users in evaluation are available on the Web.