[EARLY ROUGH ROUGH DRAFT] Including Users in Web Accessibility Evaluation
Web accessibility evaluation has often been limited to assessing conformance to accessibility standards, such as the Web Content Accessibility Guidelines (WCAG). Broadening evaluation to include people with disabilities helps ensure that technical accessibility solutions are applied effectively.
For example, take a Web developer who has never heard of a screen reader.
In trying to meet the Web accessibility guideline: "Provide text alternatives
for all non-text content", he might code:
alt="This image is a line art
drawing of a dark green magnifying glass. If you click on it, it will take
you to the Search page." Just one time watching a user listen to the
site with a screen reader will clearly show the developer that his alt text
is ineffective and
alt="search" is all that is needed.
Including people who are target "users" of your Web site throughout Web site development helps your Web developers implement accessibility more effectively, thus maximizing your investment in accessibility. When development teams, managers, and other stakeholders have an opportunity to watch users with disabilities use their site, the results are almost always a powerful increase in understanding of basic user issues and accessibility solutions, and motivation for improving accessibility.
While this document focuses on including users with disabilities in evaluation, they really should be included throughout the development process, for example, integrated into a user-centered design (UCD) process.
A comprehensive evaluationof a Web site combines evaluation expertise to assess technical conformance and the experience of people with disabilities in usability testing for accessibility issues. This document introduces including users in Web accessibility evaluation, provides guidance on some considerations for including users, and lists additional information.
Users with disabilities can be included in a wide range of evaluation activities, from limited informal evaluation focusing specifically on accessibility issues to large-scale formal usability testing. There are many options in between these extremes:
- Informal evaluation of a specific accessibility issue can be as simple as asking a person down the hall who uses a screen reader to find a bit of data in an early draft of a data table that you are developing, and observing their interaction.
- Formal usability testing of an entire Web sitefollows established protocols to provide quantitative and qualitative data from representative users performing specific tasks in order to measure parameters such as ease-of-use, time to complete a task, error rate, and user satisfaction. Usability testing is a discipline with lots of stuff to know.
Few organizations have the resources to employ experts in usability testing. However, formal usability testing not required in most cases. Significant benefit can be realized by including users with disabilities without much investment in formal testing. Below are some tips. Learn more about doing more, too!
People with disabilities are as diverse as any people. They have different experiences, different expectations, and different preferences. They use different interaction techniques, different adaptive strategies, and different assistive technology configurations. People have different disabilities: visual, auditory, physical, speech, cognitive, and neurological disabilities. Even within one category, there is extreme variation; for example, "visual disability" includes people who been totally blind since birth, people who have distortion in their central vision from age-related degeneration, and people who temporarily have blurry vision from an injury or disease.
Include as broad a variety of users with different disabilities as you can, and avoid the pitfall of only including people who are blind. In most cases evaluators have limited time and budget for evaluation and cannot include many users in evaluation. There are resources on the Web that provide additional guidance on "determining participant characteristics" for users with disabilities in accessibility evaluation.
When you are only able to have a small number of users, it is especially important to carefully consider all feedback and avoid assuming that feedback from one person with a disability applies to all people with disabilities. A person with a disability does not necessarily know how other people with the same disability interact with the Web, nor know enough about other disabilities to provide valid guidance. Don't make accessibility design decisions based only on the recommendations of one person with a disability. What works for one person might not work for everyone with that disability or other disabilities. Be careful not to assume that all users, including users with disabilities, use your product the same way.
A primary consideration in choosing users to help with evaluation is their experience interacting with the Web. Some people with disabilities use assistive technologies (AT) such as screen readers that read aloud the Web page, screen magnifiers, and different pointing devices instead of mice. Some AT is complicated and difficult to learn. A user with insufficient experience may not know how to use their AT effectively. Problems identified may be due to their lack of knowledge of the AT (for example, not knowing how to read data tables or navigate through headings using a screen reader), not problems with the Web site being evaluated. On the other hand, a very advanced user might know uncommon work-arounds to overcome problems that would be impossible for the average user to handle.
Just as with any evaluation with users, whether you include novice users, "average" users, or advanced users depends on your target users. For example, If you are developing a Web application to be used by accountants inside a company, you probably want advanced AT users; for a public Web site to apply for disability benefits, you want novice AT users.
Web accessibility depends on several components of Web development and interaction working together, including Web browsers, AT, and Web content. Users are likely to identify accessibility issues in different components; for example, when evaluating a Web site, the user might find an accessibility problem with the AT. Here are some points related to identifying and addressing issues:
- browser, assistive technology - different browsers and ATs have different features and interactions.
- user knowledge/experience - as described above, an issue might be due to the users lack of experience.
- Web site - most problems will probably be things that you can fix in the Web site.
Additionally, users with disabilities will almost certainly find general usability problems that impact all users, including users without disabilities. When the evaluation is being used to improve a Web site, it is rarely necessary to distinguish between usability and accessibility issues. However, when the evaluation data will be used to make public statements about accessibility, it can be vital to distinguish between usability and accessibility issues.
It is very important that any reporting of accessibility evaluations clearly indicate the evaluation parameters, such as the methodology and the user characteristics. The report should clearly indicate what it asserts and what it does not assert, especially when people who are not familiar with accessibility evaluation will likely read the report. For example, successful user evaluation with a limited number of users does not guarantee that the Web site is accessible to all people with disabilities, nor that it conforms to specific guidelines.
For More Information
This document just touches on a few points on topics that are vast. There is lots more information available out there on all aspects of including users in evaluation.
The other pages in the Evaluating Web Sites for Accessibility resource suite provide related information; for example, before evaluating with users, conduct a preliminary evaluation to make sure that there are no serious accessibility problems that will prevent users with disabilities from using the Web site. [Other WAI resources that provide relevant information including How People with Disabilities Use the Web.]
Additional resources are available on the Web on topics such as:
- Usability evaluation techniques - different types of usability testing; test design; developing test protocol including questionnaires, tasks, data collection; conducting pilot tests; how many participants to include in usability testing
- Planning for usability testing for accessibility - determining participant characteristics, recruiting participants, providing compensation
- Preparing for usability testing for accessibility - preparing test materials, ensuring the facility is accessible, setting up and testing the assistive technology, conducting a pilot test, using screening techniques
- Conducting a usability test for accessibility - interacting with people with disabilities, setting up the room
- Reporting usability testing for accessibility - distinguish between accessibility and usability issues, drawing conclusions, writing about people with disabilities
- Incorporating accessibility throughout a user-centered design (UCD) process