[DRAFT] Involving Users in Web Accessibility Evaluation
Web accessibility evaluation often focuses on assessing conformance to accessibility standards, such as the Web Content Accessibility Guidelines (WCAG). Broadening evaluation to involve people with disabilities helps better understand accessibility issues and implement more effective accessibility solutions.
For example, take a Web developer who does not know what it is like to use
a screen reader. To meet the Web accessibility
guideline "Provide text alternatives for all non-text content", the developer
alt="This image is a line art drawing of a dark green
magnifying glass. If you click on it, it will take you to the Search
page." However, observing a person use the site with a screen reader
will clearly show the developer that the alt text is ineffective and
alt="search" is all that is needed.
When Web developers, managers, and other project stakeholders see people with disabilities use their Web site, most are highly motivated by a new understanding of accessibility issues. Collaborating with people with disabilities who are target "users" of a Web site from early on in a project helps Web developers be more efficient in addressing accessibility, thus maximizing the results from investment in accessibility.
While involving users with disabilities in evaluation has many benefits, it alone cannot determine if a Web site is accessible. It is an integral part of a comprehensive evaluation of Web accessibility that includes:
- assessing conformance to WCAG to ensure that [technical@@basic] accessibility is provided to users with a range of disabilities and situations,
- [@@ [structured] evaluation by accessibility experts with first hand experience of how people with different disabilities interact with the Web, ("heuristic review")] and
- involving users with disabilities in evaluation.
This document introduces involving users in Web accessibility evaluation, provides guidance on some considerations for involving users, and links to additional information. It is part of a multi-page Evaluating Web Accessibility resource suite that outlines different approaches for evaluating Web accessibility.
A first step in evaluating Web accessibility is conducting a preliminary review of the Web site to check for any obvious accessibility problems. This allows you to fix any significant barriers before spending time evaluating with users with disabilities.
Users with disabilities can be included in a wide range of evaluation activities, from brief consultations to large-scale usability studies. There are many options in between these extremes:
- Informal evaluation of a specific accessibility issue can be as simple as asking a person in your office who uses a screen reader to find some data in an early draft of a data table that you are developing, observing their interaction, and discussing issues.
- Formal usability testing of a Web site follows established protocols to gather quantitative and qualitative data from representative users performing specific tasks. Formal usability tests can be optimized to focus on accessibility issues.
Conducting informal evaluations throughout development is more effective than formal usability testing at the end of a project. In most cases, including users in evaluation involves finding a few people with disabilities and including them throughout the development process to complete sample tasks on developing prototypes and discuss accessibility issues and solutions.
It is especially important when you are able to involve only a few users to carefully consider all feedback and avoid assuming that feedback from one person with a disability applies to all people with disabilities. A person with a disability does not necessarily know how other people with the same disability interact with the Web, nor know enough about other disabilities to provide valid guidance on other accessibility issues.
People with disabilities are as diverse as any people. They have different experiences, different expectations, and different preferences. They use different interaction techniques, different adaptive strategies, and different assistive technology configurations. People have different disabilities: visual, auditory, physical, speech, cognitive, and neurological disabilities — and some people have multiple disabilities. Even within one category, there is extreme variation; for example, "visual disability" includes people who have been totally blind since birth, people who have distortion in their central vision from age-related degeneration, and people who temporarily have blurry vision from an injury or disease.
Includes users with a variety of different disabilities and user characteristics. Selecting an optimum number of users with the best suited characteristics is important, challenging, and different based on the situation. In most cases evaluators have limited time and budget and cannot include many users in evaluation. There are resources on the Web that provide guidance on determining participant characteristics for a particular situation and on finding participants with disabilities.
A primary consideration in selecting users to help with evaluation is their experience interacting with the Web. For example, some assistive technologies (AT) are complicated and difficult to learn. A user with insufficient experience may not know how to use their AT effectively. Problems identified may be due to the user's lack of knowledge of the AT, not problems with the Web site being evaluated. On the other hand, a very advanced user might know uncommon work-arounds to overcome problems in the site that the "average" user would not be able to handle.
Just as with any evaluation with users, whether you include novice users, average users, or advanced users depends on your target users. For example, If you are developing a Web application to be used by accountants inside a company, you probably want advanced AT users; for a public Web site to apply for disability benefits, you want novice AT users.
Web accessibility depends on several components of Web development and interaction working together, including Web browsers, assistive technologies, and Web content. For any accessibility problems you identify, determine which components are responsible. For example, if a user has trouble with a data table, it could be because:
- the developer did not markup/code the data table properly, or
- the user's AT does not facilitate reading data tables effectively, or
- the user does not know how to use the AT's table reading feature.
In addition to revealing accessibility problems, evaluating with users with disabilities usually reveals general usability problems that impact all users, including users without disabilities.
As explained in the introduction, involving users with disabilities makes accessibility efforts more effective and more efficient. However, it alone cannot determine if a Web site is accessible; even large-scale usability studies cannot cover the diversity of disabilities, individuals, and situations. Combining user involvement with evaluating conformance to WCAG ensures that the broad range of accessibility issues are covered.
Reports should include the scope of the study and the evaluation parameters, such as the testing methods and the user characteristics. For example, if a study included only usability testing with participants who are blind, its report should clarify that it did not evaluate conformance to accessibility guidelines and it does not apply to all people with disabilities, in order to help readers draw appropriate conclusions.
This document briefly addresses a few points of a very complex topic. Many resources on other aspects of involving users in evaluation are available on the Web.
in the User-Centered Design Process provides guidance on
incorporating accessibility throughout design of Web sites and other
information technologies and communications technology products. The
chapter on Usability
Testing for Accessibility includes:
- Planning for usability testing for accessibility - determining participant characteristics, recruiting participants, providing compensation
- Preparing for usability testing for accessibility - preparing test materials, ensuring the facility is accessible, setting up and testing the assistive technology, conducting a pilot test, using screening techniques
- Conducting a usability test for accessibility - interacting with people with disabilities, setting up the room
- Reporting usability testing for accessibility - distinguishing between accessibility and usability issues, drawing conclusions, writing about people with disabilities
- Many books, articles, conference presentations, and other resources cover usability evaluation techniques, including different types of usability testing; test design; developing test protocol including questionnaires, tasks, data collection; conducting pilot tests; and how many participants to include in usability testing.
- There are organizations around the world that specialize in helping recruit people with disabilities and conduct evaluations with users with disabilities.
- user characteristics
- User characteristics typcally include things like age, job responsibilities, software, hardware, environment (for example, home, shared office, private office, shared public terminal), computer experience, and Web experience.User characteristics can also include type of disability, adaptive strategies ued, and experience with specific assisitive technologies.
- adaptive strategies
- Adaptive strategies are techniques that people with disabilities use to improve interaction with the Web, such as increasing the font size in a common browser. Adaptive strategies include techniques with mainstream browsers or with assistive technologies.
- assistive technologies
- Assistive technologies are software or equipment that is used by people with disabilities to improve interaction with the Web, such as screen readers that read aloud Web pages for people who cannot see or read text, and voice recognition software and selection switches for people who cannot use a keyboard or mouse.
- Web content
- Web "content" generally refers to the information in a Web page or Web application, including text, images, forms, sounds, and such. More specific definitions are available in the WCAG documents, which are linked from the Web Content Accessibility Guidelines (WCAG) Overview.
Note for usability professionals: When defining usability tests specifically to find accessibility issues, the protocol will be different from typical general usability test; for example:
- you would likely use a think-out-loud technique with high facilitator interaction;
- data collection would focus on understanding errors related to accessibility issues, rather than on time-on-task or user satisfaction; and
- tasks would concentrate on specific areas of concern for potential accessibility problems, rather than general site usage.