Site Navigation: W3C Home > WAI Home > WAI Web Site Redesign Project > WAI Site Evaluation Summary

Methodology

Study Overview

In the fall of 2003 the W3C Web Accessibility Initiative (WAI) and the American Institutes for Research (AIR) conducted a series of usability tests of the WAI Web site: http://www.w3c.org/wai/ on October 28th-29th, and November 12th in our usability research facility in Concord, Massachusetts.

The primary objectives of the usability tests were to: observe and evaluate the user experience as it relates to the navigation, information architecture, design, layout, structure, and information presentation across the Web site by examining users' ability to navigate the Web site and to complete key tasks. In addition, we sought to establish a baseline from which to measure the efficacy of future redesign prototypes, to gain a greater understanding of the strengths and weaknesses of the current site, and to identify ways to improve the WAI site design.

Test Environment

We conducted the eight individual usability test sessions in AIR's usability laboratory facilities in Concord, MA. The laboratory utilized for the study featured two rooms: one for observers and another for the session moderators and study participants. A one-way mirror, through which the WAI and AIR personnel conducting the study could observe the participant without being seen, separated the two rooms. The usability lab was outfitted with closed circuit and stand-alone cameras in order to record the participant's comments, posture, facial features, and interactions with the WAI Web site. During the individual usability test sessions, a moderator was present alongside the participant to facilitate their exploration of the WAI Web site.

Prior to the participants' arrival we set up a table for computer use, and placed two chairs next to it, one for the individual participants, and one for the session moderator. In addition we ensured that camera angles and lighting conditions favored the clear capture of participants' facial expressions and used a scan-converter to capture the participants' on-screen mouse movements and interactions. A TV monitor in the observation room displayed a "picture in picture" composite image from the cameras and scan-converters involved in the study, most notably, a close-up view of the participants' computer screen, with an inset showing their facial expressions and body movements.

Test Participants

Participants in the usability tests were Web developers who were demographically representative of the end-users for the WAI Web site in that they had a professional or personal interest in Web development and Web accessibility issues. The participants' understanding of accessibility issues ranged from novice to expert.

In order to identify participants who were representative of the WAI Web site's users, AIR worked closely with members of the WAI Site Task Force (WSTF) to illuminate key end-user characteristics. From these discussions, we created a recruitment screener (included in this report as an appendix), which the WSTF reviewed, modified, and subsequently approved. The recruitment screener consisted of a series of questions posed to potential participants that were designed to determine individuals' suitability for the study.

Over the course of three weeks, we recruited a total of eight persons involved in Web development to serve as usability evaluation participants. One of the participants was a Web development professional who is blind. That participant kindly brought a personal laptop to our lab space in order to facilitate exploration of the WAI site using a JAWS set-up customized to her needs. Another participant had a hearing disability. In addition, one of our participants self-reported as color blind.

For their participation in the study, participants were reimbursed $50 for their time. Participation in the usability test sessions was voluntary. Table 1 provides specific information about the participants.

Table 1 - Participant demographics
P Company Environment Occupation Title & Position Technical Skills Accessibility Knowledge Disability
1 Commercial Web Accessibility Coordinator Technical Writer & Web Accessibility Coordinator Beginner Advanced Hearing Impaired & Cognitive Disability
2 Commercial Web Development Senior User Experience Specialist Advanced Beginner  
3 Commercial Web Management Specialist Technical Specialist Beginner Beginner  
4 Educational Institution Administration of Electronics Technical Associate Intermediate Beginner Color-blind
5 Self-employed Information Architect Consultant Web Developer Advanced Expert  
6 Not-for-profit Web Manager / IT Manager Web Manager / IT Manager Intermediate Intermediate  
7 Self-employed Entrepreneur President Advanced Intermediate  
8 Not-for-profit Program Manager of Literacy Initiative Program Manager of Literacy Initiative Beginner Beginner Visual Impairment (blind)

Test Procedure

The usability test itself consisted of eight individual sessions, each of approximately one and one half-hours duration, conducted primarily over the course of two days, with one session held a third day. We recruited the test participants and scheduled them for sessions two hours apart.

Upon participants' arrival to our facility, we greeted them, oriented them to our facility's restrooms and amenities, and asked them to read and sign a consent form. (The consent form was provided in Braille as requested by the participant who is blind.) The consent form described their rights as participants, the nature of the usability test, and any possible risks and potential benefits associated with participation. It described the data capture methods we would be using, and that highlights from the study. The form also stated that the participants' images, actions, and comments, but NOT their identities or any personally identifiable information, might at a later date be presented only to members of the WAI development team. The form offered opportunities for participants to opt in, or out, of having digital photos taken, having their images and comments included in study highlights, or participating in the study. We answered any participant questions that the consent form raised.

After they signed the consent form, we invited participants into our usability laboratory space and instructed them to seat themselves at the computer table. Once the participants were seated at the table, the session moderator began the test session by following the usability test plan. This plan instructed the moderator to welcome the participant, orient them to the usability laboratory, and ask them a series of pre-task interview questions. We followed these questions with a series of tasks designed to facilitate exploration of key WAI Web site features, functions and uses. Once the participants had attempted each of these tasks, we asked them to answer a series of post-task interview questions that elicited summative evaluations of their experiences with the WAI Web site. Next we asked the participants to fill out a post-test questionnaire designed to facilitate quantitative comparisons between participants.

At the end of each session, we thanked the participants for their participation, asked if there were any additional comments they wished to make, and escorted them back to our facility's waiting area. Once there, we presented them with an honorarium check ($50).

Test Plan

Our usability test plan was the document that served as the session moderator's script and outlined the series of activities participants would undertake. Working closely with the Web site WSTF, this document was drafted during the participant recruitment period, modified in response to WSTF comments, and was ultimately approved by the WSTF before the first usability test session.

To create this document, we created a series of pre-task interview questions. These questions were designed to elicit a greater understanding of the participants' professional experiences, Web development needs, Web development experiences, and their impressions of the WAI Web site. In addition, we asked participants to identify and to discuss their impressions of Web accessibility Web sites they currently use during the course of their Web development work.

Next we collectively identified a set of "typical" tasks that Web developers were likely to attempt while using the WAI Web site. These tasks were intended to represent key Web site functions and features, and to facilitate participants' exploration of a range of WAI Web site information mechanisms. In addition to developing the tasks, we developed accompanying follow-on and probing questions that the session moderators could use to gain a greater understanding of participants' actions, impressions, and experiences.

Table 2 - Task list. Tasks were presented in counterbalanced order.
Task # Task
Task 1 First Impressions This is the homepage of a Web site dedicated to Web-related accessibility issues. Please give me your initial reactions to this page. Feel free to explore this page as you normally would. You can scroll around with your mouse, but please don't click on anything just yet.
Task 2 Free Exploration I'm going to give you five minutes to freely explore this Web site. You may go anywhere you would like to go on the Web site, but please remember to speak aloud as you do so. I will tell you when the five minutes are up.
Task 3 WAI Mission Your friend Kevin mentions hearing about something called "the Web Accessibility Initiative" but he isn't sure what it is. Using this Web site, determine whether or not it contains information that would address Kevin's question. When you feel you have completed this task, please say so.
Task 4 Quick Tips Your team at work is developing a Web site and you have some concerns about how accessible the Web site might be to people with disabilities. Using this Web site, determine whether or not it contains information about the basic things Web developers need to know about Web accessibility. When you feel you have completed this task, please say so.
Task 5 Participation A few of your colleagues are interested in finding out how to be a part of WAI's effort to develop guidelines for Web accessibility. Using this Web site, determine whether or not opportunities exist for becoming involved in WAI Web Content guideline development. When you feel you have completed this task, please say so.
Task 6 Evaluation You have just been handed a report, generated by a Web accessibility evaluation tool, which informs you that your company Web site contains complex information graphs that do not meet "Checkpoint 1.1." Using this Web site determine how you would meet Checkpoint 1.1 and make the graphs accessible. When you feel you have completed this task, please say so. Probe: Let's imagine that you share the information you found about Checkpoint 1.1 with a colleague who is responsible for coding the company Web site. He's still confused about the issue. How would you use this Web site to provide him with more specific coding information?
Task 7 Online Forms Your company is revising the online forms on its Web site. Find specific information on how to make the online forms accessible. When you feel you have completed this task, please say so.
Task 8 Training Information You have been invited to be a presenter at a local conference on Web accessibility. Find information on this Web site that you could use in your talk. When you feel you have completed this task, please say so.
Task 9 Return Exploration I'd like you to think back over your experiences with the WAI Web site. As you do so, I'm going to give you five minutes to explore this Web site. You can choose to return to something you found confusing, or simply browse the site. As you do so, please continue to talk aloud and I will let you know when the five minutes are up.

Our usability test plan is included in this document's Appendices section.

Next: Findings: Executive Summary >>

Back to top