W3C logo Web Accessibility Initiative (WAI) logo > WAI Web site redesign project

WAI Site Usability Testing Planning

Note: This Web page contains rough notes for discussion and should not be referenced or quoted under any circumstances.

Last updated $Date: 2003/10/30 16:07:18 $ by $Author: shawn $
Primary Editor: Shawn Lawton Henry. This Web page is is under development by the WAI Site Task Force of EOWG.

on this page: Background | Goals | Participants
on other pages: Screener | Pre-Test Questions, Participant Tasks, Post-Test Interview, Post-Test Survey


Background and Notes

American Institutes for Research (AIR) offered to provide in-kind support for usability testing the WAI Web site. The WSTF is incorporating usability testing with the WAI Web Site Redesign Project. AIR provided an early Draft Usability Test Plan and Protocol, which will be modified based on the information below as it is finalized.

This first test of the current WAI Web site is limited by many constraints, e.g., 8 participants in Concord, MA, USA. We expect to conduct much broader testing of early redesign prototypes with more people, with people with more disabilities, in different countries, etc.

Usability Testing Goals

The purpose of this first baseline usability test is to:

Usability Test Participants

Usability test participants are "real users" that represent a segment of the target users of the Web site. Of 8 participants, generally there should be at least 3 participants from 2 primary user groups (rather than 1 participant from 8 user groups). An early draft of the user groups is WAI Site UCD User Analysis Details.

Required

  1. user group: Web developers and content providers, or managers - numbers 1-12 in User Group List
  2. range of Web accessibility knowledge/experience:
  3. range of technical:
  4. disability:

Requested

  1. Range of experience with/use of WAI site:
  2. Misc background:
  3. Variety of roles (e.g., job titles):
  4. Variety of organizations:
  5. People with other disabilities

Participant Screener Questions

The final version of the usability testing participant screener is on a separate Web page.

Usability Test Tasks

The tasks used for baseline usability testing should reflect the most important, frequent, or difficult tasks on the WAI Web site. They can also be related to aspects of the site that we want to improve in the redesigned site. WAI Site UCD TaskAnalysis Details is an early draft of the complete task list.

The final version of the usability test tasks and questions are on a separate Web page. Previous version of tasks are available for reference.

The last version of the tasks discussed in WSTF follow.

Draft Tasks for Baseline Usability Testing

  1. First task for all: Share your first impressions of the WAI home page.
    Facilitator will say: "This is the homepage of a Web site dedicated to Web-related accessibility issues. Please give me your initial reactions to this page. Feel free to explore this page as you normally would. You can scroll around with your mouse, but please don’t click on anything just yet."
  2. Your friend Kevin recently mentioned that he heard of something called “the Web Accessibility Initiative” but he isn’t sure what it is. Using this Web site, determine whether or not it contains information that would address Kevin’s question.
  3. Your team at work is developing a Web site and you have some concerns about how accessible the Web site might be to persons with disabilities. Using this Web site, determine whether or not it contains general hints about what Web developers need to know about Web accessibility.
  4. A few of your colleagues are interested in finding out how to be a part of WAI’s effort to develop guidelines for Web accessibility. Using this Web site, determine whether or not opportunities exist for becoming involved in WAI guideline development.
  5. Based on a report generated by a Web accessibility evaluation tool, you have just learned that your company Web site does not meet Accessibility Checkpoint 1.1. Using this Web site, determine what Accessibility Checkpoint 1.1 is, and how you might adjust your company Web site to address the issue.
  6. Your company is revising the online forms featured on its Web site. You would like to provide your colleagues with information that will ensure that the revised online forms are more accessible than before. Determine whether or not this Web site contains relevant information.
  7. A company with many global divisions and Web sites in many languages has asked for your opinion on how Web accessibility laws differ around the world. Using this Web site, determine whether or not it contains relevant information.
  8. You have been invited to be a presenter at a local conference on Web accessibility. Using this Web site, determine if it contains any resources that would help you prepare your talk.
  9. The company division you work in is responsible for making sure that the corporate Web site is accessible. How can your team use the WAI Web site to determine the accessibility of the company Web site?
  10. Last task for all: Free exploration: Go anywhere you’d like to go on the Web site, and speak aloud as you do so.

Pre- and Post-Test Questionnaires

The final version of the usability test tasks and questions are on a separate Web page.

The last version of the questions discussed in WSTF follow.

Pre-Test Questionnaire

  1. Have you ever researched Web accessibility for people with disabilities?
    If yes:
    1. What resources do you use to learn about Web accessibility?
    2. What Web sites do you use to learn about Web accessibility?
    3. How often do you use them?
    4. What do you like about each of the Web sites?
    5. What do you dislike about each of the Web sites?
    6. What types of Web accessibility information do you look for on the Internet?
    7. What are the last three Web accessibilty topics or questions you researched on the Internet?
    8. What other resources (e.g., books, classes) do you use to learn about Web accessibility?
    9. What drives your interest in Web accessibility?
    10. How long have you been involved in Web accessibility?
  2. Do any of the authoring tools you use have accessibility-related features?
  3. How long have you been involved in Web development or management?
  4. If you were to envision your ideal Web accessibility Web site, what sorts of information would it contain? What would it look and act like? How would it be organized?

Post-Test Interview

  1. What are your overall impressions of the Web site?
  2. If you had to give the site a grade, from A to F, where A was exemplary and F was failing, what grade would you give it, and why?
  3. Name three words or characteristics that describe this Web site.
  4. What were the 3 things you liked best about this Web site?
  5. What were the 3 things you liked least about this Web site?
  6. If you could make one significant change to this Web site, what change would you make?
  7. Would you return to this Web site on your own in the future? Why/why not?
  8. Would you recommend this Web site to a colleague? To a friend?
  9. [@@ What do we want to know about users' information needs now that they have seen the Web site...]
  10. Do you have any other questions or comments about the Web site or your experiences with it?

Post-Test Survey

For each statement, please circle a number to indicate the level to which you agree with each statement on a scale of 1 to 7. (1 = strongly disagree, 7= strongly agree)

  1. The homepage is attractive.
  2. The overall site is attractive.
  3. The site's graphics are pleasing.
  4. The site has a good balance of graphics versus text.
  5. The colors used throughout the site are attractive.
  6. The typography (lettering, headings, titles) is attractive.
  7. The homepage's content makes me want to explore the site further.
  8. It is easy to find one's way around the site.
  9. You can get to information quickly.
  10. It is fun to explore the site.
  11. It is easy to remember where to find things.
  12. Information is layered effectively on different screens.
  13. The homepage is attention-getting.
  14. Information is easy to read.
  15. Information is written in a style that suits me.
  16. Screens have the right amount of information.
  17. The site effectively communicates the organization's image.
  18. Information is relevant.
  19. The site is designed with me in mind.
  20. The site's content interests me.
  21. The site's content would keep me coming back.
  22. The site has characteristics that make it especially appealing.
  23. The site reflects progressive, leading edge design.
  24. The site is exciting.
  25. The site is well-suited to first-time visitors.
  26. The site is well-suited to repeat visitors.
  27. The site has a clear purpose.
  28. It is always clear what to do next.
  29. It is clear how screen elements (e.g., pop-ups, scrolling lists, menu options, etc.) work.
  30. Mistakes are easy to correct.
  31. @@



Copyright 1994-2003 W3C (MIT, ERCIM, Keio), All Rights Reserved. W3C liability, trademark, document use and software licensing rules apply. Your interactions with this site are in accordance with our public and Member privacy statements.