Appendices

Usability Test Test Plan

Prepared October 2003 by:
Chris Hass, Sean Jacobs, Kevin Tu
American Institutes for Research

Prepared for:
W3C WAI EOWG

The Usability Test Plan is the moderator's script that identifies the sequence of activities and questions that each individual usability test participant will experience.

Informed Consent and Introduction

Instruct participant to read and sign informed consent while in the waiting area.

Read the following introduction after participant and administrator are sitting in the laboratory.

Thank you very much for coming in today. My name is _____ and I work for the American Institutes for Research. We are an independent organization that is hired by companies and government agencies to evaluate how easy their products are to learn and use.

My role today will be to guide you through the session and interview you about your reactions to a Web site dedicated to developing Web sites that are accessible to persons with disabilities. Keep in mind, we are not evaluating you, we are interested in evaluating the quality of the Web site. Any information that you give us about your impressions of the Web site will be used to make it better. Throughout the session, I will encourage you to freely express your opinions, to comment on what is clear and what is unclear to you, and in particular, what you find confusing or difficult to understand. I am not the person who created the Web site, so please don't worry about saying something that might hurt my feelings. It's important for you to be as honest as you can about your reactions to the Web site.

As you can see, there is a one-way mirror in the room, and there may be people behind the mirror observing us during the session. In addition, we will be videotaping the session for members of the development team who couldn't be here today.

Did you have any questions about the consent form that you signed in the waiting room?

Answer any questions now.

OK, let's get started.

Before we start using the Web site, I have a few background questions that I'd like to ask you.

Pre Test Interview

1. Do you ever research issues related to Web accessibility for people with disabilities? (If no: skip to 11)
2. What resources do you use to learn about Web accessibility? (Probe: Web sites, books, classes)
3. (May not need to ask) What Web sites do you use to learn about Web accessibility?
4. How often do you use them?
5. What do you like about each of the Web sites?
6. What do you dislike about each of the Web sites?
7. What types of Web accessibility information do you look for on the Internet?
8. What are the last three Web accessibility topics or questions you researched on the Internet?
9. What drives your interest in Web accessibility?
10. How long have you been involved in Web accessibility?
11. Do you use any Web development software programs or authoring tools when you develop Web sites? Which ones? (If no, skip to 13)
12. Do any of the tools you use have features to help make your Web sites accessible to people with disabilities?
13. How long have you been involved in Web development or management?
14. If you were to envision your ideal Web accessibility Web site, what sorts of information would it contain? What would it look and act like? How would it be organized?

"Think aloud" demonstration

Here's how the rest of the session will work: I will give you a few tasks to complete, or information to look up using the Web site. I'd like you to think out loud as you try the tasks so that I can understand and follow what you are doing. Let me show you what I mean by "thinking out loud."

Demonstrate think aloud technique

That's what I mean by "thinking aloud." After each task, I'd like you to let me know if you have any comments or feedback about that task. Please remember that I am equally interested in what you like about the Web site, and what you don't like, or don't find easy to use. Feel free to ask me any questions during the session, but in some cases I may not be able to answer them.

As you move through this Web site, I'd like you to act as you typically would when you surf the Web, and please remember to think out loud about why you're clicking on things, and whether what you see after you click on something is what you expected to see.

Do you have any questions before we begin?

Task 1: First Impressions

This is the homepage of a Web site dedicated to Web-related accessibility issues. Please give me your initial reactions to this page. Feel free to explore this page as you normally would. You can scroll around with your mouse, but please don't click on anything just yet. Computer is set to: http://www.w3c.org/wai/

15. Have you ever seen this Web site before?
16. Please give me your initial impressions about the layout of this screen and what you think of the colors, graphics, photos, etc.
17. Without clicking on anything yet, please describe the options you see on the homepage and what you think they do. Feel free to move around the page, but again I'll ask you not to click on anything right now.
18. Without clicking on anything yet, if you were exploring, what would you click on first?
19. What do you think is the purpose of this site?
20. Who do you think this site is intended for?
21. Whose Web site is this?

Task 2: Free Exploration

I'm going to give you five minutes to freely explore this Web site. You may go anywhere you would like to go on the Web site, but please remember to speak aloud as you do so. I will tell you when the five minutes are up. NOTE: Remaining tasks will be counterbalanced

Task 3: WAI Mission

Your friend Kevin mentions hearing about something called "the Web Accessibility Initiative" but he isn't sure what it is. Using this Web site, determine whether or not it contains information that would address Kevin's question.

When you feel you have completed this task, please say so.

Path: Home > About > About WAI

Post-task questions: Accomplished task? ____YES ____NO ____YES - with assistance

22. How easy or difficult was it for you to accomplish this task?
23. Was there anything about the Web site that made this task especially easy or difficult?
24. Tell me about the page you are looking at. What options are available to you?
25. If participant uses jump links: What happened when you clicked on About? Is this what you expected to happen, why or why not?

Task 4: Quick Tips

Your team at work is developing a Web site and you have some concerns about how accessible the Web site might be to people with disabilities. Using this Web site, determine whether or not it contains information about the basic things Web developers need to know about Web accessibility.

When you feel you have completed this task, please say so.

Path: Home > Quick Tips > Quick Tips To Make Accessible Web Sites

Post-task questions: Accomplished task? ____YES ____NO ____YES - with assistance

26. How easy or difficult was it for you to accomplish this task?
27. Was there anything about the Web site that made this task especially easy or difficult?
28. Tell me about the page you are looking at. What options are available to you?

Task 5: Participation

A few of your colleagues are interested in finding out how to be a part of WAI's effort to develop guidelines for Web accessibility. Using this Web site, determine whether or not opportunities exist for becoming involved in WAI Web Content guideline development.

When you feel you have completed this task, please say so.

Path: Home > Participation > WAI Working Groups and Interest Groups > Web Content Accessibility Guidelines Working Group > How to Join

Post-task questions: Accomplished task? ____YES ____NO ____YES - with assistance

29. How easy or difficult was it for you to accomplish this task?
30. Was there anything about the Web site that made this task especially easy or difficult?
31. Tell me about the page you are looking at. What options are available to you?
32. What is your opinion of how information is organized on this page?

Task 6: Evaluation

You have just been handed a report, generated by a Web accessibility evaluation tool, which informs you that your company Web site contains complex information graphs that do not meet "Checkpoint 1.1."

Using this Web site determine how you would meet Checkpoint 1.1 and make the graphs accessible.

When you feel you have completed this task, please say so.

Path: Home > Guidelines > Web Content Accessibility Guidelines 1.0 > Techniques for Checkpoint 1.1 > Long Description of Images

Post-task questions: Accomplished task? ____YES ____NO ____YES - with assistance

33. How easy or difficult was it for you to accomplish this task?
34. Was there anything about the Web site that made this task especially easy or difficult?
35. What is your opinion of how information is organized on this page?
36. Were the different options such as Techniques, Checkpoints, and Guidelines, helpful or unhelpful to you? Why?

Probe: Let's imagine that you share the information you found about Checkpoint 1.1 with a colleague who is responsible for coding the company Web site. He's still confused about the issue. How would you use this Web site to provide him with more specific coding information?

Task 7: Online forms

Your company is revising the online forms on its Web site. Find specific information on how to make the online forms accessible.

When you feel you have completed this task, please say so.

Path:
Home > Techniques > Techniques for Web Content Accessibility Guidelines 1.0 > (Scroll to relevant listings)
Home > Checkpoints > 10.2, 10.4, 12.4
Home > Guideline: Search for "Form" using browser

Post-task questions: Accomplished task? ____YES ____NO ____YES - with assistance

37. How easy or difficult was it for you to accomplish this task?
38. Was there anything about the Web site that made this task especially easy or difficult?
39. What is your opinion of how information is organized on this page?
40. Is there anything that could have helped make that task easier?
41. Was the location of this information clear or unclear to you? Why?

Task 8: Training Information

You have been invited to be a presenter at a local conference on Web accessibility. Find information on this Web site that you could use in your talk.

When you feel you have completed this task, please say so.

Path: Home > Training > Curriculum on Web Content Accessibility Guidelines

Post-task questions: Accomplished task? ____YES ____NO ____YES - with assistance

42. How easy or difficult was it for you to accomplish this task?
43. Was there anything about the Web site that made this task especially easy or difficult?
44. What is your opinion of how information is organized on this page?
45. Is there anything that could have helped make that task easier?

Return Exploration

I'd like you to think back over your experiences with the WAI Web site. As you do so, I'm going to give you five minutes to explore this Web site. You can choose to return to something you found confusing, or simply browse the site. As you do so, please continue to talk aloud and I will let you know when the five minutes are up.

Post-Test Interview

46. What are your overall impressions of the Web site?
47. If you had to give the site a grade, from A to F, where A was exemplary and F was failing, what grade would you give it, and why?
48. Name three words or characteristics that describe this Web site.
49. What are the three things you like best about the Web site?
50. What are the three things you like least about the Web site?
51. If you could make one significant change to this Web site, what change would you make?
52. Would you return to this Web site on your own in the future? Why/why not?
53. Would you recommend this Web site to a colleague? To a friend?
54. Do you have any other questions or comments about the Web site or your experiences with it?

Post-Test Survey

For our last activity I'm going to give you a short questionnaire that I would like you to fill out. The questionnaire will give you a series of statements about the Web Accessibility Initiative Web site. I would like you to rate your agreement with each statement. While you fill out the questionnaire, I will step into the observation booth to see if our observers have any further questions.

For each statement, please circle a number to indicate the level to which you agree with each statement. If you feel a statement isn't relevant to your experiences with the Web site, select the "N/A" (not applicable) for that item.

1 = Disagree strongly
7 = Agree strongly

Statement	1	2	3	4	5	6	7	n/a
The homepage is attractive
The overall site is attractive
The site's graphics are pleasing
The site has a good balance of graphics versus text
The colors used throughout the site are attractive
The typography is attractive
The homepage's content makes me want to explore the site further
It is easy to find one's way around the site
You can get information quickly
It is fun to explore the site
It is easy to remember where to find things
Information is layered effectively on different screens
The homepage is attention-getting
Information is easy to read
Information is written in a style that suits me
Screens have the right amount of information
The site effectively communicates the company's image
Information is relevant
The site is designed with me in mind
The site's content interests me
The site's content would keep me coming back
The site has characteristis that make it especially appealing
The site reflects progressive, leading edge design
The site is exciting
The site is well-suited to first-time visitors
Ths site is well-suited to repeat visitors
The site has a clear purpose
It is always clear what to do next
It is clear how screen elements work
Mistakes are easy to correct

Next: Participant Recruitment Screener >>