> WAI Web site
redesign project
WAI Site Usability
Testing Planning
Note: This Web page contains rough
notes for discussion and should not be referenced or quoted under
any circumstances.
Last updated $Date: 2003/10/30 16:07:18 $ by $Author: shawn $
Primary Editor: Shawn Lawton Henry. This Web page is is under development by
the WAI Site Task Force of EOWG.
on this page: Background | Goals | Participants
on other pages: Screener | Pre-Test Questions, Participant Tasks, Post-Test Interview, Post-Test Survey
American Institutes for Research (AIR)
offered to provide in-kind support for usability testing the WAI Web site.
The WSTF is incorporating usability testing
with the WAI Web Site Redesign Project. AIR
provided an early Draft Usability Test
Plan and Protocol, which will be modified based on the information below
as it is finalized.
This first test of the current WAI Web site is limited by many
constraints, e.g., 8 participants in Concord, MA, USA. We expect to conduct
much broader testing of early redesign prototypes with more people,
with people with more disabilities, in different countries, etc.
The purpose of this first baseline usability test is to:
- observe and evaluate the user experience as it
relates to navigation, information architecture, design, layout,
structure, and information presentation across the Web site by examining
users' ability to navigate the Web site and to complete key tasks (This
test is not to evaluate Web site content, or the design of specific
documents.)
- establish a baseline from which to measure the
efficacy of future redesign prototypes
- gain further understanding of the strengths and
weaknesses of the current site, and identify ways to improve the WAI site
design
Usability test participants are "real users" that represent a segment of
the target users of the Web site. Of 8 participants, generally there should
be at least 3 participants from 2 primary user groups (rather than 1
participant from 8 user groups). An early draft of the user groups is WAI Site UCD User Analysis Details.
Required
- user group: Web developers and content providers, or managers - numbers
1-12 in User Group List
- range of Web accessibility knowledge/experience:
- 4-5 participants: know very little or nothing about Web
accessibility
- 3-4 participants: mid-level knowledge and experience with Web
accessibility
- range of technical:
- 4-5: non-technical (e.g., doesn't know HTML)
- 3-4: technical (e.g., knows HTML, +)
- disability:
- 1: experienced screen reader user
- 1: voice recognition user (e.g., person with high level
quadriplegia)
Requested
- Range of experience with/use of WAI site:
- 2-3: never been to WAI site
- 4-5: have been to WAI site a couple times in last year
- 1-2: go to WAI site at least once per month
- Misc background:
- 1-4: know about Web accessibility (e.g., national or organization
standards/guidelines), but not WCAG
- 2-4: knows WSIWIG editor, but not <HTML> much at all
- Variety of roles (e.g., job titles):
- from Web developers and content providers, or managers - numbers
1-12 in User Group List
- Variety of organizations:
- large commercial organization, small commercial organization, Web
design firm
- government agency (prefer Federal, more local OK)
- non-profit organization
- educational institution
- People with other disabilities
The final version of the usability testing participant screener is on a separate Web page.
The tasks used for baseline usability testing should reflect the most
important, frequent, or difficult tasks on the WAI Web site. They can also be
related to aspects of the site that we want to improve in the redesigned
site. WAI Site UCD TaskAnalysis
Details is an early draft of the complete task list.
The final version of the usability test tasks and
questions are on a separate Web page. Previous
version of tasks are available for reference.
The last version of the tasks discussed in WSTF follow.
Draft Tasks for Baseline
Usability Testing
- First task for all: Share your first impressions of
the WAI home page.
Facilitator will say: "This is the homepage of a Web site
dedicated to Web-related accessibility issues. Please give me your
initial reactions to this page. Feel free to explore this page as you
normally would. You can scroll around with your mouse, but please
don’t click on anything just yet."
- Have you ever seen this Web site before?
- Please give me your initial impressions about the layout of this
page and what you think of the colors, graphics, photos, etc.
- Without clicking on anything yet, please describe the options you
see on the home page and what you think they do. Feel free to move
around the page, but again I’ll ask you not to click on
anything right now.
- Without clicking on anything yet, if you were exploring, what would
you click on first?
- What do you think is the purpose of this site?
- Who do you think this site is intended for?
- Whose Web site is this?
- Your friend Kevin recently mentioned that he heard of something called
“the Web Accessibility Initiative” but he isn’t sure
what it is. Using this Web site, determine whether or not it contains
information that would address Kevin’s question.
- Your team at work is developing a Web site and you have some concerns
about how accessible the Web site might be to persons with disabilities.
Using this Web site, determine whether or not it contains general hints
about what Web developers need to know about Web accessibility.
- A few of your colleagues are interested in finding out how to be a part
of WAI’s effort to develop guidelines for Web accessibility. Using
this Web site, determine whether or not opportunities exist for becoming
involved in WAI guideline development.
- Based on a report generated by a Web accessibility evaluation tool, you
have just learned that your company Web site does not meet Accessibility
Checkpoint 1.1. Using this Web site, determine what Accessibility
Checkpoint 1.1 is, and how you might adjust your company Web site to
address the issue.
- Your company is revising the online forms featured on its Web site. You
would like to provide your colleagues with information that will ensure
that the revised online forms are more accessible than before. Determine
whether or not this Web site contains relevant information.
- A company with many global divisions and Web sites in many languages
has asked for your opinion on how Web accessibility laws differ around
the world. Using this Web site, determine whether or not it contains
relevant information.
- You have been invited to be a presenter at a local conference on Web
accessibility. Using this Web site, determine if it contains any
resources that would help you prepare your talk.
- The company division you work in is responsible for making sure that
the corporate Web site is accessible. How can your team use the WAI Web
site to determine the accessibility of the company Web site?
- Last task for all: Free exploration: Go anywhere
you’d like to go on the Web site, and speak aloud as you do
so.
The final version of the usability test tasks and questions are on a separate Web page.
The last version of the questions discussed in WSTF follow.
Pre-Test Questionnaire
- Have you ever researched Web accessibility for people with
disabilities?
If yes:
- What resources do you use to learn about Web accessibility?
- What Web sites do you use to learn about Web accessibility?
- How often do you use them?
- What do you like about each of the Web sites?
- What do you dislike about each of the Web sites?
- What types of Web accessibility information do you look for on the
Internet?
- What are the last three Web accessibilty topics or questions you
researched on the Internet?
- What other resources (e.g., books, classes) do you use to learn
about Web accessibility?
- What drives your interest in Web accessibility?
- How long have you been involved in Web accessibility?
- Do any of the authoring tools you use have accessibility-related
features?
- How long have you been involved in Web development or management?
- If you were to envision your ideal Web accessibility Web site, what
sorts of information would it contain? What would it look and act like?
How would it be organized?
Post-Test Interview
- What are your overall impressions of the Web site?
- If you had to give the site a grade, from A to F, where A was exemplary
and F was failing, what grade would you give it, and why?
- Name three words or characteristics that describe this Web site.
- What were the 3 things you liked best about this Web site?
- What were the 3 things you liked least about this Web site?
- If you could make one significant change to this Web site, what change
would you make?
- Would you return to this Web site on your own in the future? Why/why
not?
- Would you recommend this Web site to a colleague? To a friend?
- [@@ What do we want to know about users' information needs now that
they have seen the Web site...]
- Do you have any other questions or comments about the Web site or your
experiences with it?
Post-Test Survey
For each statement, please circle a number to indicate the level to which
you agree with each statement on a scale of 1 to 7. (1 = strongly disagree,
7= strongly agree)
- The homepage is attractive.
- The overall site is attractive.
- The site's graphics are pleasing.
- The site has a good balance of graphics versus text.
- The colors used throughout the site are attractive.
- The typography (lettering, headings, titles) is attractive.
- The homepage's content makes me want to explore the site further.
- It is easy to find one's way around the site.
- You can get to information quickly.
- It is fun to explore the site.
- It is easy to remember where to find things.
- Information is layered effectively on different screens.
- The homepage is attention-getting.
- Information is easy to read.
- Information is written in a style that suits me.
- Screens have the right amount of information.
- The site effectively communicates the organization's image.
- Information is relevant.
- The site is designed with me in mind.
- The site's content interests me.
- The site's content would keep me coming back.
- The site has characteristics that make it especially appealing.
- The site reflects progressive, leading edge design.
- The site is exciting.
- The site is well-suited to first-time visitors.
- The site is well-suited to repeat visitors.
- The site has a clear purpose.
- It is always clear what to do next.
- It is clear how screen elements (e.g., pop-ups, scrolling lists, menu
options, etc.) work.
- Mistakes are easy to correct.
- @@