Informal Usability Testing Planning

Note: This is all draft information -- feel free to add, edit, comment...

General Plans

"Feedback sessions": (not "usability testing") Very informal, don't even use "usability testing", in the e-mail we called it "Feedback on New WAI Tools"
no recording: No videos, audio, screen or other recording. Do not even record name. (thus no release required)
step 1: When "participant" comes in, figure out:
- much time they have (e.g., if only a few minutes before next class vs. a long time)
- which resource(s) they are best suited for (e.g., if no accessibility background, probably mostly the Roadmap and a little QuickRef, and not Report Tool)
"Facilitation" tips:
- Encourage them to "think out loud". Model it, that's pretend you are looking at an interface and comment on what you're thinking.
- You ask questions. You don't provide information or answer questions. E.g., if the participant asks, "What does this link do?" You respond, "What do you think it does?" (After you're done then you can answer their questions and explain things they didn't understand.
- As facilitator, listening is more important than talking
Observers: EOWG participants will be encouraged to observe. No one else will be invited/allowed to observe so as not to have too many observers — exception: if the participant themselves wants specific people to observe (e.g., a few colleagues want to learn about usability testing so they want to all be there). Also note that there are advantages to having more than one participant at a time, especially if they know each other — their interactions with each other are often insightful!
Sign-up: We won't have sign-up - just show up!
Rooms and computers: We have two rooms most of the time (see Schedule) that have computers around the periphery of the room and tables in the middle.

What to say

Say to participants something like:

Thanks for taking the time to help us out with our new resources!
How much time do you have now?
First I'll ask a few quick questions to figure out which prototype for you to look at. (below are examples, probe for more details as useful -- however, don't spend too much time on it)
- What is your role, e.g., web developer, business analyst, ...?
- Have you used WCAG, the Web Content Accessibility Guidelines?
- Have you evaluated a website and done an evaluation report?
[when ready to start]
- This is a draft prototype [quickref & roadmap] | early version [report tool]. We're looking for ways we can improve on it. So if you find things that don't work well, that would be helpful for us!
- We'll start with you exploring the prototype and doing whatever you want. After a while, I might stop you and ask you to do a few specific things with the prototype.
- I'm going to just listen to what you think as you explore it, and I'm not going to talk at all for a while. While doing this, it would really help me if you could talk aloud "stream of consciousness" with all your thoughts, including questions, problems, good things, and bad things. For example, [model "This is confusing. I don't understand what that button does. This section would be good for my manager... etc.].
[later. before asking questions]
- I have a few questions for you. There are no right or wrong answers, it is just to understand how you might use this resource and how we can improve it.

Scheduling

Kevin - comfortable "facilitating"
Brent - comfortable "facilitating"
Shawn - comfortable "facilitating"
Lydia - comfortable "facilitating"
Wayne (difficult to facilitate)
Shadi
Eric
Anna Belle
Howard
Wilco
Sharron

Monday

room JBWS 165 8:30-5:30
room JBWS 161 8:30-2:00

Morning availability:

Lydia - all the time
Brent - 10:00-12:00
Shawn - probably not available

1:00-2:00 availability:

Shawn - whole time
Lydia - all the time
Brent - available
Howard - available
Kevin - available

Afternoon availability:

Kevin - available
Lydia - all the time
Brent - 4:00-5:30
Shawn - after 4:00

Tuesday

room JBWS 165 8:30-5:30
room JBWS 161 8:30-10:00, 12:00-3:45

Morning availability:

Shawn - whole time
Brent - 8:00-10:00
Kevin - available

12:00-2:00 availability:

Shawn - whole time
Brent - available
Howard - whole time
Wayne - not at all
Kevin - available

Afternoon availability:

Shawn - whole time
Brent - 4:00-5:30
Howard - available 2:15 - 3:45 if needed (otherwise will attend HTML5 class)
Kevin - available

Session write-up

Please add any write-up from sessions into the following wiki page: UT May 2015/Session Feedback

Participant info

Depending on time, you might want to note the following:

Job Title
Job Responsibilities
Accessibility Responsibilities/Role
Accessibility Experience (Novice, Intermediate, Advanced)
Disabilities, ATs used
Familiarity with WCAG 2.0 (none, little, some, detailed)
Organization Name
Organization Field/Industry
~~Name Should not record name to ensure anonymity {Kevin, 1 May 2015}~~

QuickRef redesign

Quickref resource

Pre questions:

Have you used the “old” quickref before {Eric, 08 May 2015}
When you think about accessing the WCAG 2.0 success criteria, in most cases what type of guideline information are you looking for? informative, definition, understanding, technical, categorical, etc. {Brent, 07 May 2015}
When accessing the WCAG 2.0 web page, what is the most frustrating thing about finding what you need? {Brent, 07 May 2015}
- Reword: If you use WCAG 2.0 web page at the moment, what do you think of them? {Kevin, 8 May 2015}

Things to look for:

How do participants use the filters?
What filter is most frequently access?
What features do they ask clarifying questions about or seek assistance with? {Brent, 07 May 2015}
How often do they "follow the rabbit trail" into another W3C resource as they are interacting with the QuickRef? Is it intuitively leading them to this information? {Brent, 07 May 2015}
What feature does the participant comment on the most? {Brent, 07 May 2015}
What feature(s)/functionality does the participant overlook completely? {Brent, 07 May 2015}
Document any bugs that the participant points out. {Brent, 07 May 2015}
Note any frustrations that the participant has. {Andrew, 08 May 2015}
Does the participant mention any job titles or roles that they think will benefit from the use of this tool? {Brent, 07 May 2015}

Prompt questions or tasks:

What do you think are the main objectives of the page? {Eric, 08 May 2015}
- Reword: Could you explain this page to me? {Kevin, 8 May 2015}
What would you expect to find when you click filters? {Eric, 08 May 2015}
- Reword: Could you show me how you might limit the success criteria to just Level A ones? {Kevin, 8 May 2015}
- Follow up with, Which filters do you find useful? {Eric, 08 May 2015}
Could you tell me what you find most useful on this page? and why?
Could you tell me how you might use this page?
Could you tell me what you might change on this page?
How would you find success criteria related to images?

Facilitator info

Known bugs or incomplete functionality:

@@

Planning Guide

Planning Guide resource

Pre questions:

What accessibility planning activities are you currently involved in?
What sort of information would help you in this area?
What sort of things are most challenging in this area?

Things to look for:

What section is the participant drawn to? Why?
How well does the participant engage with the categories?
Is the participant able to navigate the resource?
How likely are participants to use the Key Resources or Related activities?
What are participants' thoughts on the language used?

Prompt questions or tasks:

Could you show me how you might find out about preparing a business case?
What do you think of the four section titles? (Initiate, Plan, Implement, Monitor)
What do you think of the level of information? (For example, would you want to see more or less information?)

Post questions:

Would you want more or less information on these topics? If more, what sort of other things would you want to see?
What do you think of the layout and design?

Facilitator info

Known bugs or incomplete functionality:

Only the 'Initiating' section has content all the way to the activity level
Section indicators in the primary navigation are not fully functional

WCAG-EM Report Tool

Report tool resource

Can use the Before part of the Before and After Demo to provide participants with something to assess.

A sample file is provided (File:Sample-evaluation-report.json.zip). This contains a complete assessment and report that can be used to show how the final report looks and how a more complete data sample might function.

Pre questions:

How often do you evaluate the accessibility of websites?
Do you carry out evaluation for your own organization or as a service or as an advocate?
How would you rate your expertise in accessibility evaluation?
Do you use any evaluation tools at the moment? Which ones? Any comments on them?
What are you looking for in an evaluation report tool?

Things to look for:

How easy is it to get started using the tool?
How easy is it to find and follow the instructions for completing the fields?
Is there too much back-and-forth or lack of clarity between Step 2 and Step 3?
How clear is the functionality in Step 4 (especially entering results for "entire sample" vs "individual pages")?
How easy is it to handle Step 4 when there is lots of information (for example, to recall or understand how much of the evaluation has been carried out, which pages/criteria have been evaluated, and what still needs to be done)?
How well does the summary on Step 5 and the final report communicate any missing entries?
(if time) How easy is it to save and reload sessions?
(if time) How well does the top menu bar ("new", "open", "save", ...) work? Do people remember that it is there? Are all buttons used?
Is user using more than one tool to achieve the same function?

Prompt questions or tasks:

Could you explain this tool to me?
Could you walk through how you might check any particular criteria on the demo homepage? and how you would record the result?
[After loading up the sample results] Could you walk me through the results?
[After loading up the sample results] Could you show me how you might use this as part of a report?

Post questions:

How did the tool meet your initial expectations?
What worked well, and what worked less well?
Is there any other functionality you would want to have in the tool?
Would you use that tool for reporting on accessibility evaluation? Why?

Archived Old Info

Open issues

Should ask users to complete specific tasks for usability questions we have had for each tool {Melody, 23 Apr 2015}
call these "feedback session" or "informal usability testing" or other???
- Guerilla testing sessions {Kevin, 10 Apr 2015}
- "Penny For Your Thoughts!" {Brent, 23 Apr 2015}
- "Opinions Needed" {Brent, 24 Apr 2015}
- "Usability Crowdsourcing Session" {Brent, 24 Apr 2015}
participants and tools: Will we ask all testers to look at all 3 tools? Or, it depends on their skills, experience, role, etc. For example, if a person knows nothing about WCAG, then probably not have them look at the Quick Ref?
- Look at all three tools. Would be good to get a first impression from people that are not familiar with WCAG as well. There will be many people who are directed to the Quick Ref tool when released who know nothing about WCAG. Would like to see how intuitive it is to use when one is not even familiar with the content. {Brent, 23 Apr 2015}
- I aggree with Brent, it is good to see how well we have something described when people who don’t know about it use it. {Eric, 24 Apr 2015}
Prep needed:
- Define test parameters (pre- and post- questions, instructions, tasks, what to look for, etc...)
- Participant agreement - is this only required if we are recording? {Kevin, 3 Apr 2015}
- Poster and sign-up sheets
- Email blast to ACCCESSU participants before the event soliciting their participation and inviting them to sign up for the testing, possibly for specific time slots {Howard, 10 Apr 2015}
- Thank you cards and W3C goodies
- (In parallel with the first bullet in this list) What will the users read instructions and document their feedback on? online survey, paper questionnaire, verbal conversation, other. {Brent, 23 Apr 2015}

Draft Descriptions

for AccessU program and other outreach

draft for e-mail to AccessU attendees:
Help improve W3C WAI tools and resources as AccessU
New WAI Tools Feedback
At AccessU this year, you can help improve new W3C WAI [1] tools and resources for web developers, designers, evaluators, and managers! We have 3 different prototypes that we'd love for you to try out and give us feedback on.
WAI staff and Education and Outreach Working Group (EOWG) participants will be in Room JBWS 165 most of Monday and Tuesday. Please come by if you have 10 minutes or more to take a look at the prototypes.
used in WAI IG e-mail:
New WAI Tools Feedback
You can help improve new tools and resources for web developers, designers, evaluators, and managers. We'll have a room setup for you to try out prototypes and give us feedback. WAI staff and Education and Outreach Working Group (EOWG) participants would love your input! Sign-up for a timeslot will be available near the AccessU registration.
W3C WAI invites you to help improve web accessibility resources by participating in informal usability testing during AccessU. We will be looking at resources for web developers and designers; for evaluators; and for managers. You can reserve a timeslot now through [sign up online]. Or to learn more at the event, stop by Monday morning and chat with us at [location].
Want to help design...
Want to share your feedback...
Watching individual users perform typical tasks with accessibility tools helps uncover areas that can be improved. By continuing usability testing throughout the design process, we can collect user feedback leading to ongoing refinements—and ultimately validate and measure the improvements.
Bring WCAG 2.0 supporting resources for web developers, designers, evaluators and managers to a new level by helping W3C WAI to user test resources in development. Every participant is important. [Sign up online] right away, or see us at AccessU on Monday to learn more about the event at [location].
AccessU is excited to announce that you can be a part of the team by helping to improve W3C WAI tools and resources! We invite you to participate in an informal usability testing to try out prototypes and give us your feedback. Reserve your timeslot online or learn more at the event and chat with us at [location].