5. Usability Testing, Focus Groups and Feedback
5. Usability Testing, Focus Groups, and Feedback
5.1 Working with Users with Cognitive and Learning Disabilities
This section aims to help people work with users with cognitive and learning disabilities. It focuses on:
- special considerations and support for professionals working with this group, and
- a brief introduction support for people who are new to usability testing (as any amount of user involvement can substantially improve accessibility).
Usability testing professionals should pay extra attention to ethical considerations, as this audience is potentially more vulnerable.
It is beyond the scope of this document to provide a guide to usability testing and user-research. Note that you can find additional information about including users with disabilities at Involving Users in Evaluating Web Accessibility and other useful resources on our developer resource page.
Usability testing is the best way to know if your content and design works for real people with cognitive and learning disabilities.
Usability is important for everyone. However, if someone cannot use the content or design without help because of their disability, then the content is not accessible for them. It is important to change the design so that users with cognitive and learning disabilities can use the content independently.
Including digital accessibility throughout a project, right from the beginning, improves accessibility for all users. Include people with cognitive and learning disabilities in focus groups, user needs, design patterns (repeated designs for controls and other elements) and usability testing.
Automated testing for accessibility focuses on more technical areas of accessibility. While important, automated testing often cannot assess if people with a cognitive or learning disability can use the content. It is vital for people with cognitive and learning disabilities that development teams do not rely solely on automated accessibility testing. Development teams should
- Perform usability tests with people who have cognitive and learning disabilities. Usability testing of wireframes, conceptual drawings of the interface, or template pages can be helpful to identify challenges early in the project. You probably know people with age related forgetfulness, or a cognitive and learning disability. Asking them to help find out if your design is usable is a good first step.
- Include people with cognitive and learning disabilities in their teams and the design and development process. This also includes research that has something to do with them. They are the experts in what works for them.
Sometimes designs and content are usable for some people, but not if they have cognitive or learning impairments. Sometimes content is usable by people with one cognitive and learning disability but not a different one. For example, content with fewer words and more numbers may be perfect for some autistic and dyslexic users. However, the same content is inaccessible for people with dyscalculia who struggle with numeric information. It is important that usability testing includes a diverse set of users with different cognitive and learning disabilities, such as: people with a memory impairment, learning difficulty, attention impairment, numeric impairment, language and communication disability, and intellectual disability.
5.2 Finding People to Include
Finding people to include in usability testing who have different cognitive and learning disabilities is strongly encouraged and can be achievable, even for small groups on a low budget. If your organization already involves users, this section aims to expand that activity to include people with cognitive and learning disabilities. For developers without formal user involvement, even a small amount of user input and testing can make a large difference to usability and accessibility. Further links about user testing and usability can be on our developer resource page.
People sometimes recruit users from an organization or self-help group for people with learning difficulties. Social media groups can be a convenient resource. Small development groups can achieve a large improvement by asking people who they know, such as friends, colleagues, relatives or neighbors. Try to build a group of users who:
- are older and struggle to use computers, or have age-related forgetfulness,
- may be at an early stage of dementia,
- have a specific learning disability such as those with dyslexia, dyscalculia, or AD(H)D,
- have an intellectual disability, or
- have acquired cognitive issues (for example, due to neurological trauma).
People with acquired cognitive issues have the same challenges as people with other disabilities such as:
- having difficulty (asking a family member to help) with booking travel booking or hotel booking online,
- being unable to use online banking,
- coping with content forms and pop-up windows when errors occur.
It is helpful to find people with learning and cognitive difficulties who are also in your target group as customers or users.
If your organization has a more formal process, work with those that help employees or community members get assistive technology or other accommodations. They can put out a call for volunteers to their contacts. This helps individuals self-identify and opt-in to help.
Some organizations also use peer-researchers who have cognitive and learning disabilities. Peer-researchers understand the perspective of people with their disabilities. The researchers and developers work together with peer researchers to find solutions. Peer researchers are also involved in testing the solution with other people with cognitive and learning disabilities. Our developer resource page references projects and resources with information on finding and working with persons with learning and cognitive difficulties as co-researchers or peer researchers.
5.3 Informed Consent
It is important to get a declaration of consent from all participants involved in testing and focus groups before they start. Before they sign up, participants must know and understand the details such as:
- What the project is for.
- What they will be doing and why it is helping you.
- Any risks that need to be explained and understood.
- What personal details are collected and how any personal data may be used. (Note that their comments should be anonymized before being used in any report.)
- They do not have to participate. Participation is always voluntary and they can always stop at any time.
If your tester has a guardian, you should get informed consent from both the tester and their guardian.
Using an understandable consent form is important. Our design patterns on clear content will help you use clear language and layout. Adding icons and symbols can also help.
Make sure users understand the consent form. This can be done by asking them some questions about the consent that tests that they understand the key points. You can also adapt the example consent forms from our developer resource page.
Throughout the process, remind them that participation is always voluntary and they can stop at any time. This is particularly important if they have memory impairments and may have forgotten that it is their choice to participate. Remember to thank them for their ideas and contribution.
Different areas may require consent for more items than others. Check the legal requirements in your jurisdiction and for your type of content.
5.4 Usability Testing
One approach to usability is to measure user efficacy, efficiency, and satisfaction for key tasks. This can be done by measuring or tracking:
- successes in completing tasks while noting any errors to measure efficacy,
- time taken per task to measure efficiency. Note that the relative time between tasks is often more useful than absolute numbers, and
- user’s mood and comments to measure satisfaction.
At the end of the evaluation you should be able to answer:
- What prevents the user from completing a task?
- What creates confusion? When and why do they misinterpret the interface?
- What produces an error and an incorrect action?
- When does the user get frustrated or upset?
- When does the user misunderstand navigation, menus, and controls?
- How can these problems be avoided?
5.4.1 Differences from Usability Testing with the General Population
There are some differences when usability testing with people who have cognitive and learning disabilities:
- Ask ahead of time if they need any support for their needs. This could include a quiet room or frequent breaks.
- Ask what test methods work best for them, such as individual interviews or groups. Some people will prefer to have an interview in their home.
- Ensure participation forms are easy to understand. Confirm that they understand any key points.
- Inform the participant that they can request the information in a different format. If they make a request, ensure they receive it with enough time for them to review and ask questions.
- Have a copy of the participation forms at the session, in case questions come up before the session begins.
- Send participation forms to the participant in advance. Allow plenty of time for the participant to ask questions and fill in forms.
- Allow the participant to bring a caregiver, family member, or friend to attend with them.
- If your tester has a guardian, you should get consent from both the participant and their guardian.
- If they bring a guardian or caregiver, make sure they are not doing the tasks for them. If they give help, monitor closely what help they give, as this may be due to a design fault.
- Explain the testing method before the test.
- The questions should be short and easy to understand language.
- Provide easy methods of assessing mood, rather than just asking for the participant. Try asking them to select a smiley face, such as: Figure 1 A simple mood selector.
Figure 1 A simple mood selector - Some individuals also have challenges identifying moods from faces. Other options to consider are simple mood selectors and text-based rating scales where an individual can point to their selection. For example, I really like this, it is fine, I really don’t like this.
- Check they understand the methods used to collect the data.
- Ensure the person does not feel like they are at fault for making mistakes. While this is always important during usability testing, this scenario is even more likely for people with cognitive and learning disabilities.
- Ask them for their ideas, such as, what features they would like to see, what design they prefer and what support they find most helpful. Thank them for their contribution.
- Make clear that the tester can end the user testing session at any time.
Here are some suggestions of what to look for when conducting usability testing with people with cognitive and learning disabilities:
- Before you start, make sure the research team understands that the testers cannot do anything wrong. Research should never harm the user or make them feel bad.
- Make sure the participants and researchers know they can leave at any time. No one should feel bad if they leave!
- Check that the testers understand the task or question. Encourage your testers to “think out loud”.
- Time the task takes to complete, and note any parts where users slow down or seem to struggle. Can your testers manage each task reasonably easily and quickly? Also, note any errors that they make, including clicking on the wrong item.
- Find out if completing the task is frustrating or upsetting.
- You can ask users how they are feeling before and after the tasks or rate their mood such as selecting the smiley face which represents how they feel.
- Ask them if anything was annoying.
- Ask how you can make it better for your users (people with cognitive and learning disabilities)?
- Ask your users if they have suggestions about what would make the interface easier for them to use. This is often best at the end of the usability test.
- If the user is struggling, remind them that you are reviewing the system not them and that their insights are really helpful. Thank them for helping. Remind them that it is helpful when they find issues because it helps the team make the product better. Stop the process if users are getting distressed.
- Analyze the data collected and review the findings with the team. Remember to keep the names of individuals confidential (unless they have given permission for their identity and disability to be shared).
5.5 Test Objectives
You can test the objectives of the design guide. If they are successful, that section can be considered completed!
For each objective, make sure your user testing includes individuals with a range of cognitive and learning disabilities. Do not just ask questions, but ask the user to complete an action that demonstrates usability. Test for the following but set up the tests so that the user demonstrates their knowledge and understanding rather than answers a simple question: Are enough user groups represented?
For example, a typical project may wish to include: People living with early stage dementia, age-related forgetfulness, intellectual disabilities, different specific cognitive and learning disabilities and communication disabilities.
5.5.1 Does the User Understand What Things Are and How to Use Them?
- Does the user know what the page is about?
- Does the user know what actions they can take on a page?
- Does the user know where they are in a web site, an application or a multi-step process?
- Can the user easily find the different sections of content?
-
Identify the different activities that the user may want to complete
on the page:
- Can the user achieve the activities without asking for help?
- Does the user make errors trying to achieve the activities?
- Does the user find them easy to achieve?
Related Design Objective: Objective 1: Help Users Understand What Things are and How to Use Them .
5.5.2 Can Users Find What They Need?
- Can the user easily identify all the important information and important interactive features on the site and each page?
- Can the user use both browse and search to find things? (Check for important and commonly used information or features.)
- Can the user revert or correct any action they take when interacting? Does it use a familiar and consistent action?
Related Design Objective: Objective 2: Help Users Find What They Need.
5.5.3 Is the Content Clear and Understandable?
- Can the user find a segment or a piece of key information quickly?
- Does the user understand the text?
- Does the user understand text immediately?
- Does the user know ambiguous language?
- Is the content usable without understanding math concepts?
- Is there any representation of math by words instead of numbers?
- Is the support for slow readers helpful?
- Does the user understand the use of (familiar) symbols?
- Does the user understand the use of images and multimedia?
Related Design Objective: Objective 3: Use Clear and Understandable Content.
5.5.4 Can Users Avoid Mistakes and Easily Correct Them
- Can the user easily fill in the form without making mistakes?
- When the user goes to the wrong place can they easily get back in one click? (You can press something on the screen and ask them to go back.)
- Was it pleasant to fill out the form? Has the user’s mood changed?
- Did the user have to redo anything? Was correcting any mistakes easy?
- Ask the user if they would find this easy to do if under stress or tired.
- Ask the user if anything was hard.
- After they complete the form, ask the user to change a detail at the beginning. Did they manage without losing data? Was it hard or stressful?
- Ask the user how the form could be easier to fill out. Suggest some of the relevant design techniques from the design patterns section and ask if it would help them with this form.
Related Design Objective: Objective 4: Help Users Avoid Mistakes and Know How to Correct Them.
5.5.5 Can Users Maintain Focus?
- Can they achieve the activities easily without losing focus?
- Distract the user for a minute so that they lose focus. Can they get easily back to the task?
- Ask users if they would find this easy to do if under stress or tired?
- Ask the user what would help them remember what they are doing such as headers or breadcrumbs.
- Ask the user if anything was distracting.
Related Design Objective: Objective 5: Help Users Focus.
5.5.6 Can Users Complete Processes without Relying on Memory?
Identify the different activities that the user may want to complete on the page:
- Can they achieve the activities without asking for help?
- Does the user make errors trying to achieve the activities?
- Does the user find the activities easy to achieve?
- Can the user do the same thing later (the password may have been forgotten)?
- Ask users if they would find this easy to do if under stress or tired.
- Ask users where they might have trouble if they are under stress.
Related Design Objective: Objective 6: Ensure Processes Do Not Rely on Memory.
5.5.7 Is there Enough Help and Support?
- Can the user identify the different ways a user may “Report Issues and Problems”?
- Can the user find a way to submit their feedback without asking for help?
- Can the user submit feedback at each stage of the process? This should include the home page and any place they may get stuck.
- At each stage in the task, confirm that the feedback process is one click away (or less).
- Does the user make errors trying to submit their feedback?
- Does the user’s mood deteriorate when submitting feedback? (A sign of frustration.)
- Does the user find it easy to submit their feedback?
- Ask users if they would find this easy to do, when they are stressed or tired.
- Ask the user where they might have trouble giving feedback if they were under stress.
- Is the user able to complete the task after they give feedback?
- Does the user understand the feedback process? Use concrete ways to check that the user understands. For example: Is the user able to identify if/when they will receive a response back? Can they identify how a response may come back (e.g., email, phone)? Where the feedback goes/what happens to the feedback?
- Confirm that the feedback process does not require a lot of information that will prevent people from giving feedback.
- Confirm that when feedback is given, a process is in place for acting on it!
Related Design Objective: Objective 7: Provide Help and Support.
5.5.8 Is Adaptation and Personalization Supported
- Are personalized versions of the content supplied?
- Do content modifications match the user preferences such as: less content, adding and changing icons and symbols, or simplified text?
- Check that content variations such as text simplification have kept:
- the same meaning as the original,
- content that the user wants, and
- critical paths that still work.
- Check that autofill works correctly with all content versions.
- Do the user’s preferred extensions and tools work on the site?
- Are the personalization options easy to find and set?
- Do they find it easier with personalization options supplied?
Related Design Objective: Objective 8: Support Adaptation and Personalization.