5. Usability Testing, Focus Groups and Feedback

5.1 Usability Testing Introduction

Usability testing is the best way to know if your content and functionality works for real people with cognitive and learning disabilities.

Usability is important for everyone. However, if someone cannot use the content or design without help because of their disability, then the content is not accessible for them. It is important to change the design so that users with cognitive or learning disabilities can use the content independently.

Including digital accessibility throughout a project, right from the beginning, improves accessibility for all users. This includes using user needs for people with cognitive and learning disabilities, using design patterns which focus on the needs of those with cognitive disabilities, and when possible, usability testing with individuals with cognitive disabilities.

Automated testing for accessibility focuses on more technical areas of accessibility. While important, automated testing cannot often assess if people with a cognitive or learning disability can successfully use the content. It is vital for people with cognitive disabilities that development teams do not rely solely on automated accessibility testing. Development teams should:

  • Include cognitive and learning disabilities in persona, user needs and requirements.
  • Use the design patterns as described in section 3 when designing interfaces and writing requirements.
  • If possible, perform usability tests with people who have cognitive disabilities. Usability testing of wireframes, conceptual drawings of the interface, or template pages can be helpful to identify challenges early in the project.
  • If possible, include people with cognitive and learning disabilities in the design and development process. This also includes research that has something to do with them. They're the experts in what works for them.

Sometimes designs and content are usable for some people but not if they have cognitive or learning impairments. Sometimes content is usable by people with one learning disability but not a different one. For example, content with fewer words and more numbers may be perfect for some users with dyslexia or autism spectrum disorder, but inaccessible for people with dyscalculia who struggle with numeric information. It is important that usability testing includes a diverse set of users with different cognitive or learning disabilities, such as people with a memory impairment, learning disability, attention impairment, numeric impairment, language and communication disability and intellectual disability.

5.2 Finding People to Include

Finding people to include in usability testing who have different learning and cognitive disabilities can be relatively easy. People sometimes recruit users from an organization or self-help group for people with learning difficulties. Social media groups can be an easy and convenient resource. Alternatively, small development groups can achieve a large improvement by asking people who they know, such as friends, colleagues, relatives or neighbors who:

  • Are older and struggle to use computers, or have age-related forgetfulness;
  • May be at an early stage of dementia
  • Have dyslexia and/or dyscalculia or ADHD
  • Have a learning or intellectual disability
  • People with acquired cognitive issues (for example, due to neurological trauma) who have the same challenges as people with other disabilities such as:
    • having difficulty (asking a family member to help) with booking travel booking or hotel booking online
    • being unable to use online banking
    • coping with content forms and pop-up windows when errors occur

It is helpful to find people with learning and cognitive difficulties who are also in your target group as customers or users.

If your organization has a more formal process, work with those that help employees or community members get assistive technology or other accommodations. They can put out a call for volunteers to their contacts. This helps individuals self-identify and opt-in to help.

Some organizations also use peer-researchers who have learning or cognitive disabilities. Peer-researchers understand the perspective of people with their disabilities. The researchers and developers work together with peer researchers to find solutions. Peer researchers are also involved in testing the solution with other people with cognitive and learning disabilities. Our developer resource page references project and resources with information on finding and working with persons with learning and cognitive difficulties as co-researchers or peer researchers.

5.4 Usability Testing

It is beyond the scope of this document to provide a guide to usability testing and user-research, however, there are useful resources available on our developer resource page. As a short overview, usability could be measured based on efficacy, efficiency and satisfaction. This can be done by measuring or tracking:

  • successes in completing tasks while noting any errors to measure efficacy,
  • time taken per task to measure efficiency, note that the relative time between tasks is often more useful than absolute numbers and
  • user’s mood and comments to measure satisfaction.

At the end of the evaluation you should be able to answer:

  • What prevents the user from completing a task?
  • What creates confusion? When and why do they misinterpret the interface?
  • What produces an error and an incorrect action?
  • When does the user get frustrated or upset
  • When does the user misunderstand navigation, menus and controls?
  • How can these problems be avoided?

5.4.1 Differences from Usability Testing with the General Population

There are some differences when performing usability testing with people who have cognitive impairments:

  • Ask ahead of time if they need any support for their needs. This could include a quiet room or frequent breaks.
  • Ask what test methods work best for them, such as individual interviews or groups. Some people will prefer to have an interview in their home.
  • Ensure participation forms are easy to understand.
  • Inform the participant that they can request the information in a different format. If they make a request, ensure they receive it with enough time for them to review and ask questions.
  • Have a copy of the participation forms at the session, in case questions come up before the session begins.
  • Send participation forms to the participant in advance, and allow plenty of time for the participant to ask questions and fill in forms;
  • Allow the participant to bring a care giver, family member or friend to attend with them.
  • If your tester has a guardian, you should get an assent form from the tester and an informed consent form from their guardian;
  • If they bring a guardian or care giver, make sure they are not doing the tasks for them. If they give help, monitor closely what help they give as this may be due to a design fault. 
  • Explain the testing method before the test.
  • The questions should not be too difficult. 
  • It helps to provide easy methods of assessing mood, rather than asking for the participant to verbalize. Try asking them to select a smiley face, such as: Figure 1 A simple mood selector

    a set of 5 smiley faces from happy, through neutral, to sad.
    Figure 1 A simple mood selector
  • Some individuals also have challenges identifying moods from faces. Other options to consider are simple mood selectors and text-based rating scales where an individual can point to their selection. For example, I really like this, it is fine, I really don't like this.
  • Check they understand the methods used to collect the data. 
  • Ensure the person does not feel like they are at fault for making mistakes. While this is always important during usability testing, this scenario is even more likely for people with cognitive impairments.
  • Ask them for their ideas, such as, what features they would like to see, what design they prefer and what support they find most helpful.

Here are some suggestions of what to look for when conducting usability testing with people with learning and cognitive disabilities:

  1. Before you start, make sure the research team understands that the testers cannot do anything wrong. Research should never harm the user or make them feel bad.
  2. Make sure the participants and researchers know they can leave at any time. No one should feel bad if they leave!
  3. Check that the testers understand the task or question. Encourage your testers to “think out loud” 
  4. Can your testers manage each task reasonably easily and fast? You can time the task taken to complete, and note any parts where the users slow down or seem to struggle. Also, note any errors that they make including clicking on the wrong item.
  5. Is completing the task frustrating or upsetting?
    1. You can ask the users how they are feeling before and after the tasks or rate their mood such as selecting the smiley face which represents how they feel.
    2. Ask them if anything was annoying.
  6. How can you make it better for your users (people with learning and cognitive disabilities)?
  7. Ask your users if they have suggestions about what would make the interface easier for them to use. This is often best at the end of the usability test.
  8. If the user is struggling, remind them that you are reviewing the system not them and that their insights are really helpful. Thank them for helping. Remind them that it is helpful when they find issues because it helps the team make the product better. Stop the process if users are getting distressed.
  9. Analyze the data collected and review the findings with the team. Remember to keep the names of individuals confidential unless they have given permission to have their identity and disability shared.

(With thanks to Smart4MD and Easy Reading for this overview. These projects are co-financed by the European Union under an EU Framework Programme for Research and Innovation – Horizon 2020, with grant agreement number 643399 and 780529 and the European commission for this contribution.)

5.5 Test Objectives

You can test the objectives of the design guide. If they are successful, that section can be considered completed!

For each objective, make sure your user testing includes individuals with a range of cognitive disabilities. Do not just ask questions, but ask the user to complete an action that demonstrates usability. Test for the following but set up the tests so that the user demonstrates their knowledge and understanding rather than answers a simple question:

  • Are enough user groups represented?
  • For example, a typical project may wish to include: People living with early stage dementia, age appropriate forgetfulness, intellectual disabilities, different specific learning disabilities and communication disorders.

5.5.1 Does the User Understand What Things Are and How to Use Them?

  • Does the user know what the page is about?
  • Does the user know what actions they can take on a page?
  • Does the user know where they are in a website, an application or a multi-step process?
  • Can the user easily find the different sections of content?
  • Identify the different activities that the user may want to complete on the page:
    • Can the user achieve the activities without asking for help?
    • Does the user make errors trying to achieve the activities?
    • Does the user find them easy to achieve?

5.5.2 Can Users Find What They Need?

  • Can the user easily identify any important information or interactive feature on the site or on a specific page?
  • Can the user use both browse and search approaches to finding things?
  • Can the user revert or correct any action they take when interacting? Does it use a familiar and consistent action?

5.5.3 Is the Content Clear and Understandable?

  • Does the user understand the text?
  • Does the user understand text immediately?
  • Does the user know unambiguous language?
  • Is there any content usable without understanding math concepts?
  • Is there any representation of math by words instead of numbers?
  • Is the support for slow readers helpful?
  • Does the user understand use of (familiar) symbols?
  • Does the user understand use of images and multi-media?
  • Can the user find a segment or a piece of key information quickly?

5.5.4 Can Users Avoid Mistakes and Easily Correct Them

  • Can the user easily fill in the form without making mistakes?
  • When the user goes to the wrong place can they easily get back in one click?
  • Was it pleasant to fill out the form? How is their mood changed?
  • Did they have to redo anything? Was correcting any mistakes easy?
  • Ask the users if they would find this easy to do if under stress or tired.
  • Ask the user if anything was hard.
  • Ask the user how the form could be easier to fill out. Suggest some of the relevant design techniques bellow and ask if it would help them with this form.

5.5.5 Can the Users Maintain Focus?

  • Can they achieve the activities easily without losing focus?
  • Distract the user for a minute so that they lose focus. Can they get easily back to the task?
  • Ask the users if they would find this easy to do if under stress or tired?
  • Ask the user what would help them remember what they are doing such as headers or breadcrumbs.
  • Ask the user if anything was distracting.

5.5.6 Can Users Complete Processes without Relying on Memory?

Identify the different activities that the user may want to complete on the page:

  • Can they achieve the activities without asking for help?
  • Does the user make errors trying to achieve the activities?
  • Does the user find the activities easy to achieve?
  • Can the user do the same thing later (the password may have been forgotten)?
  • Ask users if they would find this easy to do if under stress or tired.
  • Ask users were they might have trouble if they are under stress.

5.5.7 Is there Enough Help and Support?

  • Can the user identify the different ways a user may “Report Issues and Problems?”
  • Can the user find a way to submit their feedback without asking for help?
  • Can the user submit their feedback at each stage of the process including from the home page and any place they may get stuck?
  • Does the user make errors trying to submit their feedback?
  • Does the user find it easy to submit their feedback?
  • Does the user’s mood deteriorate when submitting feedback? (A sign of frustration)
  • Ask the users if they would find this easy to do if under stress or tired.
  • Ask the user where they might have trouble giving feedback if they were under stress.
  • Does the user understand the feedback process and are they able to complete the task? Use concrete ways to check that the user understands. For example: Is the user able to identify if/when they will receive a response back? Can they identify how a response may come back (e.g. email, phone)? Where the feedback goes/what happens to the feedback?
  • Make sure it is simple to use and does not require a lot of information that will prevent people from giving feedback.
  • Confirm it is available at different stages in the process and is one click away.
  • Confirm that when feedback is given and a process is in place for acting on it!

5.5.8 Is Adaptation and Personalization Supported

  • Are personalized versions of the content are supplied?
  • Do content modifications match declared user preferences such as less content, adding and changing symbols or simplified text?
  • Check that content variations such as text simplification do not incorrectly change the meaning, that content is not lost, and that critical paths still work;
  • Forms autofill works correctly with all content versions.
  • Does the users preferred extensions and tools work on the site?
  • Are the personlizations options easy to find and set?
  • Do they find it easier with personalization options supplied?