Easy Checks analysis

note: for additional information, see the Easy Checks 2022 (-2023) Project Page

Method

Provocative Prototype

As a way to spark open discussion on the context and its subjects, the whole Easy Checks webpage was printed on paper and displayed as an immense ribbon hanging from the ceiling & draping the floor. While a rough translation of a digital type of media to a physical one of course is neither accurate or fair, it triggered conversation and enabled discussion on relevant subjects such as hierarchy, informational content, and purpose. Participants were invited to use markers to mark hierarchy and informational content.

Heuristics

To complement information from existing documentation, a limited number of experts were asked to look at the Easy Checks webpage and share their insights. Afterwards, the common heuristics used to measure the webpage were discussed.

User Feedback

To complete the analysis of the current Easy Checks Webpage, users were consulted to provide an end-user perspective on the content and user experience. 6 Participants were recruited from the Dutch Tax Agency (belastingdienst), from various departments and with varying degrees of expertise regarding web development and accessibility. Inclusion criteria included 'dealing with web content in some way in your professional everyday life'. Overall, the level of expertise from the participants was higher then initially intended, making them somewhat less representative of the Easy Checks target audience. Nevertheless, the interviews yielded valuable insights.

Participants were first asked about their current work context and level of expertise, including the way they deal with accessibility, if at all. They were then asked to remotely share their screen and look at the Easy Checks webpage through the context of their own work. During this scenario, they were instructed to 'think aloud'. The interviewer would sometimes ask questions regarding what they were thinking or why they were showing a certain type of behavior. When the scenario concluded, a number of exit questions were asked in a semi-structured interview, covering topics of audience, accessibility awareness, easy of use, guidance and content. A second interviewer was tasked to collect relevant notes and quotes of each interview.

The interviews were held remotely using Microsoft Teams and WebEx and lasted approximately 1 hour each. The notes from each interview were then coded based on an initial set of heuristics, and complemented with some additional emerging themes using Atlas.ti.

Current version notes

Below is a summary of the main issues that were identified from initial analysis, heuristic review and user feedback by Jasper, December 2022. These results verify most of the issues identified and discussed during previous development cycles.

Audience

The current animation has a tone of voice and message that suits the target audience.
It is not particularly clear for visitors if, and in what way the Easy Checks website is intended or suitable for them.

[EOWG comment: What was not clear? What might make it more clear? (SLH)]

quotes from users
“While the movie… is somewhat hidden and forms a fine introduction… I’d like to have that more in-focus."
“This is not in-depth enough for professionals, but too difficult for people without expertise."

[EOWG comment: We know that some of the checks are too difficult, e.g., the forms. Did these users find the first few checks too difficult? If so, what made them difficult? (SLH)]

Accessibility awareness

The animation near the top of the page provides a 1,5-minute basic overview on how and what web accessibility could be checked (own website, competitors, suppliers).
There could be more emphasis on why web accessibility and the specific checks are important.

[EOWG comment: Each check has why it’s important for accessibility. Why was that missed? From observing users, do you think a heading “Why it’s important" would be sufficient to make it more clear? (SLH)]

[EOWG comment: If we keep the focus on the checks themselves and not expand the content to cover more about why accessibility is important (which is covered in other resources), would a pointer to the Introduction be enough? (SLH)]

quotes from users 
“when there’s no awareness it makes no sense to start with heading levels. Does everyone view accessibility the same way?"
“and I’m lacking explanation on why I should be doing this. And for whom?"

[EOWG comment: What did the user mean by “Does everyone view accessibility the same way?" Are they referring to people with disabilities? Which check(s)?(SLH)]

[EOWG comment: Was the "I’m lacking explanation on why I should be doing this." comment for specific check(s)? If so, which ones? (SLH)]

Ease of finding, understanding and using

There is a lot of (detailed) information on the current website, obfuscating important bits and making it hard for users to quickly get an overview of, learn from, and apply the easy checks.
Current content hierarchy is not consistent and too complex (preface, what to do, what to check for, examples, tips, learn more).
The textual content is too difficult and technical. Furthermore, the English language can be an additional obstacle in grasping the written content.
The overall length and depth of the textual content makes the page too time-consuming to use. The audience should be able to quickly arrive at relevant content.
The extensible +/- sections are an effort to reduce clutter but are action-heavy interactions for users. The current webpage requires users to ‘read less’ (fold in) by default, which results in a less intuitive and cumbersome user experience.

[EOWG comment: What content did they find too technical? (SLH)]

quotes from users
“I can get started, but it also keeps me from doing so because I get everything at once."
“I really need to read all the lines to understand whta this is about… [reads the word front-loaded multiple times, then give a wrong interpretation] Then I need to guess, that’s jargon"
“holy moly, just how tekst-oriented is w3c, [dutch expression about losing courage]"
“English is too hard, there’s too much information, very technically written. Very text-heavy, It could be more interactive."

'Check' affordance

The current webpage provides a lot of detailed information, but does not offer affordance towards being an actual ‘check’list. For the checking part users are left entirely to their own devices, including making a final checksum or verdict.
The informational content itself is not clearly ordered or numbered as part of a list.

[EOWG comment: Note that it's important that this resource not be very checklist-y. (SLH)]

quotes from users
“Contents checks as questions. Conclusions like ‘if you do these things, you’re on the right track".
“What I jsut did, is that that right thing? Is that correct? It’d be nice if people could help with that."

[EOWG comment: I'm not understanding what these quotes are saying? (SLH)]

Content

The current content are dated and possibly partially irrelevant in today’s world wide web environment, due to examples using aged or discontinued browser versions.

quotes from users
“Make these screenshots up-to-date. They’re old. Makes me think, is this information from this day and age?"