Introducing the auto-wcag user input template
Web accessibility evaluations can serve a number of different purposes ranging from quality assurance and error repair, individual reports and awareness raising, to benchmarking and monitoring. Many people who would like to know the accessibility status of a web page aren’t experts in the field. In such situations they rely on tools that produce reports about (potential) errors.
Only some aspects of the Web Content Accessibility Guidelines (WCAG) 2.0 can be checked automatically. The majority of Success Criteria require human judgment.
The W3C Automated WCAG Monitoring Community Group is developing a new approach to involve non-experts in the data collection process for an accessibility study. By combining the benefits of automated and manual testing we aim to improve both the quality and the coverage of evaluation results.
Automated checker tools and human judgment
Human intervention is needed in web accessibility testing because automatic testing alone cannot cover all aspects of WCAG. Many of the tools mentioned on the W3C Web Accessibility Evaluation Tools List acknowledge that fact and report issues that can’t be tested automatically as “warnings”or “potential problems”.
The main target audience of these tools are web developers. The tools are intended for use during the creation of a web site and for subsequent quality assurance. This leads to some limitations: The output of the tools contains a lot of technical terms such as HTML element names and references to technical documentation such as the Techniques for WCAG 2.0. Therefore the tools can only be used by persons with web development expertise.
Moreover, repair instructions like “Ensure that the img element’s alt text serves the same purpose as the image.” are targeted towards improvements of the web content, and are not appropriate in the context of monitoring and status reports.
The WCAG Evaluation Methodology recommends involving users with disabilities in the evaluation of a web site. However, if this is done in an informal way without controlled setting, the results are often biased because personal opinion, individual expertise, or other factors influence the result. Especially the level of expertise of the user has a strong influence on the accuracy of the results.
This leads to the conclusion that evaluators should be grouped by their level of expertise rather than by type of disability. Clearly worded questions could lead to better answers from all users with little knowledge of web accessibility.
Structured semi-automatic evaluation approach
The objective of auto-wcag is to create a process with clear structure and instructions that are easy to understand so that even non-experts can follow. Standardized questions reduce the influence of individual opinion. A clear wording and predefined answer options instead of general statements or repair instructions lead to higher quality answers and thus to more reliable results.
Each auto-wcag test case consists of a selector and one or more test steps. There are automatic steps, which can be done by a tool, and manual steps, which require human input.
The manual steps describe tool support and instructions for non-expert users. Tool support can include highlighting the test subject, presenting alternative content that is not directly visible without special settings in the user agent, or providing other specific presentations of the content. These features allow the users to focus on the test subject. The users don’t have to identify the relevant item on the page and the distraction caused by irrelevant items it reduced.
Clear instructions and additional help text enable non-experts to answer the questions as well. The template also captures two additional properties of the test steps: the requirement of interaction and the consideration of context.
Check description
The original content and the (programmatically determined) alternative content are presented alongside each other. The question asks if the alternative describes the original content. This type applies to all kinds of non-text content such as images, audio and video as covered by Success Critereon 1.1.1 Non-text Content and Guideline 1.2 Time-based Media. For example, a paragraph of text is presented together with the programmatically determined language, the user is asked if the language is specified correctly.
Check presentation
The web content (or parts of it) is presented to the user in a specific way. For instance with resized text or in linearized form. The questions address features and problems of this presentation. This type applies for example to 1.4.4 Resize text: The text of the web page is resized to 200% and the user is asked if all content is still present.
Check interaction
The complete web page is presented to the users. In this type of test the user is instructed to interact with the web content and to make a statement about the operability. This type is used to check Success Criteria addressing operability. Moreover the behavior of focus, input, and error handling can be covered. For example, the user is asked to move the focus around the web page with the keyboard and to answer the question if the focus got trapped in any component of the page.
Manual selector
So far we have covered semi-automatic tests where the tool can determine applicability and present the preprocessed subject of the test to the non-expert user. However, there are also cases where applicability cannot be determined automatically and the user acts as a manual selector. In this type of user input the user is asked to identify content items that might cause accessibility barriers, such as use of color or other sensory characteristics to reference elements of the web page. It can also be applied to instances of flashing and auto-updating content that can’t be controlled by the user. For example, users could be asked to identify moving, blinking, or scrolling content that plays automatically and cannot be paused.
Next steps
Some participants of the auto-wcag community group are currently implementing the prototype of a User Testing Tool based on the questions developed in the structured approach described in this post. The tool runs in the user’s web browser and connects to a database storing the user input. The data can then be combined with the results from other (automatic) tools to create a report about the evaluated web content.
About the author
Annika Nietzio is a web accessibility expert working at Research Instistute Technologie and Disability in Germany. In the EIII project she is exploring new ways to combine the results from automated and manual web accessibility evaluations.
