Requirements for "Techniques for Automated and Semi-Automated Evaluation Tools"

This document provides a set of initial requirements that need to be incorporated in the document "Techniques for Automated and Semi-Automated Evaluation Tools". Further refinements of this document will occur under the scope of the Evaluation and Repair Tools Working Group (ERT WG) discussions.

This version:

http://www.w3.org/WAI/ER/WD-AERT/ED-requirements20130514

Previous published version:

http://www.w3.org/WAI/ER/WD-AERT/ED-requirements20130408

Editor:

Carlos A Velasco, Fraunhofer Institute for Applied Information Technology FIT

Copyright © 2013 W3C^® (MIT, ERCIM, Keio), All Rights Reserved. W3C liability, trademark and document use rules apply.

Purpose of the document

The document presented here gathers requirements for the document "Techniques for Automated and Semi-Automated Evaluation Tools", in the following called the document. This requirements document will present also some scenarios on the use of the main document.

The purpose of the document "Techniques for Automated and Semi-Automated Evaluation Tools" is to present typical features of web accessibility evaluation tools that will support the reader in defining different tool profiles. This may include some background information on how to implement WCAG 2.0 in evaluation tools or how to integrate accessibility evaluation in different web testing workflows.

Objectives of the document

The objectives of the document "Techniques for Automated and Semi-Automated Evaluation Tools" include the following:

1. Describe to web developers typical features of web accessibility evaluation tools and briefly describe possible implementation issues of these features. These issues may include additional information like licensing schemes, etc.
2. Define profiles of accessibility evaluation tools according to different combinations of the aforementioned features.
3. Support developers of accessibility evaluation tools in presenting results to different audiences.
4. Support developers of accessibility evaluation tools to understand the different types of techniques in WCAG 2.0 and types of web accessibility tests: automatic, semiautomatic and manual. [Editorial note: to be discussed with Working Group]
5. Present different workflows for accessibility evaluation and actors that participate in them. [Editorial note: to be discussed with Working Group]

Audience of the document

The document "Techniques for Automated and Semi-Automated Evaluation Tools" is targeted mainly to development managers and developers of accessibility evaluation tools. Under this scope, we will not distinguish between commercial and open source developers, although there are use cases and issues that could be more relevant to one group than to the other.

A secondary audience of this document are users of accessibility evaluation tools like accessibility experts or web developers.

Types of tools included

Examples of tools that are included are:

• Industrial/commercial and open source tools, which test complete web sites or web applications.
• Focused tools, which test a concrete aspect of accessibility, for instance, testing contrast of images, accessibility of forms, ARIA implementation, etc.
• Tools that support research with users or developers of specific aspects of accessibility.

Typical features of an evaluation tool

The document will contain descriptions of different features that are included in accessibility evaluation tools, which help to classify them and to identify their limitations. Typical examples include:

• ability to crawl big web sites or portals
• types of web technologies handled by the tool, for instance HTML markup, stylesheets, PDF documents, Flash applications, multimedia, etc.
• ability to integrate dynamic content generated via scripting (dynamic modification of the Document Object Model according to the user interaction with the application, etc.)
• support for testing APIs like the WebDriver API, for instance
• support for standard reporting languages like EARL
• support for different accessibility compliance environments in different countries
• integration in the web development workflow as a plug-in add-on in different Integrated Development Environments (open source or commercial)
• multilinguality and internationalisation
• etc.

Scenarios

Here we will present two or more scenarios which can put in context the recommendations of the document.

John: a development manager

John is a development manager in a small software company creating testing tools for mobile and desktop web applications. Due to increasing demand from customers, the company is evaluating the possibility to extend the software to evaluate web accessibility. John consults the document to get a general overview of typical features from accessibility evaluation tools. He also gathers information about resources that helped him to understand the implications of this new functionality and how their existing tools will map into the profiles defined in the document. He creates a matrix to compare the existing characteristics from its tool with the features of accessibility tools. With the result of this comparison, he is able to estimate the effort necessary to implement the new features of the tool and create an implementation roadmap.

Issues not covered in this document

The following issues are not covered in this document:

• Procurement and acquisition issues for this type of tools are outside of the scope of this document and are covered elsewhere
• Interpretation of WCAG 2.0 success criteria and techniques
• How to interpret standards and recommendations related to web technologies

References

1. Web Content Accessibility Guidelines (WCAG) 2.0
2. Website Accessibility Conformance Evaluation Methodology 1.0
3. Developer Guide for Evaluation and Report Language (EARL) 1.0
4. UWEM, Unified Web Evaluation Methodology version 1.2
5. Requirements for web developers and web commissioners in ubiquitous Web 2.0 design and development (January 2012)
6. ACCESSIBLE project

Table of contents

What follows is a preliminary table of contents for the document [Editorial note: urgent feedback from the working group is needed]:

1. Abstract
2. Status of this document
3. Introduction
   1. Audience of this document
   2. Document conventions
   3. Complementary resources
4. Typical features of an evaluation tool
5. Example profiles of evaluation tools
6. References
7. Appendix A: Customising results to different audiences
8. Appendix B: Integrating the evaluation procedure into the development testing workflows