[contents]
Abstract
The purpose of this document is to support developers of web accessibility
evaluation tools by identifying typical features of those tools and how to classify them according to
different combinations of those features.
Status of this document
This section describes
the status of this document at the time of its publication. Other documents may supersede this document. A
list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.
Table
of contents
- Introduction
- Audience of this document
- Complementary resources
- Typical features of an accessibility evaluation tool
- Types of document formats analyzed
- Cookies
- Authentication support
- Session tracking
- Crawling of sites
- Reporting
- Localization
- Support for different policy environments
- Evaluating document fragments
- Evaluating web applications
- Support for web testing APIs
- Support for semiautomatic and manual tests
- Customization to different audiences
- Integration in the web development workflow
- Support for repair
- Persistence of results and monitoring over time
- Example profiles of evaluation tools
- References
- Acknowledgements
- Appendix A: Customising results to different
audiences
- Appendix B: Integrating the evaluation
procedure into the development testing workflows
1 Introduction
There is a wide variety of web accessibility evaluation
tools. This document intends to support evaluation tool developers to identify their key characteristics.
To achieve that, this document:
- describes to developers of web accessibility evaluation tools the typical
features of these tools and briefly presents different perspectives on these
features;
- describes typical profiles for web accessibility evaluation tools according
to different combinations of the aforementioned features;
- supports developers of web accessibility evaluation tools to understand the different
types of web accessibility tests: automatic, semiautomatic and manual; and
- supports developers of web accessibility evaluation tools to understand how to use
WCAG 2.0 success criteria, sufficient techniques, advisory techniques, and common
failures for web accessibility testing.
1.1 Audience of this document
This document is targeted mainly to
development managers and developers of web accessibility evaluation tools.
A secondary audience of
this document are users of accessibility evaluation tools like accessibility experts or web
developers.
Examples of tools that are within the scope of the document include:
- Proprietary or open source accessibility evaluation tools.
- Accessibility evaluation tools that test complete web sites or web applications.
- Accessibility evaluation tools that test single web pages or resources.
- Focused accessibility evaluation tools, which test a concrete aspect of accessibility, for
instance, contrast of images, accessibility of forms, WAI-ARIA implementation, etc.
- Accessibility evaluation tools that support research with users or developers of specific
aspects of accessibility.
1.2 Background documents
This document must be seen in the context of
several others. It is recommended that the reader reviews the following documents:
- Web Content Accessibility Guidelines 2.0 [WCAG20]
- Techniques and Failures for Web Content Accessibility Guidelines 2.0 [WCAG20-TECHS]
- Evaluation and Report Language (EARL) 1.0 Schema [EARL10]
In the document you will find additional pointers to other resources like standards,
recommendations, technical specifications, etc., which are relevant to any developer interested in
implementing an accessibility evaluation tool.
2 Typical features of an accessibility
evaluation tool
In this section, we will describe typical features and functionalities of
accessibility evaluation tools. The features of an accessibility evaluation tool can be presented from
different perspectives: the subject being tested, the target audiences of the tool, the reporting and
presentation of the results, its configurability, etc. We have tried to be as complete as possible, but it
may be that some features of existing or future evaluation tools are omitted. The following list of
characteristics does not follow any particular order.
It is very important that you analyse and
describe for your own development process and for your customers which of those features are supported in
your tool and declare any limitations of your tool.
2.1 Types of document formats
analyzed
Although the vast majority of web documents are HTML documents, there are many other types
of resources that need to be considered when analyzing web accessibility. Of particular importance are
resources like CSS stylesheets or Javascript scripts, which allow the modification of markup
documents in the user agent when they are loaded or via user interaction, as many accessibility tests are
the result of the interpretation of those resources.
In general, we can distinguish these types of
formats:
- Markup documents. These are normally HTML [HTML4, HTML5] or XML documents. Processing
these documents requires the ability to build a Document Object Model (DOM) according
to the different specifications, which can then parsed and analysed.
- Style resources. These are presentation modifiers conformant to the different
CSS specifications [CSS2, CSS3], which are
processed by the user agents. The interpretation of stylesheets is critical for many accessibility
techniques.
- Script resources. These are components that modify at a defined point
documents and styles, be it because of user interaction or because of other functional
requirements of an application (e.g., because of dynamic updates or modifications). Nowadays these
resources play a critical role in mobile web and web 2.0 applications. Ignoring these resources
may lead to missing accessibility problems or to report non-existing errors. The scripting
language commonly used on the web is Javascript, based upon ECMAScript [ECMAScript] standardized by the Ecma International standards organization
- Multimedia resources. These are images, movies or audio tracks that are
inserted into the application. From the accessibility standpoint, the interpretation of those
resources is very important, especially for issues like colour contrast, colour blindness, or
media alternatives, for instance.
- PDF documents. Many reports and administrative information is available in
this format, especially in government websites. The format was initially developed by Adobe [PDF] and was later on standardised by ISO.
- Resources with other proprietary or open source formats. Although not very
frequent on the web, you may encounter other formats like office documents, flash movies and
applications, etc. Depending on your customers, you may need to be able to parse and evaluate this
type of resources.
Most of the accessibility evaluation tools concentrate on the markup evaluation, but the most
advanced are able to process many of the types described above.
2.2 Cookies
A
cookie is a name-value pair that it is stored in the browser of the user. Cookies contain information
relevant to the website that is being rendered and often include authentication and session information.
This information can be critical for other use cases, like for instance a crawling tool.
2.3 Authentication support
Many sites require some kind of authentication
(e.g., HTTP authentication, OpenID, etc.). An accessibility testing tool shall be able to support the
typical authentication scenarios. It is important to do so because many sites present different content to
authenticated users.
2.4 Session tracking
For security reasons, some sites
include the session ID in the URL or in a cookie. With the support of the session information, websites
can implement security mechanisms like for instance login out a user after a long inactivity period or
track typical interaction paths of the users.
2.5 Crawling of sites
A typical
difference between proprietary and open source tools is the capability to crawl web resources. That means
that the tool incorporates a web crawler [WEBCRAWLER] able to extract hyperlinks
out of web resources. It must be kept in mind that, as seen in the previous section, there are many types
of resources on the web that contain hyperlinks. The misconception that only HTML documents contain links
may lead to wrong assumptions in the evaluation process.
A web crawler defines an starting point and
a set of options. The critical features of a web crawler are related to its configuration capabilities.
Among them, we can highlight:
- Types of documents extracted.
- Capability to define inclusion and exclusion filters. Sometimes customers require analysis of
concrete parts of the website, or do not want to include others.
- Multithreaded crawling. For a large site, it may be important to optimise performance by
having a tool able to crawl in parallel threads.
- Avoidance of duplicate downloads. It is typical from web resources to link many times to the
same results. If the crawler is not able to identify such issues, it may lead to a great
performance loss.
- Session tracking. If the site is big, it may occur that the session ID changes over time,
leading to identify resources as different when they are the same.
- Authentication support.
2.6 Reporting
Especially when dealing with the evaluation of several
web resources, it is important that the tool is able to generate reports with the results of the
evaluation. Among the characteristics of these reports, we can highlight:
- Support for standard reporting languages like EARL [EARL10]. There are
use cases where customers want to exchange results, compare evaluation results with other tools,
import results (for instance, when tool A does not test a given problem, but tool B does it),
filter results, etc.
- Support for aggregation of results.
- Report customization according to different criteria.
2.7 Localization
Localization is important to address worldwide
markets. There may be cases where your customers are not able to speak English and you need to present
your user interface and your reports in other languages. To that end, you can start by looking into the
authorized translations of the Web Content
Accessibility Guidelines.
2.8 Support for different policy
environments
Although there is an international effort to harmonisation of legislation in regard to
web accessibility, there are still minor differences in the accessibility policy in different countries.
It is important that you clearly define in your tool which of those policy environments you support. Most
of the tools are focused on the implementation of their Web Content Accessibility Guidelines 2.0 [WCAG20], because it is the most common reference for those policies worldwide.
2.9 Evaluating document fragments
Nowadays, it is typical that it is necessary to
test fragments of HTML documents, coming for instance from a web editor in a Content Management System.
For those cases, the tool must be able to generate a document fragment to be tested. Furthermore, the tool
needs to filter the accessibility tests according to their relevance to the document fragment.
2.10 Evaluating web applications
Web and cloud applications are becoming very frequent
on the web. These applications present similar interaction patterns as those of the desktop applications.
Tools that evaluate such applications must emulate different user actions (e.g., activating interface
components or filling and sending forms) that modify the status of the current page or load new resources.
The user of such an application would need to define these intermediate steps that can be later on
interpreted by the tool (see the following section).
2.11 Support for web testing
APIs
When evaluating accessibility of web sites and applications it is sometimes desirable to have
own scripts that emulate some kind of user interaction. With the growing complexity of web applications,
there has been an effort to standardize such interfaces. One of them is, for instance, the WebDriver API
[WebDriver]. With such tools, it is possible to write tests that automate the
browser behaviour.
2.12 Support for semiautomatic and manual tests
According to
the Evaluation and Report Language specification [EARL10], there are three types of
modes to perform accessibility tests:
- Automatic - where the test was carried out automatically by the software tool and without any
human intervention.
- Manual - where the test was carried out by human evaluators. This includes the case where the
evaluators are aided by instructions or guidance provided by software tools, but where the
evaluators carried out the actual test procedure.
- Semi-Automatic - where the test was partially carried out by software tools, but where human
input or judgment was still required to decide or help decide the outcome of the test.
Most of the tools concentrate on the testing of accessibility requirements which can be
performed automatically, although there are some that support accessibility experts by performing the
other two types of tests.
Sometimes, the tools do not declare openly that they only perform
automatic testing. Since it is a known fact that automatic tests only cover a small set of accessibility
issues, accessibility conformance can only be ensured by supporting developers and accessibility experts
while testing in the manual and semiautomatic mode.
2.13 Customization to
different audiences
Typically, evaluation tools are targeted to accessibility experts. However,
there are also tools that allow the customization of the evaluation results to other audiences like for
instance:
- web developers with no or little knowledge of accessibility, who need detailed information on
how to correct the problems found to implement appropriate solutions; or
- web commissioners, who need an aggregated view of the evaluation results and tools to support
the monitoring of their own websites.
2.14 Integration in the web development workflow
There are tools that
evaluate accessibility as an stand-alone tool, but there are other tools which integrate more naturally
into the development process of web applications as plug-ins or extensions in web browsers or in
Integrated Development Environments (IDEs) and Content Management Systems (CMS).
The latest tools
allow an optimal workflow process because integrates accessibility into the overall quality assurance
process of a website or a web application development.
2.15 Support for
repair
2.16 Persistence of results and monitoring over time
3 Example profiles of evaluation tools
4
References
The following are references cited in the document.
- CSS2
- Cascading Style Sheets Level 2 Revision 1 (CSS 2.1) Specification. W3C Recommendation 07 June
2011. Bert Bos, Tantek Çelik, Ian Hickson, Håkon Wium Lie (editors). Available at: http://www.w3.org/TR/CSS2/
- CSS3
- CSS Current Status is available at: http://www.w3.org/standards/techs/css#w3c_all
- EARL10
- Evaluation and Report Language (EARL) 1.0 Schema. W3C Working Draft 10 May 2011. Shadi
Abou-Zahra (editor). Available at: http://www.w3.org/TR/EARL10-Schema/
- ECMAScript
- ECMAScript® Language Specification. Standard ECMA-262 5.1 Edition / June 2011. Available at:
http://www.ecma-international.org/ecma-262/5.1/
- HTML4
- HTML 4.01 Specification. W3C Recommendation 24 December 1999. Dave Raggett, Arnaud Le Hors,
Ian Jacobs (editors). Available at: http://www.w3.org/TR/html4/
- HTML5
- HTML5. A vocabulary and associated APIs for HTML and XHTML. W3C Candidate Recommendation 17
December 2012. Robin Berjon, Travis Leithead, Erika Doyle Navara, Edward O'Connor, Silvia Pfeiffer
(editors). Available at: http://www.w3.org/TR/html5/
- PDF
- PDF Reference, sixth edition. Adobe® Portable Document Format, Version 1.7, November 2006.
Adobe Systems Incorporated. Available at: http://www.adobe.com/devnet/pdf/pdf_reference_archive.html
- RFC2119
- Key words for use in RFCs to Indicate Requirement Levels. IETF
RFC, March 1997. Available at: http://www.ietf.org/rfc/rfc2119.txt
- WAI-ARIA
- Accessible Rich Internet Applications (WAI-ARIA) 1.0. W3C Candidate Recommendation 18 January
2011. James Craig, Michael Cooper (editors). Available at: http://www.w3.org/TR/wai-aria/
- WCAG20
- Web Content Accessibility Guidelines (WCAG) 2.0. W3C Recommendation 11 December 2008. Ben
Caldwell, Michael Cooper, Loretta Guarino Reid, Gregg Vanderheiden (editors). Available at: http://www.w3.org/TR/WCAG20/
- WCAG20-TECHS
- Techniques for WCAG 2.0. Techniques and Failures for Web Content Accessibility Guidelines 2.0.
W3C Working Group Note 3 January 2012. Michael Cooper, Loretta Guarino Reid, Gregg Vanderheiden
(editors). Available at: http://www.w3.org/TR/WCAG20-TECHS/
- WEBCRAWLER
- Wikipedia. http://en.wikipedia.org/wiki/Web_crawler
- WebDriver
- WebDriver. W3C Working Draft 12 March 2013. Simon Stewart, David Burns (editors). Available
at: http://www.w3.org/TR/webdriver/
Acknowledgements
Appendix A: Customising results to
different audiences
Appendix B: Integrating the evaluation procedure into
the development testing workflows