[contents]
Abstract
The purpose of this document is
to support developers of web accessibility evaluation tools by identifying
typical features of those tools and how to classify them according to
different combinations of those features.
Status of this
document
This section describes the status of this document at
the time of its publication. Other documents may supersede this document. A
list of current W3C publications and the latest revision of this technical
report can be found in the W3C technical
reports index at http://www.w3.org/TR/.
Table of
contents
- Introduction
- Audience of this document
- Complementary resources
- Typical features of an accessibility evaluation
tool
- Types of document formats analyzed
- Crawling of sites
- Support for localization
- Support for different policy
environments
- Testing applications, complete resources or
fragments
- Support for web testing APIs
- Support for standard reporting
languages
- Support for semiautomatic and manual
tests
- Report customization to different
audiences
- Integration in the web development
workflow
- Example profiles of evaluation tools
- References
- Acknowledgements
- Appendix A: Customising
results to different audiences
- Appendix B:
Integrating the evaluation procedure into the development testing
workflows
1. Introduction
There is a wide variety of web
accessibility evaluation tools. This document intends to support evaluation
tool developers to identify their key characteristics. To achieve that, this
document:
- describes to developers of web accessibility evaluation tools the
typical features of these tools and briefly presents
different perspectives on these features;
- describes typical profiles for web accessibility
evaluation tools according to different combinations of the
aforementioned features;
- supports developers of web accessibility evaluation tools to
understand the different types of web accessibility
tests: automatic, semiautomatic and manual; and
- supports developers of web accessibility evaluation tools to
understand how to use WCAG 2.0 success criteria, sufficient
techniques, advisory techniques, and common failures for web
accessibility testing.
1.1. Audience of this document
This document
is targeted mainly to development managers and developers of web
accessibility evaluation tools.
A secondary audience of this document
are users of accessibility evaluation tools like accessibility experts or
web developers.
Examples of tools that are within the scope of the
document include:
- Proprietary or open source accessibility evaluation tools.
- Accessibility evaluation tools that test complete web sites or web
applications.
- Accessibility evaluation tools that test single web pages or
resources.
- Focused accessibility evaluation tools, which test a concrete aspect
of accessibility, for instance, contrast of images, accessibility of
forms, WAI-ARIA implementation, etc.
- Accessibility evaluation tools that support research with users or
developers of specific aspects of accessibility.
1.2. Background documents
This document
must be seen in the context of several others. It is recommended that the
reader reviews the following documents:
- Web Content Accessibility Guidelines 2.0 [WCAG20]
- Techniques and Failures for Web Content Accessibility Guidelines 2.0
[WCAG20-TECHS]
- Evaluation and Report Language (EARL) 1.0 Schema [EARL10]
In the document you will find additional pointers to other
resources like standards, recommendations, technical specifications, etc.,
which are relevant to any developer interested in implementing an
accessibility evaluation tool.
2. Typical features of
an accessibility evaluation tool
In this section, we will describe
typical features and functionalities of accessibility evaluation tools. The
features of an accessibility evaluation tool can be presented from different
perspectives: the subject being tested, the target audiences of the tool,
the reporting and presentation of the results, its configurability, etc. We
have tried to be as complete as possible, but it may be that some features
of existing or future evaluation tools are omitted.
It is very
important that you analyse and describe for your own development process and
for your customers which of those features are supported in your tool and
declare any limitations of your tool.
2.1. Types of
document formats analyzed
Although the vast majority of web documents
are HTML documents, there are many other types of resources that need to be
considered when analyzing web accessibility. Of particular importance are
resources like CSS stylesheets or Javascript scripts, which
allow the modification of markup documents in the user agent when they are
loaded or via user interaction, as many accessibility tests are the result
of the interpretation of those resources.
In general, we can
distinguish these types of formats:
- Markup documents. These are normally
HTML [HTML4, HTML5] or XML documents. Processing these
documents requires the ability to build a Document Object Model
(DOM) according to the different specifications, which can
then parsed and analysed.
- Style resources. These are presentation modifiers
conformant to the different CSS specifications [CSS2, CSS3], which are processed by
the user agents. The interpretation of stylesheets is critical for many
accessibility techniques.
- Script resources. These are components that modify
at a defined point documents and styles, be it because of user
interaction or because of other functional requirements of an
application (e.g., because of dynamic updates or modifications).
Nowadays these resources play a critical role in mobile web and web 2.0
applications. Ignoring these resources may lead to missing accessibility
problems or to report non-existing errors. The scripting language
commonly used on the web is Javascript, based upon ECMAScript [ECMAScript] standardized by the Ecma International
standards organization
- Multimedia resources. These are images, movies or
audio tracks that are inserted into the application. From the
accessibility standpoint, the interpretation of those resources is very
important, especially for issues like colour contrast, colour blindness,
or media alternatives, for instance.
- PDF documents. Many reports and administrative
information is available in this format, especially in government
websites. The format was initially developed by Adobe [PDF] and was later on standardised by ISO.
- Resources with other proprietary or open source
formats. Although not very frequent on the web, you may
encounter other formats like office documents, flash movies and
applications, etc. Depending on your customers, you may need to be able
to parse and evaluate this type of resources.
Most of the accessibility evaluation tools concentrate on the
markup evaluation, but the most advanced are able to process many of the
types described above.
2.2. Crawling of sites
A
typical difference between proprietary and open source tools is the
capability to crawl web resources. That means that the tool incorporates a
web crawler [WEBCRAWLER] able to extract
hyperlinks out of web resources. It must be kept in mind that, as seen in
the previous section, there are many types of resources on the web that
contain hyperlinks. The misconception that only HTML documents contain links
may lead to wrong assumptions in the evaluation process.
A web crawler
defines an starting point and a set of options. The critical features of a
web crawler are related to its configuration capabilities. Among them, we
can highlight:
- Types of documents extracted.
- Capability to define inclusion and exclusion filters. Sometimes
customers require analysis of concrete parts of the website, or do not
want to include others.
- Multithreaded crawling. For a large site, it may be important to
optimise performance by having a tool able to crawl in parallel
threads.
- Avoidance of resource duplicates. It is typical from web resources
to link many times to the same results. If the crawler is not able to
identify such issues, it may lead to a great performance loss.
- Session persistence. Some sites include the session ID in the URL.
If the site is big, it may occur that the session ID changes over time,
leading to identify resources as different when they are in reality the
same.
- Authentication support. Many sites require some kind of
authentication (e.g., HTTP authentication, OpenID, etc.). A crawler
should be in the position to handle such use cases.
2.3. Support for
localization
Internationalization and localization are important
issue to address worldwide markets. There may be cases where your customers
are not able to speak English, and you need to generate information in other
languages. To that end, you can start by looking into the authorized
translations of the Web Content
Accessibility Guidelines.
2.4. Support for different
policy environments
Although there is an international effort to
harmonisation of legislation in regard to web accessibility, there are still
minor differences in the accessibility policy in different countries. It is
important that you clearly define in your tool which of those policy
environments you support. Most of the tools are focused on the
implementation of their Web Content Accessibility Guidelines 2.0 [WCAG20], because it is the most common reference for
those policies worldwide.
2.5. Testing applications,
complete resources or fragments
Nowadays, it is typical that it is
necessary to test fragments of HTML documents, coming for instance from a
web editor in a Content Management System. For those cases, the tool must be
able to generate a document fragment to be tested. Furthermore, the tool
needs to filter the accessibility tests according to their relevance to the
document fragment.
Within this section we include testing of web
applications as well. For them it is typical to emulate different user
actions (e.g., activating interface components or filling and sending forms)
that modify the status of the current page or load new resources. The user
of such an application would need to define these intermediate steps that
can be later on interpreted by the tool.
2.6. Support
for web testing APIs
When evaluating accessibility of web sites and
applications it is sometimes desirable to have own scripts that emulate some
kind of user interaction. With the growing complexity of web applications,
there has been an effort to standardize such interfaces. One of them is, for
instance, the WebDriver API [WebDriver]. With such
tools, it is possible to write tests that automate the browser
behaviour.
2.7. Support for standard reporting
languages
There are use cases where customers want to exchange
results, compare evaluation results with other tools, import results (for
instance, when tool A does not test a given problem, but tool B does it),
filter results, etc. For such cases, it is important to support standardised
languages like the Evaluation and Report Language [EARL10] under development within W3C.
2.8. Support for semiautomatic and manual tests
According
to the Evaluation and Report Language specification [EARL10], there are three types of modes to perform
accessibility tests:
- Automatic - where the test was carried out automatically by the
software tool and without any human intervention.
- Manual - where the test was carried out by human evaluators. This
includes the case where the evaluators are aided by instructions or
guidance provided by software tools, but where the evaluators carried
out the actual test procedure.
- Semi-Automatic - where the test was partially carried out by
software tools, but where human input or judgment was still required to
decide or help decide the outcome of the test.
Most of the tools concentrate on the testing of accessibility
requirements which can be performed automatically, although there are some
that support accessibility experts by performing the other two types of
tests.
Sometimes, the tools do not declare openly that they only
perform automatic testing. Since it is a known fact that automatic tests
only cover a small set of accessibility issues, accessibility conformance
can only be ensured by supporting developers and accessibility experts while
testing in the manual and semiautomatic mode.
2.9.
Report customization to different audiences
Typically, evaluation
tools are targeted to accessibility experts. However, there are also tools
that allow the customization of the accessibility reports to other audiences
like for instance:
- web developers with no or little knowledge of accessibility, who
need detailed information on how to correct the problems found to
implement appropriate solutions; or
- web commissioners, who need an aggregated view of the evaluation
results and tools to support the monitoring of their own websites.
2.10. Integration in the web development
workflow
There are tools that evaluate accessibility as an
stand-alone tool, but there are other tools which integrate more naturally
into the development process of web applications as plug-ins or extensions
in web browsers or in Integrated Development Environments (IDEs) and Content
Management Systems (CMS).
The latest tools allow an optimal workflow
process because integrates accessibility into the overall quality assurance
process of a website or a web application development.
3. Example profiles of evaluation tools
4. References
The following are references cited in
the document.
- CSS2
- Cascading Style Sheets Level 2 Revision 1 (CSS 2.1) Specification.
W3C Recommendation 07 June 2011. Bert Bos, Tantek Çelik, Ian Hickson,
Håkon Wium Lie (editors). Available at: http://www.w3.org/TR/CSS2/
- CSS3
- CSS Current Status is available at: http://www.w3.org/standards/techs/css#w3c_all
- EARL10
- Evaluation and Report Language (EARL) 1.0 Schema. W3C Working Draft
10 May 2011. Shadi Abou-Zahra (editor). Available at: http://www.w3.org/TR/EARL10-Schema/
- ECMAScript
- ECMAScript® Language Specification. Standard ECMA-262 5.1 Edition /
June 2011. Available at: http://www.ecma-international.org/ecma-262/5.1/
- HTML4
- HTML 4.01 Specification. W3C Recommendation 24 December 1999. Dave
Raggett, Arnaud Le Hors, Ian Jacobs (editors). Available at: http://www.w3.org/TR/html4/
- HTML5
- HTML5. A vocabulary and associated APIs for HTML and XHTML. W3C
Candidate Recommendation 17 December 2012. Robin Berjon, Travis
Leithead, Erika Doyle Navara, Edward O'Connor, Silvia Pfeiffer
(editors). Available at: http://www.w3.org/TR/html5/
- PDF
- PDF Reference, sixth edition. Adobe® Portable Document Format,
Version 1.7, November 2006. Adobe Systems Incorporated. Available at: http://www.adobe.com/devnet/pdf/pdf_reference_archive.html
- RFC2119
- Key words for use in RFCs to Indicate Requirement
Levels. IETF RFC, March 1997. Available at: http://www.ietf.org/rfc/rfc2119.txt
- WAI-ARIA
- Accessible Rich Internet Applications (WAI-ARIA) 1.0. W3C Candidate
Recommendation 18 January 2011. James Craig, Michael Cooper (editors).
Available at: http://www.w3.org/TR/wai-aria/
- WCAG20
- Web Content Accessibility Guidelines (WCAG) 2.0. W3C Recommendation
11 December 2008. Ben Caldwell, Michael Cooper, Loretta Guarino Reid,
Gregg Vanderheiden (editors). Available at: http://www.w3.org/TR/WCAG20/
- WCAG20-TECHS
- Techniques for WCAG 2.0. Techniques and Failures for Web Content
Accessibility Guidelines 2.0. W3C Working Group Note 3 January 2012.
Michael Cooper, Loretta Guarino Reid, Gregg Vanderheiden (editors).
Available at: http://www.w3.org/TR/WCAG20-TECHS/
- WEBCRAWLER
- Wikipedia. http://en.wikipedia.org/wiki/Web_crawler
- WebDriver
- WebDriver. W3C Working Draft 12 March 2013. Simon Stewart, David
Burns (editors). Available at: http://www.w3.org/TR/webdriver/
Acknowledgements
Appendix A:
Customising results to different audiences
Appendix B: Integrating the evaluation procedure into the
development testing workflows