Site Navigation: W3C Home > WAI Home > ATAG WG

ATAG 2.0 Implementation Report

Abstract

This implementation report describes how the Authoring Tool Accessibility Guidelines Working Group (ATAG WG) tested and demonstrated implementability of the Authoring Tool Accessibility Guidelines (ATAG) 2.0 specification during the W3C Process Candidate Recommendation phase.

Introduction

Publication and Test Development Dates

The Authoring Tool Accessibility Guidelines (ATAG) 2.0 went to final Last Call on 10 September 2013 and entered Candidate Recommendation on 7 November 2013. ATAG 2.0 Candidate Recommendation was republished on 4 June 2015 with updated Exit Criteria. The tests for the ATAG success criteria were mostly written from December 2012 to February 2013, and were kept updated with any changes to the guidelines after that date.

Test Harness

The Authoring Tool Accessibility Guidelines Working Group (ATAG WG) had been looking for a tool to track ATAG testing and had hoped to be included in the W3C testing project. When this was not possible, the ATAG WG asked to use the testing harness used for WCAG in 2007-2008 and adapt it to ATAG. There were many delays in this adaption mostly caused by the age of the code, the need to update it to meet security requirements of W3C Systeam, and scarcity of developer resources. The test harness first started being used for testing in October 2014, but the reporting function was not working until January 2015. The reporting function for cross-checking of individual tester's results was not working until March 2015. There are still bugs in the Test Harness that require careful cross-checking of the data. This static report is the definitive version.

Candidate Recommendation Testing

The ATAG 2.0 tests are primarily manual tests, although some of the WCAG testing can be performed with automated tools. The resources required to test an entire authoring tool necessitated a focus on simple tools that had fewer screens to test. The ATAG WG recruited some volunteer testers who were trained and supervised by the editors. In order to meet the second of the 3 original exit criteria, the tools tested in full were spread across 5 market segments:

• WYSIWYG web page editing tools
• Content management tools
• Development tool for applet, scripts, or applications
• Non-text media (e.g. video, audio, images) editing tools
• Social media content authoring tools (e.g. blogs, wikis, social networks)

In total, 25 tools were tested, of which 16 were fully tested, and 10 additional tools were tested for positive examples of specific features (as opposed to features that were met by being non-applicable).

Testing simpler tools to reduce the requirement for testing resources, and the challenge in finding tools with no WCAG bugs, required testing authoring tools beyond the initial ATAG implementers. Drupal and Wordpress were too complex to test with the limited ATAG WG resources. Other implementers were appreciative of the review feedback but did not want their product names disclosed. After the ATAG testers began evaluating some tools "in the wild" (e.g. without the knowledge of the vendors), it was decided to anonymize the authoring tool testing, except where the tool developers gave permission. Each authoring tool was assigned a random 3 digit code, in the test data, and in this report. Actual product names were provided to the W3C Director with this report.

Exit Criteria

2013 Exit Criteria

When the original exit criteria were decided in 2013, ATAG WG proposed 3 exit criteria to demonstrate that ATAG could be implemented to a high level and broadly across market segments:

1. Two authoring tools that met all applicable exit criteria to level AA.
2. Five authoring tools that met all applicable exit criteria to level A representing 5 market segments of authoring tools.
3. Two implementations of each success criteria

Thirteen of the ATAG success criteria (features) are dependent on WCAG 2.0 for their levels. WCAG 2.0 was fully tested during its Candidate Recommendation period ending in 2008, and is widely implemented, including in web-based authoring tools. ATAG WG decided to only test a sample of WCAG to demonstrate ATAG 2.0 success criteria dependent on WCAG.

While there are tools that meet the ATAG success criteria, it was more challenging to find tools that also met the ATAG 2.0 dependencies with WCAG 2.0 without exception or bugs. As the ATAG testers tested more and more tools looking for the two AA and five A tools, it became apparent that the limited testing resources would cause significant delay to completing ATAG. ATAG WG applied to the W3C Director to reduce the complexity of the ATAG exit criteria from three to one. Exit Criteria #1 and #2 were removed, leaving #3, demonstrating two implementations of each success criteria. Since W3C had already approved exit criteria for HTML5 and several CSS specs that reduced the testing burden, the simplified exit criteria section was approved. HTML5 exit criteria did not require testing of the over-80 specs referenced by HTML5, so with that precedent, ATAG 2.0 also kept the sampling of the WCAG success criteria from the original exit criteria.

2015 Exit Criteria

1. Two implementations of each success criteria (the ATAG success criteria dependent on WCAG will use the same WCAG sampling as in 2013)

Implementations by Success Criteria

This table is a static snapshot of the data collected from testing each success criteria. It provides:

• SC = success criteria number
• Handle = success criteria title
• Level = success criteria level (A, AA, AAA)
• Evaluations Passed = the number of tested tools that successfully passed this success criteria
• Implemented by = the name or the identification number of the tool selected as an example for the Implementation Report

Note on following table "Statistics by success criterion": Many tools passed success criteria. If a name does not appear on a success criteria, it does not mean that they did not pass it, merely that we did not select that tool to highlight the particular success criterion.

Success Criteria Dependent on WCAG 2.0

Thirteen of the ATAG success criteria have a level that is directly tied to their ability to meet WCAG 2.0. Since WCAG has already been tested as a W3C Recommendation and there are many examples of web sites that meet WCAG 2.0, it is not necessary to test all of WCAG to demonstrate that it is possible to meet WCAG. Consequently, a sampling of six WCAG success criteria — two of each level — were chosen to demonstrate implementability of ATAG using WCAG. Most of the tested authoring tools passed many more WCAG success criteria than the six noted below.

The following series of tables shows the sampling of WCAG success criteria that two tools passed to demonstrate that the ATAG success criteria can be passed. The following series of tables show:

• The text of ATAG 2.0 success criteria whose level is based on WCAG
• Authoring tool = the name or identification number of the tool selected
• WCAG A = the number and title of the WCAG A success criteria passed by that tool
• WCAG AA = the number and title of the WCAG AA success criteria passed by that tool
• WCAG AAA = the number and title of the WCAG AAA success criteria passed by that tool