Outdated Content!
The Protocols and Formats Working Group is no longer chartered to operate. Its work will continue in two new Working Groups:
- https://www.w3.org/WAI/APA/ Accessible Platform Architectures, to review specifications, develop technical support materials, collaborate with other Working Groups on technology accessibility, and coordinate harmonized accessibility strategies within W3C; and
- https://www.w3.org/WAI/ARIA/ Accessible Rich Internet Applications, to continue development of the Accessible Rich Internet Applications (WAI-ARIA) suite of technologies and other technical specifications when needed to bridge known gaps.
Resources from the PFWG remain available to support long-term institutional memory, but this information is of historical value only.
This Wiki page was edited by participants of the Protocols and Formats Working Group. It does not necessarily represent consensus and it may have incorrect information or information that is not supported by other Working Group participants, WAI, or W3C. It may also have some very useful information.
ARIA Test Plan
Prior to the existing of the wiki, a draft test plan was created. Content should be merged.
Contents
Objectives
There are two objectives to this test plan:
- Verify and record that WAI-ARIA and the WAI-ARIA User Agent Implementation Guide meet the requirements to advance past the Candidate Recommendation stage of W3C Process.
- Support the development community in ensuring that implementations of WAI-ARIA are interoperable.
The Candidate Recommendation exit criteria requires documentation of two interoperable implementations of each normative feature of the specifications. Each feature is considered separately, so the two tools implementing one feature do not need to be the same tools implementing another.
Interoperability of WAI-ARIA includes a number of targets:
- Mainstream user agents
- Assistive technologies
- Authoring Tools
- Conformance checking tools
- Authoring toolkits
- Authoring practice recommendations
- Web applications
Testing requirements
There must be success and failure tests for each feature tested. Often there are many ways to fail, so there may be multiple failure tests. Usually there is only one way to succeed, so there may normally only be one success test.
It's desirable to do user testing as well of ARIA-enabled tools, but this is not a formal CR testing project.
Types of tests
Unit tests
Unit tests verify proper implementation of discrete features. A unit test should contain exactly one ARIA feature, to determine how that feature is handled independent of any other feature. Some test files can be used for more than one unit test.
The following unit tests are needed:
- Proper handling of roles
- on elements without native semantics
- on elements with native semantics
- Proper handling of abstract roles
- Proper handling of unrecognized roles
- Handling of default values when values not provided for a given state or property
- Omitted required states or properties for a given role
- States and properties provided that are not supported for a given role
- Global states and properties on an element with no role
- States and properties have an invalid value (outside the defined value space)
- States and properties have a value that is out of range
- Proper handling of states and properties as element is changed.
- for example, a
treeitem
is expanded/collapsed via script. - test that its
aria-expanded
state is updated accordingly.
- for example, a
- Element with a given role does not contain required owned elements
- Element with a given role are not owned by a required context role
- Role attribute with multiple tokens
- first of which is a valid ARIA role
- one (but not first) of which is a valid ARIA role
- none of which is a valid ARIA role
Feature tests
- Proper handling of descendants of roles that should not map presentational children
- Proper calculation of accessible name
- Features that depend on two or more roles, states, or properties working together
- Live regions
- Drag and drop
- Grids
- Comboboxes
Dynamic tests
- What happens when values of states and properties change?
- What happens when role values change? (shouldn't happen)
- Reaction to events (note testing this is fairly UA- or OS- specific, need figure out how to define these tests)
Application examples
Complete applications using various aspects of ARIA together, working well.
- Yahoo Mail - Contact Victor Tsaran (get a URI)
- Lotus Connections - Contact Matt King and Richard Schwerdtfeger (get a URI)
- Pick ATs to test with and particular platforms ( should have two platforms but need to decide if two different browsers on the same platforms is acceptable )
- We need establish an acceptable compliance critera for CR - (combinations of ATs, Browser, and OS)
Test tools
- AccProbe - Instructions to use AccProbe
- AccChecker supports UIA verifications and is located on CodePlex
- Accerciser
- Speclenium - not being maintained but promising
- Accessibility Inspector
- Inspect32
- Inspect and AccEvent support UIA and they are provided in the Windows SDK.
- UIA Verify
- AViewer 2013 (the Paciello Group)
Test harness
- WCAG test samples repository
- Open Ajax Alliance Accessibility Tools Test Suite and WCAG 2 validation rules (including WAI-ARIA)
- OpenAjax Accessibility WCAG 2.0 Ruleset - Jon's test case, may be the further evolution of the above link
- Mobile Web Test Harness
- YETI - YUI East Test Interface; Introducing YETI
- @@ Javascript testing environments
- Open Ajax Alliance testing tool - ask Jon
- D.O.H. (Dojo Object Harness) for unit tests.
- also: D.O.H. robot for automated UI unit tests (emulates mouse and keyboard events).
- Licence: Dual licensed under the "New" BSD License and The Academic Free License v. 2.1.
- jQuery's Qunit test suite for unit tests.
- an advantage is that QUnit is self-contained and has no dependency on jQuery, or any other JavaScript toolkit.
- Licence: Dual licensed under the MIT and GPL Version 2 licenses.
- Selenium
- Rational test harness
Milestones
- Set up repository
- Migrate test samples from Codetalks.org
- Create unit tests and feature tests
- Identify all the CR tests to perform
- Identify and configure any testing automation tools
- Determine implementations that will be run through the test suite
- Perform tests and record test results
- Identify features that don't have two document implementations