This Wiki page is edited by participants of the Protocols and Formats Working Group. It does not necessarily represent consensus and it may have incorrect information or information that is not supported by other Working Group participants, WAI, or W3C. It may also have some very useful information.
ARIA Test Plan
From Protocols and Formats Working Group Wiki
Prior to the existing of the wiki, a draft test plan was created. Content should be merged.
There are two objectives to this test plan:
- Verify and record that WAI-ARIA and the WAI-ARIA User Agent Implementation Guide meet the requirements to advance past the Candidate Recommendation stage of W3C Process.
- Support the development community in ensuring that implementations of WAI-ARIA are interoperable.
The Candidate Recommendation exit criteria requires documentation of two interoperable implementations of each normative feature of the specifications. Each feature is considered separately, so the two tools implementing one feature do not need to be the same tools implementing another.
Interoperability of WAI-ARIA includes a number of targets:
- Mainstream user agents
- Assistive technologies
- Authoring Tools
- Conformance checking tools
- Authoring toolkits
- Authoring practice recommendations
- Web applications
There must be success and failure tests for each feature tested. Often there are many ways to fail, so there may be multiple failure tests. Usually there is only one way to succeed, so there may normally only be one success test.
It's desirable to do user testing as well of ARIA-enabled tools, but this is not a formal CR testing project.
Types of tests
Unit tests verify proper implementation of discrete features. A unit test should contain exactly one ARIA feature, to determine how that feature is handled independent of any other feature. Some test files can be used for more than one unit test.
The following unit tests are needed:
- Proper handling of roles
- on elements without native semantics
- on elements with native semantics
- Proper handling of abstract roles
- Proper handling of unrecognized roles
- Handling of default values when values not provided for a given state or property
- Omitted required states or properties for a given role
- States and properties provided that are not supported for a given role
- Global states and properties on an element with no role
- States and properties have an invalid value (outside the defined value space)
- States and properties have a value that is out of range
- Proper handling of states and properties as element is changed.
- for example, a
treeitemis expanded/collapsed via script.
- test that its
aria-expandedstate is updated accordingly.
- for example, a
- Element with a given role does not contain required owned elements
- Element with a given role are not owned by a required context role
- Role attribute with multiple tokens
- first of which is a valid ARIA role
- one (but not first) of which is a valid ARIA role
- none of which is a valid ARIA role
- Proper handling of descendants of roles that should not map presentational children
- Proper calculation of accessible name
- Features that depend on two or more roles, states, or properties working together
- Live regions
- Drag and drop
- What happens when values of states and properties change?
- What happens when role values change? (shouldn't happen)
- Reaction to events (note testing this is fairly UA- or OS- specific, need figure out how to define these tests)
Complete applications using various aspects of ARIA together, working well.
- Yahoo Mail - Contact Victor Tsaran (get a URI)
- Lotus Connections - Contact Matt King and Richard Schwerdtfeger (get a URI)
- Pick ATs to test with and particular platforms ( should have two platforms but need to decide if two different browsers on the same platforms is acceptable )
- We need establish an acceptable compliance critera for CR - (combinations of ATs, Browser, and OS)
- AccProbe - Instructions to use AccProbe
- AccChecker supports UIA verifications and is located on CodePlex
- Speclenium - not being maintained but promising
- Accessibility Inspector
- Inspect and AccEvent support UIA and they are provided in the Windows SDK.
- UIA Verify
- AViewer 2013 (the Paciello Group)
- WCAG test samples repository
- Open Ajax Alliance Accessibility Tools Test Suite and WCAG 2 validation rules (including WAI-ARIA)
- OpenAjax Accessibility WCAG 2.0 Ruleset - Jon's test case, may be the further evolution of the above link
- Mobile Web Test Harness
- YETI - YUI East Test Interface; Introducing YETI
- Open Ajax Alliance testing tool - ask Jon
- D.O.H. (Dojo Object Harness) for unit tests.
- jQuery's Qunit test suite for unit tests.
- Licence: Dual licensed under the MIT and GPL Version 2 licenses.
- Rational test harness
- Set up repository
- Migrate test samples from Codetalks.org
- Create unit tests and feature tests
- Identify all the CR tests to perform
- Identify and configure any testing automation tools
- Determine implementations that will be run through the test suite
- Perform tests and record test results
- Identify features that don't have two document implementations