This Wiki page is edited by participants of the Protocols and Formats Working Group. It does not necessarily represent consensus and it may have incorrect information or information that is not supported by other Working Group participants, WAI, or W3C. It may also have some very useful information.

ARIA Test Plan

From Protocols and Formats Working Group Wiki
Revision as of 15:45, 14 February 2011 by Cooper (Talk | contribs)

Jump to: navigation, search


Prior to the existing of the wiki, a draft test plan was created. Content should be merged.

Objectives

There are two objectives to this test plan:

  • Verify and record that WAI-ARIA and the WAI-ARIA User Agent Implementation Guide meet the requirements to advance past the Candidate Recommendation stage of W3C Process.
  • Support the development community in ensuring that implementations of WAI-ARIA are interoperable.

The Candidate Recommendation exit criteria requires documentation of two interoperable implementations of each normative feature of the specifications. Each feature is considered separately, so the two tools implementing one feature do not need to be the same tools implementing another.

Interoperability of WAI-ARIA includes a number of targets:

  • Mainstream user agents
  • Assistive technologies
  • Authoring Tools
  • Conformance checking tools
  • Authoring toolkits
  • Authoring practice recommendations
  • Web applications

Testing requirements

There must be success and failure tests for each feature tested. Often there are many ways to fail, so there may be multiple failure tests. Usually there is only one way to succeed, so there may normally only be one success test.

It's desirable to do user testing as well of ARIA-enabled tools, but this is not a formal CR testing project.

Types of tests

Unit tests

Unit tests verify proper implementation of discrete features. A unit test should contain exactly one ARIA feature, to determine how that feature is handled independent of any other feature. Some test files can be used for more than one unit test.

The following unit tests are needed:

  • Proper handling of roles
    • on elements without native semantics
    • on elements with native semantics
  • Proper handling of abstract roles
  • Proper handling of unrecognized roles
  • Handling of default values when values not provided for a given state or property
  • Omitted required states or properties for a given role
  • States and properties provided that are not supported for a given role
  • Global states and properties on an element with no role
  • States and properties have an invalid value (outside the defined value space)
  • States and properties have a value that is out of range
  • Proper handling of states and properties as element is changed.
    • for example, a treeitem is expanded/collapsed via script.
    • test that its aria-expanded state is updated accordingly.
  • Element with a given role does not contain required owned elements
  • Element with a given role are not owned by a required context role
  • Role attribute with multiple tokens
    • first of which is a valid ARIA role
    • one (but not first) of which is a valid ARIA role
    • none of which is a valid ARIA role

Feature tests

  • Proper handling of descendants of roles that should not map presentational children
  • Proper calculation of accessible name
  • Features that depend on two or more roles, states, or properties working together
    • Live regions
    • Drag and drop
    • Grids
    • Comboboxes

Dynamic tests

  • What happens when values of states and properties change?
  • What happens when role values change? (shouldn't happen)
  • Reaction to events (note testing this is fairly UA- or OS- specific, need figure out how to define these tests)

Application examples

Complete applications using various aspects of ARIA together, working well.

  • Yahoo Mail - Contact Victor Tsaran (get a URI)
  • Lotus Connections - Contact Matt King and Richard Schwerdtfeger (get a URI)
  • Pick ATs to test with and particular platforms ( should have two platforms but need to decide if two different browsers on the same platforms is acceptable )
  • We need establish an acceptable compliance critera for CR - (combinations of ATs, Browser, and OS)

Test tools

Test harness

Milestones

  • Set up repository
  • Migrate test samples from Codetalks.org
  • Create unit tests and feature tests
  • Identify all the CR tests to perform
  • Identify and configure any testing automation tools
  • Determine implementations that will be run through the test suite
  • Perform tests and record test results
  • Identify features that don't have two document implementations

Resources