How to evaluate a user agent for conformance to UAAG 1.0

Abstract

This document provides guidance for determining whether a user agent satisfies the requirements of User Agent Accessibility Guidelines (UAAG) 1.0 [UAAG10]. User agents include HTML browsers and other types of software that retrieve and render Web content. A user agent that conforms to UAAG 1.0 will promote accessibility through its own user interface and through other internal facilities, including its ability to communicate with other technologies (especially assistive technologies).

Status of this document

This document has been produced as part of work on test suites being done by the User Agent Accessibility Guidelines (UAWG) Working Group. These are initial notes by the authors; this document does not represent Working Group consensus.

This document is part of a series of accessibility documents published by the Web Accessibility Initiative (WAI) of the World Wide Web Consortium (W3C). WAI Accessibility Guidelines are produced as part of the WAI Technical Activity. The goals of the User Agent Accessibility Guidelines Working Group are described in the charter.

How to evaluate a user agent for conformance

Section 3 of UAAG 1.0 defines conformance:

A user agent conforms to this document by satisfying the requirements identified by a conformance profile.

Here are the principal steps for evaluating a user agent for conformance:

1. Choose a version of UAAG 1.0.
2. Identify the user agent: the software (and other components) that fulfill the requirements of UAAG 1.0.
3. Chose a conformance profile.
4. Determine whether the subject satisfies each requirement in the profile.
5. Build a well-formed conformance claim.

This document focuses on step four.

The UAWG recommends performing evaluations with the Web-based evaluation tool available from the Working Group's home page..

1. Choose a version of UAAG 1.0

A well-formed conformance claim must include the title and (dated) URI of UAAG 1.0. For instance:

• Title: "User Agent Accessibility Guidelines 1.0"
• URI: [Dated URI to specification]

Note that the URI is the "dated" version, not the "latest version" URI (which is http://www.w3.org/TR/UAAG10/).

2. Identify the user agent

Section 1.2 of UAAG 1.0 states:

In general, a conforming user agent will consist of several coordinated components, such as a Web browser, a multimedia player, several plug-ins, features or applications provided by the operating environment, and documentation distributed with the software or available on the Web.

Together, these components must satisfy the requirements of a chosen conformance profile. For each component, identify the relevant version information; this will be part of the well-formed claim.

3. Choose a conformance profile

Chapter 3 of UAAG 1.0 explains that a user agent is not required to satisfy every requirement in this document in order to conform. Instead, during the course of an evaluation, you can choose a conformance profile, which identifies the subset of requirements the user agent actually satisfies. Chapter three explains how to choose a conformance profile and the information each profile must include. A conformance profile includes information about:

1. Chosen UAAG 1.0 version information;
2. Chosen conformance level;
3. Chosen content requirements;
4. Whether the user agent supports a selection mechanism;
5. Whether the user agent supports device-independent event handling;
6. Whether the user agent supports input devices other than the keyboard;
7. Which checkpoints (if any) do not apply;
8. Which formats have been implemented to satisfy the requirements of the document.

Chapter three also explains how to determine which checkpoints must be satisfied given a chosen conformance profile.

The UAAG 1.0 checklist organizes the checkpoints by priority, and may be used in this step. For information about calculating which checkpoints must be satisfied to conform, refer to "Notes on discretionary behavior in UAAG 1.0" [UAAG10-DISC].

4. Determine whether the subject satisfies the checkpoints

For each checkpoint in set that results from the previous step, rate how well the user agent satisfies the requirements. Since UAAG 1.0 is neither format-specific nor operating environment specific, it will be necessary to evaluate each checkpoint in the specific context of the chosen user agent.

Please note that some checkpoints include "Normative inclusions and exclusions" that qualify the checkpoint's provisions, and may have an impact on whether the user agent satisfies each provision of the checkpoint.

4.1 Find relevant detailed information

Where one finds the relevant information depends largely on the nature of the requirement:

1. Some requirements are "content requirements," which must be interpreted for each format that is used as part of the conformance claim. Note: UAAG 1.0 does not require that a conforming user agent implement a particular format, but does include requirements for formats that promote accessibility.
2. Some requirements are difficult to verify without some assistance from the software developer (e.g., when determining whether a user agent provides complete keyboard operability per checkpoint 1.1, the claimant is likely to require information from the developer).
3. Some requirements refer to adherence to external standards (whether W3C Recommendations or operating environment conventions). To determine satisfaction of these requirements, the claimant will have to rely on external resources, test suites, etc.

The following sections suggest where to find information for each type of checkpoint. See also some detailed information for UAAG 1.0 evaluations.

4.1.1 Content requirements

Relevant content requirements will depend largely on the formats identified in the conformance profile.

1. For each format, determine which parts of the format (e.g., elements and attributes, style sheet properties, etc.) are relevant to the requirements of the checkpoint (or portion of the checkpoint). The UAAG 1.0 test suite provides information about which parts of certain formats are relevant to the content checkpoints. At this step, you may consider the applicability provision (section 3.8 of UAAG 1.0) if you think a requirement is not relevant. A checkpoint applies unless:
   1. It makes requirements for graphical user interfaces or graphical viewports and the subject of the claim only has audio or tactile user interfaces or viewports.
   2. The checkpoint refers to a role of content that the subject of the claim cannot recognize because of how the content has been encoded in the format.
   3. The checkpoint requires control of a content property that the subject cannot recognize because of how the content has been encoded in the format.
2. For each relevant part of the format, determine whether the user agent satisfies all of the relevant requirements.

4.1.2 Requirements likely to involve developer input

Satisfaction of a number of requirements, notably those for user interface features, will be difficult to verify without input from developers. When this is the case, we encourage you to:

• Talk with developers;
• Look at product documentation;
• Look at claims made on the developer's Web site.

4.1.3 Requirements involving external resources

When a requirement refers to an external resource:

• Links to external resources may be found in the Techniques Document [UAAG10-TECHS]
• Information about test suites and environment conventions resources is available in the Techniques Document.

4.3 Rate the user agent

For some of the checkpoints, a binary rating ("yes" or "no") may be possible, but for others, it is likely that the claimant will need to (or want to) give a more detailed rating to the user agent for each requirement. The User Agent Accessibility Guidelines Working Group's implementation report [REPORT] offers one rating system, but this system is not required, and it may change as the Working Group continues to develop test suites, etc.

In some cases, the checkpoint will be precise enough so that techniques necessary to satisfy the checkpoint are clear . In other cases, the checkpoint will indicate techniques that are sufficient but not necessary to satisfy the checkpoint.

Some suggestions when rating a user agent:

• Provide a rating for each requirement of each checkpoint. If the rating is the same for each provision, then it makes sense to give the checkpoint an overall rating. If some provisions are satisifed well but others are not, it is more telling to provide a rating at the provision level rather than for the checkpoint as a whole.
• In some cases, a user agent may implement a feature and expose it through a configuration file but not through the user interface. This certainly counts as implementation, but would seem to merit a lower grade than if available through the user interface. If the feature is undocumented (but available), the grade should be even lower.
• Document assumptions you have made in rating the user agent (e.g., a feature would work except for a bug that is likely to be fixed).
• If you have doubts about whether a feature satisfies a requirement:
  1. Review other evaluations.
  2. Ask the User Agent Accessibility Guidelines Working Group's opinion.

An evaluation tool is available to help you perform an evaluation.

5. Build a well-formed conformance claim

Chapter three discusses requirements for "well-formed claims." The UAWG is currently working on formalisms for making claims and for representing evaluations. Refer to the UAWG's implementation report [REPORT] for more information.

References

[UAAG10]

User Agent Accessibility Guidelines 1.0, I. Jacobs, J. Gunderson, E. Hansen, eds. The latest draft of the guidelines is available at http://www.w3.org/TR/UAAG10/.

[UAAG10-TECHS]

Techniques for UAAG 1.0, I. Jacobs, J. Gunderson, E. Hansen, eds. The latest draft of the guidelines is available at http://www.w3.org/TR/UAAG10-TECHS/.

[UAAG10-DISC]

Notes on discretionary behavior in UAAG 1.0, I. Jacobs. The latest version of this draft document is available at http://www.w3.org/WAI/UA/2001/10/ts.

[REPORT]

Summary of UAAG 1.0 evaluations