Independent User Interface for Rich Internet Applications

W3C Editor's Draft

Editor:
James Craig (jcraig@apple.com), Apple Inc.

Todo: Include link to orginal proposal? Doug also recommends incorporating parts of the Intent-based Events proposal, too.

1. Introduction

This specification is intended address the problem of operating system-, device-, and localization-independent control of widgets in accessible rich internet applications.

1.1. Problem Description

At the time of this writing, WAI-ARIA states that:

Accessibility of web content requires semantic information about widgets, structures, and behaviors, in order to allow assistive technologies to convey appropriate information to persons with disabilities. [The WAI-ARIA] specification provides an ontology of roles, states, and properties that define accessible user interface elements and can be used to improve the accessibility and interoperability of web content and applications. These semantics are designed to allow an author to properly convey user interface behaviors and structural information to assistive technologies in document-level markup.

WAI-ARIA 1.0 specifies a standardized means for web applications to convey accessibility semantics through declarative markup in the DOM. This document, the Independent User Interface specification, standardizes a method for assistive technology to assume control of customized user interface widgets, or convey the user's intent for control of the web application. Simple control over web applications can be achieved by using standard activation events, like click or DOMActivate, but full control over secondary actions (tree expansion, grid sorting, scrolling) requires additional events and interfaces, that allows for more robust (link @@ WCAG Robust) control over web applications.

Benefits include low barrier of entry for end users, and ease of development for web application authors. The Independent User Interface specification allows users of assistive technology to control web applications using the same commands they are accustomed to with native regardless of rendering engine, platform, operating system, or localization.

The specification is organized into three parts: UI Change Request Events for all user agents, Accessibility Events for assistive technology, and additions to the Navigator interface for Assistive Technology Identification and Notification.

2. UI Change Request Events

The core principle behind UI Change Request Events is that they operate on a completely backwards-compatible, opt-in basis. In other words, the web application author has to be aware of these events and register event listeners, or the user agent and assistive technology behave as they normally would.

Unlike mutation events that allow user agents or assistive technology to make direct changes to the DOM, these "change request" events do not cause any direct manipulation or mutation of the DOM. Instead, the event object conveys the user's intent to the web application, and allows the web application to make the appropriate changes to the DOM, on behalf of the user agent or assistive technology. If a web application is authored to understand the change request event, it can cancel the event using preventDefault(), which informs the user agent or assistive technology that the event has been captured and understood. If a web application does not cancel any change request event, the user agent or assistive technology can attempt fallback behavior or communicate to the user that the input has not been recognized.

Note: The UIRequestEvent interface does not inherit from AccessibilityEvent, because we believe it will ultimately be useful outside the context of assistive technology. We believe this to be a lightweight, performant alternative to certain mutation events. For example, DOMAttrChangeRequest instead of DOMAttrModified.

Event fires on document.activeElement if applicable, or otherwise document.body.

Mention that these events should be asynchronous (cannot be synchronous for multi-process browsers).

Interface UIRequestEvent

interface UIRequestEvent : UIEvent {
  
  // UA or AT notifies web app of a change request
  const unsigned short    UNDO    =  1;
  const unsigned short    REDO    =  2;
  const unsigned short    ESCAPE  =  3;
  const unsigned short    DELETE  =  4;
  // expect more type constants will be added for other event types…
  // ZOOM_IN / ZOOM_OUT ? perhaps that should just be slider widget, covered under ValueChangeRequest
  
  readonly attribute unsigned short    eventType;
  
  void initUIRequestEvent(
         in DOMString typeArg,
         in boolean canBubbleArg,
         in boolean cancelableArg,
         in views::AbstractView viewArg,
         in long detailArg,
         in unsigned short eventTypeArg
       );
};

The different types of UIRequestEvents that can occur are:

UndoRequest

Initiated when the user agent or assistive technology sends an 'undo' request to the web application.

  • Bubbles: Yes
  • Cancelable: Yes
  • Context Info: eventType
RedoRequest

Initiated when the user agent or assistive technology sends a 'redo' request to the web application.

  • Bubbles: Yes
  • Cancelable: Yes
  • Context Info: eventType
EscapeRequest

Initiated when the user agent or assistive technology sends a 'escape' request to the web application.

  • Bubbles: Yes
  • Cancelable: Yes
  • Context Info: eventType
DeleteRequest

Initiated when the user agent or assistive technology sends a 'delete' request to the web application.

  • Bubbles: Yes
  • Cancelable: Yes
  • Context Info: eventType

Examples

  1. Users, wanting to 'undo' a discrete action in a web application, can indicate their intent a number of ways, including pressing Control+Z on Windows or Linux, Command+Z on Mac OS X, and even by shaking some accelerometer- or gyroscope-enabled mobile devices. User agents understanding this intent should initiate an UndoRequest event. Web authors who have registered for this event should process the event to determine whether to cancel the event. If the 'undo' action is understood in the context of the web application, web authors should undo the user's change, and cancel the event using the event object's preventDefault() method.
  2. Users, wanting to 'escape' a web application state (for example, closing a modal dialog), can indicate their intent a number of ways, including pressing Escape on most keyboard-controlled operating systems, or by using a two-finger scrub gesture in VoiceOver on iOS or Mac OS X. User agents understanding this intent should initiate an EscapeRequest event. Web authors who have registered for this event should process the event to determine whether to cancel the event. If the 'escape' action is understood in the context of the web application, web authors should perform the appropriate action (such as closing the dialog), and cancel the event using the event object's preventDefault() method. If the event is not cancelled by the web author, user agents may pass the literal interaction event to the web application; for example, as a keypress event.

To-do: Explanation needed about event order precedence.

An area where this draft is notably lacking is in regards to the precedence of event order in mainstream user agents, especially in regards to continuous input events. For example, user interactions with starting and ending events (touchstart/touchend, shakestart/shakeend, etc.) should probably take precendence over UIRequestEvents in mainstream user agents. UIRequestEvents are only intended to be used in mainstream user agents where appropriate, but the definition of "where appropriate" is lacking from the current version of this draft. Instead, we offer the previous examples for the sake of discussion.

  1. Note that the 'shake' interaction mentioned in the first example could be sent to the web app as a shakestart event which, if not captured, could then be sent as an UndoRequest. If the the web app does not capture either event, the shake interaction would be ignored by the web application.
  2. Also note that the 'scrub gesture' interaction mentioned in the second example is specific to assistive technology (VoiceOver on iOS, which intercepts all touch events), so it's appropriate for the UIRequestEvent to take precedence, and never send a touchstart event.

Interface UIScrollRequestEvent

interface UIScrollRequestEvent : UIRequestEvent {
  
  // for custom scroll views or widgets (e.g. carousels, lists, grids)
  const unsigned short    LEFT          =  1;
  const unsigned short    UP            =  2;
  const unsigned short    RIGHT         =  3;
  const unsigned short    DOWN          =  4;
  const unsigned short    LEFT_LIMIT    =  5;
  const unsigned short    TOP_LIMIT     =  6;
  const unsigned short    RIGHT_LIMIT   =  7;
  const unsigned short    BOTTOM_LIMIT  =  8;
  
  readonly attribute unsigned short   scrollType;
  
  void initUIScrollRequestEvent(
         in DOMString typeArg,
         in boolean canBubbleArg,
         in boolean cancelableArg,
         in views::AbstractView viewArg,
         in long detailArg,
         in unsigned short eventTypeArg
         in unsigned short scrollTypeArg
       );
};

The single type of UIScrollRequestEvent that can occur is:

ScrollRequest

Initiated when the user agent or assistive technology sends a scroll request to the web application. Scroll events need only be used on custom scroll views (lists and grids showing data subsets, carousels, etc.), as user agents and assistive technologies already manage scrolling of native scroll views.

Note: The scroll type constants are more or less equivalent to user-expected behavior for PageUp/PageDown and Home/End keys on native scroll views, but also allow horizontal scrolling.

  • Bubbles: Yes
  • Cancelable: Yes
  • Context Info: scrollType

Interface UIValueChangeRequestEvent

interface UIValueChangeRequestEvent : UIRequestEvent {
  
  // value changes (e.g. ranges)
  const unsigned short    INCREMENT        =  1;
  const unsigned short    INCREMENT_SMALL  =  2;
  const unsigned short    INCREMENT_LARGE  =  3;
  const unsigned short    INCREMENT_MAX    =  4;
  const unsigned short    DECREMENT        =  5;
  const unsigned short    DECREMENT_SMALL  =  6;
  const unsigned short    DECREMENT_LARGE  =  7;
  const unsigned short    DECREMENT_MIN    =  8;
  
  readonly attribute unsigned short   changeType;
  
  void initUIValueChangeRequestEvent(
         in DOMString typeArg,
         in boolean canBubbleArg,
         in boolean cancelableArg,
         in views::AbstractView viewArg,
         in long detailArg,
         in unsigned short eventTypeArg,
         in unsigned short changeTypeArg
       );
};

The single type of UIValueChangeRequestEvent that can occur is:

ValueChangeRequest

Initiated when the user agent or assistive technology sends a value change request to the web application.

Web authors should code applications to accept all values of the changeType argument. For example, if there is no special behavior for INCREMENT_SMALL or INCREMENT_LARGE, web applications would behave as if they had received a basic INCREMENT change type.

  • Bubbles: Yes
  • Cancelable: Yes
  • Context Info: eventType

Example

Users, wanting to change the value of a custom range widget (slider, media progressbar, etc.) in a web application, can indicate their intent a number of ways, including pressing various keys (Up, Down, Left, Right, PageUp, PageDown, Home, End) on most keyboard-controlled interfaces, and through gestures on many touch-enabled interfaces. User agents understanding this intent should initiate a ValueChangeRequest event. Web authors who have registered for this event, should process the event to determine whether to cancel the event. If the value change action is understood in the context of the web application, web authors should change the value of the associated widget by an amount determined via the changeType argument, and cancel the event using the event object's preventDefault() method. If the event is not cancelled by the web author, user agents may pass the literal interaction event to the web application; in this case, in the form of a keypress or touch event.

Interface DOMAttributeChangeRequestEvent

interface DOMAttributeChangeRequestEvent : UIRequestEvent {
  
  readonly attribute DOMString    attrName;
  readonly attribute DOMString    newValue;
  
  void initDOMAttributeChangeRequestEvent(
         in DOMString typeArg,
         in boolean canBubbleArg,
         in boolean cancelableArg,
         in views::AbstractView viewArg,
         in long detailArg,
         in unsigned short eventTypeArg,
         in DOMString attrNameArg,
         in DOMString newValueArg
       );
};

The single type of DOMAttributeChangeRequestEvent that can occur is:

DOMAttrChangeRequest

Initiated when the user agent or assistive technology sends an attribute change request to the web application. In order for web applications to understand the intent of change request events from the user agent or assistive technology, these change requests should be limited to attributes for which a change in value indicates a discrete, defined action for known types of widgets and UI elements.

Note: Currently, this limitation indicates that DOMAttrChangeRequest only applies to WAI-ARIA widgets, but it has the potential to be used with future iterations of HTML5 or SVG.

  • Bubbles: Yes
  • Cancelable: Yes
  • Context Info: attrName, newValue

Examples

  1. If a user wanted to expand or collapse the current node of an ARIA tree, the assistive technology would initiate a DOMAttrChangeRequest event on the tree item element, with the attrName equal to aria-expanded, and the newValue equal to true or false.
  2. If a user wanted to sort an ARIA grid on a particular column, the assistive technology would initiate a DOMAttrChangeRequest event on the column header element, with the attrName equal to aria-sort, and the newValue equal to ascending or descending.

3. Accessibility Events

The following events are only initiated by assistive technologies, as opposed to mainstream user agents.

Interface AccessibilityEvent

interface AccessibilityEvent : UIEvent {
  void initAccessibilityEvent(
         in DOMString typeArg,
         in boolean canBubbleArg,
         in boolean cancelableArg,
         in views::AbstractView viewArg,
         in long detailArg
       );
};

The different types of AccessibilityEvents that can occur are:

AXFocusIn

Initiated when the assistive technology cursor moves to a particular DOM node.

  • Bubbles: Yes
  • Cancelable: No
  • Context Info: None
AXFocusOut

Initiated when the assistive technology cursor leaves a particular DOM node.

  • Bubbles: Yes
  • Cancelable: No
  • Context Info: None
AXDragSelect

Initiated by the assistive technology in order to inform the web application than a draggable element (e.g. an element with an explicit aria-grabbed value) should be marked for dragging.

  • Bubbles: Yes
  • Cancelable: Yes
  • Context Info: None
AXDragCancel

Editorial Note: This event may not be necessary; could use general EscapeRequest instead.

Initiated by the assistive technology in order to inform the web application than the current drag operation should be cancelled.

  • Bubbles: Yes
  • Cancelable: No … Editorial Note: Does this need to be cancelable?
  • Context Info: None
AXDragDrop

Initiated by the assistive technology in order to inform the web application than a specific drop target (e.g. an element with an explicit aria-dropeffect value) should receive the drop event of the current drag operation.

  • Bubbles: Yes
  • Cancelable: Yes
  • Context Info: None

4. Assistive Technology Identification and Notification

In certain cases, it is beneficial for web authors to be aware of the assistive technology in use. This section defines a collection of attributes that can be used to determine, from script, the kind of assistive technology in use, in order to be aware of considerations for a particular assistive technology, including the need to support the UI Change Request Events or Accessibility Events defined in previous sections. Web authors should always limit client detection to detecting known versions. Web authors should always assume future versions and unknown versions to be fully compliant.

The accessibility attribute of the Navigator interface must return an instance of the Accessibility interface, which represents the identity and state of the assistive technology, and allows web pages to optionally notify assistive technology of certain interface events that cannot be indicated through declarative markup such as WAI-ARIA roles, states, and properties.

Interface Accessibility

interface Accessibility {
  readonly attribute InterfaceObject accessibility; // Editorial Note: Is it necessary to define this attribute in the IDL?
  // objects implementing the Accessibility interface also implement the ScreenReader and AccessibilityNotifications interfaces
};
Accessibility implements AccessibilityNotifications;
Accessibility implements ScreenReader;
Accessibility implements Magnifier;
Accessibility implements Speech;

[Supplemental, NoInterfaceObject]
interface AccessibilityNotifications {
  void elementsChanged();
  void screenChanged();
};

Methods for Interface Accessibility

window.navigator.accessibility.elementsChanged()
Returns void. Allows the web author to send a supplemental notification to the screen reader that certain aspects of relevant DOM elements have changed, informing the assistive technology to update its cache of the accessibility tree. For example, web authors might call this method after an animation has completed, allowing a screen reader to update its cursor location, if the focused element was moved.
window.navigator.accessibility.screenChanged()
Returns void. Allows the web author to send an explicit notification to the assistive technology that all aspects of a web application view have changed, informing the assistive technology to update its cache of the accessibility tree. Web authors should call this method when a majority of the web view changes without triggering a full page refresh in the browser.

Interface ScreenReader

The screenreader attribute of the Accessibility interface must return an instance of the ScreenReader interface:

interface ScreenReader {
  readonly attribute InterfaceObject screenreader; // Editorial Note: Is it necessary to define this attribute in the IDL?
  // objects implementing the ScreenReader interface also implement the interfaces given below
};
ScreenReader implements ScreenReaderID;
ScreenReader implements ScreenReaderStatus;

[Supplemental, NoInterfaceObject]
interface ScreenReaderID {
  readonly attribute DOMString name;
  readonly attribute DOMString version;
};

[Supplemental, NoInterfaceObject]
interface ScreenReaderStatus {
  readonly attribute boolean active;
};

Properties for Interface ScreenReader

window.navigator.accessibility.screenreader.active
Returns a boolean indicating whether or not a screen reader is in use. The user agent must return true if a screen reader is running, or false otherwise. User agents may return false if the user has chosen to disallow sharing this information with the requesting domain.
window.navigator.accessibility.screenreader.activeElement
@@
window.navigator.accessibility.screenreader.name
Returns the application name of the screen reader in use. The user agent must return either an empty string or a string representing the name of the screen reader in detail, e.g. "Apple VoiceOver". User agents may return an empty string if the user has chosen to disallow sharing this information with the requesting domain.
window.navigator.accessibility.screenreader.version
Returns the application version of the screen reader in use. The user agent must return either an empty string or a string representing the version of the screen reader in detail, e.g. "4.0 (220.2)". User agents may return an empty string if the user has chosen to disallow sharing this information with the requesting domain.

Note: The authors recommend that user agents adopt a domain-level security policy for the ScreenReader interface that is similar to the security policy for location data or cookies. A user should be able to explicitly disallow sharing of this information altogether, or on a per-domain basis.

Methods for Interface ScreenReader

window.navigator.accessibility.screenreader.announce(string?, priority?)
We need a way for the web app to tell the screen reader to speak a string (like UIAccessibilityAnnouncementNotification does with VoiceOver on iOS), similar to live region changes, but an explicit method for announcements.

Interface Magnifier

The magnifier attribute of the Accessibility interface must return an instance of the Magnifier interface:

interface Magnifier {
  readonly attribute InterfaceObject magnifier; // Editorial Note: Is it necessary to define this attribute in the IDL?
  // objects implementing the Magnifier interface also implement the interfaces given below
};
Magnifier implements MagnifierID;
Magnifier implements MagnifierStatus;
Magnifier implements MagnifierNotifications;

[Supplemental, NoInterfaceObject]
interface MagnifierID {
  readonly attribute DOMString name;
  readonly attribute DOMString version;
};

[Supplemental, NoInterfaceObject]
interface MagnifierStatus {
  readonly attribute boolean active;
};

[Supplemental, NoInterfaceObject]
interface MagnifierNotifications {
  void focusPosition(
         in DOMElement element,
         in optional array cursorRect,
         in optional array selectionPolygon
       );
};

Properties for Interface Magnifier

window.navigator.accessibility.magnifier.active
Returns a boolean indicating whether or not a screen magnifier is in use. The user agent must return true if a screen magnifier is enabled (even if the current zoom level is 1.0), or false if a magnifier is not enabled. User agents may return false if the user has chosen to disallow sharing this information with the requesting domain.
window.navigator.accessibility.magnifier.name
Returns the application name of the screen magnifier in use. The user agent must return either an empty string or a string representing the name of the screen magnifier in detail, e.g. "Apple Universal Access Zoom". User agents may return an empty string if the user has chosen to disallow sharing this information with the requesting domain.
window.navigator.accessibility.magnifier.version
Returns the application version of the screen magnifier in use. The user agent must return either an empty string or a string representing the version of the screen magnifier in detail, e.g. "4.0 (220.2)". User agents may return an empty string if the user has chosen to disallow sharing this information with the requesting domain.
window.navigator.accessibility.magnifier.zoom
Returns a floating point number indicating the current magnification level of the screen magnifier. The user agent must return 1.0 or the current magnification level. User agents may return 1.0 if the user has chosen to disallow sharing this information with the requesting domain.

Methods for Interface Magnifier

window.navigator.accessibility.magnifier.focusPosition ( element, cursorRect, selectionPolygon )

Returns void. Allows the web author to send an explicit notification informing the screen magnifier to update its cache of the cursor location. Web authors should call this method when updating the display or cursor position of custom views.

Parameters for Magnifier.focusPosition
1. DOMElement element
Required reference to the element (e.g. canvas) that has focus.
2. optional array cursorRect
Optional array representing the rectangular cursor location relative to the [0,0] position of focused element. [ x1, y1, x2, y2 ]
3. optional array selectionPolygon
Optional array representing the selection polygon relative to the [0,0] position of focused element. [ x1, y1, x2, y2, ... xn, yn ]

Editorial Note: It may be better if the optional params are shape objects with a coords array and a type string like 'rect' or 'poly'. That would allow the author to decide if they wanted to provide the extra polygon coords, or just use a simple shape.

Note: The focusPosition() method would be necessary for screen magnifiers to work with canvas-based web apps like Bespin or the 280 North projects. This method is not necessary for text views managed by the user agent, including contentEditable views.

Interface Speech

The speech attribute of the Accessibility interface must return an instance of the Speech interface:

interface Speech {
  // TBD, placeholder for speech-controlled user agents or assistive technology
  // may need speech.synthesis and speech.recognition; each could have name/version/engine, etc. 
  // potential method: speech.synthesis.speak() to trigger TTS from web app
  // potential method: speech.recognition.updateDictionary() to capture correct pronunciation of uncommon or app-specific words
};