Canvas hit testing

Summary

A new hit testing method should be added to canvas elements to simplify hit testing and solve accessibility issues inherent to the fallback DOM concept.

Canvas authors are starting to create more complex user interface in canvas that can only could only map to several underlying elements (A spline editor, for example, would map to several range controls). To enable a better canvas+fallback DOM coding pattern, it seems prudent to enable easier canvas hit testing and event handling.

In the current HTML5 specification, authors are advised to create a fallback DOM under the canvas element to enable screen readers to interact with canvas user interfaces. The size and position of those elements are not defined, which causes problems for accessibility tools - what size/position should they report for these elements?

Because canvas elements usually respond to user input, it seems prudent to solve the hit testing and accessibility issues with the same mechanism.

Rationale

Authors will create two kind of canvas user interfaces:

[1] Simple canvas user interfaces that map well to a single regular input elements A canvas checkbox (http://www.htmlstack.com/checkbox/ ) is the quintessential example here. A canvas bitmap is represented by a single checkbox in the DOM.

[2] Complex canvas user interfaces that maps to several regular input elements Authors will create canvas UIs that contain multiple complex visual input controls that should map to several regular elements in the fallback dom.)

(Examples: See the customize-timing-function canvas UI in http://ie.microsoft.com/testdrive/Graphics/hands-on-css3/hands-on_transitions.htm.

These user interfaces don't match well to regular input controls - which might be why the author chose to use canvas.

The best approach to solving the accessibility issues in these complex canvas user interfaces would be to map the individual user interface elements in the canvas bitmap to multiple tabbable/focusable/etc input elements in the fallback DOM. We should enable authors to easily define hit testing regions on the canvas and map events to the underlying elements in the fallback dom. This will reduce author burden, and also solve the issue of reporting meaningful position+size information to accessibility tools.

Details

[Add two new methods to interface CanvasRenderingContext2D]

http://dev.w3.org/html5/2dcontext/#conformance-requirements

boolean setElementPath(in Element element); //Associates the current path with an element in the canvas DOM boolean clearElementPath(in Element element); //Disassociates any path associated with an element in the canvas DOM

[Make the following normative text]

When a canvas is interactive, authors should include focusable elements in the element's fallback content corresponding to each focusable part of the canvas.

[Add a definition of the methods]

http://dev.w3.org/html5/2dcontext/#complex-shapes-paths

boolean setElementPath(in Element element); Associates the current path with an element in the DOM

The setElementPath() method associates the current path with an element in the canvas.

If a path for an element has not been associated via setElementPath(), the size/height/width/region of a focusable element in the fallback content is the bounding rectangle of the parent canvas element.

When setElementPath() is called, the drawing path is used to form the element's defined region provided that drawing path contains a closed path. The drawing path is used to form a best fit bounding rectangle in screen coordinates. The bounding rectangle and drawing path may be used to enhance accessibility properties [ARIA] for the targeted element.

User agents should use the information set by setElementPath() to create accessible user experiences. For example, a screen reader may read the fallback element's details when the user indicates interest in that region of the canvas.

The setElementPath(element) method, when invoked, must run the following steps:

1. If the element is not a descendant of the canvas element with whose context the method is associated, then return false and abort these steps.

2. If supporting an accessibility API, user agents may use the drawing path to form a best fit rectangle in screen coordinates and apply it to the bounding rectangle of the associated accessible object. The path should be subject to the clipping region.

3. Return true.

The clearElementPath() method disassociates any path associated with an element in the canvas DOM set using setElementPath().

When clearElementPath() is called, the path associated using setElementPath() is disassociated with the element in the canvas DOM.

The clearElementPath(element) method, when invoked, must run the following steps:

1. If the element is not a descendant of the canvas element with whose context the method is associated, then return false and abort these steps.

2. User agents must remove the path associated with the element using setElementPath.

3. Return true.

[Define a new canvas hit testing method]

When the user interacts with the canvas:

1. If the subdom of the canvas contains one or more elements with an associated path, the user agent should determine if the user's interaction falls within the confines of an element's path, using the same mechanism as the isPointInPath() function. If two or more elements have overlapping paths (set via setElementPath()) the last call to setElementPath() applies. (If the subdom element associated with a path is no longer in the subdom of the canvas, that particular path should not apply in this determination.)

2. If the user interacts with a canvas bitmap region that is inside a element path (has an associated subdom element), the user agent must fire a simple event (that corresponds with the user's interaction) at the subdom element that bubbles and is cancelable. Events should only be fired at the canvas element when the interaction does not intersect with the bitmap regions (associated paths) of any canvas subdom elements.

Positive Effects

Authors will have a standard method to implement hit-testing in their applications.

Authors will have a standard method to define regions of interest on a canvas bitmap, improving accessibility for users.

Allows a screen magnifier user to locate and zoom to any element rendered on canvas

Allows a screen reader user with a Braille better fill a refreshable Braille display by using the coordinates supplied to place rendered objects on the same line facilitating a determination of what goes on line of Braille text.

Allows assistive reading tools to speak the appropriate content when the user points to items on canvas.

Negative Effects

None identified.

Conformance Classes Changes

None identified

Risks

None identified