Bug 13176 - The bounds of canvas fallback content, as rendered on the canvas, are not provided by the user agent to an assistive technology.
The bounds of canvas fallback content, as rendered on the canvas, are not pro...
Status: CLOSED FIXED
Product: HTML WG
Classification: Unclassified
Component: LC1 HTML Canvas 2D Context
unspecified
All All
: P1 enhancement
: ---
Assigned To: Jay Munro
HTML WG Bugzilla archive list
canvas RFE
: a11y, a11ytf, a11y_canvas, WGDecision
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2011-07-07 18:07 UTC by Rich Schwerdtfeger
Modified: 2012-10-23 12:45 UTC (History)
15 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Rich Schwerdtfeger 2011-07-07 18:07:07 UTC
In HTML, SVG, desktops, and mobile devices an author has the ability to bind the bounds of object drawn on the screen to the object that provides the accessibility information. The bounds of the object are defined by the path or in less complicated systems - a rectangle. No such feature is provide for natively in canvas.

In canvas, the subtree is used to represent a 1:1 mapping of objects rendered on the physical canvas for the purpose of providing keyboard navigable objects that can expose accessibility semantics to the browser to support platform accessibility APIs. What is missing is the ability to defined the bounds of the associated object drawn on the canvas. 

There are some questionable work arounds where the subtree elements can be rendered transparently, using CSS, with a higher Z order above the canvas element. Yet, this would require the author to manage: 
- bounds transformations based on the canvas transformation matrix
- adjustment of the bounds based on scrolling or moving of the browser on the screen

This is a very expensive management barrier to accessibility when the user agent could simply provide the ability to bind retained paths for bounds of the object on canvas to the subtree element and use it to also perform hit testing. This would allow the user agent to route pointer events directed at the canvas object to the same element in the canvas subtree that processes the keyboard events. It also allows user agents to supply the bounds of the object being drawing to the corresponding accessible object generated from the canvas subtree element like all GUIs today. 

Without the bounds of an object:

- mobile screen readers like VoiceOver cannot detect the existence of accessible objects when you move your finger over them on canvas
- screen magnifiers cannot find the location of the object to zoom to it
- screen readers cannot orient objects on the same line to drive Braille devices

This is a very serious problem that should not be left up to developer hacking. It needs to supported by the user agent transparently to the author which is why employing retained paths, defining the bounds of objects, to the accessible objects in the canvas subtree to support hit testing would be the easiest vehicle for developers to provide the bounds of an object to an assistive technlogy.
Comment 1 Rich Schwerdtfeger 2011-07-07 22:43:56 UTC
See also bug: http://www.w3.org/Bugs/Public/show_bug.cgi?id=13181
Comment 2 Charles Pritchard 2011-07-11 09:22:26 UTC
Related:

Argument for the approval of adding setElementPath to the Canvas 2d specification.
http://lists.w3.org/Archives/Public/public-html-comments/2011Jul/0004.html
Comment 3 Aryeh Gregor 2011-07-12 22:24:26 UTC
I don't understand the subject matter well enough to resolve this, and it seems like none of the other editorial assistants do either (or they're not interested).  I suspect the eventual resolution will be WONTFIX, so I'll raise the priority to P1 to get the process moving faster, per the PriorityRequest keyword.
Comment 4 Charles Pritchard 2011-07-12 22:41:46 UTC
(In reply to comment #3)
> I don't understand the subject matter well enough to resolve this, and it seems
> like none of the other editorial assistants do either (or they're not

Technical summary
http://lists.w3.org/Archives/Public/public-canvas-api/2011JulSep/0187.html

Authors require the ability to pass path information to the UA accessibility API in order to identify the screen location of elements in the shadow dom to assistive technologies.

The most basic information expressed is a bounding box.

An AT may navigate the DOM and select an element of interest (such as selecting a form element), for example: <canvas><button /></canvas> - it may examine the bounding box of the dom element and do something with that information such as drawing a big high contrast ring around the area or scrolling to the area.

When canvas is active, the shadow dom is not visible to the end user, but it can be navigated. Were the canvas author able to run a method, similar to drawFocusRing(element), they could inform the UA accessibility API of the bounds of the element. I've proposed setElementPath as the method name.
Comment 5 Rich Schwerdtfeger 2011-07-27 18:32:58 UTC
To Areyh, I understand you are not a subject matter expert. Consequently, I put its importance back to major. 

That said, Jonas Sicking has a proposal is close to what I had been asking for on the list. I will be working with Jonas to incorporate what he is changing into the same change proposal:

http://lists.w3.org/Archives/Public/public-canvas-api/2011JulSep/0195.html

The net of this is that the path association with the element is sufficient to allow the user agent to use the path to compute the bounds of an object and set it in the platform accessibility API implementation. Furthermore, a future API modification for web apps might be to scroll an element into view based on the bounds of the object stored in the browser. This capability will be of particular assistance to Google which is targeting AT browser extensions.
Comment 6 Laura Carlson 2011-07-28 13:17:48 UTC
Related WHATWG IRC discussion:
http://krijnhoetmer.nl/irc-logs/whatwg/20110728#l-53
Comment 7 Charles Pritchard 2011-07-28 20:04:33 UTC
Per WHATWG Bugzilla conventions, moving this to "blocker" as "major" is unused.
http://wiki.whatwg.org/wiki/Bugzilla_conventions#Severities

This bug affects Mobile Safari on iOS, making it almost unusable. It also affects vendors targeting the windows desktop, supporting IAccessible / MSAA, as their accessibility trees are lacking sufficient information to respond to accLocation requests. As such, I believe it falls under P1 "blocker" per WHATWG conventions.
Comment 8 Michael[tm] Smith 2011-08-04 05:03:20 UTC
mass-move component to LC1
Comment 9 Ian 'Hixie' Hickson 2011-08-11 04:59:40 UTC
Wouldn't SVG be the right solution to this? Surely using <canvas> when you are expecting a user to interact with the app via a tactile interface rather than through sight is just misusing canvas.
Comment 10 Charles Pritchard 2011-08-11 08:09:05 UTC
(In reply to comment #9)
> Wouldn't SVG be the right solution to this? Surely using <canvas> when you are
> expecting a user to interact with the app via a tactile interface rather than
> through sight is just misusing canvas.

Boundaries are used for both sight-oriented and eyes-free UIs. This includes ZoomText and VoiceOver. No, SVG is not an adequate solution for this.
Comment 11 steve faulkner 2011-08-11 08:29:00 UTC
(In reply to comment #9)
> Wouldn't SVG be the right solution to this? Surely using <canvas> when you are
> expecting a user to interact with the app via a tactile interface rather than
> through sight is just misusing canvas.

So are you saying that canvas interactvity should not be allowed on touch based interfaces?
If so can you get the browser vendors to remove the ability to make canvas interactive?

If developers can not use canvas to provide interactive content they will be forced to use SVG, if you truly believe that this bugs premises are based on misuse of canvas then convince vendors to remove the ability for developers to misuse it.
Otherwise users with disabilities will be the losers in your moral crusade to dictate to developers how canvas should be used. 'Just Say No' didn't work to stem drug use or promiscuity abd it won't work for canvas.
Comment 12 Rich Schwerdtfeger 2011-08-11 12:55:15 UTC
SVG is an alternate solution should someone wish to use vector graphics. Not everyone chooses to do that. SVG accessibility infrastructure is a long way off. What is important is that both SVG and canvas support platform accessibility infrastructure. I have been working both efforts in parallel. That said, due to the ubiquity of HTML it is first in the priority list for accessibility. Canvas is part of HTML.

There has already been extensive discussion on the list for this. I don't see any point in doing it again here. 

I am going to work with Jonas to provide a change proposal based on one of his posts.
Comment 13 Charles Pritchard 2011-08-11 22:46:37 UTC
(In reply to comment #9)
> Wouldn't SVG be the right solution to this? Surely using <canvas> when you are
> expecting a user to interact with the app via a tactile interface rather than
> through sight is just misusing canvas.

One more comment on this, because it occurred to me, that perhaps the WWHATWG editor does not understand accessibility testing. There is no expectation that the user will  "interact" via any particular interface, tactile or otherwise. The purpose is to expose information to the operating system / UA, that's all. From there, we certainly test usability, but that's a separate issue for app developers, not browser vendors.

Further, eyes-free use does not necessarily mean that the user, or other observers are not looking or able to look at the screen.

Circling back: drawFocusRing is not sufficient for driving screen magnification nor Apple's VoiceOver. It only works for an element that is in focus, and some magnification and touch-based interfaces do not send focus-events until after the user has initiated one. drawFocusRing works for many use cases, and should be used via onfocus events. It's not adequate for the remaining cases which rely on IAccessible accLocation and do not trigger focus. By adding a new method to the canvas api, as we have proposed a setClickableRegion(element), similar to drawFocusRing, the information can be exposed to the OS / UA, and used.

I hope that helps. The checkbox example in the WHATWG spec would benefit from setClickableRegion. Currently, it will-not work on VoiceOver in Mobile Safari. It would require only one-line, to fix that, if setClickableRegion were supported.
Comment 14 Sam Ruby 2011-08-14 23:51:52 UTC
This bug was marked as P1 over 30 days ago, and still hasn't been RESOLVED.

Editors: please RESOLVE it ASAP.  NEEDSINFO and WONTFIX are valid resolutions
for this part of the process.  We simply want to get this bug to a state where
we are prepared to accept change proposals should anybody be inclined to
produce such.
Comment 15 Ian 'Hixie' Hickson 2011-09-21 22:42:35 UTC
For touch and keyboard navigation, it's up to the author to implement the appropriate feedback mechanisms; it doesn't need extra help from the system accessibility API. For example, CSS3 UI's properties can be used to specify the elements in various navigation directions.

Zooming is handled already by the focus ring APIs.

In general, however, using <canvas> for something that can be sanely interacted with in non-visual mediums is completely counter to the point of <canvas>. If your content is media-independent, then use HTML, that's what it's for.

EDITOR'S RESPONSE: This is an Editor's Response to your comment. If you are satisfied with this response, please change the state of this bug to CLOSED. If you have additional information and would like the editor to reconsider, please reopen this bug. If you would like to escalate the issue to the full HTML Working Group, please add the TrackerRequest keyword to this bug, and suggest title and text for the tracker issue; or you may create a tracker issue yourself, if you are able to do so. For more details, see this document:
   http://dev.w3.org/html5/decision-policy/decision-policy.html

Status: Rejected
Change Description: no spec change
Rationale: Already sufficiently supported.
Comment 16 Rich Schwerdtfeger 2011-09-22 11:38:43 UTC
Ian, screen magnifiers have the requirement to zoom to any object on the canvas without changing focus. This requirement came form AI Squared - the producer of the leading screen magnifier on Windows. To do this they require the bound of an object as they have come to expect from platform accessibility API since 1995. Moving focus does not resolve this issue. 

Touch creates an additional problem in that moving your fingers across the UI, and zooming, would require an association with the bounds of the object on canvas which is also not supplied.  In the case of VoiceOver for screen reading a use is able to move their fingers across HTML content and associate the object with the rendered position so that VoiceOver may speak the object under which the user is pointing. When this occurs, the focus is NOT moved. In fact the user wants to be able to browse without moving focus. In <canvas> no such positioning information exists. Canvas is indeed used to render objects. They don't even need to be interactive, yet the user still needs to know their position.
Comment 17 Charles Pritchard 2011-09-22 16:29:17 UTC
(In reply to comment #15)
> Zooming is handled already by the focus ring APIs.

Zooming is -partially- handled by the focus ring APIs. It's counter to WCAG to require that an AT focus on an element prior to the user confirming their intent to activate that focus. While authors could plausibly loop through every interactive element, to share that information, such as solution is rather inefficient and also seems counter-indicative of current practices. Generally, when an element loses focus from drawFocusRing, I'd think that the accessibility tree would also remove its "focus" dimensions.

Let me know if that's unclear.
 
> In general, however, using <canvas> for something that can be sanely interacted
> with in non-visual mediums is completely counter to the point of <canvas>. If
> your content is media-independent, then use HTML, that's what it's for.

Begging your re-evaluation here: the purpose is for canvas to be sanely interacted with as a spatial medium. "Non-visual" is a much more broad term.

Canvas is very much a spatial language. Yes, it has paint servers. But most of the data passed around is data about spatial coordinates. These bug reports relate current defects on various ATs relating to spatial navigation. I do not think it's practical to tell AT vendors that they are "wrong" in their implementations. All accessibility trees share this spatial information as part of their data set.
Comment 18 Sam Ruby 2011-09-26 20:34:31 UTC
We had an agreement for 1 month turn around for P1 bugs, and this is well past that point.  As such, if anybody wishes to escalate this, they can do so at this time.
Comment 19 Charles Pritchard 2011-09-26 20:52:38 UTC
We confirmed that this is an issue for AT vendors in January:
"[Vendors] need the screen location of all content with or without focus... the bounding rectangle of each element in screen coordinates (or client coordinates if we know the relative top-left)"

http://lists.w3.org/Archives/Public/public-canvas-api/2011JanMar/0000.html

I spoke to several developers at the SVG F2F in Seattle about the issue.
http://lists.w3.org/Archives/Public/public-svg-wg/2011JulSep/0059.html

Perhaps the editor would have an easier time discussing this with Cameron McCormack, as he was present at the meeting. Jonas Sickling has offered a proposal similar to the one that the public-canvas-api group developed in early 2011.

I have since narrowed down that API to one method, instead of three. The very point of this API is to "use HTML", which is something the editor seems to misunderstand. We are using HTML in the Canvas subtree. We need to inform the system about the spatial positioning of those HTML elements.

This would also inform element.scrollIntoView, in effect, harmonizing the Canvas subtree with the rest of the HTML display DOM.

FWIW: scrollPathIntoView is an interesting response to setCaretSelectionRect, and one I think should be reviewed by the W3C and AT vendors.
Comment 20 Rich Schwerdtfeger 2011-09-26 21:25:58 UTC
I was avoiding an escalation as I believe that Ian closed it without having adequate information in his decision. We believe he has that now (see my response below). The task force was waiting a week for a response from Ian before we actually escalated it. 

If you think we need to escalate this to have it resolve then please consider this an escalation. 

Note: I might add that no proposal is being filed on this as I am waiting for Maciej to respond with the last proposal related to Issue 131. Maciej stated that he needs all changes to the spec. be made against the existing one in draft and he wants diffs. We are waiting on that one to be processed first. I understand Maciej is on vacation. 

Please advise.
Comment 21 Sam Ruby 2011-09-27 23:42:26 UTC
(In reply to comment #20)
> 
> If you think we need to escalate this to have it resolve then please consider
> this an escalation. 

If it is not a priority at this time, consider dropping the priority.  If it is a priority, consider writing a Change Proposal.
Comment 22 Rich Schwerdtfeger 2011-09-29 15:35:18 UTC
Note: I spoke with David Bolter at Mozilla this week and he stated that they plan on implementing a solution for this by Q1 2012. That includes defect:
http://www.w3.org/Bugs/Public/show_bug.cgi?id=13181
Comment 23 Ian 'Hixie' Hickson 2011-09-29 23:19:39 UTC
I'll speak with David about what he had in mind.
Comment 24 steve faulkner 2011-09-30 09:27:21 UTC
(In reply to comment #23)
> I'll speak with David about what he had in mind.

It would be best for all stakeholders if discussion occurs on a public mailing list not in secret.
Comment 25 Rich Schwerdtfeger 2011-09-30 13:02:27 UTC
Ian, 

Great. Specifically, Jonas Sickling has a proposal that we want to work to implement. David knows about it. What we are looking for is something like this proposal from Jonas. I plan on writing the change proposal with input from Jonas. I would like you input on it:

http://lists.w3.org/Archives/Public/public-canvas-api/2011JulSep/0195.html
Comment 26 Sam Ruby 2011-10-11 19:43:57 UTC
Reminder: a failure to escalate this bug by January 14th, 2012 will result in this bug no longer being treated as a last call comment:

  http://lists.w3.org/Archives/Public/public-html/2011Jun/0315.html
Comment 27 Charles Pritchard 2011-10-17 05:28:23 UTC
(In reply to comment #26)
> Reminder: a failure to escalate this bug by January 14th, 2012 will result in
> this bug no longer being treated as a last call comment:
> 
>   http://lists.w3.org/Archives/Public/public-html/2011Jun/0315.html


This is marked as a P1 issue: what further escalation procedures should occur? This issue is holding me up. I'd really like to go out and let Canvas developers know about the a11y layer. Many of the vendor representatives in these discussions do not use Canvas. And it shows.
Comment 28 Benjamin Hawkes-Lewis 2011-10-17 07:44:04 UTC
(In reply to comment #27)
> This is marked as a P1 issue: what further escalation procedures should occur?

Escalation as a Tracker Request as per comment #15.
Comment 29 Frank Olivier 2011-12-07 20:37:20 UTC
I would like to see a mechanism like this...
http://www.w3.org/wiki/Canvas_hit_testing
...added to the spec to address this issue; solving hit testing and the bounds issue with one spec change.
Comment 30 Ian 'Hixie' Hickson 2011-12-09 21:58:48 UTC
BTW, it would be helpful if this wiki page (or another but with the same structure) could be filled in documenting precisely what it is that needs addressing here:

   http://wiki.whatwg.org/wiki/Canvas#Limitations_of_real-world_use_cases

See also bug 13578.
Comment 31 Charles Pritchard 2011-12-13 04:23:56 UTC
(In reply to comment #30) 
>    http://wiki.whatwg.org/wiki/Canvas#Limitations_of_real-world_use_cases

The whatwg wiki to be restricted from accepting new users. Here is a summary from Tab, based on the conversations he and I had on the public-canvas-api list. He is simply summarizing, not advocating any position in this summary. He is acting more as an editor in this case, trimming down the long discussions I had with him -- the "I" in this writing is referring to me.

"""
Platform accessibility APIs only need a fairly limited amount of
information to communicate with the user.  The information we can
currently expose with a combination of the subtree, drawFocusRing, and
scrollPathIntoView fill in *almost* all of that information.  The only
remaining hole is the location of "active" regions.

This can be somewhat obtained post-facto by focusing everything in the
subtree and seeing what geometry is exposed by drawFocusRing.  This
could, unfortunately, have other side effects (such as actually
drawing focus rings), and amounts to polling, with all the undesirable
details that entails, as the clickable regions may change over time.
Actively declaring the regions up-front produces a much better story
for accessibility and performance purposes.

As with drawFocusRing and scrollPathIntoView, we can attach useful
functionality to this so that authors will use it to help themselves,
and the a11y benefits just come along for the ride.  Specifically, if
you associate a clickable region of the canvas with an element in the
subtree via setClickableRegion(element), we can make clicks within
that region dispatch to the associated element instead of the canvas,
making it easier to listen for clicks on "objects" in the canvas.  I
believe that anything authors would want to use this functionality for
is a useful thing to expose to the AT API.
"""
Comment 32 Rich Schwerdtfeger 2012-01-11 17:58:03 UTC
Provide location information to assistive technology.
Comment 33 Rich Schwerdtfeger 2012-01-12 22:28:58 UTC
Issue Title: Provide canvas location and hit testing capability to fallback content 

Issue text: Provide a facility to assign location information for rendered child of fallback content and provide a way to do hit testing within the location of the rendered child such that pointer events may be dispatched to the rendered child. The dimensions defining the location may be provided to an assistive technology, such as a magnifier, to determine the objects location on the screen. 

Both bugs 13181 and 13176 must be associated with this issue.
Comment 34 Frank Olivier 2012-01-13 23:50:55 UTC
[Copying comment from 13578; related bug]
I have written up some proposed spec text at
http://www.w3.org/wiki/Canvas_hit_testing for this change; the text includes
the rationale for this change.
Comment 35 Frank Olivier 2012-01-13 23:51:11 UTC
[Copying comment from 13578; related bug]
I have written up some proposed spec text at
http://www.w3.org/wiki/Canvas_hit_testing for this change; the text includes
the rationale for this change.
Comment 37 Sam Ruby 2012-01-18 01:55:43 UTC
Changed status per: http://lists.w3.org/Archives/Public/public-html/2012Jan/0087.html
Comment 38 Ian 'Hixie' Hickson 2012-02-29 00:06:05 UTC
Reclosing since this is marked TrackerIssue, but note that I intend to address this very shortly in a pretty comprehensive manner. See http://wiki.whatwg.org/wiki/Canvas#Regions for details.
Comment 40 rcabanie 2012-10-22 17:45:27 UTC
Issue-201 applied - http://lists.w3.org/Archives/Public/public-html/2012Aug/0453.html 

EDITOR'S RESPONSE: This is an Editor's Response to your comment. If you are
satisfied with this response, please change the state of this bug to CLOSED. If
you have additional information and would like the Editor to reconsider, please
reopen this bug. If you would like to escalate the issue to the full HTML
Working Group, please add the TrackerRequest keyword to this bug, and suggest
title and text for the Tracker Issue; or you may create a Tracker Issue
yourself, if you are able to do so. For more details, see this document:

   http://dev.w3.org/html5/decision-policy/decision-policy.html

Status: resolved as fixed
Description: Added back in the features:
dashed lines
ellipses
path objects & path methods
Additional TextMetrics features

That were removed earlier for issue-201.
Comment 41 rcabanie 2012-10-22 17:47:46 UTC
Decision applied for Issue-201
Comment 42 Rich Schwerdtfeger 2012-10-23 01:27:36 UTC
Rick, Sam, What happens if we don't get implementation of the path features for HTML5? Is this re-opened. I recall that this feature was "at risk" for HTML5. This is a process question.
Comment 43 rcabanie 2012-10-23 03:06:55 UTC
(In reply to comment #42)
> Rick, Sam, What happens if we don't get implementation of the path features
> for HTML5? Is this re-opened. I recall that this feature was "at risk" for
> HTML5. This is a process question.

As far I know, if there is no implementation, this feature will be removed from the spec that is going to LC/RC.
However, it will stay in the spec for the next version of HTML5.1 canvas since the WG resolved to have this feature. (The API's might change if there is browser/user feedback)
Comment 44 Maciej Stachowiak 2012-10-23 03:12:44 UTC
To clarify, removal of features with no implementations will likely only happen towards the end of CR, not necessarily on entry to CR. Though there is a process to request early removal.
Comment 45 Rich Schwerdtfeger 2012-10-23 12:45:09 UTC
I am satisfied with the response. I have closed the bug. Thank you.