Bug 19020 - <audio> and <video> do not have sufficient support for synchronized accessibility content
<audio> and <video> do not have sufficient support for synchronized accessibi...
Status: CLOSED FIXED
Product: HTML WG
Classification: Unclassified
Component: HTML5 spec
unspecified
All All
: P3 normal
: ---
Assigned To: This bug has no owner yet - up for the taking
HTML WG Bugzilla archive list
: a11y, a11ytf, media, PFWG, TrackerIssue
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2012-09-25 21:54 UTC by contributor
Modified: 2013-01-30 14:26 UTC (History)
15 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description contributor 2012-09-25 21:54:21 UTC
This was was cloned from bug 5758 as part of operation LATER convergence.
Originally filed: 2008-06-15 01:36:00 +0000
Original reporter: Jim Jewett <jimjjewett@gmail.com>

================================================================================
 #0   Jim Jewett                                      2008-06-15 01:36:22 +0000 
--------------------------------------------------------------------------------
The new <audio> and <video> elements do not permit fallback content.  

This is a regression from the more generic <object> (though not from <embed>).

The reasoning is that authors SHOULD use accessibility aids built into the underlying media.  While this is indeed desirable, it is not always possibe -- and relies on the browser being able to interpret the media stream, at least to some extent.

An additional fallback step is required for authors who cannot modify the original stream, and for users of browsers which do not understand the format well enough to extract accessibility information.

The fallback already available for <object> would be sufficient, as would a more specialized element such as the sometimes proposed <alt>.
================================================================================
 #1   Ian 'Hixie' Hickson                             2008-06-15 02:12:35 +0000 
--------------------------------------------------------------------------------
<video> and <audio> do support fallback content for user agents with no video and audio support. Just put the fallback contents in the elements.

We will have a format that all UAs support, and that will be the fallback to use when this is deployed. So there is no concern of lack of codec support.

For users who really don't want audio/video but want an alternative form, fallback support wouldn't work, since the UA would quite happily just show the video. Instead, authors should just provide links to transcripts (or whatever). This isn't an accessibility concern (or at least isn't exclusively a concern for disabled users, I should say), because all users sometimes want transcripts.
================================================================================
 #2   Jim Jewett                                      2008-06-15 14:21:29 +0000 
--------------------------------------------------------------------------------
(1)  The standard does not require use of the "one good codec"; other codecs are explicitly permitted.

(2)  The "one good codec" has not yet been chosen.  That is largely because there are no such codecs yet -- and that is even without an explicit accessibility requirement on the codec selection.

(3)  There will be user agents that cannot do audio or video because of hardware limits; expecting them to download the entire multimedia stream just to extract the accessibility data is perhaps too optimistic.
================================================================================
 #3   Jim Jewett                                      2008-10-17 02:20:59 +0000 
--------------------------------------------------------------------------------
Also note that it is not always possible to edit the original media.  If the media comes from a third party, the web page author may not have the either the legal authority to modify it (copyright concerns) or the technical ability (offsite hosting; proprietary formats).  The web page author must still be able to provide fallback content as appropriate to the web page.
================================================================================
 #4   Ian 'Hixie' Hickson                             2008-11-22 12:15:33 +0000 
--------------------------------------------------------------------------------
"Fallbacks" like transcripts are suitable for everyone, so it doesn't make sense to hide them when the video is available. Transcripts that are alternative media streams should just be listed as <source>s. What other kinds of fallback did you have in mind?
================================================================================
 #5   Martin Kliehm                                   2009-10-29 15:24:18 +0000 
--------------------------------------------------------------------------------
I cannot tell what the original aim of the bug reporter were, but regarding accessible fallback this is what WCAG guideline 1.2 is about.[1]

1) If a fallback element is nested within the audio or video element, in some cases the fallback element still needs to be focusable by keyboard, even when the browser supports audio/video. For example Bruce Lawson [2] nests a download link within a video element, so this should be keyboard accessible.

2) If the audio or video content itself is inaccessible due to the nature of the person's disability, accessible alternative content should be available. Accessible alternative content could be a transcript, a textual description, an audio description, captions, or a synchronized sign language video. This content could be linked or otherwise machine-readable bolted onto the element by attributes. Machine-readability is important for assistive technology like screenreaders, but also for search engines. @aria-describedby comes to my mind for text, but @alt or @longdesc could also be appropriate. Synchronized multimedia formats need other attributes to be associated with the original audio/video.

For example a link to a transcript would be sufficient, but users should be able to select if captions are displayed, choose from a range of several captions in multiple languages, and if a sign language interpretation is available this could be displayed as picture-in picture. Bruce Lawson writes that he expected the @controls to specify this, alas this is not yet the case.[2]

It should be noted that accessing alternative content via controls is important for all users, so it would be counter-productive to make them only available if assistive technology devices are present. Deaf users just need a way to enable captions, and so do users who don't have any audio equipment.

Regarding captions there are several competing formats: DFXP [3] as a W3C standard, YouTube officially supports SubViewer and SubRip, inofficially also SAMI and SMIL, but there's also DAISY, QTtext, DVB, TimedText, EBU, SCC, Kate, CMML, SSA, MicroDVD, USF, and VOBSub, among others. I would expect browser vendors to agree on a set of supported formats.

From an author's perspective there need to be easy methods to add such content; editing a link or adding an attribute that refers to another file is reasonable, but embedding captions within a video file could be beyond their abilities.

[1] http://www.w3.org/TR/WCAG/#media-equiv
[2] http://www.brucelawson.co.uk/2009/accessibility-of-html5-video-and-audio-elements/
[3] http://www.w3.org/TR/ttaf1-dfxp/
================================================================================
 #6   Martin Kliehm                                   2009-10-29 21:02:32 +0000 
--------------------------------------------------------------------------------
Example:

-  BBC video with timed text, synchronized to chapters:
   http://open.bbc.co.uk/rad/demos/html5/rdtv/episode2/index.html

But it doesn't end here. What we will see often is a combination of video and canvas. Although a video element might be present what we actually see could be video in a canvas. Just bear that in mind when solving bug #7011.

1. Content Injection:
   http://people.mozilla.com/~prouget/demos/DynamicContentInjection/play.xhtml

2. Filters:
   https://developer.mozilla.org/En/Manipulating_video_using_canvas
   http://blip.tv/file/1881590

3. Face recognition and social media layers:
   http://blip.tv/file/2265515

4. Dailymotion (huge French video site) demo in combination with JavaScript,
   SVG, CSS3, and canvas control elements (volume slider):
   http://www.dailymotion.com/openvideodemo

  
================================================================================
 #7   Michael Cooper                                  2009-11-03 21:21:15 +0000 
--------------------------------------------------------------------------------
PFWG is interested in this issue. A group has formed to explore the requirements and solutions. A preliminary meeting was held 1 November 2009 <http://www.w3.org/2009/11/01-media-minutes>. The work will continue in the HTML Accessibility Task Force. Note ISSUE-9 (currently "raised") is also relevant.
================================================================================
 #8   Laura Carlson                                   2009-11-04 00:13:48 +0000 
--------------------------------------------------------------------------------
Multimedia Accessibility <Audio> <Video> Wiki Page
http://esw.w3.org/topic/HTML/MultimediaAccessibilty
================================================================================
 #9   Ian 'Hixie' Hickson                             2010-01-04 08:33:00 +0000 
--------------------------------------------------------------------------------
(In reply to comment #5)
> 1) If a fallback element is nested within the audio or video element, in some
> cases the fallback element still needs to be focusable by keyboard, even when
> the browser supports audio/video. For example Bruce Lawson [2] nests a download
> link within a video element, so this should be keyboard accessible.

That's a user agent implementation issue. Since the children of <video> are only exposed to users of UAs that don't support <video> at all, the HTML5 spec can't do much about it anyway  if it could, the UAs would just implement <video>.


> 2) If the audio or video content itself is inaccessible due to the nature of
> the person's disability, accessible alternative content should be available.

This is possible (indeed, easy): just include such content, or links to such content, in the page (not inside the <video> element).


> Accessible alternative content could be a transcript, a textual description

Both of these are already supported, just by regular HTML alongside the <video>. No special feature is needed to enable this.


> an audio description, captions, or a synchronized sign language video.

These are features of the video formats themselves. We haven't yet resolved the format issue. If we can't resolve it, and indeed even if we can, in due course, once browsers implement what is already specced to an adequate level, we will be introducing explicit syntax for time-synchronised captions, sign language insets, etc. This isn't fallback, however, so is out of scope for this bug.


> This
> content could be linked or otherwise machine-readable bolted onto the element
> by attributes. Machine-readability is important for assistive technology like
> screenreaders, but also for search engines. @aria-describedby comes to my mind
> for text, but @alt or @longdesc could also be appropriate. Synchronized
> multimedia formats need other attributes to be associated with the original
> audio/video.

It's unclear which of the many features discussed above this is referring to. Each of the features above have different needs, so it would be inappropriate to address them all the same way.


> For example a link to a transcript would be sufficient, but users should be
> able to select if captions are displayed, choose from a range of several
> captions in multiple languages, and if a sign language interpretation is
> available this could be displayed as picture-in picture. Bruce Lawson writes
> that he expected the @controls to specify this, alas this is not yet the
> case.[2]

It's unclear exactly what is missing here. What is stopping user agents from supporting captions today, for instance?

This is out of scope of this bug, though, which is about fallback. I would encourage you to file very specific bugs on each specific issue, rather than a single bug for the entire area.


> It should be noted that accessing alternative content via controls is important
> for all users, so it would be counter-productive to make them only available if
> assistive technology devices are present. Deaf users just need a way to enable
> captions, and so do users who don't have any audio equipment.

Indeed. This is possible today, and needs no author involvement.


> Regarding captions there are several competing formats: DFXP [3] as a W3C
> standard, YouTube officially supports SubViewer and SubRip, inofficially also
> SAMI and SMIL, but there's also DAISY, QTtext, DVB, TimedText, EBU, SCC, Kate,
> CMML, SSA, MicroDVD, USF, and VOBSub, among others. I would expect browser
> vendors to agree on a set of supported formats.

Format decisions are an outstanding issue currently, and there is work ongoing to resolve it.


> From an author's perspective there need to be easy methods to add such content;
> editing a link or adding an attribute that refers to another file is
> reasonable, but embedding captions within a video file could be beyond their
> abilities.

On the medium term, we will be adding features for external captions and the like. However, it would be harm accessibility of <video> on the Web on the short and medium term if we were to add it today, before browsers have correctly implemented the existing requirements. When browser vendors work on this feature, we want them to be focused on it  we don't want it to be just one more item on a long list of features that they have to all implement at once to get <video> support.


In conclusion, I think the spec as is has suitable support for _fallback_ for <video> and <audio>, though I agree that it is far from a comprehensive solution. I do not think, however, that it is in anyone's interests for us to add more features to make this simpler at this time, since they would end up misimplemented and would unintentionally harm accessibility. This doesn't just apply to these accessibility enhancements, by the way; we've delayed a whole bunch of features that make certain effects simpler while we wait for implementations to catch up.

Also, it's worth noting that as it stands today, authors are very unlikely to make immediate extensive use of <video> anyway. We haven't even found a common codec yet. Therefore in practice the lack of easy-to-use accessibility hooks does not actually have much of an effect at this time.

Here is what I propose as a way forward for this general topic:

- UAs continue to implement <video> such that the implementations reach a high enough caliber that there are no longer major bugs that would distract from additional feature work.
- in the meantime, we experiment with finding the right API, markup, and formats for captions and other time-linked annotation data.
- for the few early-adopter authors making immediate use of <video>, we encourage either the use of scripts that layer subtitles (etc) over video, or the use of subtitles embedded within the video files as separate text tracks.
- once UAs have stable high-quality implementations of the feature set in the spec today, we add external subtitle (etc) support to the spec.


EDITOR'S RESPONSE: This is an Editor's Response to your comment. If you are satisfied with this response, please change the state of this bug to CLOSED. If you have additional information and would like the editor to reconsider, please reopen this bug. If you would like to escalate the issue to the full HTML Working Group, please add the TrackerRequest keyword to this bug, and suggest title and text for the tracker issue; or you may create a tracker issue yourself, if you are able to do so. For more details, see this document:
   http://dev.w3.org/html5/decision-policy/decision-policy.html

Status: Partially Accepted
Change Description: none at this time
Rationale: see above
================================================================================
 #10  Laura Carlson                                   2010-01-12 13:28:10 +0000 
--------------------------------------------------------------------------------
Could "Resolution: LATER" be defined for this bug? 

Is there a ballpark date? Does it mean before HTML5 last call? Or does it mean HTML6? 

Thanks.
================================================================================
 #11  Ian 'Hixie' Hickson                             2010-01-12 21:49:21 +0000 
--------------------------------------------------------------------------------
It depends on implementations, as described in detail in comment 9. The best way to help implementations reach a reliable interoperable state faster is to write test cases and file bugs on implementations.
================================================================================
 #12  Michael Cooper                                  2010-02-04 16:36:15 +0000 
--------------------------------------------------------------------------------
HTML Accessibility Task force making this a formal tracked bug for the TF.
================================================================================
 #13  Michael Cooper                                  2010-02-04 16:42:37 +0000 
--------------------------------------------------------------------------------
Removing myself from cc list, default on bug edits is to add, but I'm only acting on behalf of HTML A11Y TF which is already cc'd.
================================================================================
 #14  Silvia Pfeiffer                                 2010-02-07 01:36:15 +0000 
--------------------------------------------------------------------------------
The discussions on this bug have touched multiple topics related to media and accessibility.

I'd like to propose this bug to focus on the support of synchronised accessibility data (captions, subtitles, audio description, sign language) - both embedded in stream and external.

In particular, this bug is not about "fallbacks" (and possibly the word "fallback" should be replaced with "support" in the title).

As for all the other topics that this bug has touched upon, here is what I took out:
* choice of a baseline code - not something we can make any progress on through a bug
* video fallback in UAs where video is not supported - already possible
* non-synchronised text alternatives - already possible through putting text or a link next to the video (and possibly using aria-describedBy to link it to the video)
* access to captions/subtitles available in media files that a UA does not support (e.g. MPEG with subtitles in Firefox) - is simply not technically possible
* @alt attribute - has been discussed in WHATWG, see thread at http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2009-August/021865.html; basically, the following attributes already provide what is required: @title, @aria-label, @aria-describedby

Hope this answers some questions and lets us move to the core accessibility issues for media elements that are still open and that Ian proposed a process for how to address.
================================================================================
 #15  Maciej Stachowiak                               2010-03-14 13:14:17 +0000 
--------------------------------------------------------------------------------
This bug predates the HTML Working Group Decision Policy.

If you are satisfied with the resolution of this bug, please change the state of this bug to CLOSED. If
you have additional information and would like the editor to reconsider, please reopen this bug. If you would like to escalate the issue to the full HTML Working Group, please add the TrackerRequest keyword to this bug, and suggest title and text for the tracker issue; or you may create a tracker issue yourself, if you are able to do so. For more details, see this document:
   http://dev.w3.org/html5/decision-policy/decision-policy.html

This bug is now being moved to VERIFIED. Please respond within two weeks. If this bug is not closed, reopened or escalated within two weeks, it may be marked as NoReply and will no longer be considered a pending comment.
================================================================================
 #16  Michael[tm] Smith                               2010-03-14 13:33:02 +0000 
--------------------------------------------------------------------------------
http://www.w3.org/html/wg/tracker/issues/9
================================================================================
 #17  Jim Jewett                                      2010-03-15 13:54:46 +0000 
--------------------------------------------------------------------------------
It is formally on me to accept resolution and close, or not.

Given comment 12, I don't feel comfortable doing that.

I won't claim to be fully satisfied, but I'm not sure how to better express my lingering concerns -- so I defer to the HTML Accessibility Task.  When they're satisfied, I'll agree to closure.
================================================================================
 #18  Maciej Stachowiak                               2010-03-15 14:42:53 +0000 
--------------------------------------------------------------------------------
This bug is already marked TrackerIssue so there's no need to close it right now.
================================================================================
 #19  Michael[tm] Smith                               2010-03-18 15:45:11 +0000 
--------------------------------------------------------------------------------
Reopened per discussion on a11y TF telcon 2010-03-18
================================================================================
 #20  Maciej Stachowiak                               2010-03-18 16:43:22 +0000 
--------------------------------------------------------------------------------
Since this bug has already been escalated to the tracker, it's inappropriate to also reopen it.
================================================================================
 #21  Maciej Stachowiak                               2010-03-18 16:43:41 +0000 
--------------------------------------------------------------------------------
(Moving back to VERIFIED)
================================================================================
 #22  Laura Carlson                                   2010-03-22 16:20:52 +0000 
--------------------------------------------------------------------------------
(In reply to comment #21)
> (Moving back to VERIFIED)

Maciej set the status of this bug to VERIFIED FIXED after it was reopened through a misunderstanding of the decision process at the a11y teleconference.

Maciej's rationale is that this bug is already escalated to a tracker issue. VERIFIED with a TrackerIssue keyword is indeed the correct state for a bug that has been "resolved" by the editor but has not yet in fact been decided by the working group via the escalation process. Maciej was correct to move the bug back to VERIFIED per HTML working group process.
http://dev.w3.org/html5/decision-policy/decision-policy.html#states-and-keywords

However, Ian did not FIX this bug. It is not FIXED so VERIFIED FIXED is not appropriate. 

It was last labeled VERIFIED LATER.
http://lists.w3.org/Archives/Public/public-html-a11y/2010Mar/0395.html

Relabeling this bug VERIFIED LATER as that is was last set it to. Maybe a more appropriate label can be worked out?
================================================================================
 #23  Michael Cooper                                  2011-03-08 16:10:18 +0000 
--------------------------------------------------------------------------------
Bug Triage sub-team notes that comment 14 points out that most of the issues can be addressed. What's missing in this bug is a reference to the track element that was introduced later. This element resolves the concerns. No action is required yet as the bug can't be closed until a text format is agreed upon. Verified Later is an appropriate state at this moment.
================================================================================
Comment 1 Silvia Pfeiffer 2013-01-24 08:09:32 UTC
EDITOR'S RESPONSE: This is an Editor's Response to your comment. If you are satisfied with this response, please change the state of this bug to CLOSED. If you have additional information and would like the editor to reconsider, please reopen this bug. If you would like to escalate the issue to the full HTML Working Group, please add the TrackerRequest keyword to this bug, and suggest title and text for the tracker issue; or you may create a tracker issue yourself, if you are able to do so. For more details, see this document:
   http://dev.w3.org/html5/decision-policy/decision-policy.html

Status: Accepted

Change Description:
none at this time - all changes were made since the bug was registered

Rationale:
The introduction of WebVTT and <track> with captions, subtitle, descriptions, chapter support for embedded and external text track data, as well as the introduction of the media controller that allows audio descriptions and sign language videos to be synchronized to a media resource addresses the issues of this bug.