This is an archived snapshot of W3C's public bugzilla bug tracker, decommissioned in April 2019. Please see the home page for more details.

Bug 9452 - <video> Handling of additional tracks of a multitrack audio/video resource
Summary: <video> Handling of additional tracks of a multitrack audio/video resource
Status: RESOLVED FIXED
Alias: None
Product: HTML WG
Classification: Unclassified
Component: pre-LC1 HTML5 spec (editor: Ian Hickson) (show other bugs)
Version: unspecified
Hardware: All All
: P2 major
Target Milestone: ---
Assignee: HTML Accessibility Task Force
QA Contact: HTML WG Bugzilla archive list
URL: http://www.w3.org/WAI/PF/HTML/wiki/Me...
Whiteboard: video
Keywords: a11y, a11ytf, hasProposal, media, NE, WGDecision
: 9586 12002 (view as bug list)
Depends on:
Blocks: 12544 12545 12546 12547 12548
  Show dependency treegraph
 
Reported: 2010-04-09 08:39 UTC by Silvia Pfeiffer
Modified: 2011-06-10 20:11 UTC (History)
13 users (show)

See Also:


Attachments

Description Silvia Pfeiffer 2010-04-09 08:39:07 UTC
Section affected: media sections

http://www.whatwg.org/specs/web-apps/current-work/multipage/video.html


Problem: No support for multitrack media resources

The current audio and video elements allow to reference multitrack audio and video resources, i.e. resources that have other data tracks than the main audio and video track. Examples of such additional tracks are a caption track, a subtitle track, a audio description track (being an audio recording of a scene description), but also e.g. a second camera angle or a separate music track.

While such resources can be used in existing audio and video elements, HTML5 only deals with the main audio and video track. The problem is that HTML5 does not currently acknowledge other tracks or provide an API to interact with them. Other tracks are neither exposed in the user interface nor controllable through JavaScript.



Suggested way to solve problem:

The media subgroup of the HTML5 Accessibility Task Force has worked on the following proposal:
http://www.w3.org/WAI/PF/HTML/wiki/Media_MultitrackAPI

This proposal is the result of several discussions there. It is not yet a recommendation of the task force, but in a good enough state to be introduced into the HTML WG.


The following two core approaches are proposed:

1. Introduce a JavaScript API to access information about, activate and deactivate tracks of a multitrack media resource

AND

2. Introduce a recommendation of UI controls for displaying available tracks visually and accessibly as part of the media controls, such that users can activate and deactivate tracks interactively


Also relates to:
* Issue-9 (http://www.w3.org/html/wg/tracker/issues/9)

* Bug 5758: Insufficient accessibility fallback for <audio> or <video> (http://www.w3.org/Bugs/Public/show_bug.cgi?id=5758)

* Bug 8659: Media events to indicate captions and audio descriptions (http://www.w3.org/Bugs/Public/show_bug.cgi?id=8659)
Comment 1 Ian 'Hixie' Hickson 2010-09-10 01:47:20 UTC
The ability to control multiple internal media tracks (sign language video overlays, alternate angles, dubbed audio, etc) seems like something we'd want to do in a way consistent with handling of multiple external tracks, much like how internal subtitle tracks and external subtitle tracks should use the same mechanism so that they can be enabled and disabled and generally manipulated in a consistent way.

However, supporting that seems to me to be a feature we should defer to a later version, since otherwise we'll get too far ahead of the browser vendors very quickly.

I discussed this with nessy and she suggested reassigning it to the tf for now, while they look at this. Please feel free to reassign this to me for further consideration (which would probably take the form of approaching browser vendors and getting their take on this).
Comment 2 Ian 'Hixie' Hickson 2010-09-10 09:00:43 UTC
*** Bug 9586 has been marked as a duplicate of this bug. ***
Comment 3 Silvia Pfeiffer 2010-09-10 11:13:26 UTC
(In reply to comment #1)
> The ability to control multiple internal media tracks (sign language video
> overlays, alternate angles, dubbed audio, etc) seems like something we'd want
> to do in a way consistent with handling of multiple external tracks, much like
> how internal subtitle tracks and external subtitle tracks should use the same
> mechanism so that they can be enabled and disabled and generally manipulated in
> a consistent way.
> 
> However, supporting that seems to me to be a feature we should defer to a later
> version, since otherwise we'll get too far ahead of the browser vendors very
> quickly.
> 
> I discussed this with nessy and she suggested reassigning it to the tf for now,
> while they look at this. Please feel free to reassign this to me for further
> consideration (which would probably take the form of approaching browser
> vendors and getting their take on this).

We can try and come up with a spec proposal in the Accessibility TF. It is a good idea to think about it in the context of multiple external tracks. The SMIL <par> element comes to mind, though it is unclear as yet how that can be fitted with the existing media elements. We will need to discuss.
Comment 4 Ian 'Hixie' Hickson 2010-11-11 23:06:38 UTC
EDITOR'S RESPONSE: This is an Editor's Response to your comment. If you are satisfied with this response, please change the state of this bug to CLOSED. If you have additional information and would like the editor to reconsider, please reopen this bug. If you would like to escalate the issue to the full HTML Working Group, please add the TrackerRequest keyword to this bug, and suggest title and text for the tracker issue; or you may create a tracker issue yourself, if you are able to do so. For more details, see this document:
   http://dev.w3.org/html5/decision-policy/decision-policy.html

Status: Accepted
Change Description: It's big.
Rationale: I made the change the chairs requested.
Comment 5 Silvia Pfeiffer 2010-11-12 03:45:56 UTC
(In reply to comment #4)
>
> Status: Accepted
> Change Description: It's big.
> Rationale: I made the change the chairs requested.

I think this may be a confusion - this particular bug is about something new that doesn't exist yet, namely the handling of multiple audio and video resources in sync. I believe your fixes only applied to WebSRT and the <track> element.

We are still working on a solution for multitrack audio/video in the a11y TF, so I would prefer to keep this bug open and the duty on me or the a11y TF as before. Nothing to do for Ian until we come back with a proposal.
Comment 6 Ian 'Hixie' Hickson 2010-11-12 20:43:08 UTC
Oh, my bad. I mistook this for one of the other bugs.

It's way too early to do multitrack. If we add it now, it'll be very poorly implemented and will result in a terrible experience for anyone who uses it. Indeed, premature implementation can end up breaking the feature for years (this has happened several times in the past even with HTML"5" features, e.g. look at how we screwed up with postMessage() security or look at the pushState() arguments issue). I don't think this should be an LC blocker, and would recommend marking this LATER. However, I'll leave this bug open for the task force to do with as desired if having the bug open is helpful.
Comment 7 Maciej Stachowiak 2010-11-12 21:54:58 UTC
(In reply to comment #6)
> Oh, my bad. I mistook this for one of the other bugs.
> 
> It's way too early to do multitrack. If we add it now, it'll be very poorly
> implemented and will result in a terrible experience for anyone who uses it.

I think implementers are at the point where we are ready to add support for descriptive audio and the like, whether or not it's in the spec (in addition to captions and text-to-speech descriptive audio). It would be helpful if the spec gave guidance on how to do that implementation work.
Comment 8 Paul Cotton 2010-12-09 16:31:54 UTC
This pre-Last Call bug is assigned to the A11Y TF and has been marked with TrackerRequest since the TF is working on a Change Proposal for the bug.

/paulc
Comment 9 Sam Ruby 2011-01-05 18:34:51 UTC
http://www.w3.org/html/wg/tracker/issues/152
Comment 10 Martin Kliehm 2011-02-08 16:37:50 UTC
*** Bug 12002 has been marked as a duplicate of this bug. ***
Comment 11 Ian 'Hixie' Hickson 2011-03-03 22:17:43 UTC
http://www.w3.org/WAI/PF/HTML/wiki/Media_Multitrack_Media_API is very useful information for this issue.
Comment 12 Ian 'Hixie' Hickson 2011-03-03 23:50:33 UTC
http://krijnhoetmer.nl/irc-logs/whatwg/20110304#l-39
Comment 13 Philip Jägenstedt 2011-03-04 07:25:52 UTC
Quoting Hixie from <http://krijnhoetmer.nl/irc-logs/whatwg/20110304#l-145>: "if we have a controller + slaves model, you can imagine a model where the controller doesn't have a timeline, it just controls the velocity"

I just want to point out that for normal playback of a single resource, the timeline is in practice controlled by the rate of the sound card, not the wall clock. The reason is that the wall clock and sound card clock might not be in sync, and audio glitches are much more noticeable. Only if there is no audio stream is the wall clock used.

I haven't tried implementing sync of two resources, but I think that it would have to be done by locking both to the sound card clock. If we define a audio- and video-less controller that drives the pipeline, we need to ensure that it can still be implemented by syncing to an audio stream of one of the resources. Actually using the wall clock to drive pipelines with audio streams would likely give pretty bad results.
Comment 14 Eric Carlson 2011-03-04 17:07:48 UTC
(In reply to comment #13)
> I haven't tried implementing sync of two resources, but I think that it would
> have to be done by locking both to the sound card clock. If we define a audio-
> and video-less controller that drives the pipeline, we need to ensure that it
> can still be implemented by syncing to an audio stream of one of the resources.
> Actually using the wall clock to drive pipelines with audio streams would
> likely give pretty bad results.
>
I agree, emphatically. Syncing to the wall clock will result in audible glitches and a poor user experience.
Comment 15 Ian 'Hixie' Hickson 2011-03-21 11:01:47 UTC
I made a proposal for this bug:
   http://lists.w3.org/Archives/Public/public-html/2011Mar/0436.html
Comment 16 Glenn Adams 2011-03-21 11:46:32 UTC
+1, but I would also suggest regularizing the value type of media.textTracks to be a MultipleTrackList; further, to avoid confusion, it would be good to rename media.addTrack() to media.addTextTrack();

glenn

(In reply to comment #15)
> I made a proposal for this bug:
>    http://lists.w3.org/Archives/Public/public-html/2011Mar/0436.html
Comment 17 Sam Ruby 2011-04-23 15:19:30 UTC
We have consensus on the media controller portions that are currently commented out of the W3C copy of the draft.  Please uncomment these portions out.

Subsequent changes are being pursued as separate bug reports.  Specifically:

http://www.w3.org/Bugs/Public/show_bug.cgi?id=12544
http://www.w3.org/Bugs/Public/show_bug.cgi?id=12545
http://www.w3.org/Bugs/Public/show_bug.cgi?id=12546
http://www.w3.org/Bugs/Public/show_bug.cgi?id=12547
http://www.w3.org/Bugs/Public/show_bug.cgi?id=12548
Comment 18 Ian 'Hixie' Hickson 2011-04-26 00:52:39 UTC
Fixed, but due to a publication pipeline issue the W3C copy hasn't synced yet.
Comment 19 Sam Ruby 2011-04-26 14:49:45 UTC
(In reply to comment #18)
> Fixed, but due to a publication pipeline issue the W3C copy hasn't synced yet.

Pipeline publication issue appears to have been a temporary load condition, which has since been resolved:

http://krijnhoetmer.nl/irc-logs/whatwg/20110426#l-301
http://krijnhoetmer.nl/irc-logs/whatwg/20110426#l-420
http://krijnhoetmer.nl/irc-logs/whatwg/20110426#l-423

Additionally, I do not see this change in the SVN source (currently at revision 6030):

http://svn.whatwg.org/webapps/
Comment 20 Sam Ruby 2011-04-27 12:22:34 UTC
Checked in as WHATWG revision r6031.
Check-in comment: remove CONTROLLER and TT markers
http://html5.org/tools/web-apps-tracker?from=6030&to=6031