This is an archived snapshot of W3C's public bugzilla bug tracker, decommissioned in April 2019. Please see the home page for more details.

Bug 13625 - There is no way to pass audio and video content metadata to the user agent that is required in some cases for playback.
Summary: There is no way to pass audio and video content metadata to the user agent th...
Status: CLOSED WONTFIX
Alias: None
Product: HTML WG
Classification: Unclassified
Component: HTML5 spec (show other bugs)
Version: unspecified
Hardware: All All
: P2 normal
Target Milestone: ---
Assignee: Silvia Pfeiffer
QA Contact: HTML WG Bugzilla archive list
URL:
Whiteboard:
Keywords: a11y, a11ytf, media
Depends on:
Blocks:
 
Reported: 2011-08-03 19:38 UTC by Bob Lund
Modified: 2014-02-26 18:24 UTC (History)
15 users (show)

See Also:

Attachments

Description Bob Lund 2011-08-03 19:38:53 UTC 
Problem description:

Commercial video providers want to use an HTML5 user agent to provide user interface for media playback on devices that also support the UPnP home networking protocol [1]. In this use case, home network content is stored in a content directory where each item in the content directory contains the URL of the content item along with content metadata. This metadata is called resource encoding characteristics properties [2]. Some of this data may be needed by the user agent for content playback, specifically:

o protocolInfo - third and fourth fields [3] contain content type information
o size - provides a hint of the content item size in the case of file based content
o protection  contains additional information for content protection

There is currently no way to represent this type of metadata in the video and audio element so it is available to the user agent. These metadata items may be relevant to types of content other than UPnP.

Proposed solution:

To address the above problem, it is proposed that three new attributes be added to the HTMLMediaElement for this metadata, e.g. contentInfo, size and protectionInfo.


[1] http://upnp.org/specs/arch/UPnP-arch-DeviceArchitecture-v1.0.pdf, also ISO/IEC 29341-1:2008
[2] http://upnp.org/specs/av/UPnP-av-ContentDirectory-v4-Service.pdf, also ISO/IEC 29341-3-12:2008
[3] http://upnp.org/specs/av/UPnP-av-ConnectionManager-v3-Service.pdf, also ISO/IEC 29341-3-11:2008 section 2.5.2
Comment 1 Michael[tm] Smith 2011-08-04 05:16:36 UTC 
mass-move component to LC1
Comment 2 Philip Jägenstedt 2011-08-04 08:24:11 UTC 
Would it be possible to treat UPnP devices much like a webcam so that you get access to it via getUserMedia? That way the UA is pretty much free to do whatever is necessary to get access to the content.
Comment 3 Bob Lund 2011-08-04 16:29:00 UTC 
(In reply to comment #2)
> Would it be possible to treat UPnP devices much like a webcam so that you get
> access to it via getUserMedia? That way the UA is pretty much free to do
> whatever is necessary to get access to the content.

I looked for getUserMedia in W3C HTML5 ed draft and WHATWG but didn't see anything? Can you provide a link?

The problem is that the user agent might need this metadata for playback. If getUserMedia is something available to script then it wouldn't appear to solve this problem.

Bob
Comment 4 Philip Jägenstedt 2011-08-05 06:27:40 UTC 
http://www.whatwg.org/specs/web-apps/current-work/multipage/video-conferencing-and-peer-to-peer-communication.html#obtaining-local-multimedia-content

getUserMedia has an options parameter which could probably be used to pass along whatever initialization data is needed.
Comment 5 Bob Lund 2011-08-11 17:04:16 UTC 
(In reply to comment #4)
> http://www.whatwg.org/specs/web-apps/current-work/multipage/video-conferencing-and-peer-to-peer-communication.html#obtaining-local-multimedia-content
> 
> getUserMedia has an options parameter which could probably be used to pass
> along whatever initialization data is needed.

Thanks for the link. It looks like getUserMedia is used to expose a local media source while the use case stimulating this bug was the client accessing content from a server. It seems however that the getUserMedia object could be applied to the remote content use case in the following way:

- The options attribute would be used to convey the media URL that the local UA uses to fetch the media.

- The options attribute would also be used to convey metadata specific to that content that the local UA needs to correctly decode the content.

- Depending on the metadata, certain behavior of the MediaStream may be disabled, recording for example or forking of specific tracks.

I am assuming that when using the MediaStream (returned by getUserMedia) as a child of <video>, multiple video, audio and text tracks in the MediaStream would be exposed per HTML5 support for multiple video/audio tracks and <track>.
Comment 6 Michael Cooper 2011-09-20 15:31:17 UTC 
Bug triage sub-team thinks this is an HTML A11Y TF priority for the media sub-team.
Comment 7 Clarke Stevens 2011-11-09 23:04:59 UTC 
At the F2F meetings in Santa Clara, the HTML WG appeared to support this bug as
a way to pass parameters to media elements for support of adaptive bit rate media as well as content protection schemes. However, the HTML WG would like a concise list of common parameters for these purposes.

The Media Pipeline task force (MPTF) has agreed to review this and come up with a proposed list.

Here is a link to the relevant MPTF requirements:

http://www.w3.org/2011/webtv/wiki/MPTF/MPTF_Requirements#R7._Additional_Media_Parameters
http://www.w3.org/2011/webtv/wiki/MPTF/MPTF_Requirements#R10._Content_Protection_Parameters
Comment 8 Mark Vickers 2011-11-09 23:14:50 UTC 
(In reply to comment #7)
> At the F2F meetings in Santa Clara, the HTML WG appeared to support this bug as
> a way to pass parameters to media elements for support of adaptive bit rate
> media as well as content protection schemes. However, the HTML WG would like a
> concise list of common parameters for these purposes.
> 
> The Media Pipeline task force (MPTF) has agreed to review this and come up with
> a proposed list.

The MPTF also took an action item to provide a document detailing the mapping from UPnP to HTML5, which is the example in the original bug report. This is currently in preparation.
Comment 9 Michael[tm] Smith 2011-11-20 14:17:32 UTC 
Flagging this with the needsinfo keyword to note that resolution is waiting on further information to be provided; see comment #7 and comment #8
Comment 10 Ian 'Hixie' Hickson 2011-11-24 00:54:25 UTC 
EDITOR'S RESPONSE: This is an Editor's Response to your comment. If you are satisfied with this response, please change the state of this bug to CLOSED. If you have additional information and would like the editor to reconsider, please reopen this bug. If you would like to escalate the issue to the full HTML Working Group, please add the TrackerRequest keyword to this bug, and suggest title and text for the tracker issue; or you may create a tracker issue yourself, if you are able to do so. For more details, see this document:
   http://dev.w3.org/html5/decision-policy/decision-policy.html

Status: Did Not Understand Request
Change Description: no spec change
Rationale: see comment 9
Comment 11 Michael[tm] Smith 2011-11-24 01:14:16 UTC 
Bob, Mark, Clarke,

Moving this to resolved=needsinfo is just a housekeeping way of indicating that this is waiting for additional information. It doesn't mean anybody considers it finally resolved ("resolved" is just the closest appropriate status that bugzilla provides).  

So please go ahead and reopen this when you have the info ready that you noted in comment #7 and comment #8.
Comment 12 Bob Lund 2011-11-28 23:40:32 UTC 
(In reply to comment #11)
> Bob, Mark, Clarke,
> 
> Moving this to resolved=needsinfo is just a housekeeping way of indicating that
> this is waiting for additional information. It doesn't mean anybody considers
> it finally resolved ("resolved" is just the closest appropriate status that
> bugzilla provides).  
> 
> So please go ahead and reopen this when you have the info ready that you noted
> in comment #7 and comment #8.

In the HTML WG/Media Pipeline Task Force at TPAC CableLabs and Comcast agreed to document the how HTML5 would be used in a UPnP environment. This link http://html5.cablelabs.com/upnp/html5-upnp-integration.html#2-box-a provides additional details relevant to the UPnP use case behind this bug.

Bob
Comment 13 Michael[tm] Smith 2011-11-29 09:32:55 UTC 
(In reply to comment #12)
> In the HTML WG/Media Pipeline Task Force at TPAC CableLabs and Comcast agreed
> to document the how HTML5 would be used in a UPnP environment. This link
> http://html5.cablelabs.com/upnp/html5-upnp-integration.html#2-box-a provides
> additional details relevant to the UPnP use case behind this bug.

Excellent.

I note that this bug is now just waiting on the list of parameters mentioned in comment #7
Comment 14 Clarke Stevens 2011-12-16 06:57:05 UTC 
(In reply to comment #13)
> (In reply to comment #12)
> > In the HTML WG/Media Pipeline Task Force at TPAC CableLabs and Comcast agreed
> > to document the how HTML5 would be used in a UPnP environment. This link
> > http://html5.cablelabs.com/upnp/html5-upnp-integration.html#2-box-a provides
> > additional details relevant to the UPnP use case behind this bug.
> 
> Excellent.
> 
> I note that this bug is now just waiting on the list of parameters mentioned in
> comment #7

The Media Pipeline Task Force has submitted the following proposal in response to comment #7

http://www.w3.org/2011/webtv/wiki/MPTF/ADR_Minimal_Control_Model_Proposal
Comment 15 Clarke Stevens 2011-12-16 06:57:49 UTC 
(In reply to comment #13)
> (In reply to comment #12)
> > In the HTML WG/Media Pipeline Task Force at TPAC CableLabs and Comcast agreed
> > to document the how HTML5 would be used in a UPnP environment. This link
> > http://html5.cablelabs.com/upnp/html5-upnp-integration.html#2-box-a provides
> > additional details relevant to the UPnP use case behind this bug.
> 
> Excellent.
> 
> I note that this bug is now just waiting on the list of parameters mentioned in
> comment #7

The Media Pipeline Task Force has submitted the following proposal in response to comment #7

http://www.w3.org/2011/webtv/wiki/MPTF/ADR_Minimal_Control_Model_Proposal
Comment 16 contributor 2012-07-18 07:26:03 UTC 
This bug was cloned to create bug 17971 as part of operation convergence.
Comment 17 Silvia Pfeiffer 2012-09-28 23:58:06 UTC 
If I understand you correctly, what you are asking for is to provide an API to influence decisions that the browser makes for HTTP adaptive streaming.

[1] lists the following interfaces:

* maximumBandwidth (input) - specifies the maximum bandwidth usage for
downloading adaptive streaming content. The input is in bits/second.
* minimumBandwidth(input) - specifies the minimum throughput above which the UA
should optimize for continuous playback and below which it should optimize for
quality, even if this implies buffering. The input is in bits/second.
* startingPlaybackHint(input) - gives the UA a hint about the preferred startup
optimization, whether it be for fast video startup or for high quality. 

IIUC this is proposing to have a middle ground between the situation where the
Web Developer does everything related to adaptive streaming by hand (using [2])
and the situation where the Web Developer provides a list of resource
descriptions (likely in a DASH file) and leaves the decisions completely to the browser which bytes to get from which file.

==

In addition, this proposal [1] includes a list of suggested statistics on videos:
* downloadRate
* representationID
* droppedFrames
* decodedFrames
* bufferLength

This second half reminds me a lot of bug 12399 and bug 14970 . It would be good to get involved with the Web Performance group and arrange for some of these statistics to be standardised there. Note that they have just asked for input on what next features they should work on. [3]

==

I'm also suggesting to move this to HTML.next, since I can't see this being resolved for HTML5.


[1] http://www.w3.org/2011/webtv/wiki/MPTF/ADR_Minimal_Control_Model_Proposal
[2] http://dev.w3.org/2011/webrtc/editor/getusermedia.html
[3] http://lists.w3.org/Archives/Public/public-web-perf/2012Sep/0044.html
Comment 18 Ian 'Hixie' Hickson 2012-10-01 18:38:08 UTC 
Silvia, did you mean to assign this to me? If so, please mark it as a dupe of bug 17971. Thanks!
Comment 19 Silvia Pfeiffer 2012-10-02 00:20:28 UTC 
Ian, sorry, no I didn't (not even sure how that happened...) - forwarding your question, though, to stay in syc...

Bob, Clarke: would your needs be satisfied by bug 14970, so can we close this as a duplicate of that bug?
Comment 20 Robin Berjon 2012-11-27 15:55:20 UTC 
This is definitely interesting, but as Silvia suggests it won't make it in the 5.0 timeframe. Moving to HTML.next.
Comment 21 Robin Berjon 2013-01-21 15:59:46 UTC 
Mass move to "HTML WG"
Comment 22 Robin Berjon 2013-01-21 16:02:33 UTC 
Mass move to "HTML WG"
Comment 23 Silvia Pfeiffer 2013-04-26 10:09:05 UTC 
In the related bug 17971, Bob Lund repliad:

The intent of 13625 was to request a media element API to pass media metadata to the player. The use cases are being addressed by EME and MSE work in the HTML-WG so this bug can be closed.

I am closing this bug based on this information. If there is anything left, please re-open with a concrete proposal or add a new bug to the EME or MSE work.