This is an archived snapshot of W3C's public bugzilla bug tracker, decommissioned in April 2019. Please see the home page for more details.

Bug 11984 - <video>: Figure out the story with respect to honouring Content-Type headers vs sniffing content
Summary: <video>: Figure out the story with respect to honouring Content-Type headers ...
Status: RESOLVED FIXED
Alias: None
Product: WHATWG
Classification: Unclassified
Component: HTML (show other bugs)
Version: unspecified
Hardware: Other other
: P3 major
Target Milestone: Unsorted
Assignee: Ian 'Hixie' Hickson
QA Contact: HTML WG Bugzilla archive list
URL: http://www.whatwg.org/specs/web-apps/...
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-02-05 00:08 UTC by contributor
Modified: 2014-08-04 18:18 UTC (History)
23 users (show)

See Also:


Attachments

Description contributor 2011-02-05 00:08:06 UTC
Specification: http://www.whatwg.org/specs/web-apps/current-work/complete/video.html
Section: http://www.whatwg.org/specs/web-apps/current-work/#loading-the-media-resource

Comment:
Simplify <video> for implementors and authors by ignoring the Content-Type
HTTP header

Posted from: 85.227.154.141 by simonp@opera.com
Comment 1 Simon Pieters 2011-02-05 00:18:13 UTC
Rationale 

The current state of implementations is a mess, as summarized by Ian Hickson (http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2010-August/028291.html ): 
"Microsoft's position is, as far as I can tell, that there's no point looking at the Content-Type header, [...]" 
"Safari does crazy things right now that we won't go into; [...]" 
"Chrome right now sniffs like IE, [...]" 
"Opera does what the spec suggests. [...]" 
"Mozilla respects the Content-Type religiously, [...]" 

The currently specified handling of Content-Type has concrete negative side-effects for both implementors and authors, outlined in the following sections. 

Rather than continuing to maintain this mess, we should jump directly to the most sane of the likely outcomes, which is to always ignore Content-Type for <video>. 


Issues for Implementors 

* Because application/octet-stream is special-cased in canPlayType, the same code path cannot be used for canPlayType as for checking Content-Type and <source type>. 
* In the resource fetch algorithm, if the resource is transferred using many HTTP byte range requests, it's unclear if Content-Type should be checked on the first request, or on every request. Either would be problematic: 
   - Checking it only once is inconsistent because on subsequent requests, Content-Type may have changed, but this would be ignored. 
   - Checking it for every request could amount to very many times during the playback of a long video. If the underlying media framework must be queried each time, it could be time-consuming. Introducing caching to work around it increases the risk of bugs. 
* In order to be compatible with what is sent in Content-Type, implementations must support several synonymous MIME types. This is exposed via canPlayType, where there was no legacy to consider originally, creating unnecessary room for incompatibilities. For example, Firefox supports 4 different MIME types for WAVE, Opera supports 3 and Chrome supports 2. (http://wiki.whatwg.org/wiki/Video_type_parameters#Browser_Support ) 


Issues for Authors 

* Because of the way application/octet-stream is special-cased, the result of canPlayType is subtly different from how Content-Type and <source type> are treated, which is confusing. For example, canPlayType("application/octet-stream") returns "", while canPlayType("application/octet-stream;") returns "maybe". Yet, using either value in Content-Type should cause the user agent to use sniffing to determine the resource type and try decoding it. 
* Many servers are not configured to serve the correct Content-Type for audio and video types: 
   - Authors who are not aware of the requirements on Content-Type will go through time-consuming debugging before eventually learning how to configure their server. 
   - Authors who cannot control what Content-Type HTTP header is sent by their server will be unable to use <video> at all. 


Details 

* Remove all mention of application/octet-stream from the <video> section, especially: 
   - The special-casing of application/octet-stream from the canPlayType method. 
* Remove all mention of Content-Type from the <video> section, especially: 
   - The following case from the resource fetch algorithm: "If the media resource is found to have Content-Type metadata that, when parsed as a MIME type (including any codecs described by the codecs parameter, if the parameter is defined for that type), represents a type that the user agent knows it cannot render (even if the actual media data is in a supported format)" 


Impact 

Positive Effects 
* The behavior of <video> with respect to Content-Type is aligned with how e.g. <img> already works. 
* No more time is spent maintaining an unstable situation on which it is very unlikely implementations will converge. 
* Implementations of <video> are simplified. 
* The incentive for supporting many different equivalent MIME types in canPlayType and <source type> is removed, as they come only from legacy Content-Type issues. 
* Some legacy MIME types, such as audio/x-pn-wav, can likely be removed from implementations of canPlayType. 
* Using <video> becomes simpler for authors. 


Negative Effects 

* Video will always play in a <video> context but may fail in a top-level browsing content, depending on the Content-Type and what sniffing is used. (This is already the situation for images.) 
* Content-Type as the definitive mechanism for type information in HTTP is undermined even further. 


Conformance Classes Changes 

User Agents 


Risks 

* Failure to standardize sniffing of video types may result in poor interoperability. This is already the case, as the spec currently requires sniffing after checking Content-Type. However, the risk does increases somewhat when more implementations switch to always sniffing. 

Note: Work on standardized sniffing for audio and video types has already begun in Media Type Sniffing. http://tools.ietf.org/html/draft-ietf-websec-mime-sniff-01


(Note: This bug is basically foolip's CP for ISSUE-145 (http://www.w3.org/html/wg/wiki/ChangeProposals/NoVideoContentType ). However, I think his CP is not appropriate as a CP for ISSUE-145 since it makes a substantive change while the original bug was about an editorial issue. Therefore I think it is more appropriate for foolip's CP to be a separate bug, hence this bug.)
Comment 2 Ian 'Hixie' Hickson 2011-02-05 01:31:54 UTC
EDITOR'S RESPONSE: This is an Editor's Response to your comment. If you are satisfied with this response, please change the state of this bug to CLOSED. If you have additional information and would like the editor to reconsider, please reopen this bug. If you would like to escalate the issue to the full HTML Working Group, please add the TrackerRequest keyword to this bug, and suggest title and text for the tracker issue; or you may create a tracker issue yourself, if you are able to do so. For more details, see this document:
   http://dev.w3.org/html5/decision-policy/decision-policy.html

Status: Accepted
Change Description: see diff given below
Rationale: Concurred with reporter's comments.
Comment 3 contributor 2011-02-05 01:32:11 UTC
Checked in as WHATWG revision r5836.
Check-in comment: Bring the spec more in line with what the browsers do for video (largely, ignoring Content-Type). This will probably be tweaked further once Adam's draft has a section dedicated to sniffing media container formats.
http://html5.org/tools/web-apps-tracker?from=5835&to=5836
Comment 4 Adam Barth 2011-02-05 02:09:54 UTC
I've updated http://tools.ietf.org/html/draft-ietf-websec-mime-sniff to include a hook for sniffing video specifically.  Please let me know if you need other hooks in that document.
Comment 5 Julian Reschke 2011-02-05 10:43:27 UTC
I believe this should be reverted, see <http://lists.w3.org/Archives/Public/public-html/2011Feb/0114.html>.
Comment 6 Adrian Bateman [MSFT] 2011-02-14 20:46:28 UTC
Microsoft's position: IE9 RC strictly obeys the Content-Type returned to media elements. We believe this behaviour matches the latest Firefox 4 Beta. Based on feedback from developers and also following the "state of implementations" thread highlighted by Simon above, we changed the implementation of media elements between IE9 Beta and IE9 RC. While it required extra work in our browser engine to respect the Content-Type in this way, it turned out that this was less work than supporting a robust future-proof content-sniffing approach. Consequently we disagree that this change simplifies things for implementers and we request that the change be reverted. 

Currently, we do not support media playback for content with the application/octet-stream content type. The working group should consider application/octet-stream support as a separate issue.
Comment 7 Philip Jägenstedt 2011-02-14 21:45:27 UTC
(In reply to comment #6)
> Microsoft's position: IE9 RC strictly obeys the Content-Type returned to media
> elements. We believe this behaviour matches the latest Firefox 4 Beta. Based on
> feedback from developers and also following the "state of implementations"
> thread highlighted by Simon above, we changed the implementation of media
> elements between IE9 Beta and IE9 RC. While it required extra work in our
> browser engine to respect the Content-Type in this way, it turned out that this
> was less work than supporting a robust future-proof content-sniffing approach.
> Consequently we disagree that this change simplifies things for implementers
> and we request that the change be reverted. 
>
> Currently, we do not support media playback for content with the
> application/octet-stream content type. The working group should consider
> application/octet-stream support as a separate issue.

Uh, just when all browser vendors seemed to be OK with ditching Content-Type...

What about file:, ftp: and other protocols that don't have any equivalent of Content-Type? What about when Content-Type is missing from a HTTP respsonse?

By reverting this change we'd be bringing back application/octet-stream which you don't support. Wouldn't it be better to agree on something and change the spec to that than to bring back something which doesn't match any browser and never will?
Comment 8 Chris Double 2011-02-14 23:26:44 UTC
(In reply to comment #7)
> 
> Uh, just when all browser vendors seemed to be OK with ditching Content-Type...
>

Firefox currently doesn't sniff and respects the Content-Type. We don't plan to change this behavior at this stage. What I'd like to see is further work and testing on the content sniffing algorithm before any decision is made to change.

> What about file:, ftp: and other protocols that don't have any equivalent of
> Content-Type? What about when Content-Type is missing from a HTTP respsonse?

We have an internal mapping from file extension to content type for these purposes. 

> By reverting this change we'd be bringing back application/octet-stream which
> you don't support. Wouldn't it be better to agree on something and change the
> spec to that than to bring back something which doesn't match any browser and
> never will?

I'd also like the application/octet-stream issue to be dealt with separately as per Microsoft's request.
Comment 9 Sam Ruby 2011-02-15 00:16:52 UTC
(In reply to comment #6)
> Currently, we do not support media playback for content with the
> application/octet-stream content type. The working group should consider
> application/octet-stream support as a separate issue.

Please file a separate bug report on this.
Comment 10 David Singer 2011-02-15 03:25:54 UTC
Content-type has decreasing utility; it fails to handle files that conform to more than one standard (one file can simultaneously be valid as video/3gpp, video/mp4, and video/3gpp2, for example). Many users cannot configure their web servers to return correct content-types (it's not under their control). A a declared attribute in the markup, type has utility. As a guess from the file-extension by the web server, the utility is marginal at best; the UA can and usually does do better at deducing what the correct type of a file is. Given that you have to be prepared, at the UA, to disbelieve what you're told, or adjust it, we question whether tangling the web server's guess into the UA behavior makes life any simpler or more interoperable.
Comment 11 Philip Jägenstedt 2011-02-15 08:47:02 UTC
(In reply to comment #8)
> (In reply to comment #7)
> > 
> > Uh, just when all browser vendors seemed to be OK with ditching Content-Type...
> >
> 
> Firefox currently doesn't sniff and respects the Content-Type. We don't plan to
> change this behavior at this stage. What I'd like to see is further work and
> testing on the content sniffing algorithm before any decision is made to
> change.

I've discussed this with Robert O'Callahan, see <http://krijnhoetmer.nl/irc-logs/whatwg/20110127#l-831>. I'm not surprised that you're not a fan of the idea, though.

Should I take this that you'd consider dropping Content-Type if the solution gets spec'd, tested and shipped in another browser?

> > What about file:, ftp: and other protocols that don't have any equivalent of
> > Content-Type? What about when Content-Type is missing from a HTTP respsonse?
> 
> We have an internal mapping from file extension to content type for these
> purposes. 

Even for missing HTTP Content-Type? How about Content-Type: application/octet-stream?

> > By reverting this change we'd be bringing back application/octet-stream which
> > you don't support. Wouldn't it be better to agree on something and change the
> > spec to that than to bring back something which doesn't match any browser and
> > never will?
> 
> I'd also like the application/octet-stream issue to be dealt with separately as
> per Microsoft's request.

Would that be ISSUE-145, or the issue of whether or not to play HTTP resources with Content-Type: application/octet-stream ?
Comment 12 Maciej Stachowiak 2011-02-17 11:17:27 UTC
Since Microsoft and Mozilla do not implement the behavior resulting from this bug, and seem unwilling to do so, it may be a good idea to revert the change for now, pending further discussion. I say this even though Apple would somewhat prefer the behavior of ignoring Content-Type. Reopening bug based on new information from Mozilla and Microsoft reps.
Comment 13 Philip Jägenstedt 2011-02-17 11:57:07 UTC
(In reply to comment #12)
> Since Microsoft and Mozilla do not implement the behavior resulting from this
> bug, and seem unwilling to do so, it may be a good idea to revert the change
> for now, pending further discussion. I say this even though Apple would
> somewhat prefer the behavior of ignoring Content-Type. Reopening bug based on
> new information from Mozilla and Microsoft reps.

They also don't implement the behavior that would result from reverting this bug, and seem unwilling to do so.

Unless we can all agree to implement some specific behavior, I'd prefer to handle this as an ISSUE, to force everyone to make very specific suggestions for what the spec should be.
Comment 14 Julian Reschke 2011-02-17 12:06:16 UTC
(In reply to comment #13)
> They also don't implement the behavior that would result from reverting this
> bug, and seem unwilling to do so.

Well, not all of it. Sam's advise was to have a separate bug for that.
 
> Unless we can all agree to implement some specific behavior, I'd prefer to
> handle this as an ISSUE, to force everyone to make very specific suggestions
> for what the spec should be.

Once the change in the spec is undone, you are of course free to escalate to an issue.
Comment 15 Sam Ruby 2011-02-17 14:16:20 UTC
(In reply to comment #14)
> (In reply to comment #13)
> > They also don't implement the behavior that would result from reverting this
> > bug, and seem unwilling to do so.
> 
> Well, not all of it. Sam's advise was to have a separate bug for that.

+1

> > Unless we can all agree to implement some specific behavior, I'd prefer to
> > handle this as an ISSUE, to force everyone to make very specific suggestions
> > for what the spec should be.
> 
> Once the change in the spec is undone, you are of course free to escalate to an
> issue.

+1
Comment 16 Philip Jägenstedt 2011-02-17 15:24:54 UTC
Well, I'm hoping this will be closed as FIXED without reversion and that those who disagree with that raise it as an ISSUE, but either way the end result would be the same process.
Comment 17 Julian Reschke 2011-02-17 17:00:57 UTC
(In reply to comment #12)
> Since Microsoft and Mozilla do not implement the behavior resulting from this
> bug, and seem unwilling to do so, it may be a good idea to revert the change
> for now, pending further discussion. I say this even though Apple would
> somewhat prefer the behavior of ignoring Content-Type. Reopening bug based on
> new information from Mozilla and Microsoft reps.

"may be a good idea" is kind of fuzzy. Can we please have an official statement from the chairs? (asking because <http://www.w3.org/Bugs/Public/show_bug.cgi?id=11984#c16> indicates there's still confusion)
Comment 18 Philip Jägenstedt 2011-02-23 10:12:19 UTC
(In reply to comment #17)
> (In reply to comment #12)
> > Since Microsoft and Mozilla do not implement the behavior resulting from this
> > bug, and seem unwilling to do so, it may be a good idea to revert the change
> > for now, pending further discussion. I say this even though Apple would
> > somewhat prefer the behavior of ignoring Content-Type. Reopening bug based on
> > new information from Mozilla and Microsoft reps.
> 
> "may be a good idea" is kind of fuzzy. Can we please have an official statement
> from the chairs? (asking because
> <http://www.w3.org/Bugs/Public/show_bug.cgi?id=11984#c16> indicates there's
> still confusion)

There is no confusion, just plain old disagreement. Normally when a bug report is filed and the editor agrees and fixes it, it is those who disagree that escalate the bug to an issue. I don't see why we should do anything differently here, especially given that no one is willing to implement the previous state either.
Comment 19 Julian Reschke 2011-02-23 10:25:15 UTC
(In reply to comment #18)
> There is no confusion, just plain old disagreement. Normally when a bug report
> is filed and the editor agrees and fixes it, it is those who disagree that
> escalate the bug to an issue. I don't see why we should do anything differently
> here, especially given that no one is willing to implement the previous state
> either.

The bug has been reopened (by one of the WG chairs), so it's not in a state where it can be escalated.

As far as I can tell, the WG waits for the editor to either undo the change, or to close the bug again.
Comment 20 contributor 2011-03-01 00:19:04 UTC
Checked in as WHATWG revision r5921.
Check-in comment: revert r5836 per http://lists.w3.org/Archives/Public/public-html/2011Mar/0002.html
http://html5.org/tools/web-apps-tracker?from=5920&to=5921
Comment 21 Philip Jägenstedt 2011-03-01 08:02:12 UTC
At this point it would be most helpful if implementors at Mozilla and Microsoft stepped forward to specify exactly what they *are* willing to implement.

As an example of what is missing: do you also intend to respect the codecs parameter when given in Content-Type?
Comment 22 Philip Jägenstedt 2011-03-03 22:27:09 UTC
OK, new suggestion:

* In most cases, including at least audio/* and video/*, trust the Content-Type header and try decoding it as such. This is stricter than the current spec, which requires sniffing after inspecting Content-Type.
* Sniff for the set of Content-Type values that are usually incorrect defaults, like text/plain and application/octet-stream. The table in <http://tools.ietf.org/html/draft-ietf-websec-mime-sniff-02#section-3> looks relevant to this.
* Ignore the codecs parameter in the Content-Type header completely, since codec setup cannot be done without inspecting the data anyway.
* Get rid of all special-casing of application/octet-type in canPlayType.

Pros:
* Fixes the biggest problem of Content-Type, namely that videos served as text/plain or application/octet-stream don't play.
* Respects Content-Type in cases where second guessing likely isn't useful, like audio/* and video/*.
* Would work the same in a <video> context and in a top-level browsing context.
* Formats that are not reliably sniffable don't have to be added to the sniffing table, they just wouldn't work when served as application/octet-stream.
* Browsers that sniff to support existing content (e.g. Safari) would likely see less breakage than if moving to strict handling like Firefox/IE.
* It's still possible to move to "always sniff" from here without much damage being left behind.

Cons:
* Inconsistent with how <img> works.
* Requires considering both Content-Type and the data to determine what demuxer to plug, rather than just one or the other.
* The codecs parameter would have an effect in canPlayType but not in Content-Type. In practice the codecs parameter isn't sent in Content-Type, so it's not the end of the world.
Comment 23 Adrian Bateman [MSFT] 2011-03-04 20:12:33 UTC
(In reply to comment #22)
> OK, new suggestion:
> 
> * In most cases, including at least audio/* and video/*, trust the Content-Type
> header and try decoding it as such. This is stricter than the current spec,
> which requires sniffing after inspecting Content-Type.
> * Sniff for the set of Content-Type values that are usually incorrect defaults,
> like text/plain and application/octet-stream. The table in
> <http://tools.ietf.org/html/draft-ietf-websec-mime-sniff-02#section-3> looks
> relevant to this.
> * Ignore the codecs parameter in the Content-Type header completely, since
> codec setup cannot be done without inspecting the data anyway.
> * Get rid of all special-casing of application/octet-type in canPlayType.

We think we can live with this being what is in the spec. Unfortunately, were too late to make changes in IE9 (we announced when we shipped the release candidate that we would be shipping the final release shortly) and as usual I cant make predictions about when we will adopt things in future releases but this seems like a reasonable compromise to work towards. As you mention, there would be an inconsistency between handling of the codecs parameter between canPlayType compared to playing but this is something we think is necessary (for example to be able to choose between high and basic profile H.264 streams).
Comment 24 Philip Jägenstedt 2011-03-05 13:11:06 UTC
A few of the things in <http://www.w3.org/html/wg/wiki/ChangeProposals/NoVideoContentType#Negative_Effects> are not solved by my new suggestion, so I'd consider not fixing these cons:

* The incentive for supporting many different equivalent MIME types in canPlayType and <source type> is removed, as they come only from legacy Content-Type issues. 

* Some legacy MIME types, such as audio/x-pn-wav, can likely be removed from implementations of canPlayType.

This is damage we'll have to live with unless we always sniff, it might be OK if implementors exercise some contraint in which MIME types they add support for. WAVE might be a lost case, though.
Comment 25 Yuhong Bao 2011-03-06 07:34:33 UTC
Yea, I remember having to work with Roy Schestowitz of Techrights to fix the Content-Type when they began using the <video> tag, as one example.
Comment 26 Adrian Bateman [MSFT] 2011-03-07 17:24:52 UTC
(In reply to comment #24)
> A few of the things in
> <http://www.w3.org/html/wg/wiki/ChangeProposals/NoVideoContentType#Negative_Effects>
> are not solved by my new suggestion, so I'd consider not fixing these cons:
> 
> * The incentive for supporting many different equivalent MIME types in
> canPlayType and <source type> is removed, as they come only from legacy
> Content-Type issues. 
> 
> * Some legacy MIME types, such as audio/x-pn-wav, can likely be removed from
> implementations of canPlayType.
> 
> This is damage we'll have to live with unless we always sniff, it might be OK
> if implementors exercise some contraint in which MIME types they add support
> for. WAVE might be a lost case, though.

I didn't really understand these two points. I'm sure I'm just missing some background info. Please could you elaborate?
Comment 27 Philip Jägenstedt 2011-03-07 19:56:07 UTC
(In reply to comment #26)
> (In reply to comment #24)
> > A few of the things in
> > <http://www.w3.org/html/wg/wiki/ChangeProposals/NoVideoContentType#Negative_Effects>
> > are not solved by my new suggestion, so I'd consider not fixing these cons:
> > 
> > * The incentive for supporting many different equivalent MIME types in
> > canPlayType and <source type> is removed, as they come only from legacy
> > Content-Type issues. 
> > 
> > * Some legacy MIME types, such as audio/x-pn-wav, can likely be removed from
> > implementations of canPlayType.
> > 
> > This is damage we'll have to live with unless we always sniff, it might be OK
> > if implementors exercise some contraint in which MIME types they add support
> > for. WAVE might be a lost case, though.
> 
> I didn't really understand these two points. I'm sure I'm just missing some
> background info. Please could you elaborate?

It relates to this point from "Issues for Implementors":

* In order to be compatible with what is sent in Content-Type, implementations must support several synonymous MIME types. This is exposed via canPlayType, where there was no legacy to consider originally, creating unnecessary room for incompatibilities. For example, Firefox supports 4 different MIME types for WAVE, Opera supports 3 and Chrome supports 2. (source: <http://wiki.whatwg.org/wiki/Video_type_parameters#Browser_Support>)

If Content-Type was always ignored, this pollution of canPlayType would not happen. Actually, we could perhaps fix it in my new proposal as well by adding this point:

* Content-Type is normalized to allow for synonyms like audio/wave and audio/wav that are necessary for compat, but via canPlayType only the canonical form is exposed, so only one would return "maybe"/"probably".

I don't think it's actually worth the complexity though, I just consider it a con relative to ignoring Content-Type.
Comment 28 Adrian Bateman [MSFT] 2011-03-07 20:15:04 UTC
(In reply to comment #27)
> I don't think it's actually worth the complexity though, I just consider it a
> con relative to ignoring Content-Type.

Thanks for the explanation. I agree - I don't think it's worth trying to fix here either.
Comment 29 Michael[tm] Smith 2011-08-04 05:35:32 UTC
mass-move component to LC1
Comment 30 Ian 'Hixie' Hickson 2011-10-26 20:48:19 UTC
So what is the spec supposed to say, at this point?
Comment 31 Philip Jägenstedt 2011-10-27 08:26:34 UTC
(In reply to comment #30)
> So what is the spec supposed to say, at this point?

Everyone disagrees on that. I think it should just point to http://tools.ietf.org/html/draft-ietf-websec-mime-sniff-03#section-8 so that sniffing always takes precedence.
Comment 32 Ian 'Hixie' Hickson 2011-11-02 18:56:05 UTC
If browser vendors disagree on what should happen, then I think I will go back to strict adherence to the HTTP specs, and then let the market figure it out. Once browsers converge on a particular behaviour, or once there is agreement on what they should converge on, I will update the spec to match it.

You are welcome to try to use the escalation process to make a decision sooner, but of course if it doesn't match what finally gets implemented, it doesn't much matter what that decision is either. :-)
Comment 33 Philip Jägenstedt 2011-11-03 09:36:20 UTC
(In reply to comment #32)
> If browser vendors disagree on what should happen, then I think I will go back
> to strict adherence to the HTTP specs, and then let the market figure it out.
> Once browsers converge on a particular behaviour, or once there is agreement on
> what they should converge on, I will update the spec to match it.

If we know that browsers are implementing it differently, then I'd much prefer if the spec made it explicitly undefined and noted that it will be spec'd at some later point, rather than picking sides up front.
Comment 34 Ian 'Hixie' Hickson 2011-11-03 16:29:23 UTC
I can certainly put a warning in the spec.
Comment 35 contributor 2011-11-12 00:22:31 UTC
Checked in as WHATWG revision r6826.
Check-in comment: Add warnings about media elements and track and how to determine the type and how we don't know what hte solution is.
http://html5.org/tools/web-apps-tracker?from=6825&to=6826
Comment 36 Ian 'Hixie' Hickson 2011-11-12 00:30:26 UTC
I've put a warning in the spec. Marking this LATER for now; once browser vendors have converged on a behaviour or desired behaviour I'll revisit this. (See comment 32.)
Comment 37 Michael[tm] Smith 2013-01-24 07:19:47 UTC
This bug was cloned to create HTML WG bug 19036.
Comment 38 Ian 'Hixie' Hickson 2013-11-18 22:39:11 UTC
Ok well I guess my plan here is to test what browsers do, and spec whatever is the majority result when giving Mac Firefox trunk, Mac Safari latest release or WebKit if I have it, the most recent version of IE I can find, and Mac Chrome dev equal voting power, since it seems that there's no particular reason to believe anything else is going to change here.
Comment 39 Philip Jägenstedt 2013-11-19 06:11:19 UTC
If you have a personal preference and I happen to agree with it, I could always try to advocate for and implement it in Blink, to tilt the balance. If that feel like cheating, please proceed :)
Comment 40 Ian 'Hixie' Hickson 2013-11-19 21:45:14 UTC
I could argue a number of approaches, but I think pragmatism would favour defined sniffing based on explicitly listed content signatures.
Comment 41 Philip Jägenstedt 2013-11-20 18:06:33 UTC
Do you mean simply going with <http://mimesniff.spec.whatwg.org/#matching-an-audio-or-video-type-pattern>? What would be my preference, FWIW.
Comment 42 Gordon P. Hemsley 2013-11-20 18:33:49 UTC
(In reply to Philip Jägenstedt from comment #41)
> Do you mean simply going with
> <http://mimesniff.spec.whatwg.org/#matching-an-audio-or-video-type-pattern>?
> What would be my preference, FWIW.

Let me know if you need me to make any modifications there for you.

See also:
http://mimesniff.spec.whatwg.org/#rules-for-sniffing-audio-and-video-specifically
Comment 43 Ian 'Hixie' Hickson 2013-12-11 21:50:22 UTC
What's the status of mimesniff? It hasn't changed since June, but seems to have a lot of open issues, both in bugs and marked in the spec.
Comment 44 Ian 'Hixie' Hickson 2013-12-11 22:05:52 UTC
http://www.hixie.ch/tests/adhoc/html/video/001.html

Firefox supports WebM and Ogg, sniffs the MIME type if it's specified as application/octet-stream, and otherwise interprets the data as the given type without sniffing (so it gives a different error message for when the type doesn't match, e.g. an Ogg file labeled as WebM, and for when the type isn't supported, e.g. an Ogg file labeled as text/plain).

Chrome ignores MIME types entirely, supports WebM, H.264, and Ogg Theora.

Safari supports only H.264, but seems to entirely ignore the MIME type.

My notes say that IE used to entirely sniff. I don't have a modern IE around to test, unfortunately.

So, I guess we should just sniff.

I haven't checked how the browsers sniff.
Comment 45 Robert O'Callahan (Mozilla) 2013-12-11 22:19:03 UTC
FWIW I haven't seen much pressure to change our policy, but maybe I'm not aware of it. I'll see if I can get anyone else to comment.
Comment 46 Simon Pieters 2013-12-12 13:35:01 UTC
(In reply to Ian 'Hixie' Hickson from comment #44)
> My notes say that IE used to entirely sniff. I don't have a modern IE around
> to test, unfortunately.

See comment 6
Comment 47 Ian 'Hixie' Hickson 2014-01-02 21:02:12 UTC
Is comment 6 still accurate?

Having different browsers do different things is bad for the platform as a whole.

What's the value of having MIME types here? I honestly don't really understand why we would _not_ want to sniff here, assuming we did it consistently. It seems like a strictly superior experience for both users and authors.
Comment 48 Adrian Bateman [MSFT] 2014-01-04 02:38:45 UTC
(In reply to Ian 'Hixie' Hickson from comment #47)
> Is comment 6 still accurate?
> 
> Having different browsers do different things is bad for the platform as a
> whole.
> 
> What's the value of having MIME types here? I honestly don't really
> understand why we would _not_ want to sniff here, assuming we did it
> consistently. It seems like a strictly superior experience for both users
> and authors.

We're reviewing this topic again. I'll hopefully know more next week.

I think one problems is how feasible it is to sniff consistently. I recall that our architecture doesn't make this very convenient - if we didn't rely on the MIME type we would just pass the data stream to the media engine and let it try to play. I'm not sure we get the opportunity to efficiently intercept this and review the type of data in such a way that consistency would be easily possible. It's one of the questions I'll ask.

So I guess one question is whether it is better to ignore the MIME type but not necessarily consistently sniff the same way. After all, even if you agree on how to sniff you're not guaranteed to always be able to play the same content.
Comment 49 Ian 'Hixie' Hickson 2014-01-04 21:58:43 UTC
We should definitely sniff the same way, if we sniff.

If you can interpret the type, why couldn't you interpret the first few bytes? They're only a few bytes later on the same stream.
Comment 50 Adrian Bateman [MSFT] 2014-01-14 15:50:55 UTC
(In reply to Ian 'Hixie' Hickson from comment #49)
> We should definitely sniff the same way, if we sniff.
> 
> If you can interpret the type, why couldn't you interpret the first few
> bytes? They're only a few bytes later on the same stream.

Why? What are the consequences to not sniffing the same way? As far as I can tell, one outcome would be that legitimate content might not be detected as legitimate in some browsers if they don't sniff the same way. That's a quality of implementation problem for the browsers that chose not to play playable content. Another is that a browser thinks content is a specific content type but isn't and fails in some way. Again another quality of implementation problem.

IE doesn't do the media processing - we rely on Windows' media engine to do that. We can tell it to be strict (in which case it checks the content type and only tries to play with a decoder pipeline matching the type) or we can tell it to be lax (in which case it will iterate through the allowed decoder list and just try to play each one). We don't get an opportunity to pull out data from the stream and then push it back. Of course, we could talk to the Windows team that owns this code and ask for a change but if I am right about the consequences above then I'm not sure it is worth the additional investment.
Comment 51 Ian 'Hixie' Hickson 2014-01-14 19:09:34 UTC
What you call a "quality of implementation" issue is what I would call an "interoperability" issue. :-) The consequences are that something that works ok in one browser doesn't work ok in another.

Do you have access to the stream at all? For example, can you read the first few bytes, compare it to a table, override the Content-Type accordingly, and then pass it to the media system asking it to be strict?
Comment 52 Adrian Bateman [MSFT] 2014-01-14 19:16:56 UTC
(In reply to Ian 'Hixie' Hickson from comment #51)
> What you call a "quality of implementation" issue is what I would call an
> "interoperability" issue. :-) The consequences are that something that works
> ok in one browser doesn't work ok in another.

Except that you're only talking about being interoperably broken. There are plenty of ways in which a browser might have a bug where it can't play content that is playable elsewhere. There's little value in sharing bugs in content recognition but not in content playback (actually there's little value in sharing bugs - we should each just fix them according to our prioritization strategies).

> Do you have access to the stream at all? For example, can you read the first
> few bytes, compare it to a table, override the Content-Type accordingly, and
> then pass it to the media system asking it to be strict?

No, we don't handle the stream. The request is made from the media engine. It goes through the browser network stack but it would be a layering violation to add the sniffing there.
Comment 53 Yuhong Bao 2014-01-15 03:48:46 UTC
I think the API involved is called Media Foundation. It is extensible and there is already a WebM codec for it.
Comment 54 Ian 'Hixie' Hickson 2014-01-15 22:26:06 UTC
Adrian: We shouldn't have any bugs. I think we're agreed on this. Just like decoding should be reliable everywhere, recognising the stream should be reliable everywhere.

I'm not sure why you consider it a layering violation. It's exactly correctly layered: the network layer would be marking the stream as being the type that the specs say it should be treated as. This seems quite appropriate.

However, if what you're saying is that you won't change, then let's turn to foolip. Is there any change Blink could get fixed to follow the Content-Type header and not sniff?
Comment 55 Adrian Bateman [MSFT] 2014-01-15 23:06:13 UTC
(In reply to Ian 'Hixie' Hickson from comment #54)
> However, if what you're saying is that you won't change, then let's turn to
> foolip. Is there any change Blink could get fixed to follow the Content-Type
> header and not sniff?

I'm not saying we won't change - we're investigating changing to the lax approach where we just try to play. I don't have an answer yet - working through the conversations. If there are no bugs in the sniffing "algorithm" then this wouldn't be meaningfully different from this approach.
Comment 56 Ian 'Hixie' Hickson 2014-01-16 18:10:15 UTC
I'm not sure why the scare quotes around "algorithm", it is one:

   http://mimesniff.spec.whatwg.org/#matching-an-audio-or-video-type-pattern

The bigger problem might be in the parsing algorithms leaving error handling undefined (maybe that's where the scare quotes should be!).

I don't think just trying each decoder in turn until one doesn't give up is going to get us the level of interoperability that we should strive for here, but it's certainly closer than some browsers honouring Content-Type and others not.
Comment 57 Philip Jägenstedt 2014-01-21 03:41:58 UTC
(In reply to Ian 'Hixie' Hickson from comment #47)
> Is comment 6 still accurate?

I just tested serving an MPEG-4 file to IE as video/mp4 and application/pdf. It refused to play application/pdf.
Comment 58 Philip Jägenstedt 2014-01-21 03:42:26 UTC
(In reply to Philip Jägenstedt from comment #57)
> (In reply to Ian 'Hixie' Hickson from comment #47)
> > Is comment 6 still accurate?
> 
> I just tested serving an MPEG-4 file to IE as video/mp4 and application/pdf.
> It refused to play application/pdf.

IE11 on Win7 that is.
Comment 59 Philip Jägenstedt 2014-01-21 03:45:18 UTC
(In reply to Ian 'Hixie' Hickson from comment #54)
> Adrian: We shouldn't have any bugs. I think we're agreed on this. Just like
> decoding should be reliable everywhere, recognising the stream should be
> reliable everywhere.
> 
> I'm not sure why you consider it a layering violation. It's exactly
> correctly layered: the network layer would be marking the stream as being
> the type that the specs say it should be treated as. This seems quite
> appropriate.
> 
> However, if what you're saying is that you won't change, then let's turn to
> foolip. Is there any change Blink could get fixed to follow the Content-Type
> header and not sniff?

This is actually handled in the Chromium code and not in Blink, and unfortunately some platforms (Android) use a separate network stack for media, which makes the situation similar to what IE has with Media Foundation. However, I believe the long term goal is to at least use the same network stack on all platforms. Since sniffing is just rewriting the Content-Type, regardless of the outcome of this spec issue what one needs for interop is the ability to say to the media engine "here is the data, try to decode it as this type."

As for changing to honor Content-Type, I don't think it would be very responsible of me to champion this. In Presto I ended up removing the Content-Type check because we had customers who (rightfully) thought it was annoying that their content wouldn't play, and I don't see why they would be any less annoyed today.

However, I'm still new in Blink/Chromium, I'll post on blink-dev to see if anyone has thoughts on this.
Comment 60 Adrian Bateman [MSFT] 2014-04-25 21:11:37 UTC
(In reply to Philip Jägenstedt from comment #59)
> As for changing to honor Content-Type, I don't think it would be very
> responsible of me to champion this. In Presto I ended up removing the
> Content-Type check because we had customers who (rightfully) thought it was
> annoying that their content wouldn't play, and I don't see why they would be
> any less annoyed today.
> 
> However, I'm still new in Blink/Chromium, I'll post on blink-dev to see if
> anyone has thoughts on this.

We just shipped an update to IE11 (v11.0.7) through Windows Update that changes IE behaviour to ignore the Content-Type and always try to play the content. We pass content to the media engine with only the allowed decoders enabled and if the content can play then it does. We think this makes IE interoperable with the Chromium approach.
Comment 61 Philip Jägenstedt 2014-04-25 22:56:45 UTC
Thanks for the update, Adrian, that's very encouraging.

I see that I forgot to link to the blink-dev discussion in January, here it is:
https://groups.google.com/a/chromium.org/d/msg/blink-dev/25LOE2ZxkRk/-V391eLCb_QJ
Comment 62 Ian 'Hixie' Hickson 2014-05-07 18:55:37 UTC
So is Firefox the only browser now that doesn't do some form of sniffing?
Comment 63 Adrian Bateman [MSFT] 2014-05-07 18:58:46 UTC
(In reply to Ian 'Hixie' Hickson from comment #62)
> So is Firefox the only browser now that doesn't do some form of sniffing?

I think Firefox follows the spec and sniffs for application/octet-stream. We considered doing that decided if we were going to sniff in this case we should always do so unless the site sends a nosniff directive.
Comment 64 Ian 'Hixie' Hickson 2014-05-09 17:38:58 UTC
The spec right now doesn't say what to do one way or the other.

If we're down to just one browser not doing any sniffing, then I guess I'll update the spec to say to identify files per the MIME sniff spec and ignore Content-Type.
Comment 65 Ian 'Hixie' Hickson 2014-08-01 20:45:43 UTC
Adrian: Can you confirm comment 60? miketaylr posted some screenshots that contradict what you wrote there:

   https://cloudup.com/cdicl-Y0S0h
   https://cloudup.com/cTF_o3DUg-8
Comment 66 contributor 2014-08-01 20:56:33 UTC
Checked in as WHATWG revision r8696.
Check-in comment: Make <video> and <audio> sniff for their media data's MIME type, since most browsers seem to be going that way, and we need to pick a direction.
http://html5.org/tools/web-apps-tracker?from=8695&to=8696
Comment 67 Adrian Bateman [MSFT] 2014-08-01 21:17:01 UTC
(In reply to Ian 'Hixie' Hickson from comment #65)
> Adrian: Can you confirm comment 60? miketaylr posted some screenshots that
> contradict what you wrote there:
> 
>    https://cloudup.com/cdicl-Y0S0h
>    https://cloudup.com/cTF_o3DUg-8

Here is the result from current IE release builds:
http://pages.adrianba.net/w3c/ie11-media-content-type.png

I'm not sure why Mike isn't seeing this - I think the change was in v11.0.7.
Comment 68 Ian 'Hixie' Hickson 2014-08-04 18:18:22 UTC
Cool, thanks. In that case, I'm closing this bug. Firefox is the only browser left that honours Content-Type headers here.