(Duplicate of https://www.w3.org/Bugs/Public/show_bug.cgi?id=27249)

(Apologies for the length and possibly poor nomenclature of the below.)

Currently if you use an HTMLMediaElement:
    1) The browser automatically determines the file types / codecs of the sources and chooses the “best” source to play.
    2) The browser automatically parses the metadata of the chosen source in order to enable abilities like seeking, get the duration, etc.
    3) The browser implements some version of source buffering, but offers you know control over it. Not even at the very least starting and stopping buffering, and at most, even doing things like setting how much to buffer ahead, etc.
    4) The browser exposes *none* of this known metadata (file type, codec, file structure for seeking, bitrate, etc.) with the exception of the height, width, and duration.

If you use MSE:
    1) The browser expects you to know ahead of time if the data you’re putting into a SourceBuffer is decodable by the browser.
    2) The browser expects you to know the metadata associated with the source in order to set the duration and implement seeking *yourself*.
    3) The browser implements no buffering.

Those are your only two options. If you want to be able to do something as simple as starting or stopping buffering, you need to know all the associated metadata for your file (file type, codec, stco atom in the case of QT containers, height, width, duration, perhaps more). Your options are either to use a prepared manifest of all that required data, which is only possible if you know what content you’re to play ahead of time, or determine all that information dynamically, for which as a developer you’re now forced to use far less performant javascript, *despite the fact that the browser already implements fast compiled functions to do all this work*.


This “all or nothing” approach permeates both specs. As one example, HTMLMediaElement will tell you if it can play a source with canPlayType(), which requires a string of the format “filetype; codecs”. However it provides no way to *get* the format of a file, despite the fact that it can quite clearly determine this information (how else does it find the “best” source to play?).

Another example is that readyState change events, though fired based on buffer data, contain no snapshot of the TimeRanges buffered, despite the fact that those TimeRanges can and will change in time between the event being triggered and an event listener running and iterating through HTMLMediaElement.buffered.

MSE seems to have less of these problems, though an obvious one is the requirement for a type string with addSourceBuffer(), despite—again—no way to *get* the type of a buffer/file.


This honestly seems fairly difficult to fix since it’s a bug in the thinking behind the implementations and not just some aspect of the implementations themselves. Moreover, it requires more to be changed in the HTMLMediaElement spec than MSE’s. But I do think it’s very important that these shortcomings are addressed.

At the very least allowing greater control over the buffer (whereby “greater” I just mean the ability to start and stop the buffer, and *perhaps* control its length) would allow most developers to stick with HTMLMediaElement as opposed to being forced to use the more complex MSE.

At the most there needs to be a far greater degree of transparency in all the work the browser does regarding media. Metadata, file type, codec information, etc. should *all* be exposed to the developer, along with functions for determining them from an arbitrary SourceBuffer/ArrayBuffer/Blob/etc. The aim should be to ensure that the handling of media can continue to be dynamic (e.g. no manifests) and performant (e.g. no javascript based container parsing to get metadata) regardless of where along the continuum of HTMLMediaElement <-> MSE the developer chooses to reside.