This is an archived snapshot of W3C's public bugzilla bug tracker, decommissioned in April 2019. Please see the home page for more details.

Bug 26060 - Media Source and Web Worker
Summary: Media Source and Web Worker
Status: RESOLVED WONTFIX
Alias: None
Product: HTML WG
Classification: Unclassified
Component: Media Source Extensions (show other bugs)
Version: unspecified
Hardware: All All
: P2 normal
Target Milestone: CR
Assignee: Aaron Colwell
QA Contact: HTML WG Bugzilla archive list
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-06-11 14:14 UTC by Saran Tunyasuvunakool
Modified: 2015-02-03 18:57 UTC (History)
7 users (show)

See Also:


Attachments

Description Saran Tunyasuvunakool 2014-06-11 14:14:51 UTC
One possible use case scenario for the Media Source Extension is to construct a live video stream. In an effort to keep latency low, one may decide to make the media segments very small but have them arrive very frequently. This has the potential to cause the main thread to block and become unresponsive.

In order to alleviate this, it would be useful to be able to create the MediaSource object in a Worker running in a separate thread, and send its Blob URI back to the parent script.

As an example, with still images it is currently possible to do the following with XMLHttpRequest. The same technique may also be useful in the MediaSource scenario using WebSockets (which are already available in Workers).

----------
main.js
----------
var worker = new Worker('worker.js');
var img;

window.onload = function (event) {
	img = document.getElementById('img');
	worker.postMessage('bh.jpg');
}

worker.onmessage = function (event) {
	img.src = event.data;
}
----------

----------
worker.js
----------
var xhr = new XMLHttpRequest();
var ms = new MediaSource();

onmessage = function (event) {
	xhr.open('GET', event.data, true);
	xhr.responseType = 'blob';
	xhr.send();
}

xhr.onload = function (event) {
	postMessage(URL.createObjectURL(xhr.response));
}
----------
Comment 1 Aaron Colwell 2014-06-11 16:01:37 UTC
I'm not sure this will provide the benefits you are hoping for. Append, remove, and duration change operations explicitly change JavaScript visible HTMLMediaElement state. Even if the calls on the MediaSource object were done in the worker, you'd still have to synchronize with the main thread to make sure the HTMLMediaElement properly reflects the current state of MediaSource object. 

It isn't clear to me what "blocking" behavior you are worried about. All append and remove operations are asynchronous so a UA is completely free to do the "real work" on a separate thread if it wants to. 

The example you give only defers the loading to a worker. The actual image parsing and display still happens on the main thread so I'd be very surprised if this actually results in a significant performance win. The main benefit I can see by doing this would be that you would only try to display the image once you know you have the whole thing instead of img tag taking up space when it only has the image partially loaded.
Comment 2 Saran Tunyasuvunakool 2014-06-17 14:00:58 UTC
From my understanding, the MediaSource object itself parses the byte stream in order to extract coded frames from its container (BMFF etc.), which is what actually gets appended to the buffer, i.e. the SourceBuffer only holds the elementary media stream(s). It is this step that I thought may benefit from offloading into a worker.

While the append operations are asynchronous, is it not the case that they are still queued to be executed on the main UI thread?

Since I am not involved in implementing a UA I have no idea how much work is involved in this "demuxing" stage relative to the overall append operation. If it is negligible then perhaps indeed there is not much point in enabling Worker support.
Comment 3 Aaron Colwell 2014-06-17 15:51:01 UTC
(In reply to Saran Tunyasuvunakool from comment #2)
> From my understanding, the MediaSource object itself parses the byte stream
> in order to extract coded frames from its container (BMFF etc.), which is
> what actually gets appended to the buffer, i.e. the SourceBuffer only holds
> the elementary media stream(s). It is this step that I thought may benefit
> from offloading into a worker.
> 
> While the append operations are asynchronous, is it not the case that they
> are still queued to be executed on the main UI thread?

There is nothing that requires the parsing to occur on the main thread. The asynchronous nature of the append allows the UA to choose the main thread or a completely different one. In either case it needs to convey the JavaScript visible state to the main thread before signalling that the operation has completed.

> 
> Since I am not involved in implementing a UA I have no idea how much work is
> involved in this "demuxing" stage relative to the overall append operation.
> If it is negligible then perhaps indeed there is not much point in enabling
> Worker support.

I don't believe it is worth adding this. I believe most UA's will choose to use a different thread for parsing if they deem it performance critical. The JavaScript application should not have to worry about this.
Comment 4 Aleksei Zbiniakov 2015-01-21 11:31:11 UTC
The one of the worth scenario is when you control your media stream via SharedWorker, so all open tabs for your site get the same isolated context. In this case you can keep just one MediaSource object and allow all open tabs to access it without additional memory consumption.
Comment 5 Aaron Colwell 2015-02-03 18:57:01 UTC
(In reply to Aleksei Zbiniakov from comment #4)
> The one of the worth scenario is when you control your media stream via
> SharedWorker, so all open tabs for your site get the same isolated context.
> In this case you can keep just one MediaSource object and allow all open
> tabs to access it without additional memory consumption.

I do not believe the scenario you propose here would be common enough to be worth the extra complexity this would introduce. Allowing multiple HTMLMediaElements to reference a single MediaSource object would unnecessarily complicate the buffer eviction algorithms because different elements could be playing at different positions in the buffer. I also think this would give developers the false sense that all tabs would render the same thing because they share a buffer, but that may not actually be true if overlapping appends and evictions are occurring.