Media WG Teleconference

14 September 2021


Bernard Aboba, Chris Cunningham, Chris Needham, Cyril Concolato, Eric Carlson, Francois Daoust, Gary Katsevman, Greg Freedman, Jan-Ivar Bruaroey, Jer Noble, Joey Parrish, Matt Wolenetz, Peng Liu, Zachary Cava
Chris, Jer

Meeting minutes

A couple of media-related joint meetings during TPAC

TPAC 2021 Group meetings

cpn: focus of today MSE issues
… first AOB item: TPAC
… media ig planning meetings to assess future direction of web tech for consumer devices
… in collab w/ CTA WAVE and others
… date: 25 October, more details coming soon
… WG members invited to join

aboba: Also joint meeting w/ WebRTC group
… on 26 October

Media Source Extensions

cpn: today's focus, MSE issues
… thank you for cfc replies
… resolved: MSE in workers is ready to integrate

MSE for WebCodecs (#184)

Media Source Extensions for WebCodecs - Explainer

wolenetz: First issue, #184
… MSE for WebCodecs

wolenetz: looking for authors to try origin trial, and feedback on API
… first promise-based MSE API
… summary: MSE would take WebCodecs chunks rather than containerized bytestreams
… expecting interest from player libraries, hls.js, etc, who could use this to avoid re-muxing

MSE-in-Workers: {Audio,Video,Text}Track{,List} IDL in HTML need additional DedicatedWorker in Exposed (#280)

wolenetz: #280 how to expose HTMLMediaElement tracks in worker.
… currently, tracks interfaces exposed only in Window
… 1) do we need tracks in worker?
… 2) if yes, how to update html5 to go about it

cpn: changes to html should go to whatwg
… question: tracks exist both on HTMLMediaElement and SourceBuffer. What's the difference?

wolenetz: w/ SourceBuffer API, you can ask SB what tracks it specifically provides
… note: currently Chromium doesn't expose tracks even in Window

jer: Generally, I prefer MSE in worker to be as close to on main thread as possible. To facilitate easier adoption
… Probably not a big issue for HTML to expose tracks in Workers

Threading lifetimes (#277)

wolenetz: #277 what happens to <video> when worker is terminated
… do we remove the buffers from <video> that MSE-in-worker was contributing?
… or do we let <video> retain what was buffered
… also, should <video> get some notification that worker termination occurred
… currently, chrome's mse-in-workers drops buffers after worker dies, but not sure if it's right

jernoble: I would behave as if source object is removed from <video>. return to empty network state

wolenetz: or potentially trigger EOS behavior?

jernoble: that sounds more complicated. if source object disappears, most transparent thing is to remove the buffers
… could be detected by network state event
… similar to how <video> behaves w/ src set to empty string

cyril: can things be downloaded, buffered, then wasted if worker dies?

wolenetz: yes that could occur, but generally we expect the app to be driving the termination, so just a question about the short period of time following that

cyril: fine to remove buffers if it's just for a short period.

jernoble: "terminating" the worker would be caused by explicit terminate(), which in theory is only done by the client of the worker

cyril: can the app wait to terminate until it's used all the buffer

wolenetz: yes

cpn: what do we need to spec from interop pov

wolenetz: I like Jer's suggestion to detach as much as possible.

cpn: can you reuse media element by setting a new src / srcObject?

wolenetz: yes

Timing of initial HAVE_METADATA transition (#275)

wolenetz: Regarding #275, currently, HAVE_METADATA is spec'ed to be fired synchronously w/ arrival of init segment
… not feasible with workers. any concerns to relax it to be async
… specifically, bottom of init segment received algorithm: when all SB's have init segment received = true
… set's readyState = HAVE_METADATA

jernoble: I think the time where init segment is processed is probably already arbitrary

wolenetz: so it's probably ok to do async here as well? not blocking the worker on synchronously setting readyState?

jernoble: I agree

janivar: might be good to gather data on current browser behavior

wolenetz: will do

jernoble: do we have a wpt test for this? could use that to track differences

wolenetz: test exists, but it's flaky

jernoble: for webkit, we issue have_metadata before updated event

Object URLs vs srcObject (#156)

wolenetz: Regarding #156, chromium regrettably doesn't support srcObject yet
… the original object URL was "autorevoking"
… Chrome changed behavior to let apps explicitly revoke to improve GC ability to collect
… for MSE-in-workers, we need to keep object URLs alive long enough to be used by worker (can't immediately revoke).
… proposal: for workers, what if we don't revoke until after first use.

jernoble: I initially favored keeping the URL, but now I like srcObject. Still, usage of srcObject is very low because of missing support in other UAs

wolenetz: we haven't decided how src would work w/ MSE-in-workers yet. so far just looked at using srcObject

jernoble: for src url, we could maintain autorevoking behavior on the main thread while behaving differently for use in workers

<cpn> https://github.com/w3c/media-source/issues/28

janivar: I'm supportive of srcObject. Firefox hasn't implemented it yet.

jernoble: it seems simpler to create object in worker, do attachment on main thread, and have spec language say <video> informs the MSE-in-worker that they're now attached
… vs earlier proposals that considered creating src url in main thread

janivar: beware, as you "transfer" objects, the typical language is that the source-side is "detached"

wolenetz: Up next I'll be addressed Mark Watsons PR comments and taking a look at Spectre mitigations

cpn: to confirm, for workers, we've aligned on srcObject for attachment?

wolenetz: correct

jernoble: something to consider, should <source> elements have srcObject attribute?

Next meeting

cpn: scheduled for end of october. do we need anything sooner?

cyril: I'd like an earlier session to discuss bytestream issues

jer: how about same time in two weeks?

cyril: in the meantime, please add your thoughts to the GH issues

Minutes manually created (not a transcript), formatted by scribe.perl version 136 (Thu May 27 13:50:24 2021 UTC).