Meeting minutes
Recording: https://
Slideset: https://
Timing Control in MediaStreamTrack Insertable Media Processing using Streams 🎞︎
Issue #80: Expectations/Requirements for VideoFrame and AudioData timestamps 🎞︎
Jan-Ivar: what we want to do is to write tests to see how browsers proceed today, and ideally prescribe that
Bernard: but where? in HTML video element?
HTA: I think it'd belong there; what do video elements do when they get the same timestamps?
Youenn: each sink is supposed to define how it handles videoframes
… In Safari, we use the time at which the frame at which it is committed the sink
… webrtc-pc should define what it does when sending frames
… the HTML video element should define it, although we could do in mediacapture-main if it's easier
Bernard: that's a good point - the track could go to a peerconnection where ti would be weird to have these duplicate timestamps
Youenn: we need to test, but I suspect they're using the time at which frames are submitted
Guido: +1 to youenn
… this would also mean specifying for mediatrackrecorder, mediatrackprocessor
youenn: let's make sure to file issues for all these specs
PROPOSED: Add to mediacapturem-main extensiblity to make sure sink define their behavior on frame timestamps and file issues on sink specs accordingly
TimP: if you were to feed a track with null timestamps to a video element, would it be different if it had gone through a null transform? that would be weird
hta: a null transform would do nothing
TimP: right; but an identity transform shouldn't change how the track gets rendered
dom: as long as we're defining at the sink, this means the processing pipeline shouldn't change the outcome
youenn: it may be that we can define a default behavior for sinks (likely, changing the timestamps would have no effect)
bernard: that's my hope too; PC is an interesting case
Jan-Ivar: in the case of a transform, a Processor WOULD be a sink - so we should be careful we keep the invariant of identity
RESOLUTION: Add to mediacapture-main extensiblity consideration to make sure sink define their behavior on frame timestamps and file issues on sink specs accordingly
Issue #114: VideoTrackGenerator/MediaStream should buffer the current frame 🎞︎
Youenn: +1 on not buffering; I'm surprised that captureStream() is different
… we should check if there isn't a misunderstanding on the behavior of captureStream()
Jan-Ivar: it would be nice if we could keep them consistent, although I understand the cost of buffering
Youenn: but I don't think it's even buffering, it's only a timing question
Henrik: clearly we shouldn't buffering in the traditional sense; but in terms of remembering the last frame, in the case of a variable framerate track, is delivery guaranteed?
Bernard: that may deserve another issue
Henrik: +1 on this being the expected behavior
HTA: the <canvas> element exists no matter, so a frame can be generated at any time; I think this is expected behavior. Seeing if we can align the polyfill with this, that would be best and informative
RESOLUTION: Not buffering the last frame is the expected behavior
Guido: we can recommend to the developer that they can buffer the frame in the application and send it to the generator after the track is associated to the element
Youenn: note that this problem only occurs when using VTG in the main thread
Issue #20: Add “real-time” warning/note to `MediaStreamTrackGenerator` 🎞︎
Youenn: with VTG's writable stream, the promise could be awaited as a cue to when a new frame can be submitted
… not sure if it's needed or still web-compatible
Bernard: what happens if you oversubmit? is it only they're not being displayed by the sink?
HTA: I think they would be thrown away
Jan-Ivar: a writable stream normally comes with a queue with a highwater mark; how should the VTG infers its framerate from the pace of frames it is being fed? Maybe we should define a max framerate over which it gives backpressure
… or maybe it just drop frames over that rate if we're sticking to real-time behavior
… the stream can either be push- or pull-based; for pull-, we need to determine where the framerate gets identified
youenn: the write-frame algorithm returns a promise, which resolves the writable write(), which gives a path for backpressure
… at the moment, the promise always resolve at the end of the microtask, but we could change it to allow for backpressure
… at the high level, I don't think VTG should be lossy, while sinks might be
Guido: +1 to youenn
… if you don't wait for the promise, you have no guarantee that the frame will be processed
henrik: the promise helps with avoiding pushing too much data
… but for lossy sinks, there will still be no way to make sure the frame got processed
dom: ideally, lossy sinks would be able to transfer backpressure via VTG
youenn: this would be possible for PC; possible for recorder; not sure about media elements, although in case of too many frames they should be able to at least render the last
… I think the invariant is that VTG is not lossy, but some sinks will be lossy, and we should say that the last frame should win for renderers
jan-ivar: a writable always has a queue, even if the sink doesn't have one
… we need to understand how the framerate gets determined for a pull-based stream
Bernard: this will depend on pre-recorded vs live
Issue #86: Playback and sync of tracks created by `VideoTrackGenerator` 🎞︎
HTA: looking at the reported bug, the person reporting it had specific ideas on how the video element work, e.g. with playout points
Jan-Ivar: I don't know the answer, but writing tests would help determine if timestamp has any effect, in which case browsers are likely optimizing local playback
Youenn: Tracks are sent synchronized by the PC, send asked to render ASAP given the real-time context; likewise for local capture; I don't thnk timestamps are playing any role in it
Henrik: I thought presentationTimestamp was being used in Chrome?
Jan-Ivar: that should be testable
WebRTC API 🎞︎
Issue 229: receiver.hardwareAcceleration attribute instead of disableHardware[En|De]coding() static methods 🎞︎
Youenn: not sure whether it's worth exposing just that; there are proposals to expose webcodecs data to the PC
… I'm not sure if the implementation will be able to determine if switching to software or hardware is adapted given impact on latency, drops given the opacity of the PC flow; this is much different from the WebCodecs context where everything is happening under the control of the JS
… I would prefer we spend more time on exposing WebCodecs rather than providing this hard-to-exploit toggle to developers
… I'm not more in favor of the new API than the current one
Bernard: the original motivation was to turn-off hardware acceleration to avoid encoder bugs
… this new approach gives control which as Youenn describes wouldn't be operational
Henrik: I think this new proposal is best because it allows to have the remote party signal to the specific client they should disable their hardware encoder
HTA: I recall as Bernard that this was a toggle to remove hardware encoder on detected failure pattern
… this new proposal is more flexible; the "preferSoftware" might be too soft for the original purpose though
Jan-Ivar: one of the reason of making it softer is to allow UA to override the hint when e.g. they've been blocklisted due to issues in earlier versions of the browser
HTA: hearing two proposals: Youenn suggest deleting the old API altogether; the other is to add this new API
Bernard: with encoded sources, we're heading towards WebCodec control which will give you the granular control if you need it
Youenn: +1 to Bernard
… encoded sources bring that to the encoder side; we don't have the equivalent on decoding yet, but would be a reasonable direction to explore
Henrik: how about my suggestion to rely on the remote corruption metrics?
Youenn: you could hot-download a WASM decoder when you detect that situation
Jan-Ivar: not hearing a lot of consensus on the proposal; I expect Mozilla's position will return to negative to the existing API
Repository: w3c/
Issue #3017: Reconsider replaceTrack's blocking on the operations queue 🎞︎
HTA: +1 after putting more thoughts on this
Youenn: sounds fine, and helps getting a bit faster with limited compat issue
[Henrik: +1 on reaction]
RESOLUTION: move with forward with PR to align spec with Chrome behavior
Issue #3022: Clarify order of events when the SCTP transport is closed 🎞︎
Youenn: I think we should enqueue a task per event; in terms of order for data channels, I'm not sure if there an API with an order - there used to be a getDataChannels() ordering which we could consider reusing
… re closed transports, my intuition we should fire on the SCTP transport first, but no concrete reasoning
Jan-Ivar: we could use the orders of the internal slot, although not sure how that intersects with datachannels that are transfered
Florent: the sctpReason property gives indication about what caused the closure of the channel
TimP: I could live with getting notifications only at the transport level when the full association goes down
Jan-Ivar: if we fire first on SCTP, then you would know why all the channels close subsequently
TimP: right; in particular, you don't want to just re-open the channels if the transport itself has failed
Youenn: I think it's useful to still emit the close events on channels in particular when they've been transfered to a worker
HTA: note that there is also an error event fired
RESOLUTION: Proceed with PR to fire sctp closed on transport object and then on individual channels
Add event for CSRC change 🎞︎
Bernard: the original use case wasn't to have extensive polling; there was also no expectations to manage as large meetings as is the case today
… I think the main use case is for audio rather than video to surface where sound is coming from
… there was concerned about the volume of events that could be fired from this as well
Henrik: this would be only when the source changes, so it shouldn't a lot
Jan-Ivar: my concern is the relationship to media flowing, and the linkage to the mute event which is already being fired
Henrik: I asked and they indicated they're interested in source information
Jan-Ivar: but I'm concerned in introducing an API that would cement the interop gap we have today
Jan-Ivar: this could be expressed in the standard's position; I'm trying to make sure we distinguish implementation issues from use case discussion
Youenn: the idea is that when a CSRC or SSRC change is detected, a task would be queued to fire an event with the changed value?
… if so, seems OK with me
Jan-Ivar: I'm not sure if the csrc/ssrc names are evocative enough for the use case
Henrik: this matches the existing names of the getters
Bernard: for the video, would this be coming from the video compositor? or if the mixer mixes audio from different sources?
Henrik: it would probably send an array with the various sources?
HTA: we know there are apps that do this today via polling; what we're looking for is exposing this in a more ergonomic and efficient way
Jan-Ivar: re the use case - polling isn't necessarily bad
Henrik: you would still need polling e.g. to animate an audio volume
… but having information asap on a change to e.g. switch the main video is time critical
… we've received several reports where this polling is creating glitches and impacts performance
Bernard: it's kind of a new use case from the original one that was focused on audio - but now this also extends to video, is that common now?
Henrik: I know at least of a Google product doing this
Youenn: this aligns with usual Web expectations in terms of signaling changes to states
Jan-Ivar: my support is tentative, given my concern about the lack of interop of mute event
Henrik: we can make that clear in our intent to ship for Chromium
Youenn: implementing this new API could help improve implementations of the mute event; for Safari, we would like to align with spec and I think this would help
Henrik: fixing onunmute along with implementing this makes a lot of sense to me
… onunmute makes sense regardless since not all apps used virtual ssrcs
… the lack of support has mostly been a question of prioritization
Guido: a lot of the mute issues were linked to the ambiguity in the mediacapture spec that crept in multiple specs; this is something we want to fix
… see e.g. the proposal I made to expose additional stats to help with transition
Jan-Ivar: onunmute also useful as trigger for gathering stats
RESOLUTION: Rough consensus on moving forward with events for csrc/ssrc changes, with concerns noted on related onunmute interop
Repository: w3c/
Issue #3020: Codec negotiation and level IDs 🎞︎
Bernard: you said before negotiation, 180 throws because you have no codec; this relates to what constitutes the media format
… for H265, it's profile-id and tier only, so setting level-id to 180 should not throw
Henrik: I think the current spec says "equals" which could be clarified
… but even with that, there is still another issue that my next slide details
Jan-Ivar: I think it's OK for the code to fail before negotiation
Henrik: you're in proposal B camp then :)
Henrik: I personally prefer proposal A :)
Bernard: me too
Jan-Ivar: A makes sense after hearing your argument that it could produce unexpected results in prod
HTA: proposal A, and we should document that when you can't negotiate the codec, it should throw
RESOLUTION: Proposal A
Repository: w3c/
[Youenn departs]
Remote channelCount + Stereo Opus #3010 #3011 🎞︎
Jan-Ivar: re how to test it, with Web Audio
Henrik: still I think it would better to add it for symetry
Jan-Ivar: adding it seems OK to me
RESOLUTION: proceed with adding .channelCount on remote tracks
TimP: I'm almost sure that the SDP for Opus is always stereo
Henrik: I think the spec says it's always stereo capable; but the SDP parameter only expresses a preference
TimP: but Opus on the wire I think will always be stereo
Henrik: I think it can be mono on the wire (since I understood it can impact bitrate)
Jan-Ivar: it's hard to detect a mono signal from a stereo track; having that exposed in channelCount would be useful
… re channelCount, would this trigger onconfigurationchange?
Henrik: I haven't thought about it
HTA: I talked with Jean-Marc about mono & stereo; if you know a signal is mono you can encode/decode with less CPU power, which is good
… re testing, generate a track with different signals (e.g. a waveform on one channel and silence on the other) and see whether that's preserved
Jan-Ivar: re proposal 2, "MAY" is not great
Henrik: I suspect implementations would align with what libwebrtc does
Jan-Ivar: let's confirm it and standardize that
… if the libwebrtc behavior isn't satisfactory, we can revisit that