16:00:40 <RRSAgent> RRSAgent has joined #webrtc
16:00:44 <RRSAgent> logging to https://www.w3.org/2024/11/19-webrtc-irc
16:00:44 <Zakim> Zakim has joined #webrtc
16:00:44 <dom> RRSAgent, make log public
16:00:45 <dom> Meeting: WebRTC November 19 2024 meeting
16:00:45 <dom> Agenda: https://www.w3.org/2011/04/webrtc/wiki/November_19_2024
16:00:45 <dom> Slideset: https://docs.google.com/presentation/d/1qFWqfbOVv3mHlYCBtnmXtLdpMkk-pXf--cGoEQBCyi4/
16:00:45 <dom> Chairs: Guido, Jan-Ivar, Bernard
16:01:57 <dom> Present+ Dom, Bernard, Florent, Guido, Harald, Henrik, TimP, Youenn
16:03:57 <dom> Recording is starting
16:04:05 <dom> Present+ Carine
16:06:55 <dom> Topic: -> https://github.com/w3c/mediacapture-transform Timing Control in MediaStreamTrack Insertable Media Processing using Streams
16:06:55 <dom> Subtoic: Issue #80: Expectations/Requirements for VideoFrame and AudioData timestamps
16:06:55 <dom> [slide 11]
16:07:00 <dom> s/toic/topic/
16:09:30 <dom> Jan-Ivar: what we want to do is to write tests to see how browsers proceed today, and ideally prescribe that
16:09:48 <dom> Bernard: but where? in HTML video element?
16:10:13 <dom> HTA: I think it'd belong there; what do video elements do when they get the same timestamps?
16:10:42 <dom> Youenn: each synk is supposed to define how it handles videoframes
16:11:01 <dom> .... In Safari, we use the time at which the frame at which it is committed the sink
16:11:15 <dom> ... webrtc-pc should define what it does when sending frames
16:11:29 <dom> ... the HTML video element should define it, although we could do in mediacapture-main if it's easier
16:11:32 <dom> s/synk/sink
16:11:55 <dom> Bernard: that's a good point - the track could go to a peerconnection where ti would be weird to have these duplicate timestamps
16:12:09 <dom> Youenn: we need to test, but I suspect they're using the time at which frames are submitted
16:12:22 <dom> Guido: +1 to youenn
16:13:43 <dom> ... this would also mean specifying for mediatrackrecorder, mediatrackprocessor
16:13:59 <dom> youenn: let's make sure to file issues for all these specs
16:15:18 <dom> PROPOSED: Add to mediacapturem-main extensiblity to make sure sink define their behavior on frame timestamps and file issues on sink specs accordingly
16:15:45 <dom> TimP: if you were to feed a track with null timestamps to a video element, would it be different if it had gone through a transform? that would be weird
16:15:53 <dom> s/a transform/a null transform/
16:16:24 <dom> hta: a null transform would do nothing
16:16:56 <dom> TimP: right; but an identity transform shouldn't change how the track gets rendered
16:18:27 <dom> dom: as long as we're defining at the sink, this means the processing pipeline shouldn't change the outcome
16:18:46 <dom> youenn: it may be that we can define a default behavior for sinks (likely, changing the timestamps would have no effect)
16:19:03 <dom> bernard: that's my hope too; PC is an interesting case
16:20:13 <dom> Jan-Ivar: in the case of a transform, a Processor WOULD be a sink - so we should be careful we keep the invariant of identity
16:20:29 <dom> RESOLVED: Add to mediacapture-main extensiblity consideration to make sure sink define their behavior on frame timestamps and file issues on sink specs accordingly
16:20:42 <dom> Subtopic: Issue #114: VideoTrackGenerator/MediaStream should buffer the current frame
16:20:42 <dom> [slide 12]
16:21:50 <dom> Youenn: +1 on not buffering; I'm surprised that captureStream() is different
16:22:13 <dom> ... we should check if there isn't a misunderstanding on the behavior of captureStream()
16:22:51 <dom> Jan-Ivar: it would be nice if we could keep them consistent, although I understand the cost of buffering
16:23:19 <dom> Youenn: but I don't think it's even buffering, it's only a timing question
16:24:21 <dom> Henrik: clearly we shouldn't buffering in the traditional sense; but in terms of remembering the last frame, in the case of a variable framerate track, is delivery guaranteed?
16:24:28 <dom> Bernard: that may deserve another issue
16:24:37 <dom> Henrik: +1 on this being the expected behavior
16:25:22 <dom> HTA: the <canvas> element exists no matter, so a frame can be generated at any time; I think this is expected behavior. Seeing if we can align the polyfill with this, that would be best and informative
16:25:56 <dom> RESOLVED: Not buffering the last frame is the expected behavior
16:26:24 <dom> Guido: we can recommend to the developer that they can buffer the frame in the application and send it to the generator after the track is associated to the element
16:27:01 <dom> Youenn: note that this problem only occurs when using VTG in the main thread
16:27:34 <dom> Present+ PatrickRockhill
16:27:45 <dom> Subtopic: Issue #20: Add “real-time” warning/note to `MediaStreamTrackGenerator`
16:27:45 <dom> [slide 13]
16:30:04 <dom> Youenn: with VTG's writable stream, the promise can be awaited as a cue to when a new frame can be submitted
16:30:09 <dom> s/can/could/
16:30:18 <dom> ... not sure if it's needed or still web-compatible
16:30:57 <dom> Bernard: what happens if you oversubmit? is it only they're not being displayed by the sink?
16:31:03 <dom> HTA: I think they would be thrown away
16:32:36 <dom> Jan-Ivar: a writable stream normally comes with a queue with a highwater mark; how should the VTG infers its framerate from the pace of frames it is being fed? Maybe we should define a max framerate over which it gives backpressure
16:32:56 <dom> ... or maybe it just drop frames over that rate if we're sticking to real-time behavior
16:33:40 <dom> ... the stream can either be push- or pull-based; for pull-, we need to determine where the framerate gets identified
16:34:20 <dom> youenn: the write-frame algorithm returns a promise, which resolves the writable write(), which gives a path for backpressure
16:34:47 <dom> ... at the moment, the promise always resolve at the end of the microtask, but we could change it to allow for backpressure
16:35:15 <dom> ... at the high level, I don't think VTG should be lossy, while sinks might be
16:35:20 <dom> Guido: +1 to youenn
16:35:41 <dom> ... if you don't wait for the promise, you have no guarantee that the frame will be processed
16:36:29 <dom> henrik: the promise helps with avoiding pushing too much data
16:36:53 <dom> ... but for lossy sinks, there will still be no way to make sure the frame got processed
16:38:19 <dom> dom: ideally, lossy sinks would be able to transfer backpressure via VTG
16:39:15 <dom> youenn: this would be possible for PC; possible for recorder; not sure about media elements, although in case of too many frames they should be able to at least render the last
16:39:45 <dom> ... I think the invariant is that VTG is not lossy, but some sinks will be lossy, and we should say that the last frame should win for renderers
16:40:49 <dom> jan-ivar: a writable always has a queue, even if the sink doesn't have one
16:41:06 <dom> ... we need to understand how the framerate gets determined for a pull-based stream
16:41:17 <dom> Bernard: this will depend on pre-recorded vs live
16:41:58 <dom> Subtopic: Issue #86: Playback and sync of tracks created by `VideoTrackGenerator`
16:41:58 <dom> [slide 14]
16:46:13 <dom> HTA: looking at the reported bug, the person reporting it had specific ideas on how the video element work, e.g. with playout points
16:46:53 <dom> Jan-Ivar: I don't know the answer, but writing tests would help determine if timestamp has any effect, in which case browsers are likely optimizing local playback
16:46:59 <dom> [slide 15]
16:47:02 <dom> [slide 16]
16:54:36 <dom> Youenn: Tracks are sent synchronized by the PC, send asked to render ASAP given the real-time context; likewise for local capture; I don't thnk timestamps are playing any role in it
16:54:36 <dom> Henrik: I thought presentationTimestamp was being used in Chrome?
16:54:36 <dom> Jan-Ivar: that should be testable
16:54:41 <dom> Topic: -> https://github.com/w3c/webrtc-pc WebRTC API
16:54:41 <dom> Subtopic: -> https://github.com/w3c/webrtc-extensions/issues/229 Issue 229: receiver.hardwareAcceleration attribute instead of disableHardware[En|De]coding() static methods Issue 229: receiver.hardwareAcceleration attribute instead of disableHardware[En|De]coding() static methods
16:54:41 <dom> [slide 20]
16:54:41 <dom> Youenn: not sure whether it's worth exposing just that; there are proposals to expose webcodecs data to the PC
16:54:43 <dom> ... I'm not sure if the implementation will be able to determine if switching to software or hardware is adapted given impact on latency, drops given the opacity of the PC flow; this is much different from the WebCodecs context where everything is happening under the control of the JS
16:55:47 <dom> ... I would prefer we spend more time on exposing WebCodecs rather than providing this hard-to-exploit toggle to developers
16:56:05 <dom> ... I'm not more in favor of the new API than the current one
16:56:55 <dom> Bernard: the original motivation was to turn-off hardware acceleration to avoid encoder bugs
16:57:22 <dom> ... this new approach gives control which as Youenn describes wouldn't be operational
16:57:51 <dom> Henrik: I think this new proposal is best because it allows to have the remote party signal to the specific client they should disable their hardware encoder
16:58:23 <dom> HTA: I recall as Bernard that this was a toggle to remove hardware encoder on detected failure pattern
16:58:39 <dom> ... this new proposal is more flexible; the "preferSoftware" might be too soft for the original purpose though
16:59:12 <dom> [slide 21]
17:00:06 <dom> Jan-Ivar: one of the reason of making it softer is to allow UA to override the hint when e.g. they've been blocklisted due to issues in earlier versions of the browser
17:00:38 <dom> HTA: hearing two proposals: Youenn suggest deleting the old API altogether; the other is to add this new API
17:01:11 <dom> Bernard: with encoded sources, we're heading towards WebCodec control which will give you the granular control if you need it
17:01:30 <dom> Youenn: +1 to Bernard
17:02:21 <dom> ... encoded sources bring that to the encoder side; we don't have the equivalent on decoding yet, but would be a reasonable direction to explore
17:02:47 <dom> Henrik: how about my suggestion to rely on the remote corruption metrics?
17:03:17 <dom> Youenn: you could hot-download a WASM decoder when you detect that situation
17:03:47 <dom> Jan-Ivar: not hearing a lot of consensus on the proposal; I expect Mozilla's position will return to negative to the existing API
17:04:02 <dom> Subtopic: Issue #3017: Reconsider replaceTrack's blocking on the operations queue
17:04:02 <dom> [slide 22]
17:06:33 <dom> HTA: +1 after putting more thoughts on this
17:06:56 <dom> Youenn: sounds fine, and helps getting a bit faster with limited compat issue
17:07:02 <dom> [Henrik: +1 on reaction]
17:07:18 <dom> RESOLVED: move with forward with PR to align spec with Chrome behavior
17:07:37 <dom> Subtopic: Issue #3022: Clarify order of events when the SCTP transport is closed
17:07:37 <dom> [slide 23]
17:10:04 <dom> [slide 24]
17:10:46 <dom> Youenn: I think we should enqueue a task per event; in terms of order for data channels, I'm not sure if there an API with an order - there used to be a getDataChannels() ordering which we could consider reusing
17:11:10 <dom> ... re closed transports, my intuition we should fire on the SCTP transport first, but no concrete reasoning
17:11:35 <dom> Jan-Ivar: we could use the orders of the internal slot, although not sure how that intersects with datachannels that are transfered
17:12:37 <dom> Florent: the sctpReason property gives indication about what caused the closure of the channel
17:13:21 <dom> TimP: I could live with getting notifications only at the transport level when the full association goes down
17:13:49 <dom> Jan-Ivar: if we fire first on SCTP, then you would know why all the channels close subsequently
17:14:20 <dom> TimP: right; in particular, you don't want to just re-open the channels if the transport itself has failed
17:14:47 <dom> Youenn: I think it's useful to still emit the close events on channels in particular when they've been transfered to a worker
17:15:00 <dom> HTA: note that there is also an error event fired
17:15:41 <dom> RESOLVED: Proceed with PR to fire sctp closed on transport object and then on individual channels
17:15:57 <dom> Subtopic: -> https://github.com/w3c/webrtc-extensions/issues/230 Add event for CSRC change
17:15:57 <dom> [slide 30]
17:17:20 <dom> [slide 31]
17:18:16 <dom> Bernard: the original use case wasn't to have extensive polling; there was also no expectations to manage as large meetings as is the case today
17:18:33 <dom> ... I think the main use case is for audio rather than video to surface where sound is coming from
17:18:47 <dom> ... there was concerned about the volume of events that could be fired from this as well
17:18:57 <dom> Henrik: this would be only when the source changes, so it shouldn't a lot
17:19:28 <dom> Jan-Ivar: my concern is the relationship to media flowing, and the linkage to the mute event which is already being fired
17:19:54 <dom> Henrik: I asked and they indicated they're interested in source information
17:20:13 <dom> Jan-Ivar: but I'm concerned in introducing an API that would cement the interop gap we have today
17:20:48 <dom> Jan-Ivar: this could be expressed in the standard's position; I'm trying to make sure we distinguish implementation issues from use case discussion
17:21:26 <dom> Youenn: the idea is that when a CSRC or SSRC change is detected, a task would be queued to fire an event with the changed value?
17:21:32 <dom> ... if so, seems OK with me
17:22:03 <dom> Jan-Ivar: I'm not sure if the csrc/ssrc names are evocative enough for the use case
17:22:21 <dom> Henrik: this matches the existing names of the getters
17:23:10 <dom> Bernard: for the video, would this be coming from the video compositor? or if the mixer mixes audio from different sources?
17:23:33 <dom> Henrik: it would probably send an array with the various sources?
17:24:21 <dom> HTA: we know there are apps that do this today via polling; what we're looking for is exposing this in a more ergonomic and efficient way
17:24:45 <dom> Jan-Ivar: re the use case - polling isn't necessarily bad
17:25:03 <dom> Henrik: you would still need polling e.g. to animate an audio volume
17:25:35 <dom> ... but having information asap on a change to e.g. switch the main video is time critical
17:26:18 <dom> ... we've received several reports where this polling is creating glitches and impacts performance
17:26:58 <dom> Bernard: it's kind of a new use case from the original one that was focused on audio - but now this also extends to video, is that common now?
17:27:10 <dom> Henrik: I know at least of a Google product doing this
17:27:25 <dom> Youenn: this aligns with usual Web expectations in terms of signaling changes to states
17:28:13 <dom> Jan-Ivar: my support is tentative, given my concern about the lack of interop of mute event
17:28:27 <dom> Henrik: we can make that clear in our intent to ship for Chromium
17:29:13 <dom> Youenn: implementing this new API could help improve implementations of the mute event; for Safari, we would like to align with spec and I think this would help
17:30:28 <dom> Henrik: fixing onunmute along with implementing this makes a lot of sense to me
17:31:07 <dom> ... onunmute makes sense regardless since not all apps used virtual ssrcs
17:31:25 <dom> ... the lack of support has mostly been a question of prioritization
17:32:07 <dom> Guido: a lot of the mute issues were linked to the ambiguity in the mediacapture spec that crept in multiple specs; this is something we want to fix
17:32:41 <dom> ... see e.g. the proposal I made to expose additional stats to help with transition
17:33:49 <dom> Jan-Ivar: onunmute also useful as trigger for gathering stats
17:34:17 <dom> RESOLVED: Rough consensus on moving forward with events for csrc/ssrc changes, with concerns noted on related onunmute interop
17:34:23 <dom> RRSAgent, draft minutes
17:34:24 <RRSAgent> I have made the request to generate https://www.w3.org/2024/11/19-webrtc-minutes.html dom
17:34:46 <dom> Subtopic: Issue #3020: Codec negotiation and level IDs
17:34:46 <dom> [slide 32]
17:34:51 <dom> RRSAgent, make log public
17:35:15 <dom> [slide 33]
17:36:05 <dom> [slide 34]
17:37:42 <dom> Bernard: you said before negotiation, 180 throws because you have no codec; this relates to what constitutes the media format
17:37:59 <dom> ... for H265, it's profile-id and tier only, so setting level-id to 180 should not throw
17:38:17 <dom> Henrik: I think the current spec says "equals" which could be clarified
17:38:39 <dom> ... but even with that, there is still another issue that my next slide details
17:38:52 <dom> Jan-Ivar: I think it's OK for the code to fail before negotiation
17:39:00 <dom> Henrik: you're in proposal B camp then :)
17:39:04 <dom> [slide 35]
17:41:38 <dom> Henrik: I personally prefer proposal A :)
17:41:40 <dom> Bernard: me too
17:42:05 <dom> Jan-Ivar: A makes sense after hearing your argument that it could produce unexpected results in prod
17:42:49 <dom> HTA: proposal A, and we should document that when you can't negotiate the codec, it should throw
17:42:53 <dom> RESOLVED: Proposal A
17:43:14 <dom> Repository: https://github.com/w3c/webrtc-pc
17:43:14 <dom> Subtopic: Remote channelCount + Stereo Opus #3010 #3011
17:43:14 <dom> [slide 37]
17:43:22 <dom> s/37/36
17:44:10 <dom> [slide 37]
17:45:24 <dom> Jan-Ivar: re how to test it, with Web Audio
17:45:39 <dom> Henrik: still I think it would better to add it for symetry
17:46:09 <dom> Jan-Ivar: adding it seems OK to me
17:46:26 <dom> RESOLVED: proceed with adding .channelCount on remote tracks
17:46:32 <dom> [slide 38]
17:47:23 <dom> i/Subtopic:/[Youenn departs]
17:47:33 <dom> [slide 39]
17:49:53 <dom> TimP: I'm almost sure that the SDP for Opus is always stereo
17:50:10 <dom> Henrik: I think the spec says it's always stereo capable; but the SDP parameter only expresses a preference
17:50:28 <dom> TimP: but Opus on the wire I think will always be stereo
17:50:47 <dom> Henrik: I think it can be mono on the wire (since I understood it can impact bitrate)
17:51:26 <dom> Jan-Ivar: it's hard to detect a mono signal from a stereo; having that exposed in channelCount would be useful
17:51:42 <dom> ... re channelCount, would this trigger onconfigurationchange?
17:51:49 <dom> Henrik: I haven't thought about it
17:52:21 <dom> s/stereo/stereo track/
17:53:15 <dom> HTA: I talked with Jean-Marc about mono & stereo; if you know a signal is mono you can encode/decode with less CPU power, which is good
17:53:59 <dom> ... re testing, generate a track with different signals (e.g. a waveform on one channel and silence on the other) and see whether that's preserved
17:55:45 <dom> Jan-Ivar: re proposal 2, "MAY" is not great
17:56:05 <dom> Henrik: I suspect implementations would align with what libwebrtc does
17:56:13 <dom> Jan-Ivar: let's confirm it and standardize that
17:56:41 <dom> ... if the libwebrtc behavior isn't satisfactory, we can revisit that
17:56:57 <dom> RRSAgent, draft minutes
17:56:58 <RRSAgent> I have made the request to generate https://www.w3.org/2024/11/19-webrtc-minutes.html dom
19:30:11 <Zakim> Zakim has left #webrtc