04:50:57 <RRSAgent> RRSAgent has joined #mediawg
04:51:01 <RRSAgent> logging to https://www.w3.org/2025/11/13-mediawg-irc
04:51:01 <cpn> present+ Chris_Needham
04:51:01 <Zakim> Zakim has joined #mediawg
04:51:11 <alwu> alwu has joined #mediawg
04:51:18 <mjwilson> mjwilson has joined #mediawg
04:51:34 <tidoust> Meeting: Media and WebRTC WG joint meeting
04:51:55 <eric-carlsone> eric-carlsone has joined #mediawg
04:52:20 <dom> Present+
04:52:21 <kota1> kota1 has joined #mediawg
04:52:22 <handellm> handellm has joined #mediawg
04:52:44 <dom> Present+ Guido, Youenn, Jan-Ivar, MarkusH, Paul, Marcos, SongXu, Francois
04:53:00 <alwu> present+ Alastor Wu
04:53:01 <dom> Present+ SunShin
04:53:14 <tidoust> Agenda: https://github.com/w3c/media-wg/wiki/TPAC-2025#-thursday-13-november
04:53:16 <tidoust> scribe+
04:53:19 <song> song has joined #mediawg
04:53:35 <tidoust> Topic: MSTP for Audio / AudioTrackGenerator
04:54:00 <dom> Slideset: https://docs.google.com/presentation/d/1sd5zEnvlXO5Sk3ENQorUUIQiRz65sv0KZKxDMMYHM3I/edit?
04:54:04 <dom> [slide 101]
04:54:16 <tidoust> Guido: Proposal to add audio support in the mediacapture-transform spec.
04:54:28 <tidoust> ... The spec allows working with raw media and MediaStreamTracks.
04:54:37 <tidoust> ... Given a track, access to raw media.
04:54:38 <eric-carlsone> present+ Eric Carlson
04:54:43 <tidoust> ... Given raw media, produce a track.
04:55:10 <tidoust> ... Raw media uses interfaces from WebCodecs, with consensus to use VideoFrame.
04:55:29 <dom> -> https://github.com/w3c/mediacapture-transform/ MediaCapture Transform repo
04:55:33 <dom> [slide 102]
04:55:33 <tidoust> ... No consensus yet for AudioDate for audio, but this is shipped in Chrome and I'm going to report experience there.
04:55:55 <dom> [slide 103]
04:56:01 <Lei_Zhao> present+
04:56:13 <tidoust> Guido: VideoTrackSource allows to produce a track, given frames.
04:56:34 <dom> Present+ Harald, ChrisBlume, PeterThatcher, XIaohanWang, KenMomatsu, LeiZhao, EricCarlson
04:56:34 <tidoust> [slide 104]
04:56:48 <tidoust> Guido: Both can be combined to produce a transformed track.
04:57:23 <nigel> nigel has joined #mediawg
04:57:28 <tidoust> [slide 105]
04:57:49 <tidoust> Guido: My proposal is to try to achieve consensus on audio.
04:58:18 <tidoust> ... We don't have consensus yet because it's kind of redundant with AudioWorklet, which allows you to do audio processing.
04:58:24 <dom> s|Topic:|Topic: -> https://github.com/w3c/mediacapture-transform/issues/29
04:58:42 <cpn> present+ Ken_Komatsu, Xiaohan_Wang, Peter_Thatcher, Chris_Blume, Markus_Handell, Nishitha_Dey, Eric_Carlson, Jan-Ivar_Bruaroey
04:59:04 <tidoust> ... Audio procssing sometimes requires a realtime thread all the way through, you will see glitches.
04:59:08 <tidoust> [slide 106]
04:59:34 <tidoust> Guido: There are good use cases where the transform thread is a good fit, for example if buffering is acceptable.
04:59:43 <tidoust> ... if there is no audio playback.
05:00:05 <tidoust> ... If the processing has high variance. You may want the processing to have very low CPU.
05:00:22 <tidoust> ... In the worker, if you use buffering, you may have tolerance for variance.
05:00:44 <tidoust> ... You may want to combine audio and video processing in the same worker, which you can do with WebCodecs.
05:01:05 <dom> Present+ GabrielBrito
05:01:19 <tidoust> ... If you use directly the data that comes from AudioDate, you can access directly the input metatada such as capture timestamps.
05:01:33 <tidoust> ... It also saves you a realtime thread, which is more expensive.
05:02:40 <tidoust> ... Some examples of applications include audio analysis, encoding (leaving the main thread untouched), or transforms with no playback, for example if you want to send the result to the network.
05:02:54 <tidoust> [slide 107]
05:03:37 <tidoust> Guido: Comparing using of stream track processor, generator, and audio worklet, in general (not audio specific).
05:03:43 <tidoust> ... Both are popular APIs.
05:03:48 <tidoust> [slide 108]
05:04:41 <tidoust> Guido: If you go to medicapture-transform usage, 50% of AudioWorklet, audio is ~27% of all Processor usage. Widely used, and is used mainly without connecting to a generator.
05:04:51 <tidoust> ... E.g., to compute metrics about the audio.
05:05:26 <tidoust> ... Generator usage for audio is ~2.6%, less used than Processor. Main use case is transformations that output only to the network.
05:06:07 <tidoust> ... For video, generator is used 47% more than the generator. For audio, it's the opposite, and hundreds of percents more.
05:06:11 <tidoust> [slide 109]
05:07:00 <tidoust> Guido: Processor for Audio is widely used. Generator has more limited use cases. It's not good for playback. The use cases are more restricted, but experience suggests that people understand that.
05:07:12 <dom> q?
05:07:14 <tidoust> ... My proposal is to add it to the spec, with guidance on when to use it and when not to use it.
05:07:49 <tidoust> Youenn: I would see a use case for live recording. Sending data through WebSockets for example. In many cases, they are using Media Recording, but not always.
05:08:11 <tidoust> ... Usage to generate PCM, which works but is a bit of an abuse.
05:08:26 <Sun> Sun has joined #mediawg
05:08:28 <tidoust> ... The other is [missed].
05:08:39 <tidoust> ... It would be interesting to do measurements.
05:08:58 <tidoust> ... For generator, I feel that the only use case is doing processing before encoding the data in peer connections.
05:08:58 <cpn> s/[missed]/that avoiding realtime threads would help/.
05:09:12 <handellm> q+
05:09:13 <tidoust> ... You have your microphone, doing some processing, and then sending over.
05:09:25 <tidoust> ... If you're using generator elsewhere, that's not very gret.
05:09:29 <tidoust> s/gret/great/
05:09:46 <tidoust> Markus: Recording audio is also a use case for generator.
05:09:48 <cpn> ack h
05:10:02 <tidoust> ... Doing realtime processing on the audio.
05:10:22 <tidoust> Youenn: If you have a peer connection approach.
05:10:22 <jib> q+
05:10:51 <tidoust> ... I will concentrate on a peerconnection approach for generator.
05:11:06 <tidoust> ... We're only talking about making things from CPU-friendly. We have some time.
05:11:23 <tidoust> Guido: It's a niche use case, but it's quite good for it.
05:11:29 <dom> q?
05:11:32 <dom> ack jib
05:11:34 <tidoust> Youenn: We should put priority on the Processor first.
05:11:55 <handellm> q+
05:12:00 <tidoust> jib: The whole thing can easily be shimmed over AudioWorklet.
05:12:24 <dom> q+ guido
05:12:40 <tidoust> ... The interop gap is more problematic.
05:12:50 <tidoust> ... We don't a lot of traction as a result.
05:12:59 <dom> ack handellm
05:13:06 <tidoust> Markus: I'm not sure that you can observe timestamps in the right way.
05:13:31 <tidoust> Youenn: That's something that we can look at in the AudioWorklet context.
05:13:47 <tidoust> Markus: But then you pass a high priority thread
05:13:57 <tidoust> Youenn: For the use cases considered, that seems reasonable.
05:15:01 <tidoust> Youenn: One issue with Generator is usage in contexts when it should not be used, and one implementation may be optimized, leading to applications that ship because they work well in one particular browser, but then not as well in others.
05:15:42 <dom> ack guido
05:15:44 <tidoust> jib: Even once we get implementations, I worry that it might negatively impact progress on web types.
05:16:20 <tidoust> Guido: We saw that people were smart enough to use the processor and not the generator, so people understand the constraints.
05:16:48 <tidoust> ... The argument that not supporting a very good use case because you can also use it for other use cases when it's less good is weak.
05:16:58 <tidoust> Youenn: I'm more thinking about it in terms of priorities.
05:17:15 <cpn> q?
05:17:32 <dom> q+
05:17:42 <tidoust> Guido: Re. spec compliance, as long as we fix the issues, we're happy to align.
05:18:01 <cpn> q+ jib
05:18:02 <jib> q+
05:18:14 <tidoust> dom: We want to hear feedback from audio folks.
05:18:17 <dom> ack me
05:18:45 <tidoust> Paul: As mentioned by jib, we shipped a shim, it's easy and it works. There is no need to do something more specific.
05:18:51 <mjwilson> q+ mjwilson
05:18:54 <jib> q-
05:19:04 <hta> hta has joined #mediawg
05:19:35 <tidoust> ... In modern OS, the energy consumption gap is between 0 and 1, and there is almost no gap between 1 and 2.
05:20:06 <tidoust> ... Careful measurements have shown that improvements exist between 0 and 1, not above.
05:20:17 <handellm> q+
05:20:28 <tidoust> ... You do have a realtime thread.
05:21:03 <tidoust> Guido: If you want to do a processing that's doing something such as encoding. Why would I want to spin up a realtime thread?
05:21:33 <tidoust> Paul: If you have some audio running, you already have a realtime thread. If you have an audio context, you don't need another one.
05:22:39 <dom> q?
05:22:41 <tidoust> dom: The point is not only about performance, more about audio and video processing being combined and requiring separate contexts to run as a result.
05:22:58 <dom> Present+ VincentScheib
05:23:03 <tidoust> Youenn: In Safari, we have a limit on the number of audio contexts that you can create.
05:23:15 <tidoust> ... If we can reuse some of them, that's good.
05:23:35 <tidoust> ... This proposal for Processor would allow us to reuse, which seems good and measurable.
05:24:15 <dom> ack handellm
05:24:43 <tidoust> Markus: The type of processing we're considering here is WebGPU, WebNN.
05:24:55 <dom> ack mjwilson
05:25:01 <tidoust> Paul: In our polyfill, we send the data zero copy, that's fine.
05:25:41 <tidoust> mjwilson: [refers to a proposal worth considering as part of this discussion]
05:26:05 <GabrielBrito> GabrielBrito has joined #mediawg
05:26:21 <mjwilson> https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/main/OfflineAudioContext/explainer.md
05:26:25 <tidoust> Youenn: Having performance measurements would help convince people.
05:26:39 <tidoust> ... The Web Audio proposal would be worth hearing.
05:27:43 <cpn> Topic: Decoder failure API
05:27:47 <dom> [slide 111]
05:28:12 <dom> s|Topic: D|Topic: -> https://github.com/w3c/webrtc-extensions/issues/146 D
05:28:12 <tidoust> Steve: Edge team, talking about adding decoder fallback and failure events to WebRTC.
05:28:23 <tidoust> ... We're continuing to operate on feedback that we received since last year.
05:28:36 <tidoust> [slide 112]
05:29:22 <tidoust> Steve: Game streaming platforms realy on hardware decoding, but they don't have access to mic/cameras, so don't know when a decoder fallbacks from hardware to software.
05:29:43 <tidoust> ... We want applications to know when this happens and help them analyze what happens.
05:29:57 <tidoust> [slide 113]
05:30:16 <tidoust> Steve: We don't want to expose any vendor-specific hardware information.
05:30:21 <tidoust> [slide 114]
05:30:36 <tidoust> Steve: Current proposal is two new events on the RTCRtpReceiver.
05:30:50 <tidoust> [slide 115]
05:31:07 <tidoust> Steve: Here's the proposal for the decoder state change event.
05:31:36 <tidoust> ... The codecParameters and powerEfficientDecoder are nullable to allow implementations to determine whether to report them.
05:31:42 <tidoust> [slide 116]
05:32:18 <tidoust> Steve: You can send telemetry for example.
05:32:21 <tidoust> [slide 117]
05:32:40 <tidoust> Steve: The error event could be useful if you have an hardware decoder that cannot fall back to software.
05:32:46 <tidoust> ... Questions?
05:33:30 <tidoust> Youenn: I talked with our privacy folks. For the nullable properties, it should be pretty easy to get PING's attention through their repository.
05:33:42 <jib> q+
05:34:06 <tidoust> ... That would be a good next step to do for this proposal.
05:34:19 <dom> ack jib
05:34:39 <tidoust> jib: Some nits on the API.
05:35:00 <mjwilson> mjwilson has joined #mediawg
05:35:04 <tidoust> ... The example is not entirely clear when the event fires.
05:35:16 <tidoust> Steve: The proposal is that there will be an initial event when it starts decoding.
05:35:29 <tidoust> jib: It may be worth exposing another attribute then.
05:35:32 <hta1> hta1 has joined #mediawg
05:35:43 <dom> s/on the API./on the API. First, why on the peerconnection rather than the transceiver?/
05:35:45 <hta1> q+
05:36:13 <tidoust> jib: Agree with the need to do a privacy review.
05:36:15 <dom> ack hta
05:36:35 <tidoust> hta1: Since this is decoder, the only place it belongs is on the receiver.
05:36:45 <tidoust> Steve: That's what we did!
05:36:51 <tidoust> hta1: You did it right.
05:37:44 <tidoust> Youenn: When I'm looking at errorText, I'm wondering whether it will be platform specific. How you make use of that error code or error text. Are you trying to achieve "I should retry?" or is it more for stats purpose?
05:37:44 <lei_zhao> lei_zhao has joined #mediawg
05:38:00 <tidoust> Steve: That's a good question. We copied this from the WebRTC error.
05:38:20 <tidoust> ... It's mainly for error reporting and troubleshooting.
05:38:30 <cpn> q+
05:38:32 <tidoust> ... There may be some more errors where this could be interesting.
05:39:31 <hta1> q+
05:39:45 <hta1> q-
05:39:57 <tidoust> Nishita: Another scenario is telemetry for the app. Coding negotiations or showing the user some report.
05:40:18 <tidoust> Youenn: For telemetry, there's a proposal from Guido to use the Reporting API. That may be a useful place for this as well.
05:40:33 <dom> -> https://www.w3.org/2025/11/13-webrtc-minutes.html#f2b1 Discussion on Reporting API for WebRTC this morning
05:40:38 <tidoust> ... What can I do with it for my users might be the right question for this proposal.
05:41:22 <tidoust> cpn: Wondering whether PING already reviewed RTCRtpCodecParameters? I'm wondering whether we're opening the discussion to a much broader scope.
05:42:08 <tidoust> hta1: We switched to Media Capabilities as a result of previous discussions. What PING thinks about it right now, I do not know.
05:42:36 <tidoust> Topic: Media Playback Quality
05:42:45 <tidoust> [slide 119]
05:43:05 <tidoust> Markus: Wanted to collect feedback on a proposal for improved metrics for temporal video quality.
05:43:12 <tidoust> [slide 120]
05:43:44 <tidoust> Markus: To recap, the video conferencing problem is you capture a bunch of frames, with timestamps, and the job of the receiver is to represent that accordingly.
05:43:56 <tidoust> [slide 121]
05:44:11 <tidoust> Markus: One way to measure that is through the framerate. It doesn't use timestamps at all.
05:44:25 <tidoust> ... You don't see itches with this.
05:44:44 <tidoust> ... Good for understanding video. For screen sharing, it's not so good anymore because it's variable framerate.
05:44:51 <tidoust> ... Accuracy is very good.
05:44:54 <tidoust> [slide 122]
05:44:59 <tidoust> Markus: More complicated territory.
05:45:33 <tidoust> ... The Harmonic mean. The way it averages across frame durations.
05:45:53 <tidoust> ... This is unfortunately not available for video tiles, because we don't expose them.
05:46:26 <tidoust> ... What's good is that since the weights are the frame durations themselves, then lenghty durations will be over presented.
05:46:33 <tidoust> ... We will see more the itches.
05:46:41 <tidoust> ... But not so good for accuracy.
05:46:56 <tidoust> ... WebRTC does not know when the frames will be rendered.
05:47:02 <tidoust> ... It also cannot see frame drops.
05:47:05 <tidoust> [slide 123]
05:47:23 <tidoust> Markus: Proposal is to add getVideoPlaybackQuality()
05:47:29 <tidoust> ... Harmonic FPS.
05:47:42 <tidoust> ... And then we have the info for local video tiles.
05:47:47 <tidoust> ... Great accuracy.
05:47:54 <tidoust> ... We can still not measure frame drops.
05:48:03 <tidoust> [slide 124]
05:48:23 <tidoust> Markus: We also propose to add another measure which is reproduction jitter RMSE.
05:49:00 <tidoust> ... Difference should be 0 but fluctuates.
05:49:29 <tidoust> ... Root min square. With that, you can measure how accurate.
05:49:37 <cpn> q?
05:49:38 <tidoust> ... It can actually miss frame drops.
05:49:40 <cpn> ack c
05:49:48 <tidoust> [slide 124]
05:49:59 <tidoust> s/[slide 124]//
05:50:02 <tidoust> [slide 125]
05:50:15 <tidoust> Markus: And then add reproduction jitter metric.
05:50:38 <tidoust> ... Accuracy for all current codecs is 90kHz.
05:51:11 <tidoust> [slide 126]
05:51:24 <cpn> q+
05:51:30 <tidoust> Markus: Here's the idea, with a few new fields.
05:51:56 <tidoust> [showing vibe coded demo]
05:52:45 <tidoust> Markus: I'm doing video processing here in terms of burning the CPU.
05:52:54 <tidoust> ... I can then play with it and look at the graphs.
05:53:51 <tidoust> ... [going through different demo settings display harmonic FPS, WebRTC harmonic FPS]
05:54:47 <tidoust> ... [and jitter reproduction metric]
05:57:09 <tidoust> cpn: Last TPAC, we talked about Video Playback Quality API. Your proposal just adds more things, right?
05:57:13 <tidoust> Markus: Yes.
05:57:28 <tidoust> cpn: Do the metrics belong on this or on getStats()?
05:57:40 <tidoust> Markus: I don't think getStats() is the right place for this.
05:57:47 <dom> q?
05:57:48 <tidoust> ... E.g., Youtube playback quality.
05:57:50 <dom> ack cpn
05:58:16 <tidoust> Youenn: Using RTP timestamps, I guess? If there are none?
05:58:23 <tidoust> Markus: Then I don't [missed]
05:58:47 <tidoust> cpn: Are there comments on the definitions of the stats?
05:59:08 <tidoust> dom: Are these standard metrics for some definition of standard?
05:59:16 <tidoust> ... Are we paving the cowpath or are we trying something new?
05:59:27 <tidoust> Markus: A bit of both.
05:59:48 <tidoust> Youenn: Some A/V stats may be computed by native apps.
05:59:54 <tidoust> Markus: Only Chrome does it, I think.
06:00:11 <tidoust> ... There is a performance hit as well as you need every frame.
06:00:47 <tidoust> ... In getStats(), there's "jitter squared duration" or something.
06:01:51 <dom> https://w3c.github.io/webrtc-stats/#dom-rtcinboundrtpstreamstats-totalsquaredinterframedelay
06:01:56 <tidoust> jib: I don't think provisional stats have gone through a standardization process.
06:02:12 <tidoust> dom: totalSquaredInterFrameDelay, it is.
06:02:48 <tidoust> cpn: There's a question around the Media Playback Quality spec as it is.
06:02:57 <tidoust> ... A few years ago, we agreed to hand it over to the WHATWG.
06:03:46 <tidoust> ... What you're proposing now is to make it a more lively document.
06:03:57 <tidoust> ... And the question is whether the Media WG should handle it.
06:04:05 <tidoust> ... That requires discussion with the WHATWG.
06:04:53 <tidoust> Francois: Charter suggests that Media WG will not do any actual work on the spec over than to transition it as well.
06:05:13 <tidoust> cpn: Let's discuss as part of the Media Playback Quality repository.
06:05:33 <dom> https://w3c.github.io/media-playback-quality/
06:05:39 <dom> https://github.com/w3c/media-playback-quality/
06:05:39 <tidoust> RRSAgent, draft minutes
06:05:40 <RRSAgent> I have made the request to generate https://www.w3.org/2025/11/13-mediawg-minutes.html tidoust
06:21:56 <nigel> nigel has joined #mediawg
06:22:30 <nigel> nigel has joined #mediawg
06:23:27 <nigel> nigel has joined #mediawg
06:23:48 <nigel> nigel has joined #mediawg
06:28:02 <dom> RRSAgent, make log public
07:03:45 <nigel> nigel has joined #mediawg
07:07:22 <nigel> nigel has joined #mediawg
07:12:23 <nigel> nigel has joined #mediawg
07:29:38 <nigel> nigel has joined #mediawg
07:50:32 <nigel> nigel has joined #mediawg
08:09:25 <Zakim> Zakim has left #mediawg
08:11:12 <nigel> nigel has joined #mediawg
08:22:45 <dom> dom has left #mediawg
08:29:54 <nigel> nigel has joined #mediawg
08:52:50 <nigel> nigel has joined #mediawg
23:53:55 <RRSAgent> RRSAgent has joined #mediawg
23:53:55 <RRSAgent> logging to https://www.w3.org/2025/11/13-mediawg-irc
23:53:57 <Zakim> Zakim has joined #mediawg
23:54:05 <tidoust> RRSAgent, make log public
23:54:37 <tidoust> Meeting: Media Working Group TPAC meeting
23:54:46 <tidoust> Chair: Chris, Marcos
23:54:51 <tidoust> Agenda: https://github.com/w3c/media-wg/wiki/TPAC-2025
23:54:57 <tidoust> RRSAgent, draft minutes
23:54:59 <RRSAgent> I have made the request to generate https://www.w3.org/2025/11/13-mediawg-minutes.html tidoust
23:58:18 <Lei_Zhao> Lei_Zhao has joined #mediawg
23:58:29 <nigel> nigel has joined #mediawg
23:58:51 <tidoust> present+ Chris_Needham, Francois_Daoust, Markus_Handell, Nigel_Megitt, Dom_HazaelMassieux, Wolfgang_Schildbach, Bernd_Czelhan
00:00:12 <tidoust> present+ Mark Foltz, Alastor Wu
00:00:12 <cpn> cpn has joined #mediawg
00:00:25 <nigel> scribe+ nigel
00:01:36 <alwu> alwu has joined #mediawg
00:01:44 <tidoust> present+ Andy Estes, Xiaohan Wang, Fredrik Hubinette, Eric Carlson, Jean-Yves Avenard, Gabriel Brito
00:01:51 <alwu> present+ Alastor Wu
00:02:05 <tidoust> RRSAgent, draft minutes
00:02:06 <RRSAgent> I have made the request to generate https://www.w3.org/2025/11/13-mediawg-minutes.html tidoust
00:04:10 <tidoust> RRSAgent, bye
00:04:10 <RRSAgent> I see no action items