14:25:19 RRSAgent has joined #webrtc 14:25:23 logging to https://www.w3.org/2023/03/21-webrtc-irc 14:25:23 Zakim has joined #webrtc 14:25:30 Meeting: WebRTC March 2023 meeting 14:25:30 Agenda: https://www.w3.org/2011/04/webrtc/wiki/March_21_2023 14:25:30 Slideset: https://lists.w3.org/Archives/Public/www-archive/2023Mar/att-0004/WEBRTCWG-2023-03-21.pdf 14:25:30 Chairs: HTA, Jan-Ivar, Bernard 15:03:08 Present+ Florent, Dom, Bernard, Jan-Ivar, Fippo, Tove, Henrik, Youenn 15:04:27 Present+ Carine, Harald 15:04:34 Present+ SunShin 15:04:38 Recording is starting 15:05:29 scribe+ 15:06:42 Topic: -> https://github.com/w3c/webrtc-extensions WebRTC Extensions 15:06:42 Subtopic: PR #148 Delete Removed Features Section 15:06:42 [slide 11] 15:07:08 Bernard: brought by Florent - features removed from webrtc-pc that are now causing errors in the extensions doc 15:07:18 Florent: it's easy to add them back if/when needed anyway 15:07:42 RESOLVED: Remove removed features from webrtc-extensions (section 13) 15:07:47 Subtopic: PR #147 Add RTCRtpEncodingParameters.codec to change the active codec 15:07:47 [slide 12] 15:08:22 Henrik: at TPAC and at the January meeting, we discussed adding a codec setting to RTCParameters to allow mixed codec use cases and allow to change codecs without round-trip time if previously negotiated 15:08:38 ... and one API to specific codec and scalabilityModel 15:08:42 ... #147 is providing that 15:08:54 [slide 13] 15:09:19 Present+ PaulAdenot 15:09:28 Present+ PeterThatcher 15:10:36 Florent: the pull request also describes error conditions, similar to SVC 15:10:55 ... please bring feedback on the pull request 15:11:15 [support from fippo and harald] 15:11:52 Topic: -> https://github.com/w3c/webrtc-stats/ WebRTC Stats 15:11:52 Subtopic: Issue #742 Assorted comments on RTCAudioPlayoutStats 15:11:52 [slide 14] 15:12:35 Paul: I discovered new stats landed in the stats repo; I and my co-editor were unsure about they were and initially misunderstood them 15:12:49 ... now with greater clarity, we're unsure why they're needed 15:13:05 [slide 15] 15:13:26 Paul: I understand that Chromium has multiple paths to play audio out depending on where you put the mediastream 15:13:59 ... in case of the media element, a mixer gets into play with a ringer buffer that sometimes cause issues 15:14:20 ... the Stats API allows to detect issues that happen at that particular boundary 15:14:31 Henrik: this covers the path from webrtc to the playout device 15:14:54 ... metrics in webaudio measure what's happening in the graph, but it cannot detect the path after that to the device 15:15:08 ... this metrics is also added when web audio isn't used 15:15:18 ... it also measures a webrtc-specific thing, the synthesized sample 15:15:35 ... it's the symetric of the media-source stat 15:16:07 Paul: the Web Audio quality API can tell you everything that happens - I can't imagine anything more reported 15:16:22 Henrik: how would you get a playout delay or a synthetized sample? 15:16:47 Paul: the jitter buffer has multiple purposes: conceal packet loss and @@@ 15:17:14 ... the issue is that are two crossing of domain clocks - the mixer and the connection to the audio device 15:17:36 ... this feels like an internal implementation concern - e.g. in firefox we don't have that at all 15:17:56 ... we don't buffer and we avoid clock crossings since that require long convergence 15:18:16 ... pulling directly from the audio device to the jitter buffer solves everything 15:18:43 ... this is addressed by stats one level higher in the WebRTC stack 15:18:46 Present+ Varun 15:19:23 ... from the point of view of the playout system, inbound streams, we have stats already 15:19:39 henrik: we considered adding something to the HTML media element, but we got push back on the challenge this would be 15:20:08 ... also considered adding metrics to the mediastreamtrack - this was added recently there, so some of this could be moved there 15:20:17 ... but webrtc has value for metrics beyond what's available 15:20:50 ... we don't really care about where we get them, but metrics on synthetized samples are webrtc specific 15:20:57 [slide 16] 15:21:31 jan-ivar: this addition was merged in the spec by editors, but without consensus from the WG; I think it should be removed 15:21:36 Henrik: I disagree 15:21:43 Harald: let's continue discussion on the issue 15:22:23 Subtopic: Issue #730 The HW exposure check does not solve Cloud Gaming use cases 15:22:23 [slide 16] 15:22:35 SunShin: follow up to discussions from last month 15:22:39 [slide 17] 15:23:07 SunShin: two proposals were brought forward, but we couldn't get consensus on how to expose hardware capability 15:23:49 ... the discussion on github focused on the option in PR #725, exposing a boolean metrics on decoder fallback 15:25:27 ... it only turns to true when the software fallback is actually triggered, so doesn't expose user agent capbilitilies 15:25:27 Bernard: can this go back and forth quickly? will this be poll based? 15:25:27 ... without events, you wouldn't know when or why it happens 15:25:27 SunShin: correct 15:25:27 Bernard: what would the app do with this? 15:25:34 SunShin: when we detect fallback, we try to recover to return back to hardware e.g. by reducing resolution or reinitiating the session 15:25:46 [slide 18] 15:25:49 geheimnis` has joined #webrtc 15:25:55 SunShin: any feedback on moving forward with this? 15:26:31 youenn: I think it creates a side-information channel - it would be good to prevent this 15:26:36 Present+ PatrickRockhill 15:27:01 youenn: the goal is to solve a performance issue e.g. with a reduced resolution - maybe that's what the UA could expose (e.g. "you're using too much energy") 15:27:15 ... that might be better to expose switches from hardware<->software 15:27:36 Bernard: yes - fallback information on its own isn't very actionable 15:27:48 SunShin: right now, the only way to detect is to measure the decode time 15:27:55 ... but that comes with heuristics errors 15:28:05 ... e.g. due to network glitches 15:28:19 ... having an explict flag would be an improvement 15:29:03 Henrik: it's performance you want to act on, maybe this is about power efficiency rather than hardware/software 15:29:14 ... it looks like we know what we want to have but it exposes information 15:29:30 ... this PR allows to avoid hardware info 15:29:55 Bernard: my overall question is about the exact purpose - the Media WG was about exposing actionable information for the app 15:30:08 ... do we know what that is? or is it just for stat? 15:31:44 henrik: I imagine it's about changing resolution or scalability 15:32:05 dom: feels like relying stats as a way to trigger a change feels like an anti-pattern 15:32:36 ... also, the issue youenn was raising is not fingerprinting, but side channel communication where site A & B could detect when they're used together in the same browser 15:32:47 henrik: re stats - this could be moved to an event, that would be fine 15:33:19 ... re side channel, maybe we could explicitly exclude the situation where the error is due by multiple usage across site? 15:33:30 ... or in general, exclude side channel situations? 15:34:17 harald: how would the side channel work? one site would get it and the other not 15:34:25 henrik: not sure - maybe due to changes over time 15:35:06 youenn: if you start with hw decoder and another site tries to grab it, its UA dependent whether the new one gets it or not 15:35:36 ... I like the description of the use case you had - it would be great to describe it on the github issue, it had been difficult so far what exposing the flag would solve 15:36:10 ... having events that are actionable by the app feels much more convincing than stats which aren't expected to bring direct user benefits 15:36:33 Henrik: so reconvene with an event proposal that avoids side-channel communications 15:36:57 Jan-ivar: the issue is cross-site channel correlation, but also use grabbing/releasing hardware to message cross-origins 15:37:07 Topic: -> https://github.com/w3c/webrtc-pc WebRTC-pc 15:37:07 Subtopic: Issue #2820 / PR #2829 setParameters/insertDtmf/replaceTrack should reject on [[Stopping]] as well as [[Stopped]]? 15:37:07 [slide 19] 15:37:56 jan-ivar: transceiver.stop() puts the transceiver in stopping state 15:38:34 ... WPT and Chromium haven't caught up with that change in the spec 15:38:52 ... we're proposing to align with these implementations, and align with the precedent we have for setting direction 15:39:25 Harald: in all cases, this isn't reversible, right? 15:39:31 henrik: yes, it's irreversible 15:39:53 harald: so +1 15:40:07 RESOLVED: no objection to PR#2829 15:40:13 Subtopic: Issue #2827 / PR #2828: Hard to tell if there are state gaps in connectionState algorithm 15:40:13 [slide 20] 15:40:46 jan-ivar: this is a bit editorial - to make clear the tables capture all states 15:41:39 youenn: +1 15:41:53 RESOLVED: PR #2828 can be merged 15:42:06 Subtopic: Issue #2835: Section 4.4.2: createOffer() and setLocalDescription() resource handling 15:42:07 [slide 21] 15:42:44 Bernard: 4.4.2 claims that resources are reserved during the setLocalDescription promise 15:43:00 ... it appears this hasn't been implemented that way - resources aren't really acquired 15:43:39 henrik: it's accurate - we delay creating encoders and decoders till the last minute for performance 15:43:47 ... so handling fallback later sounds better 15:43:49 Present+ TimPanton 15:44:25 Bernard: imagine a codec only available in hw and the hardware is no longer available after the negotiation resolves? 15:44:43 henrik: right, but the alternative is hogging on all resources 15:44:59 Bernard: so yes, we should change the text, and it may be connected to the software fallback situation 15:45:06 henrik: and another motivation to add the event 15:45:14 Bernard: so let's try to make a matching PR 15:45:25 RESOLVED: #2828 is ready for pull request 15:45:34 Topic: -> https://github.com/w3c/webrtc-encoded-transform/issues/172 WebRTC Encoded Transform: Negotiating Custom Codecs 15:45:34 [slide 24] 15:46:29 Harald: when we first designed encoder transform, it was clear I hadn't completed the job - one of the dangling pieces we had found a way to change frames format on fly which meant they no longer matched what had been negotiated 15:46:58 ... that's OK when the endpoints can agree out of bands, but not when parties rely on SDP for this, the exact motivation for SDP 15:47:01 [slide 25] 15:47:23 Harald: there are elements on the way (SFU, packetizers) that kind of expect the negotiation not to lie about this 15:47:57 ... e.g. if you encrypt the first bytes of an H264 buffer, the packetizer can't recognize it 15:48:13 ... the right approach is to negotiate what you send and send what you negotiate 15:48:15 [slide 26] 15:48:50 Harald: a browser has certain capabilities that the SDP negotiation module exposes and agrees with in the negotiation, before informating the packetizer and the encoder 15:48:52 [slide 27] 15:49:45 Harald: we need to expose control to the app to say what to SDP negotiate, including what transform it is applying so that the other party can detect whether or not it supports it 15:49:59 ... and likewise, to inform the encoder / packetizer to adapt how it works 15:50:18 ... it wasn't obvious how to expose this in SDP via a JS API 15:50:23 [slide 28] 15:50:31 [slide 29] 15:50:51 Harald: I now have a proposed minimal API to achieve this 15:51:07 ... one new parameter in codecCapbility - the packetization mode 15:51:14 ... to say what we're sending and how to packetize it 15:52:32 ... this matches closely what Florent is proposing for layer encodings - we should align with what gets merged for that 15:52:34 [slide 30] 15:53:54 [slide 31] 15:55:16 TimPanton has joined #webrtc 15:55:27 [slide 32] 15:56:12 Harald: not ready for adoption, but I intend to gather feedback and experiment with implementation to check if it achieves its goals 15:56:34 ... I'll iterate in a spec branch on my repo, and hope to come back to the WG in a month or two 15:57:04 I am on a train - but If someone could voice this for me... 15:57:19 I really don't like embedding SDP in javascript objects. 15:57:36 undefined sdp too. 15:57:41 jan-ivar: I like the problem being solved; have a few nits on the API, but will need to check with my SDP guy 15:57:45 ... but it looks helpful 15:57:56 The problem needs solving of course. 15:58:10 Harald: if Byron comes to IETF, tell him to come say hi 15:58:51 Dom: [voicing TimPanton above^] 15:59:11 Thanks. 15:59:24 Bernard: this looks like how SKEEP (sp?) work which will be discussed at IETF next week 16:00:09 Harald: I have thought using it for the packetization mode indeed - they have packetization rules of some kind 16:00:23 Youenn: we'll try to make progress on encrypted packetization in AVCore 16:00:46 ... I wonder once supported if we need to make any SDP change 16:01:25 ... This API can be shimmed by doing SDP munging - except that depacketizers won't be able to inspect buffers 16:01:39 ... but that feels like encrypted packetization is addressing this 16:01:46 s/AVC/AVTCore/ 16:02:09 Harald: I thought about drafting a proposal for AVTCore focused only on packetization 16:02:17 ... it only needs to figure out to frames into packets 16:02:36 Youenn: the generic packetizer Sergio and Youenn proposed is close to that 16:02:49 ... I have an upcoming proposal in that space as well 16:02:57 Harald: looking forward to reviewing these drafts 16:03:10 ... Hear support for the problem and experimenting, and coming back after more experience 16:03:25 Topic: Ice Controller API 16:03:25 [slide 35] 16:03:33 Present+ Samir 16:03:56 [slide 36] 16:04:26 Samir: there are a few WebRTC NV use cases for which this proposal is relevant 16:05:07 [slide 37] 16:05:37 [slide 38] 16:05:57 [slide 40] 16:06:24 s/AVTCoreore/AVTCore/ 16:06:29 RRSAgent, draft minutes 16:06:31 I have made the request to generate https://www.w3.org/2023/03/21-webrtc-minutes.html dom 16:06:41 RRSAgent, make log public 16:07:12 s/40/39 16:07:22 [slide 40] 16:08:07 [slide 41] 16:08:44 Samir: the proposal is an incremental API - the browser does as usual until the app asks to intervene by using the preventDefault() method on events 16:09:08 ... it sounds like a good pattern to override the default behavior 16:09:33 ... it limits what the app has to implement in terms of the ICE logic, relying on the browser for most of the work 16:09:55 ... what the app can do is exposed via methods on the interface - more methods can be added as needed over time 16:10:07 ... e.g. gathering is not part of the initial proposal, but could be added later 16:10:09 [slide 42] 16:10:29 [slide 43] 16:10:44 Peter: some of the feedback last time was that the API shape could be improved 16:11:01 ... also tried to think about how to integrate more of the NV use cases, incl forking and gathering 16:11:11 ... leading to a proposal I call "WebICE" 16:11:20 ... but first explaining the difference between web and native in this space 16:11:23 [slide 44] 16:11:43 [slide 45] 16:11:59 [slide 46] 16:12:31 [slide 47] 16:12:52 Peter: most browsers (except maybe FF) have additional ICE optimizations to speed things up 16:13:04 ... closely related to what's desired in WebRTC NV Use casees 16:13:06 [slide 48] 16:13:23 [slide 49] 16:13:48 Peter: there is a gap between web & native; developers have asked for more capabilities for years 16:13:53 ... can we close this gap? 16:13:56 [slide 50] 16:14:12 Peter: yes - there is a wide spectrum of possibilities 16:14:29 ... FlexICE was such a proposal; ICE controller bring even more flexibility 16:14:50 ... WebICE would close the gap completely - optional A & B are two versions of that, more or less manual 16:14:52 [slide 51] 16:15:43 [slide 52] 16:16:32 [slide 53] 16:17:07 (slide 54] 16:17:37 Peter: key different between Option A & B is that A is manual - the app has responsibility to send the checks 16:17:52 ... the app needs to decide which candidate pairs to send checks on, and which is better 16:18:09 ... this is currently built-in logic in the browser - here it would be required for the app to implement this 16:18:12 [slide 55] 16:18:54 Peter: similar to what was suggested for Ice Controller - Option B allows to opt-in to manual controls (defaulting to automatic behaviors) 16:18:57 [slide 56] 16:20:06 Peter: this simplifies simple examples quite a bit - you only have code for what you want to customize 16:20:13 [slide 57] 16:20:56 Peter: for controlling gathering, WebICE allows to exclude a number of candidates during gathering 16:21:05 ... candidates can also be pruned after gathering 16:21:11 [slide 58] 16:21:30 Peter: there has not been proposals for ICE forking so far - but WebICE supports this 16:22:07 [slide 59] 16:22:22 Peter: selecting candidate pair is made very easy with this 16:22:35 [slide 60] 16:23:07 Peter: networkCost can be used to select candidates with WebICE 16:23:11 [slide 61] 16:23:55 [slide 62] 16:24:16 [slide 63] 16:25:23 Henrik: in terms of API shape, the main difference is that you have a cake with pre-defined slices you can ask for, whereas ICE controller lets you override specific events 16:25:47 Peter: two difference: capabilities (e.g. forking and gathering - they could be added to ICE Controller but aren't part of the initial proposal) 16:26:23 ... and a difference of style - WebICE gives full control (either fully manual or semi-manual) 16:27:42 Henrik: I do see more features; but I'm concerned it may be challenging to iterate on adding features over time with WebICE - it's clearer how to feature-detect with ICE controller 16:28:15 Samir: one example - an app opts out to do something, but suddenly a new slice is brought in - does that break the app? 16:28:28 Henrik: also, how do you detect components defined e.G. in dictionaries? 16:29:18 Peter: that probably matters mostly for option A; one solution might be that the default is automatic; if you opt-out from automatic, the responsibility is clear; if more behaviors can be opted out over time, then it's clear where the responsibility is 16:29:22 {sorry I'm losing quite a bit of audio here - unsurprisingly but here are some thoughts} 16:29:24 I have a problem with Peter’s option B - it requires an exhaustive list of things you might want to do - which can be derived from nv-usecases but seems to me not to allow for innovation.

I am warming to Samir’s cancelable events - except that they too require us to know which events might be cancellable (and implicitly defines what you are cancelling - which means we are documenting _all_ the ice stacks.) Also some of this looks [CUT] 16:29:35 Henrik: but given the large API surface, it would be nice to be able to ship things incrementally 16:30:17 Harald: [looking at option B example on slide 56] 16:31:19 Harald: splitting constructing an IceTransport without a peer connection and feeding it afterwards feels complicated 16:32:00 Peter: we had a chromium implementation of that years ago; but there are other ways to combine the two objects - the goal was to avoid having the PC create another ICE transport by doing it ahead of time 16:32:47 Jan-Ivar: my feedback is on scope - I'm worried about feature creep 16:33:00 ... we already use RTCIceTransport on PeerConnection 16:33:11 ... for all intent and purposes, it's the ICE Agent 16:33:25 ... my proposal was that the ICE Controller didn't need to exist separately from the PC 16:33:39 ... that would be my advice for ICE Controller 16:33:56 ... for this new proposal... would IceTransport be really RTCIceTransport? 16:34:07 Peter: in terms of scope, this is about addressing NV Use cases 16:34:23 ... it's hard to see how to support forking / gathering while relying the PC Ice Agent 16:34:38 Jan-Ivar: but only forking requires a separater gatherer, right? 16:34:57 Peter: without forking, you may be able to combine gatherer and transport 16:35:25 Jan-Ivar: in this API, you don't use event.preventDefault() approach used in ICEController 16:35:40 Peter: right - we could sitll make the default be automatic in this proposal though 16:35:53 ... you have to be explicit about opting in in automatic 16:36:02 Jan-Ivar: I would suggest scoping down 16:36:22 Peter: but this is scoped to WebRTC NV and real-world applications that improve on ICE 16:36:39 ... there can be discussions on shape & scope 16:36:53 ... on scope, the question is which part of the gap we're not going to support 16:37:29 Jan-Ivar: is there a way to converge on a single proposal - is there alignment between proposers? 16:37:40 Samir: two kind of differences: capabilities and shape 16:38:03 ... except for ICE forking, I think both proposals can evolve to have the same scope 16:38:10 ... ICE forking require a separate ICE gatherer 16:38:36 ... stylistically, I think asking the app to take fully over the ICE agent seems like a big ask 16:38:56 ... probalby not so far between Option B & ICE Controller 16:39:32 youenn: +1 to henrik on iterative shipping - figuring out the MVP with an API that can evolve towards an end goal (e.g. create an RTCIceTransport without a PC) 16:39:49 ... which of these 2 APIs are closest to the native implementation (libwebrtc of FF's) 16:40:24 Peter: I like the idea to have a roadmap for incremental shipping 16:41:11 ... in terms of implementation - I'm familiar with the ICE implementation in libwebrtc - either of these are implementable; there is a slight asterisk for that with regard to ICE forking, but I think this would be doable in libwebrtc with fairly small changes based on past implementation experience 16:41:31 Samir: there is quite a bit that was added in libwebrtc and chromium to prototype ICEController 16:42:00 ... the ICEController interface in libwebrtc would work for this; would probably also work for WebICE - some implementation is already there 16:42:34 Bernard: coming back to the use cases (slide 39) - which ones are the most important for the group to consider? 16:43:08 ... N14 for instance comes from IoT due to the cost on battery performance 16:43:21 ... some were linked to mobile networks performance 16:43:34 ... N15 had come up in a number of services wrt reliability 16:43:57 ... The forking use case came up in situations of gaming IIRC 16:44:18 ... any sense of prioritization in terms of importance? 16:44:53 I'm keen on N14 ! 16:44:57 Peter: the ones I hear people ask about are mostly N01 and N04 - control which local candidates are used and when, which things are paired 16:45:02 ... and N14 16:45:35 Harald: the impetus that led to this proposal for N04: when network conditions change, you need to be able to move to a different network without a full ICE restart 16:45:43 ... which needs to keep quality metrics 16:45:52 Bernard: right, that's part of the mobile use case 16:46:25 Peter: I would agree that ICE forking is lower priority, although I wouldn't want to paint us in a corner where we can't support it 16:46:49 Jan-Ivar: in the spirit of incremental approaches, could these be built upon the existing RTCIceTransport object? 16:46:58 ... I brought this up on the issue 16:47:13 Samir: the initial implementation was built on RTCIceTransport 16:48:22 ... it makes it easier to deal with bundling; but there is a question of ensuring that candidates can be pruned 16:48:29 ... this could be solved by specific configuration options 16:48:49 Jan-Ivar: same question for you Peter, assuming we forego ICE forking for now 16:49:13 Peter: yes, I think that should be possible; e.g. preventing to re-use IceGatherer 16:49:26 ... not sure if the proposal works without an IceGatherer object at all 16:50:11 ... it might be possible to make it so that if you don't construct an IceGatherer, one is done automatically for you 16:50:41 Henrik: wrt moving events and methods to the existing RTCIceTransport interface and the risks of a race… 16:51:12 Jan-Ivar: I commented on the issue on this - I don't think there can be a race; if you add your event handler directly after sLD, you won't have a race 16:51:57 s|comment on the issue|[commented on the issue](https://github.com/sam-vi/webrtc-icecontroller/issues/7#issuecomment-1477085943) 16:52:23 Harald: we have prototype code for ICE Controller 16:52:36 ... it's likely that ICE Controller could be implemented on top of WebICE though 16:53:06 ... so maybe this needs more experimentation 16:53:28 Peter: I don't think we need to choose quite yet - I think coming up an incremental plan based on feedback we received today would be a good next step 16:54:07 Harald: I'm hearing consensus on the problems to be solved, and that these are reasonable approaches to experiment with 16:55:37 s|Topic: Ice Controller API|Topic: -> https://github.com/sam-vi/webrtc-icecontroller Ice Controller API/ 16:55:43 RRSAgent, draft minutes 16:55:45 I have made the request to generate https://www.w3.org/2023/03/21-webrtc-minutes.html dom 18:14:05 Zakim has left #webrtc