14:57:52 <RRSAgent> RRSAgent has joined #webrtc
14:57:52 <RRSAgent> logging to https://www.w3.org/2022/10/18-webrtc-irc
14:58:10 <dom> Meeting: WebRTC October 2022 meeting
14:58:12 <dom> Agenda: https://www.w3.org/2011/04/webrtc/wiki/October_18_2022
14:58:14 <dom> Slideset: https://lists.w3.org/Archives/Public/www-archive/2022Oct/att-0001/WEBRTCWG-2022-10-18.pdf
14:58:16 <dom> Chairs: HTA, Jan-Ivar, Bernard
15:01:20 <fippo> fippo has joined #webrtc
15:01:28 <dom> Present+ Dom, Harald, Jan-Ivar, Youenn, Bernard, Florent, Ben, Tonni
15:02:18 <dom> Present+ fippo
15:02:30 <dom> Present+ Tove
15:02:52 <dom> Present+ Henrik
15:03:11 <dom> Present+ MikeEnglish
15:03:22 <dom> Recording is starting
15:03:29 <dom> scribe+
15:03:42 <dom> Present+ Elad
15:05:48 <BenWagner> BenWagner has joined #webrtc
15:05:54 <dom> Topic: -> https://github.com/w3c/webrtc-encoded-transform Encoded Transform - Overflow from TPAC
15:06:35 <dom> Harald: during TPAc, we discussed the concept of a packet API, with an explainer, use cases and architecture - not yet done
15:06:41 <dom> ... other issues didn't get covered
15:06:44 <dom> Subtopic: Issue #109 & #118 Depacketization order
15:06:45 <dom> [slide 12]
15:06:53 <dom> s/118/119/
15:07:26 <dom> Harald: packets don't arrive order on the network (they get lost or retransmitted)
15:07:45 <dom> ... frames need to be in order for the decoder
15:08:05 <dom> ... in general, a transformation is simpler when happening in decoding order
15:08:14 <dom> ... this requires a jitter buffer in front of the decoder
15:08:23 <dom> ... if the transformer itself introduces jitter, it doesn't get compensated
15:08:35 <dom> ... currently Chromium has the jitter after the transfer
15:08:53 <dom> s/transfer/transformer
15:09:07 <dom> s/Tonni/Tony
15:09:16 <dom> Bernard: this isn't the only place where we're encoutnering this problem
15:09:48 <dom> ... are you imagining an explicit API for jitter buffer - e.g. a jitter buffer provided as a transform stream?
15:10:22 <dom> Harald: we could say "frames arrive in the order they arrive in", vs "the UA reorder them, incl waiting for frames" (probably not good), with a flag allowing one or the other
15:10:39 <dom> Youenn: I recall we discussed this previously
15:10:52 <dom> ... iirc, we thought that in-order matched the Web developers expectations
15:11:01 <dom> ... it may make it harder to implement for UA
15:11:17 <dom> ... we should look at use cases where having out-of-order would be a benefit
15:11:43 <dom> ... it's a possible footgun; if there are good use cases for it, then we should look for a solution, but otherwise, we should stick with in-order as in the spec
15:11:59 <dom> Bernard: for the crypto use case, is out-of-order even doable?
15:12:04 <dom> Youenn: for SFrame yes
15:12:14 <dom> ... the counter may not be monotonic in that situation
15:12:21 <dom> ... which would lead to dropped frames
15:12:33 <dom> ... but it shouldn't be an issue from a decryption perspective
15:12:41 <dom> Bernard: so is out-of-order a speed concern?
15:12:50 <dom> hta: my worry about is in-order is in the case of lost frames
15:13:04 <dom> ... without nack, rtx - you have to give up at some point
15:13:23 <dom> ... if we accept in-order frames, we accept that lost frames will cause delays of some magniture
15:13:26 <dom> s/ture/tude
15:14:31 <dom> dom: the wait-for-loss delay could be provided by the developer?
15:14:48 <dom> youenn: having both options would create complexity for developers
15:15:06 <dom> ... if the transform is taking sometimes 2ms and sometimes much longer, a jitter buffer would then be beneficial
15:15:20 <dom> ... for decryption or metadata passing, it should be fairly stable
15:15:23 <hta> hta has joined #webrtc
15:15:33 <dom> ... not sure of the value of a jitter buffer positioned after
15:15:48 <dom> hta: sounds like we need more time on use cases
15:16:19 <dom> Tony: moving the jitter buffer earlier means increased packet loss (given that it removes the processing time from the jitter buffer)
15:16:40 <dom> ... there will be delays introduced from operating in a worker (rather than say a real time worklet)
15:17:00 <dom> youenn: currently chrome & safari implementations do out-of-order, which don't match the spec
15:17:21 <dom> ... is Chrome planning to move to in-order? if implementations don't intent to align with the spec, that's also a consideration
15:17:33 <dom> hta: switching to in-order would require a compelling argument
15:18:06 <dom> jib: unless the transform has side effects (time-dependent), it shouldn't matter too much
15:18:21 <dom> ... use cases would be helpful
15:18:41 <dom> ... out-of-order seems a footgun - why should developers worry about that?
15:19:05 <dom> hta: if delay matters, in-order is a footgun
15:19:41 <dom> youenn: so we should use cases for both in-order and out-of-order
15:19:46 <dom> Subtopic: Issue #143 generateKeyFrame
15:19:46 <dom> [slide 13]
15:19:46 <dom> -> https://www.w3.org/2022/09/13-webrtc-minutes.html#t07 TPAC discussion
15:20:10 <dom> Present+ PatrickRockhill
15:20:46 <dom> Fippo: I wanted to suggest a 4th proposal - an empty return value, but allow the app to pass any subset of the rids to generate keyframes
15:21:40 <dom> ... some encoders can generate keyframes from individual rids, others can't - it depends on the codecs
15:21:54 <dom> hta: the argument list API would thus be strictly more powerful without additional implementor burden
15:23:03 <dom> youenn: at TPAC our conclusion was one rid was good & simple enough; we didn't have use cases for 2 layers hitting the same frame
15:23:58 <dom> ... an encoder-behavior dependent API isn't so helpful, but I agree it isn't a big burden to add either
15:24:24 <dom> hta: medium objection to single value, no strong objection to array - should we go with the array args?
15:24:34 <dom> RESOLVED: pass an array arguments to generateKeyframes
15:24:38 <dom> fippo: I'll do the PR
15:24:44 <dom> Subtopic: Issue #158 / PR #140: add mimeType to metadata
15:24:44 <dom> [slide 14]
15:25:12 <dom> HTA: figuring the meaning of a payload requires parsing the SDP to figure out what was negotiated
15:25:23 <dom> ... the UA already knows which mime type is associated with which payload type
15:26:07 <dom> Fippo: another argument for it is that we don't specific how the data is structured
15:26:16 <dom> ... being able to specify it as depending on the mime type would be good
15:26:28 <dom> youenn: thanks, this provides a good use case
15:26:40 <dom> ... I think that's a pattern we already apply elsewhere
15:26:40 <dom> Fippo: in stats, indeed
15:27:01 <dom> Florent: isn't that available via getParameters? that exposes the list of payload types
15:27:07 <dom> HTA: but only if you have the PC
15:27:12 <dom> Fippo: that's harder in workers
15:27:29 <dom> RESOLVED: Add mimeType to metadata
15:27:33 <dom> Subtopic: Issue #154: add rtp seqNum to inbound audio
15:27:33 <dom> [slide 15]
15:27:51 <dom> Fippo: we have a custom decoder that relies on the rtp sequence number to detect loss in the audio
15:28:01 <dom> ... relatively easy to add to incoming frames for audio
15:28:10 <dom> ... more complicated for video, or for outgoing frames
15:28:22 <dom> HTA: for incoming audio, you have one packet resulting in one set of samples
15:28:40 <dom> youenn: coming back to in/out-of order, this would expose that
15:28:52 <dom> ... if we're not doing in-order, this may create confusion
15:29:15 <dom> Fippo: in our use case, we have our custom JS jitter buffer; we don't reenqueue the frame into the pipeline
15:29:37 <dom> HTA: so that's also a use case for out-of-order: bring your own jitter buffer
15:29:45 <dom> Fippo: I can that written up as input to the other discussion
15:30:03 <dom> HTA: are we happy to expose this only for audio incoming frames, as a non required dictionary?
15:31:01 <dom> jib: I think it would still be interesting to understand better this one-ended use cases
15:31:33 <dom> HTA: ok, so let's wait for the use cases before proceeding then
15:31:59 <dom> Subtopic: Issue #131: Packetization API
15:31:59 <dom> [slide 16]
15:32:13 <dom> HTA: any more comment on the packetization API beyond what was discussed at TPAC?
15:32:26 <dom> Youenn: we could start with things like MTU
15:32:30 <dom> HTA: in the frame API?
15:32:45 <dom> Present+ Varun
15:33:08 <dom> Fippo: MTU is mostly an issue for audio; I don't think we hit that threshold even with redundancy
15:33:19 <dom> ... it becomes an issue with transform that changes size largely
15:33:54 <dom> Youenn: I don't think adding the MTU to the frame API would make sense - more at the context level, with changes signaled via events
15:34:18 <dom> ... the frame is coming from the encoder, that's not where the MTU info lives
15:34:40 <dom> Topic: -> https://github.com/w3c/mediacapture-extensions/ Media Capture Extensions
15:34:46 <dom> Subtopic: PR #77: Add MediaStreamTrack framesCaptured and framesEmitted
15:34:46 <dom> [slide 18]
15:35:08 <dom> Henrik: `track.getSettings().frameRate` tells the configured, but not actual frame rate
15:35:45 <dom> ... knowing the actual frame rate and the dropped frames would be useful
15:36:07 <dom> ... some of that are exposed in stats, or in media playback metrics
15:36:30 <dom> ... but the measurements are happening later in the pipeline - e.g. if the frame is dropped as soon as it is produced, it won't show up
15:36:45 <dom> ... and we shouldn't force a webrtc PC to get track specific info
15:36:48 <dom> [slide 19]
15:37:15 <dom> henrik: my proposal is to add a frame counter to track API, with a `getStats()` method
15:38:13 <dom> youenn: all APIs that are using an MST will allow you to get the number of frames that you're actually receiving
15:38:26 <dom> ... Media capture transform gives you the count of frames, likewise for WebRTC & HTMLMediaElement
15:38:47 <dom> ... what you want is focused between the sink & the source
15:39:16 <dom> ... not sure I understand the diff between emitted and captured - that feels a bit specific to a specific pipeline
15:39:37 <dom> ... in our model, it's not clear it would be easy to specific an interoperable way to distinguish captured from emitted
15:39:49 <dom> ... so maybe focusing first on captured?
15:40:01 <dom> henrik: that makes sense; captured is the main gap in any case
15:40:40 <dom> jan-ivar: framesCaptured makes sense with a low-lighting camera use case (although we could revisit the constraint model for that)
15:40:57 <dom> ... share Youenn's concerns for emitted, which feels implementation dependent
15:41:40 <dom> ... I'm not sure about `getStats()` vs a constraint
15:41:48 <dom> Bernard: next step?
15:42:06 <dom> Henrik: I'm hearing support for framesCaptured in some form, and leave emitted for later
15:42:26 <dom> HTA: framesEmitted makes sense for consistency, but I see the argument that it may be redundant
15:42:39 <dom> ... so let's start with framesCaptured as accepted
15:42:49 <dom> RESOLVED: move forward with framesCaptured only for now
15:42:55 <dom> Topic: -> https://github.com/w3c/webrtc-pc/ WebRTC & Simulcast
15:42:59 <dom> Subtopic: Issue #2732: Inconsistent rules for rid in RTCRtpEncodingParameters
15:42:59 <dom> [slide 23]
15:43:22 <dom> jib: following up to our discussions started in TPAC about rid length
15:43:50 <dom> ... limiting RID length to 16 characters would help with web compat
15:44:08 <dom> ... an errata has been published on RFC8851 removing - and _ characters
15:44:25 <dom> ... feedback on restricting the length would be hard as an erratum, but could be done in a -bis
15:45:32 <dom> hta: note that the empty string is outlawed by the BNF
15:47:19 <dom> dom: if we wait for -bis, are implementations going to be updated to match the allowed lengths?
15:47:38 <dom> florent: it should be possible to update chrome in that direction if we think if it's a good idea
15:47:52 <dom> hta: we don't know of any use case where 17 characters are necessary
15:48:08 <dom> youenn: we could limit to 16 characters with a note mentioning ongoing IETF discussion
15:48:31 <dom> jib: we could also have a separate decision on addTransceiver vs accepting incoming offers and answers
15:50:28 <dom> dom: I don't think it goes against the protocol to limit what the API accepts to generate rids (we should definitely accept any valid rid in O/A)
15:50:42 <dom> jib: but then you have an API that doesn't let you set values that you accept from a remote description
15:50:52 <dom> Subtopic: Issue #2764: What is the intended behavior of rollback of remote simulcast offer?
15:50:52 <dom> [slide 25]
15:53:06 <dom> RESOLVED: proceed with the proposed clarification
15:53:13 <dom> Subtopic: Issue #2737 / PR #2788: Modifications to [[SendEncodings]] from setParameters and sLD/sRD can be racy
15:53:13 <dom> [slide 26]
15:55:05 <dom> hta: should that addition also be guarded by "if remote is true"?
15:55:16 <dom> jib: it would have to also have a "is an answer" gate - I can update the PR
15:55:41 <dom> henrik: if you restart and apply the steps again, wouldn't you implicitly rollback anything changed by the in-parallel operations?
15:56:04 <dom> ... to do that correctly, you would have to wait until the SDP is applied
15:56:19 <dom> jib: this is run before we call the success callback
15:56:40 <dom> ... we would wait until all setParameters are settled
15:56:48 <dom> ... similar to if a remote description came right after
15:56:57 <dom> henrik: so this is done before the SDP process?
15:56:58 <dom> jib: right
15:57:52 <dom> Subtopic: Issue #2762: Simulcast: Implementations do not fail (and that seems good)
15:57:52 <dom> [slide 27]
15:58:08 <dom> RRSAgent, draft minutes
15:58:08 <RRSAgent> I have made the request to generate https://www.w3.org/2022/10/18-webrtc-minutes.html dom
15:58:11 <dom> RRSAgent, make log public
16:01:10 <dom> [Varun, Youenn depart]
16:01:25 <dom> [slide 28]
16:02:51 <dom> RESOLVED: close #2762 as is
16:02:59 <dom> Topic: -> https://github.com/w3c/webrtc-extensions WebRTC Extensions: Data Channels
16:03:08 <dom> Subtopic: Issue #114: RTCDataChannel transfer and maxMessageSize
16:03:08 <dom> [slide 32]
16:03:53 <dom> florent: RTCDataChannels are transferable; maxMessageSize in RTCSctpTransport needs to be checked before sending data over a channel
16:04:26 <dom> ... with a channel transferred to a worker, the maxMessageSize may be renegotiated on the main thread, which wouldn't be visible to the worker trying to send data
16:04:32 <dom> [slide 33]
16:05:15 <dom> Florent: we could prevent changing the maxMessageSize during renegotiation - doesn't really happen in practice
16:05:37 <dom> ... then that value could be kept in the transferred rtcdatachannel and keep the send algorithm as is
16:06:12 <dom> ... the other aspect to consider is that the datachannel might have been transferred before the initial negotiation
16:07:22 <dom> ... updating that value of maxMessageSize could be done as part of the "announcing a data channel as open" algorithm
16:08:12 <dom> dom: how confident are we that maxMessageSize can be frozen in renegotiation is web compatible?
16:08:46 <dom> florent: we would want to confirm that indeed
16:09:13 <dom> ... sending too much data closes the data channel, so developers already need to pay attention
16:09:32 <dom> Bernard: the only time you would see this is in some weird maintenance scenarios - it should be very rare
16:09:42 <dom> florent: we can add some measurement in Chrome to see if that happens
16:10:31 <dom> dom: +1 to these solutions if they're web compatible
16:11:56 <dom> florent: so we can start with copying the value in opening, and measure web-compatibility of rejecting a renegotiated size
16:12:13 <dom> jib: would maxMessageSize end up being exposed on the data channel?
16:12:23 <dom> florent: we could do that, but that's not part of this proposal
16:12:47 <dom> ... this wasn't useful in the context of running everything in the same context as peerconnection
16:13:03 <dom> ... but with transferred channels, this makes more sense to consider
16:14:24 <dom> dom: it would be clunky not to expose it
16:14:38 <dom> Subtopic: Issue #115: Need to specify behavior of detached RTCDataChannel objects
16:14:38 <dom> [slide 34]
16:15:29 <dom> florent: we need to document a [[Detached]] internal slot per the HTML spec for transferable platform objects
16:15:56 <dom> ... we would keep [[isTransferable]] for the a datachannel that has already sent
16:16:14 <dom> [no objection]
16:16:40 <dom> jib: it remains unclear what happens to data channels when they're transfered in the main thread
16:16:48 <dom> [slide 35]
16:17:34 <dom> florent: should transfered data channels be garbage collectable in the main thread? they're "closed" which make them collectable without a strong reference
16:17:54 <dom> ... we could add a new state "detached" on top of opening, open, closed etc
16:18:45 <dom> Bernard: I prefer  Proposal 2
16:19:59 <dom> jib: transferable objects are more like a clone, leaving an unoperative a clone
16:20:27 <dom> ... so the broader question is how a [[Detached]] data channel should behave, how it should affects the existing algorithms
16:20:52 <dom> florent: because they're closed, this already impacts the methods close() and send()
16:21:48 <dom> florent: hearing some support to introducing a "detached" state, and a  [[Detached]] internal slot
16:22:07 <dom> hta: what about garbage collection?
16:22:15 <dom> florent: let's discuss on github
16:22:20 <dom> Topic: -> https://github.com/w3c/mediacapture-handle Capture Handle
16:22:20 <dom> [slide 39]
16:22:31 <dom> RRSAgent, draft minutes
16:22:31 <RRSAgent> I have made the request to generate https://www.w3.org/2022/10/18-webrtc-minutes.html dom
16:23:17 <dom> [slide 40]
16:24:01 <dom> [slide 41]
16:25:08 <dom> [slide 42]
16:26:27 <dom> [slide 43]
16:27:06 <dom> [slide 44]
16:27:15 <dom> [slide 45]
16:27:30 <dom> [slide 46]
16:28:15 <dom> [slide 47]
16:29:33 <dom> [slide 48]
16:29:47 <dom> [slide 49]
16:30:03 <dom> Elad: the proposal is to add some structure to capture handle
16:30:23 <dom> ... fo crop targets (possibly with specific content hints)
16:31:03 <dom> jib: what about a messageport?
16:31:20 <dom> elad: still not structured, so leads to tight coupling
16:32:08 <dom> jib: I'm not sure we want to specific all the different things that application might need to agree
16:32:30 <dom> ... per #11, I don't think we should re-invent postMessage
16:33:28 <dom> elad: a messageport informs the capturee they're being captured
16:33:56 <dom> ... capture handle is a unidirectional message port
16:34:15 <dom> ... being able to update the handle is useful given that the captured content is going to change
16:34:26 <dom> ... a messageport can be useful in general, but for different use cases
16:34:49 <dom> jib: can we take a step back to understand the requirements we have?
16:35:08 <dom> ... what API surface would be expose here?
16:35:32 <dom> elad: adding structure for a crop target in the capture handle instead of a simple string
16:36:14 <dom> ... croptarget would have contenthints, and also add a messageport as a separate suggestion
16:38:05 <dom> dom: I think maybe a unidirectional messageport would work for what we want?
16:38:27 <dom> elad: several suggestions: move from string to object in capture handle - already needed for tightly coupled apps
16:38:58 <dom> ... for loosely coupled apps, similar to what capture actions already allow, adding explicit support for croptargets / contenthints would go a long way to help
16:39:37 <dom> elad: what about the first suggestion - moving from a string to an object?
16:39:55 <dom> jib: would re-iterate #11 - let's not reinvent postMessage
16:40:40 <dom> elad: but this adds ability to decouple capturees/capturer
16:40:58 <dom> jib: but adding this to the browser API when it facts it's down to the app to use it or not
16:41:08 <dom> ... that's odd
16:41:29 <dom> elad: it's similar to capture actions, not really more formalized in semantics
16:42:17 <dom> hta: are there establishing standardized protocols over messageport already?
16:42:25 <dom> dom: don't know off the top of my head, would have to check
16:43:40 <dom> hta: if we were to have to come up with that, this feels scary
16:45:32 <dom> dom: re going with an objects, would that be for serializable objects?
16:45:43 <dom> elad: yes
16:46:29 <dom> jib: the original purpose for handle was an identifier; now we're talking about passing objects, that changes the nature of the API
16:47:06 <dom> [slide 51]
16:47:39 <dom> elad: a messageport doesn't address all the use cases - it's not structured
16:47:56 <dom> ... I'm hearing support for the use cases, and not seeing an alternative proposal
16:48:22 <dom> jib: I remain a bit lost on the requirements we're solving with this API
16:48:49 <dom> ... e.g. it could be a separate field instead of being part of the handle
16:49:12 <dom> ... I'm not sure why should allow random web sites to specific crop targets
16:49:55 <dom> elad: slide 47 illustrates how this could be a purely user-driven process to avoid any user tricking
16:50:43 <dom> jib: but I'm not sold we need to allow this for random web sites
16:50:58 <dom> harald: what criteria would a web site eligible to this?
16:51:08 <dom> jib: with a messageport?
16:51:32 <dom> elad: but that makes it more likely to create situations where a web site might want to trick another provider?
16:52:26 <dom> jib: I still don't see a compelling case for making handle an object
16:52:59 <dom> elad: is the video provider / vc collaboration use case compelling?
16:53:09 <dom> jib: yes - we should figure a better way
16:53:30 <dom> elad: what way though?
16:54:19 <dom> hta: I'm hearing 2 proposals: make handle with some pre-defined fields for specific purposes; and a messageport for tightly coupled apps
16:54:34 <dom> s/purposes/purposes (e.g. listing croptargets)
16:55:01 <dom> ... these are 2 independent proposals that should be evaluated separately
16:56:53 <dom> jib: would allow any serializable object be safe to expose to the capturer? that seems problematic
16:57:12 <dom> elad: the security properties are similar (or even somewhat safer) than a messageport
16:57:36 <dom> Ben: with arbitrary objects, could that raise OOM concerns?
16:58:18 <dom> elad: the capturing page would expose itself to such an attack
16:58:31 <dom> ben: are there objects that could create risks for the receiver?
16:58:40 <dom> elad: not that I'm aware
17:03:51 <BenWagner> BenWagner has left #webrtc
17:03:54 <dom> dom: I think the chairs will have to propose steps to unblock this conversation
17:04:08 <dom> ... maybe an explainer would help figure out all the considerations that need to be taken into account
17:04:15 <dom> hta: the chairs will do so
17:04:19 <dom> RRSAgent, draft minutes
17:04:19 <RRSAgent> I have made the request to generate https://www.w3.org/2022/10/18-webrtc-minutes.html dom