15:40:06 <RRSAgent> RRSAgent has joined #webrtc
15:40:06 <RRSAgent> logging to https://www.w3.org/2022/11/15-webrtc-irc
15:40:09 <Zakim> Zakim has joined #webrtc
15:40:25 <dom> Meeting: WebRTC November 2022 meeting
15:40:25 <dom> Agenda: https://www.w3.org/2011/04/webrtc/wiki/November_15_2022
15:40:26 <dom> Slideset: https://lists.w3.org/Archives/Public/www-archive/2022Nov/att-0000/WEBRTCWG-2022-11-15.pdf
15:40:28 <dom> Chairs: HTA, Jan-Ivar, Bernard
15:56:15 <tovep> tovep has joined #webrtc
15:57:57 <TuukkaToivonen> TuukkaToivonen has joined #webrtc
16:02:36 <dom> Present: Florent, PatrickRockhill, Youenn, TOve, PhilippHancke, Tuukka, Elad, Dom, Jan-Ivar, Hugo, PeterThatcher, Harald
16:03:43 <dom> Present+ Eero
16:04:06 <eehakkin> eehakkin has joined #webrtc
16:04:48 <dom> Present+ Cullen
16:05:12 <dom> Present+ Bernard
16:06:15 <dom> Recording is starting
16:07:39 <dom> Present+ Carine
16:07:55 <fippo> fippo has joined #webrtc
16:08:45 <dom> Topic: -> https://github.com/w3c/webrtc-encoded-transform Encoded Transform
16:08:45 <dom> [slide 10]
16:09:42 <dom> Harald: I offered to use the IETF Hackathon to experiment with encoded transform (on my own, for lack of participants)
16:09:47 <dom> [slide 11]
16:11:05 <dom> [slide 12]
16:12:23 <dom> Harald: developed 2 demos to evaluate the API (but not for signals)
16:12:25 <dom> [slide 13]
16:12:55 <dom> Harald: I had initially thought I needed both producers and consumers, but writing the demos, only the producers seemed necessary
16:13:30 <dom> [slide 14]
16:14:30 <dom> Harald: the processing is done via a user-defined JS class that you insert in the processing pipeline, but without requiring a single PC used in both end of the pipe
16:14:47 <dom> ... this led to the conclusion that the API could be used
16:15:33 <dom> ... Peter worked separately on how that one-way API approach could be done with the existing two-ways APIs
16:15:36 <dom> [slide 15]
16:15:58 <dom> Peter: I got it working with transport, codec
16:16:05 <dom> [slide 16]
16:16:47 <dom> Peter: a constructor would help
16:16:55 <dom> ... also missing signals for congestion control
16:16:57 <dom> [slide 17]
16:17:14 <dom> Peter: pretty straightforward on the receiver side
16:17:17 <dom> [slide 18]
16:17:53 <dom> Peter: again, missing a way to control e.g. the encoder bitrate based on congestion control
16:17:56 <dom> [slide 19]
16:18:29 <dom> Peter: for a Decoder, we would again want a constructor for the encoded video frame, and signals to detect the need for a key frame
16:18:35 <dom> [slide 20]
16:19:35 <dom> Peter: Harald's approach would satisfy these needs
16:20:06 <dom> Youenn: with regard to these 5 gaps, there is already a solution for the keyframe problem
16:20:32 <dom> ... for constructors, I'm not sure why we need something on top of what WebCodecs provide from raw data; what's the point of using PC for incoming data?
16:21:06 <dom> Peter: WebCodecs doesn't have a built-in jitter buffer, whereas this would
16:21:21 <dom> Youenn: but we've been discussing letting the app define the jitter buffer
16:21:30 <dom> ... so it's not clear that there is a benefit
16:21:50 <dom> Peter: it would still allow to get the same behavior that you get from WebRTC without having to write your own jitter buffer
16:22:06 <dom> Youenn: I think this would benefit from clearer use cases
16:22:23 <dom> Harald: one of the use cases that needs this is getting an incoming video frame and passes it out to a different peer connection
16:22:31 <dom> ... or passing it to 2 peer connections
16:22:38 <dom> Youenn: to re-forward it?
16:22:41 <dom> harald: possibly, yes
16:22:52 <dom> Youenn: this may be mostly about seralization, rather than a constructor
16:23:00 <dom> s/sera/seria/
16:23:11 <dom> harald: metadata may need rewriting
16:23:17 <dom> ... let's see about use cases
16:23:57 <dom> Jan-Ivar: what's the high level problem we're solving? would this be instead of encoded transform? re-imagining it? identifying issues with it?
16:24:09 <dom> ... we have readable and writable streams on mediastreamtracks
16:24:26 <dom> ... so I can already receive a track and forward it
16:24:36 <dom> ... what's the difference?
16:24:54 <dom> Harald: this relates to the use cases discussed at TPAC
16:25:25 <dom> ... there were compelling arguments that this could not be addressed without substantive changes of the webrtc encoded transform API
16:25:40 <dom> ... not clear if this should replace or extend it - depending on where the shape lands
16:25:57 <dom> Peter: you cannot forward well without bandwidth estimation
16:26:34 <dom> ... you could re-use the encoded(audio|video)frame to forward them as is, but you probably need to re-packetize which you can't do without a constructor
16:27:17 <dom> Jan-ivar: OK; still unclear how this would affect the API shape
16:27:37 <dom> Peter: I was focused on identifying the gaps at this stage
16:28:00 <dom> Harald: I explicitly shied away from presenting an API shape, to focus on use cases and requirements at this stage
16:28:33 <dom> ... this is to stimulate the discussions
16:28:52 <dom> Peter: my impression is that this could be added with fairly minimal changes (constructors, signals)
16:28:56 <dom> ... not a big delta from what we have
16:29:31 <dom> Present+ BrianBaldino
16:29:44 <dom> Harald: so next step is to enumerate use cases a bit more before making a change proposal
16:29:49 <dom> ... Peter and I will continue to iterate on this
16:30:09 <dom> Topic: -> https://github.com/w3c/webrtc-pc WebRTC PC
16:30:25 <dom> Subtopic: Issue #2795: Missing URL in RTCIceCandidateInit
16:30:25 <dom> [slide 24]
16:30:45 <dom> Youenn: this follows from discussion at the previous meeting
16:31:22 <dom> ... the server URL used to be exposed in the event, and it has been proposed to move it to the candidate object itself
16:31:47 <dom> ... but we didn't discuss whether it would survive JSON serialization / deserialization
16:32:10 <dom> ... so far serialization/deserialization has been without information less
16:32:23 <dom> ... should that apply to the URL attribute?
16:32:32 <dom> [slide 25]
16:33:03 <dom> Youenn: this impacts whether it gets submitted to remote parties by default (although this is only about defaults, not about protecting the info in general since it remains available to JS)
16:33:30 <dom> ... in general, do we want to keep the invariant of non-lossiness on this object?
16:34:00 <dom> ... Personally, I don't think there are good use cases to pass the url to remote parties, and we should keep the model consistent with regard to lossiness
16:34:15 <dom> ... so we should keep the url attribute to the event rather than the object
16:34:34 <dom> ... it can be shimmed easily from one to the other
16:34:40 <dom> [slide 26]
16:35:08 <dom> fippo: toJSON conveys information that is needed for ICE
16:35:49 <dom> ... additional properties were added to avoid having developers parsing data out of the canddiate string
16:35:55 <dom> ... e.g. to determine the network topology
16:35:58 <dom> [slide 27]
16:37:35 <dom> youenn: the question is about convenience / POLA
16:37:53 <dom> fippo: exposing the data on candidate is best to avoid having to go through stats
16:38:12 <dom> ... you can't correlate you event with stats except through IP address matching
16:38:40 <dom> Jan-Ivar: I'm hearing that the candidates already has information that aren't exposed in toJSON
16:38:49 <dom> Fippo: right, e.g. relayProtocol
16:39:07 <dom> Jan-Ivar: so that already breaks the supposed invariant on non-lossiness
16:39:29 <dom> Youenn: if so, that goes against the spirit of the spec
16:39:46 <dom> ... if that's not the case, this may require clarifying the spec or aligning it with the invariant
16:39:57 <dom> fippo: the problem is that we're trying to treat local and remote candidates the same
16:40:06 <dom> ... but local candidates can have more info
16:40:25 <dom> youenn: that's why I thought the event was a good way to expose local information
16:40:34 <dom> Fippo: in stats we distinguish a lot between local & remote
16:40:59 <dom> jan-ivar: my preference is to not send it to remote parties, and so not include it in toJSON
16:41:22 <dom> ... the design pattern for events is also not to expose properties on the event when it can be exposed on the underlying object
16:41:35 <dom> ... so I lean towards exposing it in the object
16:42:03 <dom> youenn: we then at least to change the constructor
16:42:23 <dom> harald: the candidate is behaving like a data object, without inherent behavior
16:42:51 <dom> ... people expect to copy data objects, and they would expect toJSON() to allow this - breaking such a pattern is a bad idea
16:43:15 <dom> ... we have a backwards compatibility problem since toJSON is used to send data to remote parties
16:43:35 <dom> ... I think it was a mistake to use toJSON for transmission
16:43:46 <dom> ... I think putting the url data on the candidate is right
16:45:12 <dom> ... I think the right direction would be to add a method that only exposes the right info for the remote party
16:45:28 <dom> Youenn: another approach would be to distinguish local vs remote candidates
16:45:36 <dom> Harald: that's an interesting idea
16:45:58 <dom> Jan-Ivar: I agree we have a wart here, but I don't think we should chase technical purity
16:46:22 <dom> ... subclassing would not solve the backwards compat issue
16:46:45 <dom> Youenn: let's iterate on this discussion on github
16:47:10 <dom> Subtopic: Issue #2796: A simulcast transceiver saved from rollback by addTrack doesn’t re-associate, but unicast does
16:47:10 <dom> [slide 28]
16:47:30 <dom> Jan-Ivar: more corner cases, esp to with rollbacks
16:51:18 <dom> Harald: proposal seems reasonable to me
16:51:22 <dom> Bernard: +1
16:51:29 <dom> Harald: Jan-Ivar will propose a PR
16:51:41 <dom> Subtopic: Issue #2724: The language around setting a description appears to prohibit renegotiation of RIDs
16:51:41 <dom> [slide 29]
16:52:10 <dom> Jan-Ivar: see also PR #2794
16:53:36 <dom> [slide 30]
16:54:12 <dom> Jan-Ivar: this would match Chrome & Safari, although there is a remaining inconsistency identified in Chrome
16:54:43 <dom> Harald: this is the one where you discovered Chrome disabled layers rather than removing them
16:54:58 <dom> ... this sounds reasonable given our previous agreement on this
16:55:10 <dom> Jan-Ivar: it's a small change that doesn't introduce new behaviors, but extend them
16:55:25 <dom> Harald: I think this works
16:55:26 <dom> ... Will you add tests to?
16:55:37 <dom> Jan-Ivar: yes, along with FF implementation of setParameters
16:55:41 <dom> s/to?/too?/
16:56:31 <dom> Topic: -> https://github.com/w3c/webcodecs Timing Model & WebCodecs
16:56:50 <dom> Bernard: the gorup created a videoframemetadata registry with a process
16:57:09 <dom> ... an example of that is the request to register human face metadata #607
16:57:24 <dom> ... this also relates to the requestVideoFrameCallback spec (being merged in HTML)
16:57:38 <dom> ... which also exposes metadata and whether they should be exposed there as well
16:58:17 <dom> ... the rVFC spec exposes timing info at all aspects of the pipeline (captureTime, rtpTimestamp, receiveTime, processingDuration, expectedDisplayTime, presentationTime)
16:58:38 <dom> ... it mixes codec-related timing but also rtp-related info
16:58:57 <dom> ... this brings up a number of questions: where is the metadata exposed in our APIs (e.g. mediacapture transform)
16:59:16 <dom> ... should I expect .captureTime to be visible in a videoframe
16:59:42 <dom> ... likewise, there are assumptions on whatthings should happen in WebRTC (e.g. setting the rtpTimestamp)
17:00:25 <dom> ... is metadata passed through the pipeline: converting a video frame with mediacapture transofrm and pass it to webrtc - is this still visible at the end in rVFC? in encoded transform?
17:00:36 <dom> ... in WebCodecs encoded chunks?
17:01:01 <dom> ... do we need to file related issues?
17:01:37 <dom> Youenn: I filed some of these issues - captureTime etc are planned to move to videoframemetadata
17:01:58 <dom> ... that should bring consistency throughout the pipeline
17:02:21 <dom> ... mediacapture transform will not perserve it magically - if you clone the frame, metadata will be clone along with it
17:02:31 <dom> ... likewise if it goes through WebRTC PC
17:02:57 <dom> ... encoded chunks doesn't expose that metadata - maybe we should; we haven't heard feedback or use cases for that yet
17:03:47 <dom> ... in terms of what the WG may need to discuss: how do we compute presentationTime? VideoTrackGenerator allows to set timestamp, but we're not defining what happens on rendering (e.g. re jitter buffer)
17:04:29 <dom> Harald: if a processing element has metadata defined both as part of input & output, should we have a general rule about metadata it doesn't understand?
17:04:54 <dom> ... for the metadata info it knows about (e.g. width and height for an encoder), it won't remain unchanged
17:05:13 <dom> ... but for metadata that isn't understood, should have a rule to leave it unchanged?
17:05:36 <dom> Bernard: the registry rule is that this is up to the registry-linked spec to define
17:05:48 <dom> ... not sure we can have a rule that is imposed to all WGs
17:06:06 <dom> ... a rule would have to be proposed to be enforced
17:07:02 <dom> Youenn: individual metadata spec could describe how they're handled by processors
17:07:23 <dom> Bernard: next step would be to file specific issues on specific specs
17:07:33 <dom> Youenn: the main remaining issue might be on rendering time
17:07:37 <dom> ... in media capture main
17:07:59 <dom> Topic: -> https://github.com/w3c/mediacapture-extensions/pull/78 Face Detection
17:07:59 <dom> [slide 33]
17:08:37 <dom> Tuukka: the face detection proposal now uses videoframemetadata object
17:10:55 <dom> [slide 34]
17:13:39 <dom> slide 35]
17:16:24 <dom> [slide 36]
17:18:43 <dom> [slide 3è]
17:18:46 <dom> s/è/7
17:19:39 <dom> [slide 38]
17:21:17 <dom> Tuukka: looking for feedback on the general direction
17:21:35 <dom> Youenn: thanks - looks like a great improvement, and exciting to see this moving forward
17:21:52 <dom> ... dictionary members probably don't need to be nullable, but some may need to be marked as required
17:22:19 <dom> ... re center points vs bounding box vs best possible contours: I'm not sure if a sequence is best vs different fields
17:22:44 <dom> ... not sure about faceDetectionMaxCountourPoints - do we really need this now? can we leave this for later? or have a hint?
17:23:10 <dom> ... if developers just want a bounding box, maybe we should let developers express it, and send back a detailed contour otherwise
17:23:24 <dom> ... the example may need an update wrt @@@
17:23:37 <dom> ... I guess this means the proposal will be split across webcodecs and mediacapture-extensions
17:23:52 <dom> Tuukka: the metadata and the constraints are both specified in mediacapture extensions
17:24:00 <dom> ... are you suggesting the former should be done in webcodecs?
17:24:11 <dom> Youenn: not sure - I guess this is testing the registry process
17:24:32 <dom> ... the registry entry could either define the metadata or link to the mediacapture extensions spec
17:25:15 <dom> Tuukka: the constraints and metadata are co-dependent
17:25:25 <dom> ... they need to be maintained together
17:25:55 <dom> youenn: that makes sense; webcodecs has been asking to be able to review metadata when they change, so it may be best to have something in webcodecs space
17:26:06 <dom> ... we can iterate with webcodecs folks on the details
17:26:14 <dom> timp: I like this - looks useful & interesting
17:26:39 <dom> ... it would be good to document the lifespan and meaning of the id - in particular, that it doesn't allow to correlate faces across streams
17:27:26 <dom> ... re contour & bounding box, I agree with Youenn that they're not the same and should be handled separately, not rely on 4 items == bounding box
17:28:12 <dom> tuukka: the goal here was to avoid cluttering metadata as new contour approaches emerge
17:28:36 <dom> jan-ivar: looking at the broader question of merging this
17:29:01 <dom> ... from a privacy perspective, it looks like it doesn't add any concerns over having the detection done in JS
17:29:11 <dom> ... this looks good to me
17:30:05 <dom> Youenn: let's see a PR that editors can iron out and then run a CfC?
17:30:43 <dom> Topic: -> https://github.com/w3c/mediacapture-handle/issues/70 MessagePort on Capture Handle
17:30:43 <dom> [slide 44]
17:31:55 <dom> [slide 45]
17:32:49 <dom> [slide 46]
17:34:22 <dom> [slide 47]
17:34:57 <dom> [slide 48]
17:36:21 <dom> [slide 49]
17:38:03 <dom> [slide 50]
17:39:04 <dom> Youenn: having a message channel between capturer and capture makes sense
17:39:42 <dom> ... a few things off in the API shape that we can iterate on (e.g. event handler in a dictionary - they're usually on objects)
17:39:52 <dom> ... I'm not sure about the "supportsMessagePort" boolean
17:40:04 <dom> ... I would prefer we start from a minimal API surface
17:40:19 <dom> ... also for messageportinvalidated - we should discuss this with the HTML spec folks
17:40:39 <dom> ... this underlying behavior already exists with other messageports
17:41:21 <dom> ... I would prefer a name different "getMessagePort" given its side effects
17:41:29 <dom> ... I like the integration with capture handle
17:41:47 <dom> elad: +1 to "openMessagePort" instead of get...
17:42:13 <dom> ... I'm happy to discuss reduction of API surface
17:42:37 <dom> ... s/handle/controller
17:43:06 <dom> ... my proposal deals both with capture handle and controller - how do you feel about integration with handle?
17:43:18 <dom> Youenn: event handler in a dictionary feels wrong
17:43:30 <dom> ... don't have strong feelings on handle vs mediaDevices in general
17:44:30 <dom> elad: the link to capture handle happens both on capturer & capturee
17:44:53 <dom> ... you commented on only one side?
17:45:05 <dom> youenn: on the other side, I would move it to capture controller
17:46:05 <dom> jan-ivar: I really like the 1st part of the presentation - agree on use cases & requirements
17:46:13 <dom> ... would like to iterate on github on the API shape
17:46:29 <dom> ... generally would agree with youenn to move it to controller rather than track
17:46:44 <dom> ... I think the direction you're presenting makes sense as a starting point
17:47:16 <dom> Elad: so next steps is to surface similar events following that pattern on capture controller
17:48:05 <dom> ... we should revisit this a the next meeting
17:48:14 <dom> Topic: -> https://github.com/w3c/mediacapture-main/pull/912 enumerateDevices & Focus
17:48:14 <dom> [slide 53]
17:49:52 <dom> jan-ivar: PR #912 allows the behavior in Safari by relaxing the focus requirements a little bit
17:49:56 <dom> [slide 54]
17:50:49 <dom> [slide 55]
17:52:50 <dom> Youenn: I like this proposal; LGTM
17:53:24 <dom> Harald: my reading is that it waits after the gUM prompt has been replied to?
17:53:41 <dom> jan-ivar: after it has shown up, not responded to (since that requires focus in any case)
17:54:14 <dom> harald: I'll re-read the PR carefully to make sure it doesn't introduce issues
17:54:55 <dom> Elad: can you clarify the "anti-spying" behavior?
17:55:11 <dom> Jan-Ivar: the PR doesn't change the focus requirement, only its timing
17:55:18 <dom> Elad: ok, I'll bring the question on github then
17:55:25 <dom> [slide 56]
17:56:04 <dom> Jan-Ivar: we also had developers complaining that enumerateDevices() block when there is no focus (which is marked an optional behavior)
17:56:16 <dom> ... the PR proposes to make it tied to visibility, not focus
17:56:48 <dom> ... this helps backwards compat, and still satisfies the anti-fingerprinting requirement (anti-spying only applies to getUserMedia)
17:57:21 <dom> ... this would make the check deterministic as requested by the developer
17:58:52 <dom> [slide 57]
18:00:05 <dom> Youenn: so the goal is to reduce friction for developers and align user agents behaviors - that's a good goal
18:00:18 <dom> ... do you foresee compat issues in implementing this?
18:01:16 <dom> ... will it fix existing firefox issues that developers were complaining about or does that require developers adoption before it does?
18:01:44 <dom> jan-ivar: they would have to add the visibilityState check to avoid being "blocked"
18:02:10 <dom> elad: I could use more time to review this
18:02:49 <dom> youenn: I think it would be good to get feedback from other UAs and developers
18:02:56 <dom> Bernard: do we need a CfC?
18:03:49 <dom> Jan-Ivar: developers should be happy given that it relaxes the behavior
18:03:57 <dom> Dom: does this need an updated privacy review?
18:04:06 <dom> jan-ivar: I don't think so since the behavior was already optional
18:04:25 <dom> ... and the fuzzing advice is already in the spec
18:05:04 <dom> harald: I'll have to review this in details
18:05:32 <dom> Dom: so we can delegate this for final review by Harald, Elad & Youenn?
18:05:34 <dom> JIB: SGTM
18:05:40 <dom> RRSAgent, draft minutes
18:05:40 <RRSAgent> I have made the request to generate https://www.w3.org/2022/11/15-webrtc-minutes.html dom
18:05:50 <dom> RRSAgent, make log public
18:06:51 <dom> s/slide 35/[slide 35/
18:06:52 <dom> RRSAgent, draft minutes
18:06:52 <RRSAgent> I have made the request to generate https://www.w3.org/2022/11/15-webrtc-minutes.html dom
18:30:35 <Zakim> Zakim has left #webrtc