14:47:00 <RRSAgent> RRSAgent has joined #webrtc
14:47:00 <RRSAgent> logging to https://www.w3.org/2022/04/26-webrtc-irc
14:47:01 <Zakim> Zakim has joined #webrtc
14:47:07 <dom> Meeting: WebRTC April 2022 VI
14:47:16 <dom> Chair: Harald, Jan-Ivar, Bernard
14:47:16 <dom> Recording: https://www.youtube.com/watch?v=qSlXLqouxCs
14:47:29 <dom> Agenda: https://www.w3.org/2011/04/webrtc/wiki/April_26_2022
14:49:44 <dom> Slideset: https://lists.w3.org/Archives/Public/www-archive/2022Apr/att-0004/WEBRTCWG-2022-04-26.pdf
14:50:33 <dom> agenda+ -> https://github.com/webmachinelearning/webnn/issues/226 WebNN Integration with real-time video processing
14:51:01 <dom> agenda+ -> https://github.com/w3c/webrtc-extensions WebRTC Extensions
14:51:23 <dom> agenda+ WebRTC-PC Simulcast Issues
14:52:03 <dom> agenda+ -> https://github.com/w3c/mediacapture-extensions/issues/47 Voice Isolation Constraint
14:52:33 <dom> agenda+ -> https://github.com/w3c/mediacapture-handle/issues/35 support for contentHint in Capture Handle
14:52:52 <dom> agenda+ -> https://github.com/w3c/mediacapture-screen-share/issues/219 Avoid user-confusion by avoiding offering undesired audio sources
14:53:17 <dom> agenda+ -> https://github.com/w3c/mediacapture-region Region Capture
14:53:37 <dom> agenda+ CaptureController
15:01:15 <dom> Present+ Sergio, Jan-Ivar, TimP, Elad, PatrickRockhill, Anssi, Ningxin, Dom
15:01:32 <dom> Recording is starting
15:02:03 <dom> Present+ Youenn
15:02:11 <dom> Present+ Harald
15:02:40 <dom> Present+ PhilippHancke
15:03:33 <ningxin_hu> ningxin_hu has joined #webrtc
15:03:46 <jib> jib has joined #webrtc
15:04:05 <youenn> youenn has joined #webrtc
15:04:19 <dom> Zakim, next item
15:04:19 <Zakim> agendum 1 -- -> https://github.com/webmachinelearning/webnn/issues/226 WebNN Integration with real-time video processing -- taken up [from dom]
15:05:01 <dom> [slide 10]
15:05:05 <dom> [slide 11]
15:06:03 <dom> [slide 12]
15:06:04 <youenn> [slide 12]
15:06:23 <dom> Present+ Florent
15:06:32 <youenn> [slide 13]
15:07:00 <youenn> [slide 15]
15:08:15 <youenn> ningxin: slide 15 is high level pipeline to build a background blur video pipeline
15:09:22 <dom> Present+ MichaelSeydl
15:09:42 <dom> Present+ Carine
15:10:24 <youenn> Two implementations: WebGL and WebGPU/WebNN.
15:10:37 <youenn> texture uploads to GPU in both cases.
15:12:40 <youenn> Last shader is taking 3 input: original image, blurred image, and computed segmentation map.
15:15:05 <TimPanton> TimPanton has joined #webrtc
15:15:46 <TimPanton> q+
15:17:12 <youenn> Description of the perf issues, in particular CPU usage and GC.
15:18:52 <youenn> Bernard: is there a copy on the output at offscreencanvas level?
15:18:57 <youenn> Ningxin: not sure.
15:19:29 <dom> ack TimPanton
15:19:49 <youenn> Tim: is the perf acceptable? or do we need to make massive improvements?
15:20:17 <youenn> ningxin: we need to measure battery impact
15:21:46 <youenn> dom: we are doing this prototype to evaluate what HW acceleration can bring us. And identify potential roadblocks when trying to do video processing on media capture
15:22:28 <youenn> for instance cooler conversion or pixel format.
15:22:34 <youenn> s/cooler/color/
15:22:39 <dom> scribe+
15:23:13 <dom> youenn: looking at 20% CPU on GC - can that be fixed by implementations, or is it an architectural issue with having lots of objects created per frame?
15:23:23 <dom> ... on native, there is usually a buffer pool to help with that
15:23:41 <dom> ... does that need to be surfaced to the JS, or can that be dealt solely by the UA?
15:25:01 <youenn> ningxin: GPUBuffers are created beforehand. Some objects are created for every frame, like textures.
15:25:17 <youenn> There are ways to avoid many object allocations.
15:25:44 <youenn> at JS level. Maybe UA optimisations might help.
15:26:01 <youenn> dom: what are the next steps for this project?
15:26:18 <youenn> ningxin: 1. enable WebGPU backend.
15:27:14 <youenn> 2. new APIs that allow import frames as GPU textures and see whether that will improve efficiency.
15:27:43 <youenn> 3. Improve VideoFrame GC PR: we will try out when it is merged in Chrome.
15:28:11 <dom> Present+ ChrisCunningham
15:28:37 <dom> youenn: re CPU efficiency - this is moving between main thread and worker thread, that may have a small perf impact
15:28:50 <dom> ... doing everything in the worker might be helpful once that's possible
15:30:52 <dom> Zakim, next item
15:30:52 <Zakim> agendum 2 -- -> https://github.com/w3c/webrtc-extensions WebRTC Extensions -- taken up [from dom]
15:31:02 <dom> [slide 19]
15:31:12 <youenn> Issue 95
15:31:46 <dom> -> https://github.com/w3c/media-capabilities/issues/185 Media Capabilities issue 185
15:32:16 <dom> [slide 20]
15:33:00 <dom> [slide 21]
15:33:48 <dom> [slide 22]
15:34:03 <dom> [slide 23]
15:34:21 <dom> [slide 24]
15:34:42 <youenn> Bernard: question to WG is: is it a goal for MC to deprecate getCapabilities?
15:35:44 <dom> youenn: my understanding is that media capabilities is really about audio/video capabilities
15:35:53 <dom> ... so it doesn't make sense to expose e.g. CN there
15:36:03 <dom> ... they should stay in WebRTC getCapabilities
15:36:37 <dom> ... getCapabilities() being sync is problematic; that's less of an issue for software capabilities such as CN
15:36:56 <dom> ... so deprecated getCapabilities fully is not a goal, but partially, yes
15:39:12 <dom> Florent: +1 on the approach usability of resulting split is a concern
15:39:43 <dom> chris: seems fine to use that split; do we want to return rtc codec info from media capabilities?
15:40:00 <dom> ... if so, please take at look at https://github.com/w3c/media-capabilities/issues/185
15:40:35 <dom> youenn: +1 on disambiguating the outcome of this situation
15:40:47 <dom> ... listing all codecs is a non goal
15:41:03 <dom> ... an SFU is typically only interested in a few codecs
15:41:25 <dom> ... for P2P, setCodecPreferences is probably not needed in the first place - you can deal with a generic codec negotiation
15:41:47 <dom> s/all codecs/all codecs in just one call/
15:43:00 <dom> jib: would be good to clarify if we want to deprecate "real" codecs from getCapabilities? this sounds like a good long term goal for me
15:43:26 <dom> harald: I worry that RTX/RED/FEC info needs to be available somewhere
15:43:40 <dom> ... getCapabilities has known problems and would be the only way to get it
15:44:11 <dom> ... changing getCapabilities is actually harder to deprecating it
15:45:23 <dom> ... in the long run, it's best to deprecated getCapabilities and replace it with a better dedicated API
15:46:17 <dom> Florent: two different scenarios for setCodecPreferences: talking with an SFU in which case you can make specific codec queries; in a P2P scenario, if you can't enumerate all the codecs, you won't be able to call setcodecpreferences
15:46:29 <dom> ... this would require hardcoding a list of codecs
15:46:44 <dom> ... is there a way to make getCapabilities evolve in a shape that would satisfy everyone?
15:47:10 <dom> ... getCapabilities+setCodecPreferences has a lot of current usage, will be hard to deprecated
15:47:19 <youenn> Issue 100
15:47:22 <youenn> [slide 25]
15:47:46 <dom> s/Issue 100/Subtopic: Issue #100
15:48:04 <dom> s/Issue 95/Subtopic: Issue #95
15:48:32 <dom> [slide 27]
15:49:09 <dom> youenn: might be fine, but I worry about the defaults? would they be the same across browsers?
15:49:22 <dom> ... there are current codecs that are defaults, but that may need to evolve over time
15:49:28 <dom> ... this could create Web compat issues
15:49:48 <dom> Sergio: some of the codecs are receive-only
15:50:21 <dom> ... the list would be based on common sense, but I don't have a strong opinion
15:50:46 <dom> youenn: my worry is about P2P - if the defaults aren't same across UAs, the negotiation will fail
15:51:09 <dom> sergio: my suggestion was to use defaults in the offer, and adapt the answer based on the offer
15:52:07 <dom> harald: two interfaces needed: the list of codecs currently willing to offer, the set of codecs you can offer
15:52:22 <dom> ... the 1st one might be getCapabilities, the proposal on the slide for the 2nd
15:53:04 <dom> ... in terms of interop, MTI codecs should be the safety net, and they should be in the mandatory-to-offer
15:53:38 <dom> florent: the proposal seems ot have a lot of overlap with setCodecPreferences / getParameters - could we improve these instead of coming up with new API
15:53:50 <dom> [Philipp supports this on the chat]
15:54:42 <dom> Sergio: would be fine; I started from the rtp header extensions, maybe that should apply there?
15:54:55 <dom> florent: the difference is that there is already an API to set codec preferences
15:55:03 <dom> sergio: but header extensions could be added there too?
15:55:44 <dom> Bernard: let's continue the discussion in the issue
15:55:49 <dom> ... or work on a matching PR
15:56:09 <dom> Zakim, pick agendum 4
15:56:09 <Zakim> I don't understand 'pick agendum 4', dom
15:56:16 <dom> Zakim, take agendum 4
15:56:16 <Zakim> I don't understand 'take agendum 4', dom
15:56:20 <dom> Zakim, take item 4
15:56:20 <Zakim> I don't understand 'take item 4', dom
15:56:24 <dom> Zakim, item 4
15:56:24 <Zakim> I don't understand 'item 4', dom
15:56:28 <dom> Zakim, agendum 4
15:56:28 <Zakim> I don't understand 'agendum 4', dom
15:56:48 <dom> Topic: https://github.com/w3c/mediacapture-extensions/issues/47 Voice Isolation Constraint
15:56:56 <dom> [slide 41]
15:57:07 <dom> [slide 42]
15:57:33 <youenn> Resolution for issue 95: mark issue as ready for PR
15:58:11 <dom> [slide 43]
15:59:33 <dom> [slide 44]
16:00:24 <dom> [slide 45]
16:01:09 <dom> youenn: it makes sense; reasonable to ignore `noiseSuppression`
16:01:28 <dom> ... there is also `echoCancellation` in the audio pipeline
16:01:43 <dom> ... does it make sense to do `echoCancellation` when this is set?
16:02:03 <dom> harald: I think it's mostly orthogonal
16:02:23 <dom> youenn: so `echoCancelation: false` is compatible with `voiceIsolation: true`
16:02:45 <dom> ... it may be challenging for some implementations to support these combinations
16:03:44 <dom> jan-ivar: I like this too; what should the default be? that may bring concerns
16:03:49 <dom> harald: we can discuss this in the PR
16:04:03 <dom> ... conservatively, the default should be the current behavior (false)
16:04:53 <youenn> dom: instead of boolean, we could use strings for extra flexibility.
16:05:39 <youenn> Resolution: mark issue as ready for PR
16:05:53 <dom> s/issue/voiceIsolation issue #47/
16:05:59 <dom> Zakim, take up item 5
16:05:59 <Zakim> agendum 5 -- -> https://github.com/w3c/mediacapture-handle/issues/35 support for contentHint in Capture Handle -- taken up [from dom]
16:06:09 <dom> [slide 48]
16:07:31 <dom> [slide 49]
16:07:55 <dom> [slide 50]
16:09:39 <dom> [slide 51]
16:11:06 <dom> youenn: setting the track hint is unnecessary - if the capturer is setting the hint on its side, the UA knows that the track being captured is text - there is no need to transmit it to the capturer
16:11:16 <dom> ... except maybe if WebCodecs is the picture
16:11:43 <dom> ... having the UA taking care of this seems preferable
16:12:31 <dom> elad: the suggestion would be that the captured content self-declare its type and the UA uses it?
16:12:59 <dom> ... but that removes the liberty of the capturer to decide whether to use the hint or not
16:13:44 <dom> ... which could be based on e.g an allowlist
16:14:04 <dom> ... autodetection by the UA would have its own limitation
16:14:46 <dom> bernard: re the WebCodecs case - contentHint is not automatically consumed by WebCodecs, it's up to the app to apply it as codec setting
16:15:56 <dom> jib: I agree with youenn that the UA is in good place to shortcircuit the capturer part
16:16:13 <dom> ... the proposal could be useful for the capturee side
16:17:04 <dom> ... exposing further metadata to the controller might be an interesting addition to my capturecontroller proposal
16:17:23 <dom> youenn: it could be exposed at the videoframe level
16:17:32 <dom> jib: I see agreement on the need, not yet on the API shape
16:17:43 <dom> Zakim, next item
16:17:43 <Zakim> agendum 2 -- -> https://github.com/w3c/webrtc-extensions WebRTC Extensions -- taken up [from dom]
16:17:46 <dom> Zakim, next item
16:17:46 <Zakim> agendum 2 was just opened, dom
16:17:55 <dom> Zakim, take up item 6
16:17:55 <Zakim> agendum 6 -- -> https://github.com/w3c/mediacapture-screen-share/issues/219 Avoid user-confusion by avoiding offering undesired audio sources -- taken up [from dom]
16:18:03 <dom> [slide 54]
16:18:55 <dom> [slide 55]
16:19:25 <dom> [slide 56]
16:21:26 <youenn> Tim: is this only applicable for echo management?
16:21:56 <youenn> elad: it could be that an application is interested in recording a specific tab, no more than that.
16:22:21 <youenn> Tim: this use case does not seem address: identifying the desired tab would be needed.
16:24:50 <youenn> Elad: some VC applications usually do not want to capture system audio.
16:25:38 <youenn> Jan Ivar: supportive, how about reusing displaySurface constraint here?
16:25:43 <youenn> Elad: Might work for me.
16:26:12 <youenn> Jan Ivar: I would like to remove monitor from here.
16:28:04 <youenn> dom: if we do not include monitor here, audio: true might capture system audio. But applications would not be able to explicitly ask for system audio.
16:28:38 <youenn> dom: displaySurface would be a strange name for audio.
16:33:08 <dom> youenn: let's enumerate the different approaches: avoidSystemAudio, displaySurface, sources
16:33:16 <youenn> youenn: scope is unclear, we need to clarify this before going to PR.
16:34:08 <dom> youenn: different properties allow to do feature detection on what kind of recording the UA can do
16:34:47 <dom> elad: my focus is only limiting access to system audio, but I also think flexibility is helpful
16:35:12 <dom> timp: back to my echocancellation point - the constraint could be linked to whether the source can be echocancelled
16:35:47 <youenn> Harald: source being echo cancellable is a second concern. Biggest point is avoiding system audio.
16:35:52 <youenn> Tim: as well as window audio.
16:35:57 <dom> harald: echoCancellation is a secondary concern - capturing system audio could disclose info from a 3rd party
16:39:57 <dom> Zakim, take up item 7
16:39:57 <Zakim> agendum 7 -- -> https://github.com/w3c/mediacapture-region Region Capture -- taken up [from dom]
16:40:00 <youenn> Resolution: continue discussions on GitHub.
16:40:15 <dom> [slide 59]
16:40:39 <dom> youenn: #11 is an issue on the shape of the CropTarget API
16:40:53 <dom> ... given current chrome implementation work, feels it's useful to converge on the API shape
16:41:17 <dom> [slide 60]
16:41:41 <dom> youenn: do we want to attach the API to element or to MediaDevices?
16:44:04 <dom> ... element feels like a better path
16:44:15 <dom> jib: +1
16:44:54 <dom> elad: I prefer mediaDevices given its linkage to screen capture
16:45:29 <dom> youenn: cropTarget is linked to MediaStreamTrack, not mediaDevices
16:45:34 <dom> ... and it's really tied to an element
16:46:03 <dom> elad: it can be used through an object you get from getDisplayMedia
16:47:50 <dom> youenn: but with a detached mediaDevices, you can't reject the promise
16:49:31 <youenn> dom: prefer element option.
16:52:32 <dom> youenn: next question is attribute vs method
16:52:45 <dom> ... slight pref for attribute, but no strong feeling
16:53:19 <dom> elad: there is a cost to minting a crop target - we mark the element in the rendering pipeline in specific ways that we shouldn't abuse
16:53:33 <dom> youenn: I thought you were going to use a lazy approach to reduce that cost
16:54:29 <dom> elad: lazy tagging might help, but this needs more thinking
16:55:12 <dom> jib: +1 to attribute
16:55:29 <dom> ... developers value trump implementators value
16:55:50 <dom> elad: I don't think it matters much to developers in the first place
16:56:22 <dom> harald: disagree with messing with the element interface, and on hiding the fact that the operation has a cost
16:57:36 <dom> ... also async (promises) may be needed for some implementations
16:57:45 <dom> ... let's not hide the reality of the situation
16:58:03 <dom> jib: the cost seems to be Chrome-specific
16:58:17 <dom> ... the real goal of this API is a transferable reference
16:59:11 <dom> youenn: +1
17:00:10 <dom> ... other APIs in the past have re-used the element interface, have made similar decisions on methods / attributes, async vs sync
17:00:24 <dom> ... we should follow existing implemented platterns
17:01:24 <dom> dom: is there any other API that may be use this tranferable reference?
17:02:31 <dom> youenn: that's something I bring up in the issue
17:02:44 <dom> elad: this may create unsafe usage for this well-defined targett
17:02:53 <dom> s/platt/patt/
17:03:00 <dom> s/targett/target/
17:03:16 <dom> jan-ivar: this could be evaluated
17:03:31 <dom> hta: but this shouldn't block progress on the specific narrow goal we have
17:07:07 <dom> youenn: my focus is aligning with current API patterns for this API
17:08:15 <dom> elad: the TAG will chime in; but if they don't give a clear specific suggestion
17:09:00 <dom> ... we could move with the current design that can be polyfilled
17:12:14 <dom> RRSAgent, draft minutes
17:12:14 <RRSAgent> I have made the request to generate https://www.w3.org/2022/04/26-webrtc-minutes.html dom
17:12:24 <dom> RRSAgent, make log public
17:12:49 <dom> i/Meeting:/ScribeNick: youenn
17:12:51 <dom> RRSAgent, draft minutes
17:12:51 <RRSAgent> I have made the request to generate https://www.w3.org/2022/04/26-webrtc-minutes.html dom