13:41:21 <RRSAgent> RRSAgent has joined #webrtc
13:41:21 <RRSAgent> logging to https://www.w3.org/2022/06/23-webrtc-irc
13:41:49 <dom> Meeting: WebRTC June 2022 VI
13:41:49 <dom> Chair: HTA, Bernard, Jan-Ivar
13:41:49 <dom> Slideset: https://lists.w3.org/Archives/Public/www-archive/2022Jun/att-0006/WEBRTCWG-2022-06-23.pdf
13:41:49 <dom> Agenda: https://www.w3.org/2011/04/webrtc/wiki/June_23_2022
13:59:21 <dom> Regrets+ Carine
14:01:22 <eehakkin> eehakkin has joined #webrtc
14:01:39 <dom> Present+ Bernard, Dom, Elad, Florent, Harald, Jan-ivar, Patrick_Rockhill, Eero_Hakkinen
14:01:48 <dom> Present+ TuukkaT
14:01:53 <dom> Present+ TimPanton
14:02:49 <dom> Recording is starting
14:02:58 <dom> Present+ Guido
14:03:07 <dom> Present+ Youenn
14:04:16 <eehakkin_> eehakkin_ has joined #webrtc
14:04:19 <dom> scribe+
14:04:32 <dom> Present+ Riju
14:05:31 <dom> Topic: -> https://github.com/w3c/webrtc-charter/ WebRTC WG Re-Charter
14:05:31 <dom> -> http://w3c.github.io/webrtc-charter/webrtc-charter.html Draft updated charter for the WebRTC WG
14:05:31 <dom> [slide 9]
14:09:22 <steely-glint> steely-glint has joined #webrtc
14:09:32 <dom> dom: we need a new charter, and to discuss issue #70 (migrating work to another group)
14:09:42 <dom> youenn: would this require rechartering Media WG?
14:09:44 <dom> dom: not sure, possibly
14:10:00 <dom> aboba: we have a joint meeting with Media WG at TPAC
14:11:05 <dom> dom: but we can't wait until then to come to a conclusion
14:11:37 <dom> youenn: migrating items between charters is always challenging from our side, in administrative aspecs
14:12:47 <dom> bernard: do we need meetings with the media wg to discuss this?
14:13:06 <dom> dom: if we want to migrate the work, yes - but we first need to decide whether we want to do that
14:13:30 <dom> bernard: the fact that we have dependencies on videoframe makes it an interesting question to consider
14:13:43 <dom> elad: how busy is the Media WG? would it increase or decrease the pace of our work?
14:15:18 <dom> harald: that would have to be something to discuss with the chairs
14:15:43 <dom> dom: bringing it to media wg would bring more of a media perspective when this group comes more with a transmission perspective
14:16:35 <dom> harald: so the chairs will discuss this with the Media WG chairs
14:16:43 <dom> dom: no objection from the group in exploring this?
14:16:44 <dom> [none]
14:17:02 <dom> Topic: -> https://github.com/w3c/mediacapture-region Region Capture Issues
14:17:02 <dom> Subtopic: Issue #17: the case for making CropTarget Sync
14:17:02 <dom> [slide 13]
14:18:04 <dom> Jan-Ivar: this is about the cropping API - issue #17 is about whether it should be sync vs async
14:18:23 <dom> ... long discussion on the issue - I'll be presenting arguments why it should be sync
14:19:14 <dom> ... The TAG design principles include encouragement to use sync APIs when appropriate, with some exceptions (incl cross-process communications)
14:19:16 <dom> [slide 14]
14:19:52 <dom> Jan-Ivar: the API currently in the spec (that doesn't have consensus) is async
14:20:14 <dom> ... so you have to `await CropTarget.fromElement(element)`
14:20:29 <dom> ... I'm proposing it doesn't need to be async,
14:20:48 <dom> ... the purpose of the operation is associating  an identifier with an element
14:21:07 <dom> ... as currently specified, it can't fail
14:21:39 <dom> ... the goal of the crop target is to share it over postMessage across documents
14:21:47 <dom> [slide 15]
14:22:09 <dom> Jan-Ivar: multiple actions needs to happen before we're cropping anything
14:22:32 <dom> ... cropTarget can be done ahead of time or later
14:23:11 <dom> ... if it gets postMessaged to the top-level document, the said document can offer to the user to crop to that target
14:23:36 <dom> ... it's only at the end of this process that there is a clear intent to crop
14:24:09 <dom> ... UA can optimize this by running some of the underlying tasks early, but that creates risks in case this doesn't go through
14:24:21 <dom> ... the complexity of that situation shouldn't be exposed to developers
14:24:35 <dom> [slide 16]
14:25:02 <dom> Jan-Ivar: #48 is asking to allow failure from the minting process due to resource exhaustion
14:25:16 <dom> ... which seems linked to optimizations implemented in Chrome
14:25:34 <dom> ... The issue is that it allows random document to exhaust cropping resources
14:25:59 <dom> ... since the API is not gated by permissions - creating DOS risks
14:26:35 <dom> ... and may expose apps that don't deal well with that failure
14:26:52 <dom> ... if resource allocation is moved to the cropTo step, this risk disappears
14:28:00 <dom> ... similar to mediaSource.getHandle
14:28:53 <dom> [slide 17]
14:30:51 <dom> Jan-Ivar: I believe my proposed API is faster, simpler and still optimizable
14:31:08 <dom> ... I don't think we need to block on inter-process communication
14:31:09 <dom> [slide 18]
14:32:17 <dom> jan-ivar: failing of optimization shouldn't imply a failure of the operation
14:32:38 <dom> ... optimizations that influence API design decision tend to generate further issues
14:32:44 <dom> ... because optimizations create new side effects
14:33:07 <dom> ... there is no developer benefits to this API being async
14:33:44 <dom> ... and there are general developer costs to async APIs - they create pre-emption points which risk data races
14:34:16 <dom> ... multiplying failure points for rare error cases is a footgun
14:34:28 <dom> ... and async is slower as I've shown
14:35:30 <dom> elad: the TAG offers design principles, but also meta principles - not sure it's productive to discuss other browsers implementation
14:35:51 <dom> ... the fingerprinting risk is reasonable, but the spec doesn't force to surface this
14:36:06 <dom> ... a promise doesn't have to fail in your implementation either
14:36:52 <dom> jan-ivar: the API should be designed on principles
14:37:27 <dom> ... Mozilla is here to push a better API for the Web
14:37:34 <dom> Subtopic: Issue #17: the case for making CropTarget Async
14:37:34 <dom> [slide 20]
14:37:51 <dom> Elad: this API is used in production through origin trial
14:37:59 <dom> ... we know the API works and makes developers and customers happy
14:38:12 <dom> ... we've learned a lot of lessons by implementing and shipping this
14:38:16 <dom> [slide 21]
14:38:34 <dom> Elad: the question is whether minting a token needs to be sync or async
14:38:54 <dom> ... I'll explain the Chrome decision and that it doesn't impact negatively anyone else
14:39:01 <dom> [slide 22]
14:39:44 <dom> Elad: our implementation keeps track of which apps have a crop target
14:40:29 <dom> ... once it's postMessage'd, this allows Chrome to optimize the time-to-cropping when the cropTo method is actually invoked
14:40:52 <dom> ... that makes it simpler and more performant, in particular in case of CPU congestion in the capturing document
14:40:55 <dom> [slide 23]
14:41:20 <dom> [slide 24]
14:41:46 <dom> Elad: Chrome needs this to be async, whether Mozilla prefers it to be sync
14:42:09 <dom> ... what's the harm of having an async API?
14:42:11 <dom> [slide 25]
14:42:40 <dom> Elad: Priority of consistuencies goes through users, developers, implementers, spec writers
14:42:44 <dom> [slide 26]
14:42:55 <dom> Elad: from a user perspective, seems mostly orthogonal
14:43:31 <dom> ... from a developer perspective - what we've heard is that they don't care as it doesn't change much in their huge existing codebase
14:43:51 <dom> ... implementors - as an implementor, we see this as an imperative for us
14:43:58 <dom> [slide 27]
14:44:33 <dom> Elad: what is the negative impact then? Is this theoretical purity?
14:44:42 <dom> ... given that IPC is involved, async makes sense
14:44:42 <dom> [slide 28]
14:45:07 <dom> Elad: the TAG actually insists that theoretical purity doesn't trump implementers needs
14:45:10 <dom> [slide 29]
14:45:19 <dom> Elad: the TAG discussed this API
14:45:43 <dom> ... they were satisfied with the API
14:45:49 <dom> ... they haven't seen much ergonomic gain for sync
14:46:09 <fbeaufort> fbeaufort has joined #webrtc
14:46:16 <dom> ... they also highlighted that interop should drive the work of the group
14:46:18 <dom> [slide 30]
14:46:40 <dom> Elad: I've shown that consistuencies either don't care about sync vs async, and that at least one implementor needs async
14:46:59 <dom> ... also, it's easy to go from async to sync, while the other way is difficult
14:47:18 <dom> Subtopic: #17 discussion
14:47:41 <dom> Bernard: by making it async, you're saying that this implies that when you have a cropTarget, you know it's ready to use
14:47:58 <dom> ... we've had situations in the WebRTC WG where we've found that sync APIs needed in fact to be async
14:48:08 <dom> Elad: cf slide 22
14:50:27 <dom> Youenn: I'm surprised you're saying this is MUST - I thought this was implementable as a sync API, but that you favored the trade-off that async allows
14:51:10 <dom> ... as I've pointed out, this trade-off creates a fingerprinting and interop issue - so a footgun
14:51:41 <dom> ... so I thought both were implementable but sync would be more complex in Chrome
14:51:52 <dom> Elad: I don't agree with that characterization
14:52:09 <dom> Youenn: I'm surprised that both approaches are faster - both can't be true
14:52:17 <dom> s/are faster/are claiming to be faster/
14:52:50 <dom> Youenn: usually sync APIs are more efficient, except when they creating a blocking situation, which I'm not hearing is the case here
14:52:58 <dom> ... a sync API helps developers, at least a bit
14:54:10 <dom> Elad: resource allocation design decision is orthogonal to sync vs async
14:55:02 <dom> ... I don't think the resource mitigation limits fingerprinting risks can be mitigated through a per-iframe limit
14:55:50 <dom> ... in terms of performance, what needs to be fast is cropTo - anything before that, the user doesn't notice
14:55:52 <anssik> anssik has joined #webrtc
14:57:19 <dom> youenn: sync cropTarget minting is faster, but you're saying this is not a relevant optimization compared to cropTo
14:57:53 <dom> elad: anything that comes before cropTo is irrelevant to user perceived performance (and mostly negligible in any case)
14:58:24 <dom> TimP: as a developer, I have a mild preference to keep it sync as it's easier to use
14:58:33 <dom> ... I don't think a developer benefit to making it async
14:58:49 <dom> ... there may be a user benefit in terms of the crop transition UX
14:59:06 <dom> ... that may convince me of the value if it can be shown
15:00:01 <dom> ... from the developer perspective, managing interesting failures on obtaining target would also be convincing, but I haven't heard that
15:00:26 <dom> Elad: it's a particular choice of trade-offs that require async, and async isn't going to harm other consistuencies
15:00:31 <dom> TimP: developers will suffer
15:00:37 <dom> Elad: I claim it's negligible
15:01:04 <dom> TimP: I don't agree - this can generate non-trivial changes, although it's certainly doable
15:02:04 <dom> Jan-Ivar: re no downside - nobody claiming that Chrome optimizations aren't useful
15:02:13 <dom> ... I don't understand why these optimizations can't be done with a sync API
15:02:24 <dom> ... why can't you implement a fallback?
15:02:51 <dom> ... other downsides include fingerprinting, DOS, proliferation of async, failure management for developers
15:03:08 <dom> ... why does it need to be async?
15:03:36 <dom> Harald: code complexity itself is a risk; this particular implementation has been used and tested
15:04:28 <dom> ... anyone that depends on cropTo and doesn't notice it failing as an issue
15:04:52 <dom> ... it's time to stop this discussion - we have seen that Chrome claims that a sync implementation would be make it significantly more complex
15:05:08 <dom> ... the impact on developers is irritating but not fatal
15:05:23 <dom> ... we have not seen compelling arguments that we need to change what has been proposed
15:05:58 <dom> ... I don't see consensus for change, there is an implementation of the current spec - I suggest we declare the API to be async and move on
15:06:40 <dom> Jan-Ivar: I hear ease of implementation - complexity in the API trumps complexity of implementation
15:06:49 <dom> ... because of the priority of consistuencies
15:07:15 <dom> Harald: a more complex API that does the right thing is better than a simple API that does the wrong thing
15:08:18 <dom> Elad: I'm not seeing consensus, but I don't think there are remaining benefits to discuss this
15:09:34 <dom> dom: we could either run a vote, or wait for more implement experience
15:10:17 <dom> s/implement/implementation/
15:10:52 <dom> TimP: I would be inclined to say that getting other implementation is most important
15:11:03 <dom> ... sync would be more elegant, but it doesn't look like we're going to get that
15:11:20 <dom> Subtopic: Issue #18: Is CropTarget name too generic?
15:11:20 <dom> [slide 33]
15:12:41 <dom> Youenn: "crop" as at term isn't used too broadly so far, so probably OK
15:12:43 <dom> ... not sure that "target" helps
15:12:48 <dom> [slide 34]
15:13:45 <dom> [slide 35]
15:14:48 <dom> Youenn: "object whose sole purpose is to be given cropTo" - may not be limited to elements
15:14:59 <dom> ... the term CropRegion might work better to represent
15:15:11 <dom> ... and would align with the spec name ("region capture")
15:15:57 <dom> ... thoughts?
15:16:39 <dom> TimP: the name should reflect that it is a token, not a region or a target itself
15:16:45 <dom> ... if it's opaque, it should say so
15:16:57 <dom> Youenn: it may not remain opaque
15:17:10 <dom> TimP: a region sounds like something you could do math on, e.g. calculate its surface area
15:17:17 <dom> ... which you can't
15:17:50 <dom> elad: similar reservation - region feels something with coordinates; also, a cropTarget isn't static, it can move, which cropregion makes more misleading
15:18:19 <dom> harald: this is bikeshedding; I don't see benefit in changing it
15:18:44 <dom> Bernard: +1 that CropRegion is confusing; I prefer the current name
15:19:16 <dom> Youenn: any interest in clarifying the definition (i.e. whether it's a reference to an element or something more generic)
15:19:24 <dom> ... I guess that can be done later
15:20:00 <dom> RESOLVED: close #18 without changing the name of CropTarget
15:20:06 <dom> RRSAgent, draft minutes
15:20:06 <RRSAgent> I have made the request to generate https://www.w3.org/2022/06/23-webrtc-minutes.html dom
15:20:12 <dom> Subtopic: Issue #63: Cropping non-self-capture tracks
15:20:12 <dom> [slide 38]
15:20:25 <dom> Elad: currently you can crop only to current tab
15:20:35 <dom> ... I suggest we allow cropping arbitrary tabs
15:20:54 <dom> [slide 39]
15:21:28 <dom> [slide 40]
15:22:46 <dom> Jan-Ivar: a concern is that it might allow sites to censor themselves when captured
15:23:03 <dom> Elad: the capturing app can ignore crop targets
15:24:35 <dom> ... in fact, cropping automatically would make no sense
15:24:46 <dom> Jan-Ivar: I want the group to be aware of the risk
15:24:50 <dom> Elad: but is it likely?
15:25:52 <dom> Jan-Ivar: I won't predict the future; I don't see other issues with this
15:26:05 <dom> harald: please write this up in the github issue; I don't understand the risk
15:26:23 <dom> TimP: I support allowing this beyond self-capture
15:27:44 <dom> Elad: can we agree that by next meeting we agree to expand this unless a compelling case is made against it?
15:28:45 <dom> Jan-Ivar: imagine a bank wanting to redact what would get shared over screen capture
15:29:10 <dom> ... can write this up by next meeting
15:29:14 <dom> Harald: I support this too
15:29:17 <dom> dom: me too
15:29:24 <dom> Subtopic: Making CropTargets stringifiable
15:29:24 <dom> [slide 41]
15:29:54 <dom> Elad: making croptargets stringifiable would help e.g. for communication over capture handle
15:30:03 <dom> ... not sure I understand the risks
15:30:43 <dom> Youenn: a string makes it much more difficult to garbage-collect a croptarget
15:35:19 <dom> Jan-Ivar: +1 to Youenn
15:35:43 <dom> ... I haven't heard use cases that justify this
15:35:50 <dom> ... having GCable croptarget is good to keep
15:37:49 <dom> youenn: if a croptarget comes with resource allocation, being able to end these allocations is a good thing
15:38:09 <dom> elad: you can't just associate the string to the element, and when gc'ing the croptarget, remove that association
15:38:45 <dom> TimP: this removes the opacity that I relied in my previous support
15:38:58 <dom> ... stringifying makes it harder to reason about the safety of this
15:39:35 <dom> elad: the only difference between the two is equality
15:39:44 <dom> jan-ivar: there are differences in garbage collection
15:39:54 <dom> TimP: from a developer perspective, there is a difference
15:40:11 <dom> ... there are very limited number of paths to get a cropTarget
15:40:26 <dom> ... once it's a string, many more paths can be used
15:41:02 <dom> Youenn: can you bring that argument to the github?
15:41:29 <dom> Elad: would like to get resolution to this; we can skip the predictable errors
15:42:16 <dom> harald: part of the issue seems to be about reconstructing a croptarget from a string (not about stringifying per se)
15:42:30 <dom> Topic: -> https://github.com/w3c/mediacapture-extensions/pull/48 Face Detection
15:42:31 <dom> -> https://github.com/riju/faceDetection/blob/main/explainer.md Face Detection explainer
15:42:31 <dom> [slide 45]
15:44:31 <dom> [smode 46]
15:45:58 <dom> riju: this shows our proposal helps reduce power consumption - almost 2x at 15fps compared to using TF.js
15:46:03 <dom> s/smode/slide
15:46:45 <dom> youenn: the 1st column has no face detection, and the 2nd is doing face detection in the driver?
15:46:51 <dom> tuukka: right
15:47:32 <dom> youenn: in some OSes, the two might be equal if the camera is doing it systematically
15:47:46 <dom> riju: indeed, in Android this might be the case
15:47:56 <dom> [slide 47]
15:48:27 <dom> Riju: having a persistant id is very important for face tracking
15:48:46 <dom> ... re keeping the probably optional - sure, but all platforms provide this
15:49:13 <steely-glint_> steely-glint_ has joined #webrtc
15:49:15 <dom> ... a developer may use this to decide to apply further processing (e.g. funny hat)
15:49:27 <dom> ... but open to making it optional
15:49:49 <dom> ... Re VideoFrameMetadata, you suggest coordination on WebCodecs?
15:50:05 <dom> Youenn: yes, we need to engage with them to find the right construact
15:50:09 <dom> s/act/ct
15:50:29 <dom> Riju: re API surface, we started with a minimal set, increased it based on feedback
15:50:37 <dom> ... but we can re-reduce it for the MVP
15:50:44 <dom> ... e.g. remove the mesh parts
15:50:55 <dom> ... the contour was Harald's request
15:51:26 <dom> ... face landmarks are usually important in post-processing, think we should keep in MVP
15:51:39 <dom> youenn: my point was in terms of priorities & focus
15:51:48 <dom> ... e.g. for the next 6 months
15:52:03 <dom> riju: removing mesh, but keep landmarks
15:52:24 <dom> harald: I still have a problem with the API
15:52:33 <dom> ... the power consumption improvement is nice-to-have
15:52:41 <dom> ... attachment to the videoframe is nice
15:52:47 <dom> ... but still unclear what to use it for
15:52:59 <dom> ... the explainer doesn't help much with it
15:53:10 <dom> ... what can I do with the output of that API?
15:53:16 <dom> ... what the MVP would be viable for?
15:53:49 <dom> riju: e.g. landmarks would be used for post-processing, e.g eye-gaze correction
15:54:36 <dom> ... the platforms only give bounding boxes at this stage
15:54:51 <dom> Harald: I would like to a more complete use case
15:55:10 <dom> Youenn: one use case is that some encoders optimize based on specific bounding boxes
15:55:49 <dom> Bernard: +1 to youenn - segmentation helps with encoding
15:57:02 <dom> Jan-Ivar: the explainer talks about attaching to videoframe, but the API is still anchored in mediastreamtrack (e.g. for capabilities)
15:57:14 <dom> ... how would this API be usable on non-camera sources?
15:57:24 <dom> ... e.g. on recorded videos
15:57:45 <dom> riju: we couldn't use the same platform APIs to get the power consumption benefits
15:58:44 <dom> youenn: I think cameras should be our primary target
15:58:56 <dom> ... for recorded videos, you could add this through a media capture transform
15:59:25 <dom> jan-ivar: adding these metadata through the transform?
15:59:27 <dom> youenn: ye
15:59:31 <dom> s/ye/yes
16:00:07 <dom> eero: our proposal has support for setting custom metadata in videoframe
16:00:25 <dom> harald: the constraints are used to instruct the driver to produce the info, which is then attached to the videoframe
16:00:29 <dom> ... that makes sense to me
16:00:51 <dom> ... but writing up the enhanced encoding use case would help making compelling
16:01:04 <dom> riju: any support for prototyping this?
16:01:13 <dom> harald: yes - we need to find compelling applications
16:01:51 <dom> riju: also heard support from youenn
16:02:06 <dom> jan-ivar: I still have some concerns whether this would reveal difference across platforms
16:02:21 <dom> ... would suggest raising an issue on Mozilla's standard positions
16:02:41 <dom> bernard: useful to prototype; the metadata discussion should be brought to the WebCodecs falks
16:02:43 <dom> s/falks/folks/
16:02:52 <dom> riju: will follow up accordingly
16:02:55 <dom> RRSAgent, draft minutes
16:02:55 <RRSAgent> I have made the request to generate https://www.w3.org/2022/06/23-webrtc-minutes.html dom
16:03:25 <dom> RRSAgent, make log public