IRC log of webrtc on 2022-06-23

Timestamps are in UTC.

13:41:21 [RRSAgent]
RRSAgent has joined #webrtc
13:41:21 [RRSAgent]
logging to https://www.w3.org/2022/06/23-webrtc-irc
13:41:49 [dom]
Meeting: WebRTC June 2022 VI
13:41:49 [dom]
Chair: HTA, Bernard, Jan-Ivar
13:41:49 [dom]
Slideset: https://lists.w3.org/Archives/Public/www-archive/2022Jun/att-0006/WEBRTCWG-2022-06-23.pdf
13:41:49 [dom]
Agenda: https://www.w3.org/2011/04/webrtc/wiki/June_23_2022
13:59:21 [dom]
Regrets+ Carine
14:01:22 [eehakkin]
eehakkin has joined #webrtc
14:01:39 [dom]
Present+ Bernard, Dom, Elad, Florent, Harald, Jan-ivar, Patrick_Rockhill, Eero_Hakkinen
14:01:48 [dom]
Present+ TuukkaT
14:01:53 [dom]
Present+ TimPanton
14:02:49 [dom]
Recording is starting
14:02:58 [dom]
Present+ Guido
14:03:07 [dom]
Present+ Youenn
14:04:16 [eehakkin_]
eehakkin_ has joined #webrtc
14:04:19 [dom]
scribe+
14:04:32 [dom]
Present+ Riju
14:05:31 [dom]
Topic: -> https://github.com/w3c/webrtc-charter/ WebRTC WG Re-Charter
14:05:31 [dom]
-> http://w3c.github.io/webrtc-charter/webrtc-charter.html Draft updated charter for the WebRTC WG
14:05:31 [dom]
[slide 9]
14:09:22 [steely-glint]
steely-glint has joined #webrtc
14:09:32 [dom]
dom: we need a new charter, and to discuss issue #70 (migrating work to another group)
14:09:42 [dom]
youenn: would this require rechartering Media WG?
14:09:44 [dom]
dom: not sure, possibly
14:10:00 [dom]
aboba: we have a joint meeting with Media WG at TPAC
14:11:05 [dom]
dom: but we can't wait until then to come to a conclusion
14:11:37 [dom]
youenn: migrating items between charters is always challenging from our side, in administrative aspecs
14:12:47 [dom]
bernard: do we need meetings with the media wg to discuss this?
14:13:06 [dom]
dom: if we want to migrate the work, yes - but we first need to decide whether we want to do that
14:13:30 [dom]
bernard: the fact that we have dependencies on videoframe makes it an interesting question to consider
14:13:43 [dom]
elad: how busy is the Media WG? would it increase or decrease the pace of our work?
14:15:18 [dom]
harald: that would have to be something to discuss with the chairs
14:15:43 [dom]
dom: bringing it to media wg would bring more of a media perspective when this group comes more with a transmission perspective
14:16:35 [dom]
harald: so the chairs will discuss this with the Media WG chairs
14:16:43 [dom]
dom: no objection from the group in exploring this?
14:16:44 [dom]
[none]
14:17:02 [dom]
Topic: -> https://github.com/w3c/mediacapture-region Region Capture Issues
14:17:02 [dom]
Subtopic: Issue #17: the case for making CropTarget Sync
14:17:02 [dom]
[slide 13]
14:18:04 [dom]
Jan-Ivar: this is about the cropping API - issue #17 is about whether it should be sync vs async
14:18:23 [dom]
... long discussion on the issue - I'll be presenting arguments why it should be sync
14:19:14 [dom]
... The TAG design principles include encouragement to use sync APIs when appropriate, with some exceptions (incl cross-process communications)
14:19:16 [dom]
[slide 14]
14:19:52 [dom]
Jan-Ivar: the API currently in the spec (that doesn't have consensus) is async
14:20:14 [dom]
... so you have to `await CropTarget.fromElement(element)`
14:20:29 [dom]
... I'm proposing it doesn't need to be async,
14:20:48 [dom]
... the purpose of the operation is associating an identifier with an element
14:21:07 [dom]
... as currently specified, it can't fail
14:21:39 [dom]
... the goal of the crop target is to share it over postMessage across documents
14:21:47 [dom]
[slide 15]
14:22:09 [dom]
Jan-Ivar: multiple actions needs to happen before we're cropping anything
14:22:32 [dom]
... cropTarget can be done ahead of time or later
14:23:11 [dom]
... if it gets postMessaged to the top-level document, the said document can offer to the user to crop to that target
14:23:36 [dom]
... it's only at the end of this process that there is a clear intent to crop
14:24:09 [dom]
... UA can optimize this by running some of the underlying tasks early, but that creates risks in case this doesn't go through
14:24:21 [dom]
... the complexity of that situation shouldn't be exposed to developers
14:24:35 [dom]
[slide 16]
14:25:02 [dom]
Jan-Ivar: #48 is asking to allow failure from the minting process due to resource exhaustion
14:25:16 [dom]
... which seems linked to optimizations implemented in Chrome
14:25:34 [dom]
... The issue is that it allows random document to exhaust cropping resources
14:25:59 [dom]
... since the API is not gated by permissions - creating DOS risks
14:26:35 [dom]
... and may expose apps that don't deal well with that failure
14:26:52 [dom]
... if resource allocation is moved to the cropTo step, this risk disappears
14:28:00 [dom]
... similar to mediaSource.getHandle
14:28:53 [dom]
[slide 17]
14:30:51 [dom]
Jan-Ivar: I believe my proposed API is faster, simpler and still optimizable
14:31:08 [dom]
... I don't think we need to block on inter-process communication
14:31:09 [dom]
[slide 18]
14:32:17 [dom]
jan-ivar: failing of optimization shouldn't imply a failure of the operation
14:32:38 [dom]
... optimizations that influence API design decision tend to generate further issues
14:32:44 [dom]
... because optimizations create new side effects
14:33:07 [dom]
... there is no developer benefits to this API being async
14:33:44 [dom]
... and there are general developer costs to async APIs - they create pre-emption points which risk data races
14:34:16 [dom]
... multiplying failure points for rare error cases is a footgun
14:34:28 [dom]
... and async is slower as I've shown
14:35:30 [dom]
elad: the TAG offers design principles, but also meta principles - not sure it's productive to discuss other browsers implementation
14:35:51 [dom]
... the fingerprinting risk is reasonable, but the spec doesn't force to surface this
14:36:06 [dom]
... a promise doesn't have to fail in your implementation either
14:36:52 [dom]
jan-ivar: the API should be designed on principles
14:37:27 [dom]
... Mozilla is here to push a better API for the Web
14:37:34 [dom]
Subtopic: Issue #17: the case for making CropTarget Async
14:37:34 [dom]
[slide 20]
14:37:51 [dom]
Elad: this API is used in production through origin trial
14:37:59 [dom]
... we know the API works and makes developers and customers happy
14:38:12 [dom]
... we've learned a lot of lessons by implementing and shipping this
14:38:16 [dom]
[slide 21]
14:38:34 [dom]
Elad: the question is whether minting a token needs to be sync or async
14:38:54 [dom]
... I'll explain the Chrome decision and that it doesn't impact negatively anyone else
14:39:01 [dom]
[slide 22]
14:39:44 [dom]
Elad: our implementation keeps track of which apps have a crop target
14:40:29 [dom]
... once it's postMessage'd, this allows Chrome to optimize the time-to-cropping when the cropTo method is actually invoked
14:40:52 [dom]
... that makes it simpler and more performant, in particular in case of CPU congestion in the capturing document
14:40:55 [dom]
[slide 23]
14:41:20 [dom]
[slide 24]
14:41:46 [dom]
Elad: Chrome needs this to be async, whether Mozilla prefers it to be sync
14:42:09 [dom]
... what's the harm of having an async API?
14:42:11 [dom]
[slide 25]
14:42:40 [dom]
Elad: Priority of consistuencies goes through users, developers, implementers, spec writers
14:42:44 [dom]
[slide 26]
14:42:55 [dom]
Elad: from a user perspective, seems mostly orthogonal
14:43:31 [dom]
... from a developer perspective - what we've heard is that they don't care as it doesn't change much in their huge existing codebase
14:43:51 [dom]
... implementors - as an implementor, we see this as an imperative for us
14:43:58 [dom]
[slide 27]
14:44:33 [dom]
Elad: what is the negative impact then? Is this theoretical purity?
14:44:42 [dom]
... given that IPC is involved, async makes sense
14:44:42 [dom]
[slide 28]
14:45:07 [dom]
Elad: the TAG actually insists that theoretical purity doesn't trump implementers needs
14:45:10 [dom]
[slide 29]
14:45:19 [dom]
Elad: the TAG discussed this API
14:45:43 [dom]
... they were satisfied with the API
14:45:49 [dom]
... they haven't seen much ergonomic gain for sync
14:46:09 [fbeaufort]
fbeaufort has joined #webrtc
14:46:16 [dom]
... they also highlighted that interop should drive the work of the group
14:46:18 [dom]
[slide 30]
14:46:40 [dom]
Elad: I've shown that consistuencies either don't care about sync vs async, and that at least one implementor needs async
14:46:59 [dom]
... also, it's easy to go from async to sync, while the other way is difficult
14:47:18 [dom]
Subtopic: #17 discussion
14:47:41 [dom]
Bernard: by making it async, you're saying that this implies that when you have a cropTarget, you know it's ready to use
14:47:58 [dom]
... we've had situations in the WebRTC WG where we've found that sync APIs needed in fact to be async
14:48:08 [dom]
Elad: cf slide 22
14:50:27 [dom]
Youenn: I'm surprised you're saying this is MUST - I thought this was implementable as a sync API, but that you favored the trade-off that async allows
14:51:10 [dom]
... as I've pointed out, this trade-off creates a fingerprinting and interop issue - so a footgun
14:51:41 [dom]
... so I thought both were implementable but sync would be more complex in Chrome
14:51:52 [dom]
Elad: I don't agree with that characterization
14:52:09 [dom]
Youenn: I'm surprised that both approaches are faster - both can't be true
14:52:17 [dom]
s/are faster/are claiming to be faster/
14:52:50 [dom]
Youenn: usually sync APIs are more efficient, except when they creating a blocking situation, which I'm not hearing is the case here
14:52:58 [dom]
... a sync API helps developers, at least a bit
14:54:10 [dom]
Elad: resource allocation design decision is orthogonal to sync vs async
14:55:02 [dom]
... I don't think the resource mitigation limits fingerprinting risks can be mitigated through a per-iframe limit
14:55:50 [dom]
... in terms of performance, what needs to be fast is cropTo - anything before that, the user doesn't notice
14:55:52 [anssik]
anssik has joined #webrtc
14:57:19 [dom]
youenn: sync cropTarget minting is faster, but you're saying this is not a relevant optimization compared to cropTo
14:57:53 [dom]
elad: anything that comes before cropTo is irrelevant to user perceived performance (and mostly negligible in any case)
14:58:24 [dom]
TimP: as a developer, I have a mild preference to keep it sync as it's easier to use
14:58:33 [dom]
... I don't think a developer benefit to making it async
14:58:49 [dom]
... there may be a user benefit in terms of the crop transition UX
14:59:06 [dom]
... that may convince me of the value if it can be shown
15:00:01 [dom]
... from the developer perspective, managing interesting failures on obtaining target would also be convincing, but I haven't heard that
15:00:26 [dom]
Elad: it's a particular choice of trade-offs that require async, and async isn't going to harm other consistuencies
15:00:31 [dom]
TimP: developers will suffer
15:00:37 [dom]
Elad: I claim it's negligible
15:01:04 [dom]
TimP: I don't agree - this can generate non-trivial changes, although it's certainly doable
15:02:04 [dom]
Jan-Ivar: re no downside - nobody claiming that Chrome optimizations aren't useful
15:02:13 [dom]
... I don't understand why these optimizations can't be done with a sync API
15:02:24 [dom]
... why can't you implement a fallback?
15:02:51 [dom]
... other downsides include fingerprinting, DOS, proliferation of async, failure management for developers
15:03:08 [dom]
... why does it need to be async?
15:03:36 [dom]
Harald: code complexity itself is a risk; this particular implementation has been used and tested
15:04:28 [dom]
... anyone that depends on cropTo and doesn't notice it failing as an issue
15:04:52 [dom]
... it's time to stop this discussion - we have seen that Chrome claims that a sync implementation would be make it significantly more complex
15:05:08 [dom]
... the impact on developers is irritating but not fatal
15:05:23 [dom]
... we have not seen compelling arguments that we need to change what has been proposed
15:05:58 [dom]
... I don't see consensus for change, there is an implementation of the current spec - I suggest we declare the API to be async and move on
15:06:40 [dom]
Jan-Ivar: I hear ease of implementation - complexity in the API trumps complexity of implementation
15:06:49 [dom]
... because of the priority of consistuencies
15:07:15 [dom]
Harald: a more complex API that does the right thing is better than a simple API that does the wrong thing
15:08:18 [dom]
Elad: I'm not seeing consensus, but I don't think there are remaining benefits to discuss this
15:09:34 [dom]
dom: we could either run a vote, or wait for more implement experience
15:10:17 [dom]
s/implement/implementation/
15:10:52 [dom]
TimP: I would be inclined to say that getting other implementation is most important
15:11:03 [dom]
... sync would be more elegant, but it doesn't look like we're going to get that
15:11:20 [dom]
Subtopic: Issue #18: Is CropTarget name too generic?
15:11:20 [dom]
[slide 33]
15:12:41 [dom]
Youenn: "crop" as at term isn't used too broadly so far, so probably OK
15:12:43 [dom]
... not sure that "target" helps
15:12:48 [dom]
[slide 34]
15:13:45 [dom]
[slide 35]
15:14:48 [dom]
Youenn: "object whose sole purpose is to be given cropTo" - may not be limited to elements
15:14:59 [dom]
... the term CropRegion might work better to represent
15:15:11 [dom]
... and would align with the spec name ("region capture")
15:15:57 [dom]
... thoughts?
15:16:39 [dom]
TimP: the name should reflect that it is a token, not a region or a target itself
15:16:45 [dom]
... if it's opaque, it should say so
15:16:57 [dom]
Youenn: it may not remain opaque
15:17:10 [dom]
TimP: a region sounds like something you could do math on, e.g. calculate its surface area
15:17:17 [dom]
... which you can't
15:17:50 [dom]
elad: similar reservation - region feels something with coordinates; also, a cropTarget isn't static, it can move, which cropregion makes more misleading
15:18:19 [dom]
harald: this is bikeshedding; I don't see benefit in changing it
15:18:44 [dom]
Bernard: +1 that CropRegion is confusing; I prefer the current name
15:19:16 [dom]
Youenn: any interest in clarifying the definition (i.e. whether it's a reference to an element or something more generic)
15:19:24 [dom]
... I guess that can be done later
15:20:00 [dom]
RESOLVED: close #18 without changing the name of CropTarget
15:20:06 [dom]
RRSAgent, draft minutes
15:20:06 [RRSAgent]
I have made the request to generate https://www.w3.org/2022/06/23-webrtc-minutes.html dom
15:20:12 [dom]
Subtopic: Issue #63: Cropping non-self-capture tracks
15:20:12 [dom]
[slide 38]
15:20:25 [dom]
Elad: currently you can crop only to current tab
15:20:35 [dom]
... I suggest we allow cropping arbitrary tabs
15:20:54 [dom]
[slide 39]
15:21:28 [dom]
[slide 40]
15:22:46 [dom]
Jan-Ivar: a concern is that it might allow sites to censor themselves when captured
15:23:03 [dom]
Elad: the capturing app can ignore crop targets
15:24:35 [dom]
... in fact, cropping automatically would make no sense
15:24:46 [dom]
Jan-Ivar: I want the group to be aware of the risk
15:24:50 [dom]
Elad: but is it likely?
15:25:52 [dom]
Jan-Ivar: I won't predict the future; I don't see other issues with this
15:26:05 [dom]
harald: please write this up in the github issue; I don't understand the risk
15:26:23 [dom]
TimP: I support allowing this beyond self-capture
15:27:44 [dom]
Elad: can we agree that by next meeting we agree to expand this unless a compelling case is made against it?
15:28:45 [dom]
Jan-Ivar: imagine a bank wanting to redact what would get shared over screen capture
15:29:10 [dom]
... can write this up by next meeting
15:29:14 [dom]
Harald: I support this too
15:29:17 [dom]
dom: me too
15:29:24 [dom]
Subtopic: Making CropTargets stringifiable
15:29:24 [dom]
[slide 41]
15:29:54 [dom]
Elad: making croptargets stringifiable would help e.g. for communication over capture handle
15:30:03 [dom]
... not sure I understand the risks
15:30:43 [dom]
Youenn: a string makes it much more difficult to garbage-collect a croptarget
15:35:19 [dom]
Jan-Ivar: +1 to Youenn
15:35:43 [dom]
... I haven't heard use cases that justify this
15:35:50 [dom]
... having GCable croptarget is good to keep
15:37:49 [dom]
youenn: if a croptarget comes with resource allocation, being able to end these allocations is a good thing
15:38:09 [dom]
elad: you can't just associate the string to the element, and when gc'ing the croptarget, remove that association
15:38:45 [dom]
TimP: this removes the opacity that I relied in my previous support
15:38:58 [dom]
... stringifying makes it harder to reason about the safety of this
15:39:35 [dom]
elad: the only difference between the two is equality
15:39:44 [dom]
jan-ivar: there are differences in garbage collection
15:39:54 [dom]
TimP: from a developer perspective, there is a difference
15:40:11 [dom]
... there are very limited number of paths to get a cropTarget
15:40:26 [dom]
... once it's a string, many more paths can be used
15:41:02 [dom]
Youenn: can you bring that argument to the github?
15:41:29 [dom]
Elad: would like to get resolution to this; we can skip the predictable errors
15:42:16 [dom]
harald: part of the issue seems to be about reconstructing a croptarget from a string (not about stringifying per se)
15:42:30 [dom]
Topic: -> https://github.com/w3c/mediacapture-extensions/pull/48 Face Detection
15:42:31 [dom]
-> https://github.com/riju/faceDetection/blob/main/explainer.md Face Detection explainer
15:42:31 [dom]
[slide 45]
15:44:31 [dom]
[smode 46]
15:45:58 [dom]
riju: this shows our proposal helps reduce power consumption - almost 2x at 15fps compared to using TF.js
15:46:03 [dom]
s/smode/slide
15:46:45 [dom]
youenn: the 1st column has no face detection, and the 2nd is doing face detection in the driver?
15:46:51 [dom]
tuukka: right
15:47:32 [dom]
youenn: in some OSes, the two might be equal if the camera is doing it systematically
15:47:46 [dom]
riju: indeed, in Android this might be the case
15:47:56 [dom]
[slide 47]
15:48:27 [dom]
Riju: having a persistant id is very important for face tracking
15:48:46 [dom]
... re keeping the probably optional - sure, but all platforms provide this
15:49:13 [steely-glint_]
steely-glint_ has joined #webrtc
15:49:15 [dom]
... a developer may use this to decide to apply further processing (e.g. funny hat)
15:49:27 [dom]
... but open to making it optional
15:49:49 [dom]
... Re VideoFrameMetadata, you suggest coordination on WebCodecs?
15:50:05 [dom]
Youenn: yes, we need to engage with them to find the right construact
15:50:09 [dom]
s/act/ct
15:50:29 [dom]
Riju: re API surface, we started with a minimal set, increased it based on feedback
15:50:37 [dom]
... but we can re-reduce it for the MVP
15:50:44 [dom]
... e.g. remove the mesh parts
15:50:55 [dom]
... the contour was Harald's request
15:51:26 [dom]
... face landmarks are usually important in post-processing, think we should keep in MVP
15:51:39 [dom]
youenn: my point was in terms of priorities & focus
15:51:48 [dom]
... e.g. for the next 6 months
15:52:03 [dom]
riju: removing mesh, but keep landmarks
15:52:24 [dom]
harald: I still have a problem with the API
15:52:33 [dom]
... the power consumption improvement is nice-to-have
15:52:41 [dom]
... attachment to the videoframe is nice
15:52:47 [dom]
... but still unclear what to use it for
15:52:59 [dom]
... the explainer doesn't help much with it
15:53:10 [dom]
... what can I do with the output of that API?
15:53:16 [dom]
... what the MVP would be viable for?
15:53:49 [dom]
riju: e.g. landmarks would be used for post-processing, e.g eye-gaze correction
15:54:36 [dom]
... the platforms only give bounding boxes at this stage
15:54:51 [dom]
Harald: I would like to a more complete use case
15:55:10 [dom]
Youenn: one use case is that some encoders optimize based on specific bounding boxes
15:55:49 [dom]
Bernard: +1 to youenn - segmentation helps with encoding
15:57:02 [dom]
Jan-Ivar: the explainer talks about attaching to videoframe, but the API is still anchored in mediastreamtrack (e.g. for capabilities)
15:57:14 [dom]
... how would this API be usable on non-camera sources?
15:57:24 [dom]
... e.g. on recorded videos
15:57:45 [dom]
riju: we couldn't use the same platform APIs to get the power consumption benefits
15:58:44 [dom]
youenn: I think cameras should be our primary target
15:58:56 [dom]
... for recorded videos, you could add this through a media capture transform
15:59:25 [dom]
jan-ivar: adding these metadata through the transform?
15:59:27 [dom]
youenn: ye
15:59:31 [dom]
s/ye/yes
16:00:07 [dom]
eero: our proposal has support for setting custom metadata in videoframe
16:00:25 [dom]
harald: the constraints are used to instruct the driver to produce the info, which is then attached to the videoframe
16:00:29 [dom]
... that makes sense to me
16:00:51 [dom]
... but writing up the enhanced encoding use case would help making compelling
16:01:04 [dom]
riju: any support for prototyping this?
16:01:13 [dom]
harald: yes - we need to find compelling applications
16:01:51 [dom]
riju: also heard support from youenn
16:02:06 [dom]
jan-ivar: I still have some concerns whether this would reveal difference across platforms
16:02:21 [dom]
... would suggest raising an issue on Mozilla's standard positions
16:02:41 [dom]
bernard: useful to prototype; the metadata discussion should be brought to the WebCodecs falks
16:02:43 [dom]
s/falks/folks/
16:02:52 [dom]
riju: will follow up accordingly
16:02:55 [dom]
RRSAgent, draft minutes
16:02:55 [RRSAgent]
I have made the request to generate https://www.w3.org/2022/06/23-webrtc-minutes.html dom
16:03:25 [dom]
RRSAgent, make log public