21:59:20 <RRSAgent> RRSAgent has joined #mediawg
21:59:25 <RRSAgent> logging to https://www.w3.org/2024/11/12-mediawg-irc
21:59:25 <Zakim> Zakim has joined #mediawg
22:01:21 <cpn> Meeting: Media WG
22:02:00 <cpn> present+ Chris_Needham, Eric_Carlson, Francois_Daoust, Jean-Yves_Avenard, Bernard_Aboba
22:02:30 <tidoust> Agenda: https://github.com/w3c/media-wg/blob/main/meetings/2024-11-12-Media_Working_Group_Teleconference-agenda.md
22:02:43 <tidoust> RRSAgent, make logs public
22:03:38 <cpn> present+ Mark_Foltz
22:04:04 <eugene> eugene has joined #mediawg
22:04:15 <cpn> present+ Bryant_Chandler, Nic_Champagne_Williamson, Eugene_Zemtsov
22:05:02 <cpn> present+ Marcos_Caceres
22:05:07 <cpn> Chair: Chris, Marcos
22:05:54 <marcos> marcos has joined #mediawg
22:06:09 <marcos> TOPIC: Agenda bashing
22:06:34 <tidoust> scribe+ tidoust
22:06:52 <cpn> scribe+ cpn
22:07:02 <markafoltz> markafoltz has joined #mediawg
22:07:08 <cpn> Topic: Camera Effect Status API
22:07:42 <eric_carlson> eric_carlson has joined #mediawg
22:07:53 <tidoust> -> https://github.com/w3c/webcodecs/issues/839 issue #839
22:10:06 <cpn> Byrant: The goal is to let developers monitor changes to platform blur, or in future, other effects
22:10:19 <cpn> ... We put it on VideoFrameMetadata so it can be frame accurate
22:10:33 <cpn> ... It's not for controlling platform effects, e.g., to turn them on or off
22:10:41 <cpn> ... Not for platforms that don't have those effects
22:11:06 <cpn> ... The API is a field on VideoFrameMetadata, a MediaEffectInfo object, which BackgroundBlur inherits
22:11:31 <cpn> ... The object allows for future expansion, e.g., we might want to expose info about where the effect was implemented, or how intense the blur is
22:12:19 <cpn> ... Example usage, you implement a TransformStream so you can see each Frame coming through. The metadata returns a dictionary, check if the backgroundBlur is present
22:13:03 <cpn> ... There's a PR against WebCodecs, feedback that we want a W3C spec link, so I'm working on adding it to MediaCapture-Extensions, so I'll update the PR to link to that
22:13:19 <cpn> ... We have links to the explainer
22:13:46 <cpn> Mark: We have broader ambitions, but after getting feedback from people at Apple and at TPAC, we scoped it down to additional info on VideoFrameMetadata
22:14:00 <cpn> ... If we can get this out, we'd look at adding other effects, depending on developer interest
22:14:21 <cpn> ... Bryant today created a PR for MediaCapture-Extensions. Looking forward to feedback there
22:14:33 <cpn> ... We'll link that from the WebCodecs PR
22:14:39 <cpn> q?
22:15:01 <tidoust> q+
22:15:18 <cpn> Eugene: Seems in line with what we did for human face segmentation
22:15:30 <cpn> ... Seems little downside to exposing it
22:15:52 <cpn> ... Any web applications asking for it, e.g., Meet?
22:16:08 <cpn> Mark: They're interested, part of conversation with them about preventing double effects
22:16:21 <cpn> Bryant: Helping users know where the effects are coming from
22:16:45 <cpn> Francois: In the proposed IDL there's a dictionary with a readonly member, I don't think that's allowed in WebIDL
22:17:16 <cpn> ... An application could apply the blur effect itself and set the value?
22:17:58 <cpn> Mark: That sounds correct. We dont' have in scope changing the state of the video frame, as the blur is applied beforehand, so setting wouldn't do anything
22:18:05 <cpn> Bryant: It's a typo
22:18:40 <cpn> Eugene: The metadata() method returns a dictionary. You can set any fields, specced or not, but that doesn't change the frame itself
22:18:51 <jernoble> jernoble has joined #mediawg
22:19:20 <cpn> ... The value comes from the capture pipeline or WebRTC stream, or when you create a frame via the constructor, you could have a way to amend metadata
22:19:37 <cpn> q?
22:19:40 <cpn> ack tid
22:20:23 <cpn> Marcos: You could simplify the code using nullish coalescing
22:20:52 <cpn> Chris: Are there other implementations interested in this?
22:21:25 <cpn> Marcos: Once we have a concrete proposal, we could look from WebKit point of view
22:22:16 <cpn> Chris: And adding the registry entry conditional on multi-implementer support?
22:22:24 <cpn> Marcos: Is it shipping in Chrome?
22:22:39 <cpn> Mark: It's work in progress
22:22:56 <cpn> Marcos: There should be interest from another implementer
22:23:07 <cpn> Mark: Ok, so as we move ahead we'll ask WebKit position
22:23:38 <cpn> ... As a side-effect of this work we're adding the support for metadata in general, so easier to ship metadata in future
22:24:00 <tidoust> q+
22:24:02 <cpn> ... We're also intersted in the face segmentation metadata, just not had time to get into it yet
22:25:40 <cpn> Francois: Related to the human face segmentation metadata, the MediaCapture-Streams spec suggests you can set the value in the web app, or it can be set by the UA
22:26:09 <cpn> ... But right now, setting the data has no effect, and you can't pass in to the VideoFrame constructor
22:27:12 <cpn> Eugene: Both VideoFrameInit and VideoFrameBufferInit lets you set it
22:27:32 <cpn> Mark: For blur, I don't think it's meaningful, not sure for other metadata
22:28:08 <cpn> Francois: The TransformStream could let you apply the blur and set the metadata
22:28:18 <cpn> Mark: If you're creating a new VideoFrame, yes
22:28:47 <cpn> Bernard: It's a general problem with these registry entries, they create data, they don't imply any API can do anything with it, it's just a collection of data
22:29:57 <cpn> ... It just links to a spec, MCE, but the registry isn't a spec, just a pointer. Whether it's a coherent story, the registry doesn't require
22:30:09 <cpn> ... For example, no implication that WebCodecs does anything with it
22:30:46 <cpn> Eugene: It's just a way to surface additional info about the VideoFrame, so the app doesn't try to do extra effects or the web app can use the face metadata, or delivery timestamps the app can use
22:31:03 <cpn> ... WebCodecs doesn't depend on it, it's surfaced for apps to use
22:31:38 <cpn> ... We discussed adding it vs surfacing via a side channel, but we don't have any side channels
22:32:06 <cpn> ... Apps such as Teams or Zoom can do something useful with it
22:32:35 <cpn> Bernard: It's not actionable (face data) not useful for avoiding double action
22:32:37 <Marcos> Marcos has joined #mediawg
22:33:02 <cpn> Bryant: This is a different kind of double action, just avoiding repeat processing
22:34:05 <cpn> Eugene: If you create a VideoFrame from a canvas, and you know you already did background blur, you can set it in VideoFrameInit
22:35:42 <tidoust> cpn: You could set this value on the VideoFrameInit but the browser implementation would not pay attention to that, it would just set the value. I'm wondering about cases where the metadata property could conflict with a property that the user agent might set itself
22:36:10 <cpn> Eugene: The user set value should override the implementations
22:37:11 <cpn> Bernard: You have a track from a camera, get a stream of video frames, the backgroundBlur is set set on frames from that source
22:37:24 <cpn> ... The application could set the value, but it just messes up the signal it gets
22:37:49 <cpn> Mark: Hers's a use case. Suppose an app doesn't want to double blur. It detects if blur is applied or not. If not applied, it applies its own
22:38:11 <cpn> ... Then, it wants to copy metadata from one frame to another, then update the blur to note the fact it applied its own
22:38:28 <cpn> ... It's not from the UA now, just something the app keeps track of
22:38:41 <cpn> Bernard: And it doesn't get used in the WebCodecs encoder
22:38:48 <cpn> Mark: Right, it's just for application's bookkeeping
22:39:24 <cpn> Topic: Media Capabilities MIME Types
22:39:29 <tidoust> -> https://github.com/w3c/media-capabilities/pull/222 Fixup MIME usage
22:39:56 <cpn> Marcos: We've been investigating the MIME type processing in the MC API spec
22:40:02 <cpn> ... We found interop issues
22:40:18 <cpn> ... Chris has had a go at rewriting algorithms, and Mark reviewing
22:40:32 <cpn> ... As chairs we're doing a bit too much editing....
22:40:49 <cpn> ... But it's in a pretty good place, can take what we have and finish it off
22:41:05 <cpn> ... The tests that are there don't necessarily match the spec. Need to decide what to do with those
22:41:36 <cpn> ... We could feed them into ChatGPT to generate test cases, but also decide which implementations should win
22:41:38 <cpn> q+
22:41:41 <cpn> ack t
22:42:03 <cpn> ... How much interop pain will there be, how much breakage should we tolerate to get correct behaviour
22:42:27 <cpn> ... We don't want multiple MIME parsers, mimesniff should be the one used throughout the platform
22:42:44 <cpn> ... We take the input, pass through mimesniff to parse it to give a structure
22:42:55 <cpn> ... Take the component parts and do things with them
22:43:19 <cpn> ... How to deal with different behaviours, parameter handling, some implementations reject, and some ignore
22:43:52 <tidoust> cpn: I can show some examples.
22:44:03 <tidoust> ... [shares screen]
22:45:18 <tidoust> ... First example is a string that according to MIME sniff is not a valid MIME type string "audio/mpeg;". Chrome rejects on this, Firefox resolves on it, Safari also resolves on it but says it's not supported.
22:46:06 <tidoust> ... Per the current spec, Chrome is doing the right thing, but what we're proposing in the PR is to use the parse MIME type algorithm, in which case Firefox and Safari are actually doing the right thing for that new algorithm.
22:46:16 <tidoust> ... We need to decide what is the right way to do it.
22:47:10 <tidoust> ... The second example is that we have language on "does the MIME type imply a single codec?". The example is "video/webm" or "video/mp4", which are just containers and don't say much about the contents.
22:47:23 <tidoust> ... And can support multiple codecs.
22:47:51 <tidoust> ... Same thing, implementations differ. In this example, the string should be rejected.
22:48:34 <tidoust> ... Third example is multiple parameters in the MIME type string. "audio/webm; codecs="opus"; foo="bar"'. Different behavior.
22:48:46 <tidoust> ... Should be rejected per spec, but Chrome resolves.
22:49:21 <tidoust> ... Final example is multiple codecs in the codecs parameters. Chrome resolves, but Firefox and Safari reject. Per spec, it should be rejected.
22:49:57 <tidoust> ... Some variability in implementations. I'm trying to figure out what behavior we want to specify, and which implementations need to change to match whatever we end up with.
22:50:31 <tidoust> markafoltz: These are good finds! Thanks for the all the work.
22:50:36 <jernoble> jernoble has joined #mediawg
22:51:00 <tidoust> ... As editor, my next step would be to create test cases that action some of these so that we know where implementations are.
22:51:55 <tidoust> ... My second comment is that we've been working on this PR for some time and it's quite complicated. I think that, as an editor, my goal is to make the PR self-consistent, with the acknowledgment that there may remain some issues to resolve later.
22:52:14 <tidoust> cpn: Even if that means that we end up with a different set of requirements.
22:52:44 <tidoust> markafoltz: We should file interop issues for these and link them from the spec. I want to land the structure because it is an improvement over the status quo.
22:53:23 <tidoust> cpn: The difficulty is that what we have in the PR is still not quite right. "MIME type support" can still return a failure case, even though it shouldn't because failures should have been captured earlier.
22:53:47 <tidoust> markafoltz: That's what I mean by self-consistent. It should not have bugs but may not reflect reality of implementations.
22:54:31 <tidoust> cpn: The actual check for support is pretty simple, we can probably delegate that to MIME Sniff.
22:54:43 <tidoust> markafoltz: The problem is that there are different steps for audio and video.
22:55:01 <tidoust> cpn: If we break up the WebRTC steps from the rest, perhaps that would simplify things.
22:55:46 <tidoust> markafoltz: The existing spec does not really explain whether the WebRTC codec that you need to use is because of support or because of validation.
22:56:18 <tidoust> ... That's another question we could ask. I'm inclined to keep what both implementation currently do (Firefox does not implement that part).
22:56:58 <tidoust> ... My proposal is to update the validation to approximate what browsers currently do with some issues noting that there are interop issues.
22:58:02 <tidoust> Marcos: That seems fine. The only need that I had was around the internal slot attached to a dictionary. That seems a premature optimization there. Apart from that, I think it's generally looking good enough, with issues.
22:58:18 <tidoust> markafoltz: Some specs use slots a lot to pass data around, others don't.
22:58:47 <tidoust> Marcos: I use them quite a lot too. Generally, we only put them on interfaces, not on dictionaries.
22:59:10 <tidoust> ... It doesn't save much in the spec to use an internal slot.
22:59:27 <tidoust> ... Let's keep the spec as dumb and simple as possible so that it's easier to fix and update as needed.
22:59:42 <tidoust> Marcos: I'll think about removing the slot.
23:00:13 <tidoust> cpn: And maybe I'll sent a PR to Web Platform Tests with these cases for you to review.
23:00:36 <tidoust> RRSAgent, draft minute
23:00:36 <RRSAgent> I'm logging. I don't understand 'draft minute', tidoust.  Try /msg RRSAgent help
23:00:37 <tidoust> RRSAgent, draft minutes
23:00:39 <RRSAgent> I have made the request to generate https://www.w3.org/2024/11/12-mediawg-minutes.html tidoust
23:01:29 <tidoust> s/TOPIC: Agenda bashing//
23:01:41 <tidoust> RRSAgent, draft minutes
23:01:43 <RRSAgent> I have made the request to generate https://www.w3.org/2024/11/12-mediawg-minutes.html tidoust