IRC log of mediawg on 2022-11-08

Timestamps are in UTC.

21:58:57 [RRSAgent]
RRSAgent has joined #mediawg
21:58:57 [RRSAgent]
logging to https://www.w3.org/2022/11/08-mediawg-irc
21:59:01 [Zakim]
Zakim has joined #mediawg
21:59:50 [tidoust]
tidoust has joined #mediawg
22:01:38 [cpn]
Meeting: Media WG
22:01:45 [cpn]
Chair: Chris_Needham
22:01:56 [tidoust]
RRSAgent, make logs public
22:02:01 [tidoust]
RRSAgent, draft minutes
22:02:01 [RRSAgent]
I have made the request to generate https://www.w3.org/2022/11/08-mediawg-minutes.html tidoust
22:02:33 [Matt_Wolenetz]
Matt_Wolenetz has joined #mediawg
22:02:55 [Matt_Wolenetz]
present+ Matt_Wolenetz
22:03:01 [cpn]
Present: Chris_Needham, Sushanth_Rajasankar, Jer_Noble, Francois_Daoust, Matt_Wolenetz, Greg_Freedman, Joey_Parrish
22:03:08 [tidoust]
Present: Francois_Daoust, Chris_Needham, Greg_Freedman, Matt_Wolenetz, Sushanth_Rajasankar, Karl_Tomlinson
22:03:13 [tidoust]
s/Present: Francois_Daoust, Chris_Needham, Greg_Freedman, Matt_Wolenetz, Sushanth_Rajasankar, Karl_Tomlinson//
22:03:17 [cpn]
scribe+ cpn
22:03:30 [cpn]
present+ Frank_Liberato
22:04:02 [tidoust]
Agenda: https://github.com/w3c/media-wg/blob/main/meetings/2022-11-08-Media_Working_Group_Teleconference-agenda.md#agenda
22:04:07 [jernoble]
jernoble has joined #mediawg
22:04:28 [cpn]
present+ Karl_Tomlinson
22:04:37 [cpn]
present+ Eric_Carlson
22:04:42 [jernoble]
present+ jernoble
22:05:07 [gregwf]
gregwf has joined #mediawg
22:05:22 [cpn]
present+ Mark_Watson
22:05:33 [cpn]
present+ Jean-Yves_Avenard
22:06:32 [tidoust]
scribe+ tidoust
22:06:36 [tidoust]
Topic: Agenda bashing
22:07:05 [tidoust]
cpn: Main topic is MSE Interoperability/implementation conformance issues, including perhaps reflections on changes that needed to be brought to MSE in worker
22:07:14 [tidoust]
... Also topic on media capabilities towards the end of the call
22:07:18 [tidoust]
Topic: MSE Interoperability/implementation conformance issues
22:07:31 [tidoust]
Subtopic: Gapless audio support feature detection
22:07:43 [tidoust]
ghurlbot, this is w3c/media-source
22:07:43 [ghurlbot]
tidoust, OK
22:07:51 [tidoust]
issue #37
22:07:52 [ghurlbot]
https://github.com/w3c/media-source/issues/37 -> Issue 37 Support sample accurate audio splicing using timestampOffset/appendWindowStart/appendWindowEnd (Melatonin64) feature request, needs author input, TPAC-2022-discussion
22:07:52 [Github]
https://github.com/w3c/media-wg/issues/37 : October 11 Media WG meeting
22:08:56 [tidoust]
Matt_Wolenetz: When the presentation coded frame of two audio fragments overlap, I believe the spec describes a cross-fade. Chrome used to have that, but the widespread proliferation of incorrect timing and codec formats
22:09:05 [tidoust]
... meant that we now truncate.
22:09:28 [tidoust]
... We couldn't perceive any benefit with cross-fade.
22:09:46 [tidoust]
... Meanwhile, chromium implementation allows to do frame accurate splicing.
22:10:12 [tidoust]
... In order to do that, you must not drop everything that was decoded from both audio frames.
22:10:27 [tidoust]
... And perform trimming.
22:10:55 [tidoust]
... Issue is whether we can standardize this feature: frame-accurate audio frame post-decoding splicing
22:11:24 [tidoust]
... And also can we add feature detection for whether this is supported by the browser
22:12:04 [tidoust]
jean-yves: In Firefox, the data will be trimmed once decoded. More accuracy than the 1048 sample.
22:12:34 [tidoust]
Matt_Wolenetz: You need to feed more than just the frame so that at the splice point, this is stable.
22:12:46 [tidoust]
... Yes, that is what I'm talking about.
22:13:32 [tidoust]
jean-yves: Multiple encodings use different flags. You end up with conflicts, e.g. with ADTS.
22:13:55 [tidoust]
... Is it something that will be within MSE, like a message to send, or contained within the binary format that we append.
22:14:46 [tidoust]
Matt_Wolenetz: In terms of specs, yes. In terms of the binary format, the decoded sequence needs to be appended in a well-formed sequence by the app, so that the splicing point gets clearly set.
22:15:14 [tidoust]
... The app needs to add enough of the B frames so that the splicing can happen smoothly.
22:16:20 [tidoust]
jean-yves: The way MSE is designed is that it's always dealing with compressed frames rather than decoded frames.
22:17:03 [tidoust]
Matt_Wolenetz: We check in Chromium append calls for positions where splicing might append, especially for audio.
22:17:10 [tidoust]
... And we keep track of them.
22:18:06 [tidoust]
... We decode across the splice point faster than real-time so that we can adjust things. That enables the frame-accurate gapless audio scenario.
22:18:15 [tidoust]
... It's been in Chromium for a long time.
22:18:30 [tidoust]
... We haven't had the cross-fade feature in Chromium as a result.
22:19:01 [tidoust]
... [more technical details not captured by scribe]
22:19:27 [tidoust]
... Not all implementations are required to have faster-than-real-time decoding capabilities, which may make it a concern for some devices and platforms.
22:19:50 [tidoust]
jernoble: Back to cross-fade, I don't think webkit does it either. Firefox?
22:20:09 [tidoust]
jean-yves: I don't think there is cross-fading either.
22:20:35 [tidoust]
Matt_Wolenetz: The spec says: if you don't do cross-fade, you need to drop. We don't do either, since we do frame-accurate splicing.
22:20:57 [tidoust]
jean-yves: I have only seen frame-accurate splicing in demo apps, not real ones.
22:21:09 [tidoust]
Matt_Wolenetz: There used to be a Google gapless music app.
22:22:01 [tidoust]
jernoble: Webkit will mark windows and overlaps, with samples to throw. That has led to some discontinuities.
22:22:41 [tidoust]
... I wonder if there's something that we can do in the case where the append is smaller than the codec window.
22:23:48 [tidoust]
... A microsecond is a gap. You'll hear a click when that happens.
22:24:14 [tidoust]
... We have gotten feedback from the hardware team as some of the recent hardware is pretty sensitive about that.
22:24:37 [tidoust]
... That's what the cross-fade was intended to address.
22:24:58 [tidoust]
Matt_Wolenetz: You can have both frame-accurate and cross-fade.
22:25:37 [tidoust]
karlt: Normative text says you need to drop the whole block and then in a non-normative note it says you can do something else.
22:26:19 [tidoust]
... It seems that all of the implementations keep blocks, perhaps that should be what the normative part of the spec says.
22:26:32 [tidoust]
Matt_Wolenetz: I agree.
22:26:47 [tidoust]
... We don't know whether TV devices
22:27:13 [tidoust]
... keep or drop the frames.
22:27:38 [tidoust]
jean-yves: No implementation ever stops on gaps.
22:28:14 [tidoust]
cpn: So we've got consistent behavior. Question about TV devices. Is that something that we should take up to TV people?
22:28:25 [tidoust]
... That would provide input on the feature detection part of the feature.
22:29:29 [tidoust]
jean-yves: Should we consider that feature detection as part of media capabilities instead of adding a mechanism in MSE that currently does not exist?
22:29:49 [tidoust]
... Media Capabilities has a way to test if MSE supports a particular system.
22:30:41 [tidoust]
Matt_Wolenetz: Question on software decoder. Media Capabilities covers more than MSE. If MSE can answer the question for itself, it seems preferable to do it in MSE.
22:31:24 [tidoust]
... The whole collection of issues that we're discussing are priority 2. That's because they're hard to test, hard to find the right behavior, etc.
22:32:06 [tidoust]
jernoble: If the use case is just "gapless labeling", that doesn't seem like a super compelling use case to me.
22:32:43 [tidoust]
Mark_Watson: gapless Audio splicing is also useful in a number of other scenarios.
22:33:28 [tidoust]
... What is happening today is not ideal, but it's better to leave it at what it is in order not to introduce differences of behavior.
22:33:49 [tidoust]
jernoble: Seems to match what Will Law has been asking for some time.
22:34:01 [tidoust]
... Making things observable without changing the behavior.
22:35:10 [tidoust]
Mark_Watson: It depends on whether we're talking about exposing more information about what browsers are already doing today, then yes that's useful, but I somehow already know that, even though it's better to know if that behavior is guaranteed.
22:35:34 [tidoust]
... Or whether we're talking about exposing information about a behavior that changed in browsers.
22:36:51 [tidoust]
Matt_Wolenetz: My assumption was that Chromium was the only browser doing frame-accurate splicing, but that seems wrong. So feature detection was meant to detect new browsers supporting the feature, but that may not be needed anymore.
22:37:58 [tidoust]
... For lots of splicing scenarios, it depends on the codec and within the codecs whether you're at the start or in the middle of a frame.
22:38:23 [tidoust]
... It doesn't seem that it would be a breaking change if an implementation starts doing this.
22:38:54 [tidoust]
... but it might be useful to tell the app that a browser supports this feature upfront.
22:39:22 [tidoust]
... If there are multiple behavior allowed by the spec, it's probably better to have a mechanism to detect the feature.
22:39:35 [tidoust]
cpn: It sounds like next step is around testing.
22:39:56 [tidoust]
Matt_Wolenetz: Yes, we seem to all be saying that we're doing some form of frame-accurate splicing.
22:40:21 [tidoust]
Subtopic: Relax timing constraint of initial HAVE_METADATA and later HAVE_CURRENT_DATA
22:40:30 [tidoust]
issue #275
22:40:30 [ghurlbot]
https://github.com/w3c/media-source/issues/275 -> Issue 275 Consider relaxing timing of initial HAVE_METADATA transitioning (wolenetz) agenda, TPAC-2022-discussion
22:40:32 [tidoust]
issue #215
22:40:32 [ghurlbot]
https://github.com/w3c/media-source/issues/215 -> Issue 215 Spec is too rigid on requiring initial HAVE_CURRENT_DATA transition occur synchronously within coded frame processing (possibly ditto for HAVE_METADATA and init segment received processing) (wolenetz) interoperability, TPAC-2022-discussion
22:41:06 [tidoust]
Matt_Wolenetz: In Chromium, we have a separate thread that holds decoding, buffering and so on.
22:41:45 [tidoust]
... To not block the main thread for some actions, we don't block the update and delivery of event scheduling while we wait for transitioning statuses.
22:42:48 [tidoust]
... In Segment parsing loop, I think, you're supposed to wait until HAVE_METADATA. We don't do that in Chrome. I'd like to relax the constraints in order not to do undue blocking of the main thread.
22:43:10 [tidoust]
jernoble: That seems reasonable to me.
22:43:25 [tidoust]
... Especially for workers.
22:44:19 [tidoust]
cpn: Last time we discussed, there was a question about gathering information on what different implementations actually do. Do we need that? It seems to me that if we're just relaxing the constraints, we can go ahead.
22:44:40 [tidoust]
Matt_Wolenetz: Question is whether it will bug applications if more implementations do what Chrome already does.
22:45:23 [tidoust]
... You may get duration information faster than ready information. That can surprise apps, but then Chrome has been doing that forever.
22:46:24 [tidoust]
... There's been some timing hiccups in the tests in WPT.
22:46:52 [tidoust]
... It sounds like it may be something to propose in a PR.
22:47:11 [tidoust]
... If you can see any kind of regression that this may create, please raise it.
22:47:23 [tidoust]
cpn: Checking in with some of the major player libraries might be useful.
22:47:53 [tidoust]
... So next step on this is a PR.
22:48:09 [tidoust]
Subtopic: API for interoperable gap playback/tolerance
22:48:13 [tidoust]
issue #160
22:48:14 [ghurlbot]
https://github.com/w3c/media-source/issues/160 -> Issue 160 Support playback through unbuffered ranges, and allow app to provide buffered gap tolerance (davemevans) feature request, TPAC-2022-discussion
22:48:38 [tidoust]
Matt_Wolenetz: My time on MSE is limited right now.
22:49:33 [tidoust]
... As part of the HLS native implementation in Chrome that we're building on top of MSE concepts, we need to implement an internal API for interoperable gap playback/tolerance.
22:49:55 [tidoust]
... The problem right now is that we have some stalls happening based on various tolerance settings.
22:50:36 [tidoust]
jernoble: Some of the issues around gaps should be handled by modifying the output timeline, which is something we do for small gaps.
22:52:05 [tidoust]
Matt_Wolenetz: Now there's also out of order audio codecs.
22:53:23 [tidoust]
... This API came in discussions recently at FOMS, which I was unable to attend.
22:54:01 [tidoust]
jernoble: Every time people attempt to implement HLS in MSE, we discover new features that MSE is missing, so interested in your exploration.
22:54:32 [tidoust]
Subtopic: Reflections on MSE in Worker and WG review process
22:55:07 [tidoust]
Matt_Wolenetz: The issue that was encountered was late phase based on comments from Mozilla
22:55:23 [tidoust]
... transition from MediaSource Handle to a property.
22:56:04 [tidoust]
... I made the assumption that [missed] could be retained on the property. That was not true. We ended up with regression.
22:56:16 [tidoust]
... What we would be good would be more thorough reviews.
22:56:32 [tidoust]
... The more eyes on such features, the better.
22:56:40 [tidoust]
... No one can claim to be an expert on the whole web platform.
22:57:11 [tidoust]
... Internally, we've been reflecting on binding generators that could warn us when there are problems.
22:57:24 [tidoust]
cpn: I see there's like a TAG issue that you raised about this
22:57:31 [tidoust]
-> TAG review issue https://github.com/w3ctag/design-principles/issues/400
22:57:45 [tidoust]
s/TAG review/TAG design principle
22:58:25 [tidoust]
Matt_Wolenetz: That's about it for MSE. I will be focusing on the HLS implementation in MSE.
22:58:51 [tidoust]
... Getting pre-emptive media source in the meantime would be great to unblock MSE in iOS.
22:59:05 [tidoust]
cpn: What would you like to do about remaining issues?
22:59:29 [tidoust]
Matt_Wolenetz: It depends on how much incoming feedback and comments we get on these.
22:59:56 [tidoust]
cpn: OK, we'll check with you before next call.
23:00:05 [tidoust]
Topic: Media Capabilities editors
23:00:28 [tidoust]
cpn: Mounir and Chris moved on to other things. So we need new editors.
23:00:41 [tidoust]
... I think Vi from Microsoft is the only one still around.
23:00:49 [tidoust]
jean-yves: I can help with some of that.
23:01:16 [tidoust]
cpn: Thanks for the offer. I'll send the call around to see if anyone is willing to help you.
23:01:35 [tidoust]
RRSAgent, draft minutes
23:01:35 [RRSAgent]
I have made the request to generate https://www.w3.org/2022/11/08-mediawg-minutes.html tidoust