21:58:57 RRSAgent has joined #mediawg 21:58:57 logging to https://www.w3.org/2022/11/08-mediawg-irc 21:59:01 Zakim has joined #mediawg 21:59:50 tidoust has joined #mediawg 22:01:38 Meeting: Media WG 22:01:45 Chair: Chris_Needham 22:01:56 RRSAgent, make logs public 22:02:01 RRSAgent, draft minutes 22:02:01 I have made the request to generate https://www.w3.org/2022/11/08-mediawg-minutes.html tidoust 22:02:33 Matt_Wolenetz has joined #mediawg 22:02:55 present+ Matt_Wolenetz 22:03:01 Present: Chris_Needham, Sushanth_Rajasankar, Jer_Noble, Francois_Daoust, Matt_Wolenetz, Greg_Freedman, Joey_Parrish 22:03:08 Present: Francois_Daoust, Chris_Needham, Greg_Freedman, Matt_Wolenetz, Sushanth_Rajasankar, Karl_Tomlinson 22:03:13 s/Present: Francois_Daoust, Chris_Needham, Greg_Freedman, Matt_Wolenetz, Sushanth_Rajasankar, Karl_Tomlinson// 22:03:17 scribe+ cpn 22:03:30 present+ Frank_Liberato 22:04:02 Agenda: https://github.com/w3c/media-wg/blob/main/meetings/2022-11-08-Media_Working_Group_Teleconference-agenda.md#agenda 22:04:07 jernoble has joined #mediawg 22:04:28 present+ Karl_Tomlinson 22:04:37 present+ Eric_Carlson 22:04:42 present+ jernoble 22:05:07 gregwf has joined #mediawg 22:05:22 present+ Mark_Watson 22:05:33 present+ Jean-Yves_Avenard 22:06:32 scribe+ tidoust 22:06:36 Topic: Agenda bashing 22:07:05 cpn: Main topic is MSE Interoperability/implementation conformance issues, including perhaps reflections on changes that needed to be brought to MSE in worker 22:07:14 ... Also topic on media capabilities towards the end of the call 22:07:18 Topic: MSE Interoperability/implementation conformance issues 22:07:31 Subtopic: Gapless audio support feature detection 22:07:43 ghurlbot, this is w3c/media-source 22:07:43 tidoust, OK 22:07:51 issue #37 22:07:52 https://github.com/w3c/media-source/issues/37 -> Issue 37 Support sample accurate audio splicing using timestampOffset/appendWindowStart/appendWindowEnd (Melatonin64) feature request, needs author input, TPAC-2022-discussion 22:07:52 https://github.com/w3c/media-wg/issues/37 : October 11 Media WG meeting 22:08:56 Matt_Wolenetz: When the presentation coded frame of two audio fragments overlap, I believe the spec describes a cross-fade. Chrome used to have that, but the widespread proliferation of incorrect timing and codec formats 22:09:05 ... meant that we now truncate. 22:09:28 ... We couldn't perceive any benefit with cross-fade. 22:09:46 ... Meanwhile, chromium implementation allows to do frame accurate splicing. 22:10:12 ... In order to do that, you must not drop everything that was decoded from both audio frames. 22:10:27 ... And perform trimming. 22:10:55 ... Issue is whether we can standardize this feature: frame-accurate audio frame post-decoding splicing 22:11:24 ... And also can we add feature detection for whether this is supported by the browser 22:12:04 jean-yves: In Firefox, the data will be trimmed once decoded. More accuracy than the 1048 sample. 22:12:34 Matt_Wolenetz: You need to feed more than just the frame so that at the splice point, this is stable. 22:12:46 ... Yes, that is what I'm talking about. 22:13:32 jean-yves: Multiple encodings use different flags. You end up with conflicts, e.g. with ADTS. 22:13:55 ... Is it something that will be within MSE, like a message to send, or contained within the binary format that we append. 22:14:46 Matt_Wolenetz: In terms of specs, yes. In terms of the binary format, the decoded sequence needs to be appended in a well-formed sequence by the app, so that the splicing point gets clearly set. 22:15:14 ... The app needs to add enough of the B frames so that the splicing can happen smoothly. 22:16:20 jean-yves: The way MSE is designed is that it's always dealing with compressed frames rather than decoded frames. 22:17:03 Matt_Wolenetz: We check in Chromium append calls for positions where splicing might append, especially for audio. 22:17:10 ... And we keep track of them. 22:18:06 ... We decode across the splice point faster than real-time so that we can adjust things. That enables the frame-accurate gapless audio scenario. 22:18:15 ... It's been in Chromium for a long time. 22:18:30 ... We haven't had the cross-fade feature in Chromium as a result. 22:19:01 ... [more technical details not captured by scribe] 22:19:27 ... Not all implementations are required to have faster-than-real-time decoding capabilities, which may make it a concern for some devices and platforms. 22:19:50 jernoble: Back to cross-fade, I don't think webkit does it either. Firefox? 22:20:09 jean-yves: I don't think there is cross-fading either. 22:20:35 Matt_Wolenetz: The spec says: if you don't do cross-fade, you need to drop. We don't do either, since we do frame-accurate splicing. 22:20:57 jean-yves: I have only seen frame-accurate splicing in demo apps, not real ones. 22:21:09 Matt_Wolenetz: There used to be a Google gapless music app. 22:22:01 jernoble: Webkit will mark windows and overlaps, with samples to throw. That has led to some discontinuities. 22:22:41 ... I wonder if there's something that we can do in the case where the append is smaller than the codec window. 22:23:48 ... A microsecond is a gap. You'll hear a click when that happens. 22:24:14 ... We have gotten feedback from the hardware team as some of the recent hardware is pretty sensitive about that. 22:24:37 ... That's what the cross-fade was intended to address. 22:24:58 Matt_Wolenetz: You can have both frame-accurate and cross-fade. 22:25:37 karlt: Normative text says you need to drop the whole block and then in a non-normative note it says you can do something else. 22:26:19 ... It seems that all of the implementations keep blocks, perhaps that should be what the normative part of the spec says. 22:26:32 Matt_Wolenetz: I agree. 22:26:47 ... We don't know whether TV devices 22:27:13 ... keep or drop the frames. 22:27:38 jean-yves: No implementation ever stops on gaps. 22:28:14 cpn: So we've got consistent behavior. Question about TV devices. Is that something that we should take up to TV people? 22:28:25 ... That would provide input on the feature detection part of the feature. 22:29:29 jean-yves: Should we consider that feature detection as part of media capabilities instead of adding a mechanism in MSE that currently does not exist? 22:29:49 ... Media Capabilities has a way to test if MSE supports a particular system. 22:30:41 Matt_Wolenetz: Question on software decoder. Media Capabilities covers more than MSE. If MSE can answer the question for itself, it seems preferable to do it in MSE. 22:31:24 ... The whole collection of issues that we're discussing are priority 2. That's because they're hard to test, hard to find the right behavior, etc. 22:32:06 jernoble: If the use case is just "gapless labeling", that doesn't seem like a super compelling use case to me. 22:32:43 Mark_Watson: gapless Audio splicing is also useful in a number of other scenarios. 22:33:28 ... What is happening today is not ideal, but it's better to leave it at what it is in order not to introduce differences of behavior. 22:33:49 jernoble: Seems to match what Will Law has been asking for some time. 22:34:01 ... Making things observable without changing the behavior. 22:35:10 Mark_Watson: It depends on whether we're talking about exposing more information about what browsers are already doing today, then yes that's useful, but I somehow already know that, even though it's better to know if that behavior is guaranteed. 22:35:34 ... Or whether we're talking about exposing information about a behavior that changed in browsers. 22:36:51 Matt_Wolenetz: My assumption was that Chromium was the only browser doing frame-accurate splicing, but that seems wrong. So feature detection was meant to detect new browsers supporting the feature, but that may not be needed anymore. 22:37:58 ... For lots of splicing scenarios, it depends on the codec and within the codecs whether you're at the start or in the middle of a frame. 22:38:23 ... It doesn't seem that it would be a breaking change if an implementation starts doing this. 22:38:54 ... but it might be useful to tell the app that a browser supports this feature upfront. 22:39:22 ... If there are multiple behavior allowed by the spec, it's probably better to have a mechanism to detect the feature. 22:39:35 cpn: It sounds like next step is around testing. 22:39:56 Matt_Wolenetz: Yes, we seem to all be saying that we're doing some form of frame-accurate splicing. 22:40:21 Subtopic: Relax timing constraint of initial HAVE_METADATA and later HAVE_CURRENT_DATA 22:40:30 issue #275 22:40:30 https://github.com/w3c/media-source/issues/275 -> Issue 275 Consider relaxing timing of initial HAVE_METADATA transitioning (wolenetz) agenda, TPAC-2022-discussion 22:40:32 issue #215 22:40:32 https://github.com/w3c/media-source/issues/215 -> Issue 215 Spec is too rigid on requiring initial HAVE_CURRENT_DATA transition occur synchronously within coded frame processing (possibly ditto for HAVE_METADATA and init segment received processing) (wolenetz) interoperability, TPAC-2022-discussion 22:41:06 Matt_Wolenetz: In Chromium, we have a separate thread that holds decoding, buffering and so on. 22:41:45 ... To not block the main thread for some actions, we don't block the update and delivery of event scheduling while we wait for transitioning statuses. 22:42:48 ... In Segment parsing loop, I think, you're supposed to wait until HAVE_METADATA. We don't do that in Chrome. I'd like to relax the constraints in order not to do undue blocking of the main thread. 22:43:10 jernoble: That seems reasonable to me. 22:43:25 ... Especially for workers. 22:44:19 cpn: Last time we discussed, there was a question about gathering information on what different implementations actually do. Do we need that? It seems to me that if we're just relaxing the constraints, we can go ahead. 22:44:40 Matt_Wolenetz: Question is whether it will bug applications if more implementations do what Chrome already does. 22:45:23 ... You may get duration information faster than ready information. That can surprise apps, but then Chrome has been doing that forever. 22:46:24 ... There's been some timing hiccups in the tests in WPT. 22:46:52 ... It sounds like it may be something to propose in a PR. 22:47:11 ... If you can see any kind of regression that this may create, please raise it. 22:47:23 cpn: Checking in with some of the major player libraries might be useful. 22:47:53 ... So next step on this is a PR. 22:48:09 Subtopic: API for interoperable gap playback/tolerance 22:48:13 issue #160 22:48:14 https://github.com/w3c/media-source/issues/160 -> Issue 160 Support playback through unbuffered ranges, and allow app to provide buffered gap tolerance (davemevans) feature request, TPAC-2022-discussion 22:48:38 Matt_Wolenetz: My time on MSE is limited right now. 22:49:33 ... As part of the HLS native implementation in Chrome that we're building on top of MSE concepts, we need to implement an internal API for interoperable gap playback/tolerance. 22:49:55 ... The problem right now is that we have some stalls happening based on various tolerance settings. 22:50:36 jernoble: Some of the issues around gaps should be handled by modifying the output timeline, which is something we do for small gaps. 22:52:05 Matt_Wolenetz: Now there's also out of order audio codecs. 22:53:23 ... This API came in discussions recently at FOMS, which I was unable to attend. 22:54:01 jernoble: Every time people attempt to implement HLS in MSE, we discover new features that MSE is missing, so interested in your exploration. 22:54:32 Subtopic: Reflections on MSE in Worker and WG review process 22:55:07 Matt_Wolenetz: The issue that was encountered was late phase based on comments from Mozilla 22:55:23 ... transition from MediaSource Handle to a property. 22:56:04 ... I made the assumption that [missed] could be retained on the property. That was not true. We ended up with regression. 22:56:16 ... What we would be good would be more thorough reviews. 22:56:32 ... The more eyes on such features, the better. 22:56:40 ... No one can claim to be an expert on the whole web platform. 22:57:11 ... Internally, we've been reflecting on binding generators that could warn us when there are problems. 22:57:24 cpn: I see there's like a TAG issue that you raised about this 22:57:31 -> TAG review issue https://github.com/w3ctag/design-principles/issues/400 22:57:45 s/TAG review/TAG design principle 22:58:25 Matt_Wolenetz: That's about it for MSE. I will be focusing on the HLS implementation in MSE. 22:58:51 ... Getting pre-emptive media source in the meantime would be great to unblock MSE in iOS. 22:59:05 cpn: What would you like to do about remaining issues? 22:59:29 Matt_Wolenetz: It depends on how much incoming feedback and comments we get on these. 22:59:56 cpn: OK, we'll check with you before next call. 23:00:05 Topic: Media Capabilities editors 23:00:28 cpn: Mounir and Chris moved on to other things. So we need new editors. 23:00:41 ... I think Vi from Microsoft is the only one still around. 23:00:49 jean-yves: I can help with some of that. 23:01:16 cpn: Thanks for the offer. I'll send the call around to see if anyone is willing to help you. 23:01:35 RRSAgent, draft minutes 23:01:35 I have made the request to generate https://www.w3.org/2022/11/08-mediawg-minutes.html tidoust