20:56:54 RRSAgent has joined #mediawg 20:56:54 logging to https://www.w3.org/2021/09/28-mediawg-irc 20:56:57 Zakim has joined #mediawg 20:57:03 RRSagent, make logs public 20:57:18 Meeting: Media WG Teleconference 20:57:39 Agenda: https://github.com/w3c/media-wg/blob/main/meetings/2021-09-28-Media_Working_Group_Teleconference-agenda.md 21:00:38 cyril has joined #mediawg 21:00:38 present+ Chris_Needham, Jan-Ivar_Bruaroey, Francois_Daoust, Cyril_Concolato 21:01:18 present+ Yuhao_Fu, Bernard_Aboba 21:02:01 markw has joined #mediawg 21:02:55 present+ markw 21:03:39 present+ Peng_Liu 21:05:08 pengliu has joined #mediawg 21:05:41 present+ Eric_Carlson, Jer_Noble 21:09:49 scribenick: cpn 21:09:52 Topic: TPAC planning: WebRTC WG / Media WG joint meeting 21:10:12 Chris: Topics for WebRTC joint meeting? 21:10:36 Jer: Media capabilities and harmonising with RTC 21:11:09 Jan-Ivar: Main topic is MediaCapture Transform or its replacement, about exposing real-time audio and video between MediaStreamTrack and JS 21:11:25 ... Now it seems will be Streams based. Some issues open around that 21:11:40 ... Specific to WebCodecs are video frame lifetime, close and clone methods 21:12:02 ... GC cleanup. If you do a tee to branch a stream into two, there's no cloning by default 21:12:26 ... When one branch closes the video frame it stops working from the other. Want to discuss also audio issues 21:12:36 ... We invited the Audio WG to also join 21:13:12 Bernard: TPAC is a good time for overviews or relationships between things 21:13:29 ... So the overall direction we're going with media, using Streams to create pipelines to process media 21:13:54 ... Future of content protection. S-Frame in WebRTC is an encrypted content form, doesn't work with WebCodecs 21:14:32 ... Questions about transports. Looking at the overview, what's missing, how does it all fit together for developers? 21:14:54 ... We get lots of questions from developers. How to render: WebGPU, Canvas, etc. Any recommendations? 21:15:00 ... Coherent view on Workers 21:15:29 Chris: Anything on WebTransport specifically? 21:16:03 Bernard: WT supports workers, RTCDataChannel doesn't, but that's proposed, it's an extension spec. PeerConnection doesn't support workers 21:16:47 ... So lots of different things, good to look at the overall picture. What are we not doing? 21:17:43 Jan-Ivar: Opportunity to look at the overall picture, Alternative to stand up an alternative to WebRTC using the other APIs. Looking at how that fits together and what does and doesn't work 21:18:32 Bernard: Developers ask how all the parts fit together. Could present what we think the overview is, see where there's agreement. Does it make sense? 21:19:24 Jan-Ivar: We'll make slides in WebRTC WG. Media WG can contribute to that 21:19:45 ... A question could be: If a source and sink are sink-based, why not also the bit in the middle? 21:20:01 ... Why use streams instead of promises for media capture transform? 21:20:26 ... Why isn't encode promise based? 21:20:36 ... The streams model would be able to handle it. 21:21:14 Bernard: We could present the story, issues, questions people ask. Streaming and RTC are converging, low latency streaming 21:22:05 ... Using EME with WebRTC doesn't work today. Could put some examples in 21:22:19 Jer: Can you come up with the overview? 21:23:04 Bernard: That's for the first hour, then audio for the second hour 21:24:18 Topic: MSE Bytestream Format 21:24:35 Cyril: I opened 4 issues 21:25:01 ... Generally, the intent was to read and understand the spec, check for conflicts with other specs, if any 21:25:29 ... And check interop across browsers. I also wrote some unit tests, hand-editing MP4 files and feeding to an MSE based player 21:25:38 ... I found some surprising results 21:25:43 ... I'm migrating the tests to WPT 21:25:50 ... Issue #4 ftyp box 21:25:54 -> https://github.com/w3c/mse-byte-stream-format-isobmff/issues/4 Use of ftyp box 21:26:19 ... Wording in the BSF spec says the ftyp box is part of the init segment, and the UA should run the error algorithm 21:26:37 ... if there's a mismatch. It seems to ask a lot from browsers 21:27:16 ... Requires the browser to validate. In my tests, I found the ftyp box is ignored. If I use an init segment with a moof box it's fine 21:27:36 ... I can put anything in the ftyp box and everything is accepted 21:27:54 ... My understanding of what's implemented today is that we shouldn't say anything about the ftyp box 21:28:03 MattWolenetz has joined #mediawg 21:28:07 ... Just say an init segment it just a moof box with some constraints 21:28:25 ... There's a similar statement about the segment type box that could be safely ignored 21:29:09 Jer: Wasn't there a move in the MEIG to have a restrictive ftyp that would throw errors in cases where extra boxes would be ignored. If we remove this, will there be a request to add it back? 21:29:31 ChrisN: The CMAF BSF discussion 21:29:51 Cyril: I argued against that. If all browsers implement the same thing, why take it out? 21:30:09 Jer: I agree. It's that they didn't have separate mime type so wanted to use ftyp 21:30:36 ... I think if we can parse the file, we should, and not throw errors. Being relaxed in error handling is consistent with the web approach 21:30:44 ... I'm fine with adding it to the list of boxes to ignore 21:30:54 ... But concerned that others will object 21:31:20 Cyril: I doubt browsers will verify conformance to brands. Browsers and players try to do their best with the content 21:31:40 Jer: Making WPTs would answer these questions. Update the spec to match browser behaviour 21:32:02 Cyril: If a box contains a compatible brand the UA doesn't support, it should fail. That's the opposite of what ISO BMFF says 21:32:20 Jer: I wonder if what's written was the opposite of the intent 21:32:33 Matt: I agree that the ftyp box is superfluous in Chrome 21:32:41 present+ Matt_Wolenetz 21:32:55 ... Is there a case where folks want to play streams but they want capability detection based on ftyp? 21:34:02 ... As no browsers filter on this, we should remove it from the spec. It's in the list of boxes that should be skipped in implementations 21:34:21 Cyril: Next issue, edit lists 21:34:35 -> https://github.com/w3c/mse-byte-stream-format-isobmff/issues/5 Support for edit lists 21:34:39 ... The spec currently says browsers must support one type of edit lists 21:34:55 ... An offset edit list, which offsets the composition time when you have B-frames 21:35:08 ... The edit list maps the non-zero composition time to a presentation time of zero 21:35:15 ... The spec is silent on other types of edit lists 21:35:39 ... Can you use fractional or zero rate? Can you use empty or multiple entries in edit list? 21:36:00 ... Rare to have interop in those tests. Mostly browsers ignore edit lists not supported. I think it should fire an error 21:36:04 ... Could lead to A/V sync issues 21:36:27 Matt: I have a concern about raising a decode or parse error on content that previously played successfully 21:36:48 ... Could be a note for clarification on which edit lists have interoperable support, and others would be ignored 21:37:13 Cyril: I tested fractional rates, empty edit list (should fill the timeline with a gap) 21:37:40 ... Maybe deprecation first, then removal in a future edition 21:38:24 Matt: Are there components of these edit lists used with other parts of MSE, timestampOffset, playbackRate, so MSE couldn't afford applications polyfilling 21:38:34 Jer: Hard to polyfill with muxed tracks 21:39:03 Matt: Do we have any stats on existence of these kinds of edit lists? 21:39:54 Cyril: There's one that must be supported 21:40:11 Matt: So a note to say the others should be ignored by implementations 21:40:29 Cyril: I'd prefer to say "may be" ignored, and content providers "should not" use 21:40:39 Matt: Makes sense, also gather data 21:41:05 Jer: Other uses? Offset is needed for B-frames. What about multiple playback rates, other use cases? 21:41:29 Cyril: Empty edits could be used, when you want to align audio and video, you can either remove some video content to start at the audio start 21:41:43 ... or say the audio has a gap, and the player should play video without audio until audio starts 21:42:16 Jer: Another option, if we find that these are being used in the wild, could add a "should" statement for the empty edits, or others with valid use cases 21:42:45 Matt: We don't have enough data now. If there are use cases that can't be solved ergonomically in the MSE API, can address at a later time 21:43:07 ... If we don't see people complain that playback isn't working, should we then add telemetry? 21:43:23 Cyril: I think it's important to document what content creators can rely on 21:44:04 Matt: so documenting it may be ingored, and content providers should not be used. File a github issue so people can reply to bring to our attention 21:44:29 Cyril: Next is #6, support for unknown boxes. Boxes accepted and ignored 21:44:47 ... Not sure what is meant by valid top-level boxes 21:45:14 Matt: If you put an out of order box, such as a moof before a moov. The spec handles that, but are there other cases? 21:45:20 ... That's the next issue... 21:45:47 s/... That's the next issue.../Cyril: ... That's the next issue... 21:46:02 Cyril: I tested a unkn box 21:46:18 Matt: Is that in the spec, how do we know its a top-level box? 21:46:28 Cyril: From where it's placed in the stream 21:46:48 Matt: If it's not defined as a top-level box in the normative spec 21:47:05 Mark: All the boxes that the ISO spec says are allow to appear at the top level 21:47:17 Cyril: Anyone can add other boxes at the top level if they want 21:47:47 Matt: If we need to bind the MSE spec more closely to ISO BMFF, we can 21:48:07 Cyril: Is the intent to ignore unknown boxes at the top level? 21:48:15 Matt: It could indicate the stream is malformed 21:48:41 Cyril: Concerned it doesn't scale. Each time ISOBMFF spec changes, you'd have to change implementation 21:48:48 ... It has happened before 21:49:13 Jer: If a box not defined at top level is found at top level, could through an error. But it's OK to skip an unknown box 21:49:36 Cyril: Just consume the bytes and continue parsing 21:50:08 Matt: Could there be a malformed stream that causes the implementation to hold onto large blocks of data? 21:50:14 Jer: Seems like an implementation detail 21:50:47 Matt: Some implementations may see 2 gigabytes as too large and couldn't skip 21:51:20 ... We have quota exceeded mechansim. Just thinking through implementation based considerations 21:52:04 In terms of API usage, there was one user-defined box, proposed by the BSF, but that source was unaware of pre-existing top level boxes they could have used, JS level parsing 21:52:13 s/In/... In/ 21:52:23 Cyril: The unkn box is one I invented 21:52:55 ... ISO BMFF recently introduced compressed boxes (gzip). The sidx can be replaced with !sdx, defined in ISO BMFF 21:53:25 Cyril: Let's discuss how to make the spec changes 21:53:52 ... Should I open an issue for each problem, then a PR. Or propose a rewrite as a draft and review the whole thing? 21:54:11 Matt: Depends on scale. If just a few, disuss as one offs 21:54:21 ... Design principles about not wanting to regress 21:54:58 Cyril: What about small issues? I'd like to be able to rewrite the text and review as a whole 21:55:48 ... Agree on the intent of the issues, than make a PR 21:56:02 Jer: Seems reasonable to close a number of issues in one PR 21:56:19 Matt: An issue per item sounds good, and a PR that addresses multiple 21:57:02 Cyril: The BSF is a Note. Why is it not on the Rec track? If there are tests and implementations can be compliant to it, why a Note? 21:57:33 Matt: We focused on testing MSE itself and not so much the BSFs. An implementation must support *a* BSF, so implementations may support different ones 21:57:49 ... Allowed more flexibility at the time 21:58:32 Francois: Another reason it's a note relates to patents. It more directly relates to codecs, so didn't need to ask about the royalty free patent policy 22:00:23 ChrisN: What about Process 2021, provides a structure for registries and entries? 22:00:56 Matt: WebCodecs has taken a similar approach to MSE for registries. Anything we can learn from that? 22:01:22 Francois: Similar reasons, would make sense to have them as Rec track specs 22:01:40 Matt: It's been easy to propose and support new entries fairly quickly 22:02:04 Cyril: I think it's fine if the entries aren't all at same maturity level 22:02:18 Matt: I'd need to check with colleagues on that 22:02:53 Cyril: I created WPT for this spec. It may need some more work. Is there a link between WPT and this WG? 22:03:28 Matt: Existing tests were bound to the API itself, not so much a specific BSF. The tests are mostly testing the API for a supported format 22:03:44 Spawn has joined #mediawg 22:03:49 ... Testing BSF format more deeply is good, put into a subfolder 22:04:56 Cyril: I'll start a PR. How do you detect an error? Buffer range, error event, etc? 22:05:11 Matt: There's a proposed introspection API that could help with that 22:13:08 Cyril: Thank you 22:13:58 Matt: FPWD of MSE v2 and short name 22:14:17 Francois: It'll be published on Thursday, no need for a new CfC 22:14:32 Matt: Update the SoTD? 22:14:52 Francois: Yes, feel free to do that. In future we'll switch to audomatic publishing to /TR 22:15:37 RRSAgent, draft minutes 22:15:37 I have made the request to generate https://www.w3.org/2021/09/28-mediawg-minutes.html tidoust 22:15:54 Chris: Second screen WG would like a joint meeting to talk through some issues around capability detection 22:16:12 ... Will schedule that for an upcoming call, possibly next time? 22:16:18 [adjourned] 22:17:16 rrsagent, draft minutes 22:17:16 I have made the request to generate https://www.w3.org/2021/09/28-mediawg-minutes.html cpn 22:17:25 rrsagent, make log public