IRC log of webcodecs-serialization on 2023-09-13

Timestamps are in UTC.

06:58:02 [RRSAgent]

RRSAgent has joined #webcodecs-serialization

06:58:06 [RRSAgent]

logging to https://www.w3.org/2023/09/13-webcodecs-serialization-irc

06:58:06 [Ian]

RRSAgent, do not leave

06:58:06 [Ian]

RRSAgent, make logs public

06:58:07 [Ian]

Meeting: WebCodecs Serialization Format

06:58:07 [Ian]

Chair: Peter Thatcher

06:58:08 [Ian]

Agenda: https://github.com/w3c/tpac2023-breakouts/issues/78

06:58:10 [Ian]

clear agenda

06:58:12 [Ian]

agenda+ Pick a scribe

06:58:14 [Ian]

agenda+ Reminders: code of conduct, health policies, recorded session policy

06:58:16 [Ian]

agenda+ Goal of this session

06:58:20 [Ian]

agenda+ Discussion

06:58:22 [Ian]

agenda+ Next steps / where discussion continues

06:58:31 [Ian]

Ian has left #webcodecs-serialization

07:15:38 [Ian]

Ian has joined #webcodecs-serialization

07:15:42 [Ian]

Ian has left #webcodecs-serialization

13:16:47 [RRSAgent]

RRSAgent has joined #webcodecs-serialization

13:16:47 [RRSAgent]

logging to https://www.w3.org/2023/09/13-webcodecs-serialization-irc

14:00:40 [cpn]

cpn has joined #webcodecs-serialization

14:01:17 [cpn]

Meeting: WebCodecs Serialization Format

14:01:23 [cpn]

scribe+ cpn

14:03:10 [cpn]

Present+ Peter_Thatcher, Eugene_Zemtsov, Harald_Alvestrand, Xiaohan_Wang, Eric_Carlson, Youenn_Fablet, Paul_Adenot

14:03:14 [cpn]

rrsagent, draft minutes

14:03:15 [RRSAgent]

I have made the request to generate https://www.w3.org/2023/09/13-webcodecs-serialization-minutes.html cpn

14:03:19 [cpn]

rrsagent, make log public

14:03:58 [cpn]

present+ Rijubrata_Bhaumik

14:04:23 [cpn]

Peter: In the IETF MoQ group, there's interest in sending media over QUIC

14:04:38 [cpn]

... How do you serialize the media into bytes so you can send over QUIC stream or datagrams

14:04:56 [cpn]

... Involves attaching metadata. Looked at different codecs, which have different modes

14:04:58 [Xiaohan]

Xiaohan has joined #webcodecs-serialization

14:05:04 [cpn]

... WebCodecs registry is a place we can point to

14:05:22 [cpn]

... People will want to make MoQ endpoints using web clients, using WebCodecs and WebTransport

14:05:37 [Riju]

Riju has joined #webcodecs-serialization

14:05:43 [cpn]

... Want to take the output from WebCodecs, serialize, then send to a decoder on the other side, which may be WebCodecs

14:06:10 [cpn]

... I did something similar in Second Screen WG, on Open Screen Protocol

14:06:23 [cpn]

... At that time we chose CBOR, a kind of binary JSON

14:06:59 [cpn]

... What is the interest from others? What would you want to do with the serialization, store it, send over QUIC?

14:07:30 [cpn]

Harald: IIUC this is about making sure we have both encoded data and metadata for a frame on the other side of the connection

14:08:06 [cpn]

... So it's like the RTP encapsulation format

14:08:13 [cpn]

... We don't have a 1:1 mapping of it

14:08:13 [cpn]

... Which parts of this should be different from RTP formats, and why?

14:08:23 [cpn]

Bernard: This would include things normally in RTP extensions as well

14:08:59 [cpn]

... Applies to the RTP packet, timestamp, things related to SVC in IETF are always packeted

14:09:16 [cpn]

... Would become patent encumbered in IETF

14:09:36 [cpn]

Peter: Different questions: where should this happen? How different should this be to work already done?

14:09:45 [cpn]

... IETF looking at how to do RTP over QUIC

14:10:15 [cpn]

??: I thought the output of the encoder was already serialized and you could plug it into the decoder. What additional metadata is needed?

14:11:46 [cpn]

Peter: For example, the output of the encoder includes EncodedVideoChunk, which includes the bytes, timestamp, keyframe flag, metadata such as how SVC is structured

14:11:46 [cpn]

... Information about size, color space, and which codec it is

14:12:09 [cpn]

... In header extensions, things like frame dependencies. In RTP payload header, there's metadata

14:12:24 [cpn]

... There's metadata tied to the unencoded media, e.g., audio level

14:12:41 [cpn]

... So if you look at RTP metadata, it's the same kind of metadata

14:12:59 [cpn]

??: Is the metadata already exposed to JS, just needs packaging, or are there internal pieces needed?

14:13:19 [cpn]

Peter: Want it done in a standardised way, but they don't want to use RTP

14:14:05 [cpn]

... There could be internal metadata, but a good start is to take everything in the API and serialize that

14:14:24 [cpn]

Youenn: Are you thinking of extensions?

14:14:59 [cpn]

??: It's whether the problem can be solved in JavaScript, it sounds like I could convert to JSON and parse on the other side. Would that work?

14:15:24 [cpn]

Peter: You could, yes. Make your own non-standard format. The question is what interest is there in having a standard format?

14:15:31 [cpn]

... There's interest in the MoQ WG for that

14:15:46 [cpn]

... Is there interest here?

14:16:05 [cpn]

Bernard: The standards interest comes from the caching. MoQ uses a cache, relays need metadata to do forwarding

14:16:14 [padenot_]

padenot_ has joined #webcodecs-serialization

14:16:20 [xueyuan]

s/??/Henrik/

14:16:34 [cpn]

Peter: Relays are designed to be agnostic to the media format, and some metadata is opaque to the relay

14:17:02 [cpn]

Bernard: Not sure that's true, there's preferential forwarding, may be ready by caches

14:17:19 [cpn]

Peter: That would have to be defined by MoQ WG, as it's MoQ specific

14:17:37 [cpn]

... The serialization can be done in a separate doc, as it's opaque to relays, that's my understanding

14:18:15 [cpn]

... Are people interested in being able to serialize outside of MoQ, for other use cases

14:18:27 [cpn]

Bernard: There have been issues raised about support for DRM

14:18:49 [cpn]

Xiaohan: How is that related to this question for a need to serialize WebCodecs?

14:19:25 [cpn]

Harald: We have a proposal to define a generic packetizing format for RTP so you can send any content you want over the wire

14:20:27 [cpn_]

cpn_ has joined #webcodecs-serialization

14:20:32 [cpn_]

scribenick: cpn_

14:20:50 [cpn_]

Peter: Content of S frames are opaque

14:21:12 [cpn_]

Harald: This came up in a side discussion about SDP munging for WebRTC encoded transform

14:21:50 [cpn_]

Peter: One way to think about it is, if we were to start from scratch and solve the same thing, serialize the payload, and accumulate all the header extensions, what would we do?

14:22:33 [cpn_]

... Similar question in the Second Screen group, we used CBOR and came up with types. It was straightforward, easier to use a CBOR parser

14:23:36 [cpn_]

... Let's come to Bernard's question about where this should be worked on

14:23:36 [Riju]

Riju has joined #webcodecs-serialization

14:23:38 [cpn_]

... There's IPR issues for some of the metadata

14:24:01 [cpn_]

Bernard: The concern would be if you take the WebCodecs spec you might start encumbering WebCodecs, that would be awful

14:24:24 [cpn_]

Peter: What about trying to define the serialization format in W3C, would there be enough people interested in working on it to do it here?

14:25:09 [cpn_]

Bernard: Media WG already has an ISO BMFF format for MSE

14:25:14 [cpn_]

Harald: A note of caution on IPR. IETF runs on the principle that every participant should disclose their IPR, but doesn't force anyone to give their rights to anything

14:25:24 [cpn_]

... At W3C all participants make a commitment when joining the WG

14:25:37 [cpn_]

... If you don't join the WG, no promises made

14:26:49 [cpn_]

... So unless all contributors are in the W3C WG and agree to remain members when the work item is adopted, that's what's required for W3C

14:27:21 [cpn_]

... Getting all people who're asserting IPR to join might be the issue

14:27:26 [cpn_]

Youenn: I'd tend to pick the venue where more people are interested

14:28:04 [cpn_]

Peter: Gauging level of interest is the purpose of this

14:28:25 [cpn_]

Xiaohan: What's the use case you're imagining?

14:29:01 [cpn_]

Peter: It could be used for any purpose. For the MoQ WG, it's for real time and streaming use cases, e.g., live broadcast video conference

14:29:49 [cpn_]

Xiaohan: There are already solutions not using WebCodecs out there. If it's for streaming then CMAF can be the format, but too heavy for real time

14:29:58 [cpn_]

... Why not adopt an existing format?

14:30:21 [cpn_]

Peter: MoQ WG separated the two, so the format can be CMAF based

14:31:03 [cpn_]

... Others who want to do things web oriented, you don't want to containerise to CMAF, more straightforward to do what JSON does, but more efficient, hence CBOR

14:31:40 [cpn_]

... For realtime or very low latency streaming it would be a better DX to do something new, which is why there's interest in MoQ WG for a low overhead container

14:32:02 [cpn_]

Xiaohan: Format of MediaStream in WebRTC?

14:32:28 [cpn_]

Peter: That uses RTP, as Harald described. It could be done over QUIC, there's work in IETF, but like CMAF it carries baggage

14:33:52 [cpn_]

Chris: Wouldn't MoQ WG need to define a format anyway, absent any work here?

14:34:13 [cpn_]

Peter: Yes, they're working on the CMAF based format, but the question is about the low latency format

14:34:55 [cpn_]

Bernard: The Media WG meeting heard about the IETF 117 WebCodecs presentation

14:35:47 [cpn_]

... https://datatracker.ietf.org/meeting/117/materials/slides-117-moq-webcodes-container-00

14:36:42 [cpn_]

Peter: WebCodecs already has the metadata defined that we'd want to serialize

14:36:57 [cpn_]

... What we define might be usable in different situations

14:37:52 [cpn_]

... Continue in IETF or bring to W3C

14:38:44 [cpn_]

Chris: Interest expressed at IETF to work on it?

14:38:54 [cpn_]

Peter: Yes, three of us at least are interested, and others

14:42:20 [cpn_]

Chris: We are developing a browser based video editor, have our own format, so we have the general need

14:42:29 [cpn_]

Peter: Need for interop around that?

14:42:38 [cpn_]

Chris: I'd have to follow up to get precise requirements

14:43:02 [cpn_]

Bernard: A frequent request we get on WebCodecs is a standard API for container formats

14:43:50 [cpn_]

Peter: I have a similar situation, where I have my own non-standard serialization, there might be potential for a standard solution

14:44:23 [cpn_]

Peter: So if not this, what are people interested in?

14:44:46 [cpn_]

Henrik: Using WebCodecs with the RTP transport in WebRTC, don't know if packetization needs to be standardised or not

14:45:58 [cpn_]

Peter: So if you were to use WebCodecs with RTP transport, how would you use it?

14:46:35 [cpn_]

Henrik: I see value in coupling app-specific metadata to video frames. Haven't done enough homework to know more

14:47:13 [cpn_]

Youenn: MoQ is trying to do real time for QUIC, server to client. Why not using the MoQ approach, over WebTransport

14:47:53 [cpn_]

Harald: Ability to include non-standard or proprietary metadata. That might be an interesting feature to explore further

14:48:09 [cpn_]

... How would you do that, for the interoperable web?

14:48:43 [cpn_]

Peter: Youenn said it would be to use the same format over MoQ or other channel

14:48:56 [cpn_]

... Then some interest in having P2P QUIC come back

14:49:37 [cpn_]

... If you model it that way, if you add app-specific metadata, you would rather put it in the payload than metadata extensions, treating RTP transport as a generic transport than RTP specific

14:50:32 [cpn_]

... To Harald's question about experimental metadata, it would be easy to put what you want in the format. CBOR is very flexible, key/value, you can have custom keys for custom values

14:50:40 [cpn_]

s/metadata extensions/header extensions/

14:50:53 [cpn_]

... So we could do similar to header extensions with this

14:51:54 [cpn_]

Bernard: Things useful for evaluating experiments, e.g., capture time, you wouldn't send on the wire but useful for experiments

14:51:56 [cpn_]

Harald: We also use capture time for reasons

14:52:03 [cpn_]

Peter: Since P2P and QUIC were brought up, is anyone interested in a P2P QUIC API?

14:52:38 [cpn_]

Randall: P2P QUIC would be more useful for WebRTC like things

14:53:31 [cpn_]

... Why was it dropped?

14:53:31 [cpn_]

Peter: To focus on client/server initially, then come to it later

14:54:23 [cpn_]

... Anything else to discuss?

14:55:06 [cpn_]

Harald: I was hoping to learn new use cases. I'll monitor what MoQ is doing

14:55:48 [cpn_]

Henrik: I was hoping to see if the serialization use case can be done, and it sounds like it's possible to do today

14:55:52 [cpn_]

Peter: Yes, it is possible

14:56:11 [cpn_]

... And some are already doing it, so question is about need for interop

14:56:42 [cpn_]

Henrik: If the format and API is flexible enough it may be preferable to doing it yourself

14:57:01 [cpn_]

... But risk if there's anything you can't do in the proposed format, people would do their own thing anyway

14:57:51 [cpn_]

Xiaohan: There's lots of container formats, so depends on use cases. There's a profile Peter mentioned. If requires super low latency, what should be done?

14:58:20 [cpn_]

Peter: If it's standardised and pick up and use, it might be a default people would reach for rather than start from scratch

14:59:01 [cpn_]

Xiaohan: If we get to the point where the browser can do the muxing. Is the CMAF concern to do with the size of the generated stream, or compexity?

14:59:17 [cpn_]

Peter: Size, if you're sending each as a separate message

14:59:40 [cpn_]

... At last for depacketizing, you hand a CMAF chunk to MSE

15:02:01 [cpn_]

Chris: Interest in WebCodecs for MSE. We're in effect polyfilling the video element with buffering and a canvas

15:03:44 [cpn_]

Peter: You can hand over decoded VideoFrames to MSE

15:03:55 [cpn_]

... Integration with MSE a good thing to think about

15:04:18 [cpn_]

Youenn: Would be interesting to know what's missing in MSE for you to migrate to that, what features we could add

15:04:42 [cpn_]

Peter: Thanks for coming. If you have renewed interest or use case, let us know

15:04:45 [cpn_]

[adjourned]

15:04:47 [cpn_]

rrsagent, draft minutes

15:04:49 [RRSAgent]

I have made the request to generate https://www.w3.org/2023/09/13-webcodecs-serialization-minutes.html cpn_

15:16:21 [xueyuan]

xueyuan has joined #webcodecs-serialization

15:16:59 [xueyuan]

xueyuan has left #webcodecs-serialization

15:19:43 [Xiaohan]

Xiaohan has left #webcodecs-serialization