IRC log of webcodecs-serialization on 2023-09-13

Timestamps are in UTC.

06:58:02 [RRSAgent]
RRSAgent has joined #webcodecs-serialization
06:58:06 [RRSAgent]
logging to
06:58:06 [Ian]
RRSAgent, do not leave
06:58:06 [Ian]
RRSAgent, make logs public
06:58:07 [Ian]
Meeting: WebCodecs Serialization Format
06:58:07 [Ian]
Chair: Peter Thatcher
06:58:08 [Ian]
06:58:10 [Ian]
clear agenda
06:58:12 [Ian]
agenda+ Pick a scribe
06:58:14 [Ian]
agenda+ Reminders: code of conduct, health policies, recorded session policy
06:58:16 [Ian]
agenda+ Goal of this session
06:58:20 [Ian]
agenda+ Discussion
06:58:22 [Ian]
agenda+ Next steps / where discussion continues
06:58:31 [Ian]
Ian has left #webcodecs-serialization
07:15:38 [Ian]
Ian has joined #webcodecs-serialization
07:15:42 [Ian]
Ian has left #webcodecs-serialization
13:16:47 [RRSAgent]
RRSAgent has joined #webcodecs-serialization
13:16:47 [RRSAgent]
logging to
14:00:40 [cpn]
cpn has joined #webcodecs-serialization
14:01:17 [cpn]
Meeting: WebCodecs Serialization Format
14:01:23 [cpn]
scribe+ cpn
14:03:10 [cpn]
Present+ Peter_Thatcher, Eugene_Zemtsov, Harald_Alvestrand, Xiaohan_Wang, Eric_Carlson, Youenn_Fablet, Paul_Adenot
14:03:14 [cpn]
rrsagent, draft minutes
14:03:15 [RRSAgent]
I have made the request to generate cpn
14:03:19 [cpn]
rrsagent, make log public
14:03:58 [cpn]
present+ Rijubrata_Bhaumik
14:04:23 [cpn]
Peter: In the IETF MoQ group, there's interest in sending media over QUIC
14:04:38 [cpn]
... How do you serialize the media into bytes so you can send over QUIC stream or datagrams
14:04:56 [cpn]
... Involves attaching metadata. Looked at different codecs, which have different modes
14:04:58 [Xiaohan]
Xiaohan has joined #webcodecs-serialization
14:05:04 [cpn]
... WebCodecs registry is a place we can point to
14:05:22 [cpn]
... People will want to make MoQ endpoints using web clients, using WebCodecs and WebTransport
14:05:37 [Riju]
Riju has joined #webcodecs-serialization
14:05:43 [cpn]
... Want to take the output from WebCodecs, serialize, then send to a decoder on the other side, which may be WebCodecs
14:06:10 [cpn]
... I did something similar in Second Screen WG, on Open Screen Protocol
14:06:23 [cpn]
... At that time we chose CBOR, a kind of binary JSON
14:06:59 [cpn]
... What is the interest from others? What would you want to do with the serialization, store it, send over QUIC?
14:07:30 [cpn]
Harald: IIUC this is about making sure we have both encoded data and metadata for a frame on the other side of the connection
14:08:06 [cpn]
... So it's like the RTP encapsulation format
14:08:13 [cpn]
... We don't have a 1:1 mapping of it
14:08:13 [cpn]
... Which parts of this should be different from RTP formats, and why?
14:08:23 [cpn]
Bernard: This would include things normally in RTP extensions as well
14:08:59 [cpn]
... Applies to the RTP packet, timestamp, things related to SVC in IETF are always packeted
14:09:16 [cpn]
... Would become patent encumbered in IETF
14:09:36 [cpn]
Peter: Different questions: where should this happen? How different should this be to work already done?
14:09:45 [cpn]
... IETF looking at how to do RTP over QUIC
14:10:15 [cpn]
??: I thought the output of the encoder was already serialized and you could plug it into the decoder. What additional metadata is needed?
14:11:46 [cpn]
Peter: For example, the output of the encoder includes EncodedVideoChunk, which includes the bytes, timestamp, keyframe flag, metadata such as how SVC is structured
14:11:46 [cpn]
... Information about size, color space, and which codec it is
14:12:09 [cpn]
... In header extensions, things like frame dependencies. In RTP payload header, there's metadata
14:12:24 [cpn]
... There's metadata tied to the unencoded media, e.g., audio level
14:12:41 [cpn]
... So if you look at RTP metadata, it's the same kind of metadata
14:12:59 [cpn]
??: Is the metadata already exposed to JS, just needs packaging, or are there internal pieces needed?
14:13:19 [cpn]
Peter: Want it done in a standardised way, but they don't want to use RTP
14:14:05 [cpn]
... There could be internal metadata, but a good start is to take everything in the API and serialize that
14:14:24 [cpn]
Youenn: Are you thinking of extensions?
14:14:59 [cpn]
??: It's whether the problem can be solved in JavaScript, it sounds like I could convert to JSON and parse on the other side. Would that work?
14:15:24 [cpn]
Peter: You could, yes. Make your own non-standard format. The question is what interest is there in having a standard format?
14:15:31 [cpn]
... There's interest in the MoQ WG for that
14:15:46 [cpn]
... Is there interest here?
14:16:05 [cpn]
Bernard: The standards interest comes from the caching. MoQ uses a cache, relays need metadata to do forwarding
14:16:14 [padenot_]
padenot_ has joined #webcodecs-serialization
14:16:20 [xueyuan]
14:16:34 [cpn]
Peter: Relays are designed to be agnostic to the media format, and some metadata is opaque to the relay
14:17:02 [cpn]
Bernard: Not sure that's true, there's preferential forwarding, may be ready by caches
14:17:19 [cpn]
Peter: That would have to be defined by MoQ WG, as it's MoQ specific
14:17:37 [cpn]
... The serialization can be done in a separate doc, as it's opaque to relays, that's my understanding
14:18:15 [cpn]
... Are people interested in being able to serialize outside of MoQ, for other use cases
14:18:27 [cpn]
Bernard: There have been issues raised about support for DRM
14:18:49 [cpn]
Xiaohan: How is that related to this question for a need to serialize WebCodecs?
14:19:25 [cpn]
Harald: We have a proposal to define a generic packetizing format for RTP so you can send any content you want over the wire
14:20:27 [cpn_]
cpn_ has joined #webcodecs-serialization
14:20:32 [cpn_]
scribenick: cpn_
14:20:50 [cpn_]
Peter: Content of S frames are opaque
14:21:12 [cpn_]
Harald: This came up in a side discussion about SDP munging for WebRTC encoded transform
14:21:50 [cpn_]
Peter: One way to think about it is, if we were to start from scratch and solve the same thing, serialize the payload, and accumulate all the header extensions, what would we do?
14:22:33 [cpn_]
... Similar question in the Second Screen group, we used CBOR and came up with types. It was straightforward, easier to use a CBOR parser
14:23:36 [cpn_]
... Let's come to Bernard's question about where this should be worked on
14:23:36 [Riju]
Riju has joined #webcodecs-serialization
14:23:38 [cpn_]
... There's IPR issues for some of the metadata
14:24:01 [cpn_]
Bernard: The concern would be if you take the WebCodecs spec you might start encumbering WebCodecs, that would be awful
14:24:24 [cpn_]
Peter: What about trying to define the serialization format in W3C, would there be enough people interested in working on it to do it here?
14:25:09 [cpn_]
Bernard: Media WG already has an ISO BMFF format for MSE
14:25:14 [cpn_]
Harald: A note of caution on IPR. IETF runs on the principle that every participant should disclose their IPR, but doesn't force anyone to give their rights to anything
14:25:24 [cpn_]
... At W3C all participants make a commitment when joining the WG
14:25:37 [cpn_]
... If you don't join the WG, no promises made
14:26:49 [cpn_]
... So unless all contributors are in the W3C WG and agree to remain members when the work item is adopted, that's what's required for W3C
14:27:21 [cpn_]
... Getting all people who're asserting IPR to join might be the issue
14:27:26 [cpn_]
Youenn: I'd tend to pick the venue where more people are interested
14:28:04 [cpn_]
Peter: Gauging level of interest is the purpose of this
14:28:25 [cpn_]
Xiaohan: What's the use case you're imagining?
14:29:01 [cpn_]
Peter: It could be used for any purpose. For the MoQ WG, it's for real time and streaming use cases, e.g., live broadcast video conference
14:29:49 [cpn_]
Xiaohan: There are already solutions not using WebCodecs out there. If it's for streaming then CMAF can be the format, but too heavy for real time
14:29:58 [cpn_]
... Why not adopt an existing format?
14:30:21 [cpn_]
Peter: MoQ WG separated the two, so the format can be CMAF based
14:31:03 [cpn_]
... Others who want to do things web oriented, you don't want to containerise to CMAF, more straightforward to do what JSON does, but more efficient, hence CBOR
14:31:40 [cpn_]
... For realtime or very low latency streaming it would be a better DX to do something new, which is why there's interest in MoQ WG for a low overhead container
14:32:02 [cpn_]
Xiaohan: Format of MediaStream in WebRTC?
14:32:28 [cpn_]
Peter: That uses RTP, as Harald described. It could be done over QUIC, there's work in IETF, but like CMAF it carries baggage
14:33:52 [cpn_]
Chris: Wouldn't MoQ WG need to define a format anyway, absent any work here?
14:34:13 [cpn_]
Peter: Yes, they're working on the CMAF based format, but the question is about the low latency format
14:34:55 [cpn_]
Bernard: The Media WG meeting heard about the IETF 117 WebCodecs presentation
14:35:47 [cpn_]
14:36:42 [cpn_]
Peter: WebCodecs already has the metadata defined that we'd want to serialize
14:36:57 [cpn_]
... What we define might be usable in different situations
14:37:52 [cpn_]
... Continue in IETF or bring to W3C
14:38:44 [cpn_]
Chris: Interest expressed at IETF to work on it?
14:38:54 [cpn_]
Peter: Yes, three of us at least are interested, and others
14:42:20 [cpn_]
Chris: We are developing a browser based video editor, have our own format, so we have the general need
14:42:29 [cpn_]
Peter: Need for interop around that?
14:42:38 [cpn_]
Chris: I'd have to follow up to get precise requirements
14:43:02 [cpn_]
Bernard: A frequent request we get on WebCodecs is a standard API for container formats
14:43:50 [cpn_]
Peter: I have a similar situation, where I have my own non-standard serialization, there might be potential for a standard solution
14:44:23 [cpn_]
Peter: So if not this, what are people interested in?
14:44:46 [cpn_]
Henrik: Using WebCodecs with the RTP transport in WebRTC, don't know if packetization needs to be standardised or not
14:45:58 [cpn_]
Peter: So if you were to use WebCodecs with RTP transport, how would you use it?
14:46:35 [cpn_]
Henrik: I see value in coupling app-specific metadata to video frames. Haven't done enough homework to know more
14:47:13 [cpn_]
Youenn: MoQ is trying to do real time for QUIC, server to client. Why not using the MoQ approach, over WebTransport
14:47:53 [cpn_]
Harald: Ability to include non-standard or proprietary metadata. That might be an interesting feature to explore further
14:48:09 [cpn_]
... How would you do that, for the interoperable web?
14:48:43 [cpn_]
Peter: Youenn said it would be to use the same format over MoQ or other channel
14:48:56 [cpn_]
... Then some interest in having P2P QUIC come back
14:49:37 [cpn_]
... If you model it that way, if you add app-specific metadata, you would rather put it in the payload than metadata extensions, treating RTP transport as a generic transport than RTP specific
14:50:32 [cpn_]
... To Harald's question about experimental metadata, it would be easy to put what you want in the format. CBOR is very flexible, key/value, you can have custom keys for custom values
14:50:40 [cpn_]
s/metadata extensions/header extensions/
14:50:53 [cpn_]
... So we could do similar to header extensions with this
14:51:54 [cpn_]
Bernard: Things useful for evaluating experiments, e.g., capture time, you wouldn't send on the wire but useful for experiments
14:51:56 [cpn_]
Harald: We also use capture time for reasons
14:52:03 [cpn_]
Peter: Since P2P and QUIC were brought up, is anyone interested in a P2P QUIC API?
14:52:38 [cpn_]
Randall: P2P QUIC would be more useful for WebRTC like things
14:53:31 [cpn_]
... Why was it dropped?
14:53:31 [cpn_]
Peter: To focus on client/server initially, then come to it later
14:54:23 [cpn_]
... Anything else to discuss?
14:55:06 [cpn_]
Harald: I was hoping to learn new use cases. I'll monitor what MoQ is doing
14:55:48 [cpn_]
Henrik: I was hoping to see if the serialization use case can be done, and it sounds like it's possible to do today
14:55:52 [cpn_]
Peter: Yes, it is possible
14:56:11 [cpn_]
... And some are already doing it, so question is about need for interop
14:56:42 [cpn_]
Henrik: If the format and API is flexible enough it may be preferable to doing it yourself
14:57:01 [cpn_]
... But risk if there's anything you can't do in the proposed format, people would do their own thing anyway
14:57:51 [cpn_]
Xiaohan: There's lots of container formats, so depends on use cases. There's a profile Peter mentioned. If requires super low latency, what should be done?
14:58:20 [cpn_]
Peter: If it's standardised and pick up and use, it might be a default people would reach for rather than start from scratch
14:59:01 [cpn_]
Xiaohan: If we get to the point where the browser can do the muxing. Is the CMAF concern to do with the size of the generated stream, or compexity?
14:59:17 [cpn_]
Peter: Size, if you're sending each as a separate message
14:59:40 [cpn_]
... At last for depacketizing, you hand a CMAF chunk to MSE
15:02:01 [cpn_]
Chris: Interest in WebCodecs for MSE. We're in effect polyfilling the video element with buffering and a canvas
15:03:44 [cpn_]
Peter: You can hand over decoded VideoFrames to MSE
15:03:55 [cpn_]
... Integration with MSE a good thing to think about
15:04:18 [cpn_]
Youenn: Would be interesting to know what's missing in MSE for you to migrate to that, what features we could add
15:04:42 [cpn_]
Peter: Thanks for coming. If you have renewed interest or use case, let us know
15:04:45 [cpn_]
15:04:47 [cpn_]
rrsagent, draft minutes
15:04:49 [RRSAgent]
I have made the request to generate cpn_
15:16:21 [xueyuan]
xueyuan has joined #webcodecs-serialization
15:16:59 [xueyuan]
xueyuan has left #webcodecs-serialization
15:19:43 [Xiaohan]
Xiaohan has left #webcodecs-serialization