21:02:25 RRSAgent has joined #webrtc 21:02:30 logging to https://www.w3.org/2024/09/26-webrtc-irc 21:02:30 riju has joined #webrtc 21:02:30 Zakim has joined #webrtc 21:02:34 jesup7 has joined #webrtc 21:02:35 Topic: Introduction 21:02:41 RRSAgent, make logs public 21:02:45 scribe+ cpn 21:02:45 eric_carlson has joined #webrtc 21:02:54 padenot has joined #webrtc 21:02:55 RRSAgent, draft minutes 21:02:57 I have made the request to generate https://www.w3.org/2024/09/26-webrtc-minutes.html tidoust 21:03:04 present+ 21:03:24 Bernard: We'll talk about new things today, involving WebCodecs, what additional things people want, gaps, enable wider use 21:03:31 Orphis has joined #webrtc 21:03:33 ... We had breakouts yesterday 21:03:48 ... We'll cover those topics here 21:04:14 .... RtpTransport breakout discussed custom congestion control 21:04:30 ... WebCodecs and RtpTransport on the encoding side 21:04:54 ... Sync on the Web session. Interest to sync things like MIDI 21:05:18 ... Any comments? 21:05:21 (nothing) 21:05:29 Topic: Reference Control in WebCodecs 21:05:56 Erik: Reference Frame Control, to repeat the breakout session. And Corruption Detection 21:05:59 rahsin has joined #webrtc 21:06:21 ... Reference Frame control, the coal is to implement any referene structure we want. As simple API as possible 21:06:28 ... Make the encoder as dumb as possible 21:06:43 ... Use a s few bits as possible, don't get into how to do feedbacl etc 21:07:12 Eugene: We propose a new way to spec scalability modes for SVC 21:07:22 ... This allows any kind of pattern of dependencies betwen frames 21:07:34 ... Most encoders have a frame buffer abstraction 21:07:43 ... For saving frames for future use 21:07:57 ... getAllFrameBuffers() returns a sequence of all the FBs 21:08:05 ... No heavy underlying resources 21:08:23 ... Lets us extend video encode options, so say which frame goes to which framebuffer 21:08:35 ... Signals in whcih slot the frame should be saved 21:08:44 ... And dependencies between them 21:08:51 felipc0 has joined #webrtc 21:08:57 ... This is only available under a new "manual" scalability mode 21:09:22 ... Chromium implemnted behind a flag for libav1, hopeful for libvpx, HW accel on Windows under DirectX12 21:09:39 Erik: Concrete example of how to use it. Three temporal layers 21:09:49 ... Dependencies are always downwards 21:10:22 ... We create a VideoEncoder with "manual", check the encoder supports this mode, the check the list of reference buffers, then start encoding 21:10:40 ... There are 4 cases in the switch statement. 21:10:53 baboba has joined #webrtc 21:11:05 ... To make this work, we had to make simplifications and tradeoffs 21:11:05 +q 21:11:30 ... We limit it to only use CQP 21:11:31 youenn has joined #webrtc 21:11:38 Bernard: Can I do per-frame QP? 21:11:40 Erik: Yes 21:11:55 ... You have to do per-frame QP at the momemnt, CBR is a follow up 21:12:11 ... If the codec implements fewer reference buffers than the spec 21:12:24 ... Don't support spatial SVC or simulcast 21:12:40 ... We limit to updating only a single refernece buffer for a single frame today 21:13:05 ... H264 and H265 have more complex structure how they reference things. We model only with long-term references 21:13:16 ... We have some limitations aroudn frame dropping 21:13:31 ... To summarise the breakout, most people seem supportive 21:14:03 ... We want to take this a step further, support more codecs, user needs to understand the limitations, so need to query, need to discuss an isConfigSupported() or MC API 21:14:24 ... Fingerprinting surface, not really new, just a more structured way to look at data already there 21:14:47 q+ 21:14:47 ... Need examples. Can do L1T3 today, need examples for what you can't do today 21:14:49 +q 21:14:53 ack b 21:15:00 ack j 21:15:11 q+ baboba 21:15:14 chrisguttandin has joined #webrtc 21:15:19 hta has joined #webrtc 21:15:28 Jan-Ivar: There's a videoframebuffer id? 21:16:02 Eugene: Wanted to make it more explicit from a type point of view. The spec in future can say take buffers from a particular encoder instance. Can't take from strings 21:16:03 hta has joined #webrtc 21:16:16 ... It's a closed cycle 21:16:36 Jan-Ivar: Just a bikeshed, strange to havesomething called a buffer that isn't actually a buffer 21:16:46 Eugene: Open to renaming, e.g., add Ref at the end? 21:17:05 Erik: It represents a slot where you can put something, not the content 21:17:09 q? 21:17:20 ack baboba 21:17:28 Bernard: The reference has to be the same resolution? 21:17:48 Eugene: Don't have anything for spatial scalaibilty, each will have a separate buffer 21:18:02 ... We wanted to have this interface, introduce spatial scalability in future 21:18:21 Bernard: Can do simulcast, but in the same way as WebCodecs, creating multiplke encoders 21:18:35 ... WebRTC can have one encoder do multiple resolutions 21:18:37 q? 21:18:54 Topic: Corruption Likelihood Metric 21:19:22 Erik: Detecting codec corruptions, during transport etc that lead to visible artifacts, pixels on screen with bad values 21:19:47 ... Add a new measuremtn that tries to capture this, using as little bandwith and CPU as possible 21:20:09 ... One implementation in mind, use an RTP header extension as side channel 21:20:32 ... You randomly selct a number of samples in the image and put into an extension header 21:20:51 ... The receiver takes the same locations in the image, look a t the sampe values they see. If they differ, you have a corruption 21:21:06 ... Not just a single sample value. You'll have natural distortions from compression, want to filter those out 21:21:13 ... With QP, take an average around a location 21:21:42 ... Don't want stats value to be coupled to this partucular implementation 21:22:00 ... Allows us to do completely receive side, e.g, with an ML model 21:22:38 ... Proposal to put it in the inbound RTP RTC states. Could be put in VideoPlaybackQuality. Same thing could apply to any video transmission system 21:22:39 q+ 21:22:43 hta has joined #webrtc 21:22:51 ... Looking for feedback or input 21:22:52 q? 21:22:54 ack q 21:22:57 ack f 21:22:58 q+ 21:23:40 Cullen: Sympathetic to this use case, concerned about the details. Concern about RTP header extension, doesn't get same security processing as the video, could reveal a lot of info, e.g., guess what the video was 21:23:42 ... Privacy concern 21:24:02 +q 21:24:03 Erik: That's correct. We'll rely on RFC6904 to do encryption of header extension in the initial impl 21:24:19 ... Other wise you leak a small portion of the frame 21:24:31 hta has joined #webrtc 21:24:38 q? 21:25:03 Cullen: If you trry to sample a screenshare video, large regions of black or white. Metrics for video quality, considered other options than just a few sampling points? 21:25:28 Erik: Yes, screen content is difficult, doesn't generalise as well. 21:25:46 ... With 13 samples/frame it's good at finding errors 21:25:52 tantek has joined #webrtc 21:25:55 Cullen: How many samples are you thinking of using? 21:25:58 present+ 21:26:09 Erik: 13 actual samples we transmit 21:26:10 q? 21:26:14 ack fluffy 21:26:30 Harald: Thought about adding to VideoFrameMetadata instead of Stats? 21:27:02 Erik: That's the issue of exposing up to the application level. Won't do on all frames, maybe 1 frame / second. Could involve a GPU to CPU copy, so want to limit that 21:27:20 ... Open to ideas on how to surface to the user after calculation 21:27:23 Harald: Sounds like we need to experiment 21:27:25 q? 21:27:29 ack h 21:27:54 Bernard: The implemetations didn't work, header extensions sent in clear, so privacy issue if not fixed 21:28:07 +q 21:28:28 ... Want to think beyond WebRTC - MoQ, etc. Think about making meteadata, e.g., playout quality, to get it multiple ways 21:28:30 ack b 21:28:47 Erik: The 6904 is a stop-gap to start experimenting 21:28:58 hta2 has joined #webrtc 21:29:03 ... Not sure how to transmit in a general way the samples end to end 21:29:11 q? 21:29:19 Bernard: Previous discussion on segment masks, metadata attacked to the VideoFrame 21:29:31 Erik: Please coment in GitHub 21:29:58 Youenn: Hearing it's good to experiment. This can be shimmed, transform to get the data you want 21:30:12 ... Considered doing that first, and would that be good enough? 21:30:26 Erik: Considered doing encoded transform, the QP is missing 21:30:42 ... On a native level you can adapt the thresholds to get better signal to noise 21:31:13 ... We do local experiments in the office, but want to see from actual usage 21:31:14 q? 21:31:16 ack y 21:31:26 Topic: Audio Evolution 21:31:45 Paul: We'll talk about a few new things, some are codec-specific, some not 21:32:08 ... Two items to discuss. New advances with Opus codec - 1.5 released this year, has new booleans we should take advantage of 21:32:16 sprang has joined #webrtc 21:32:28 ... And we can improve spatial audio capabilities. For surround sound, music applications, etc 21:32:44 ... Link to blog post that talks about the new features 21:33:00 ... Some are ML techniques to improve quality under heavy packet loss 21:33:25 ... With LBRR+DRED you get good quality with 90% packet loss 21:33:40 +q 21:34:03 ... To use recent Opus quality improvements, there's a decoder complexity number. In Opus codec you can trade CPU power for higher quality 21:34:22 ... If you have quality (0-10), if >=5 you get Deep PLC, very high quality PLC 21:34:32 ... If 6 you get LACE, improves speech quality 21:34:42 ... NoLace is more expensive on CPU 21:35:08 ... Need a few megabytes. Not complex, geared to realtime usage 21:35:21 ... Only works with 20ms packets and wideband bandwidth 21:35:30 ... You'd have a complexity config 21:35:54 ... It's decode-side only, no compatibilty issue 21:36:34 +q 21:36:36 ... DRED - Deep Redundancy, you put latent information in every packet, can use the data in packet received to get the data you should have received 21:37:32 ... Increase jitter buffer size, then decoder reconstructs. Requires change of API on encoder side. Reconstruct PCM from what you didn't receive 21:38:06 ... New parameters when you encode the packet. Bitstream is versioned, so it will be ignored safely and not crash the decoder 21:38:10 PeterThatcher has joined #webrtc 21:38:48 Bernard: It's not trivially integrated in WebCodecs. What to do? 21:39:20 Paul: Add a second parameter to decode, with a dictionary, to enable this recovery scheme. It would be backwards compatible 21:39:21 ack b 21:39:48 Harald: Does this introduce additional delay? 21:39:49 solis has joined #webrtc 21:40:08 Paul: The second technique that can reconstruct heavy packet loss .. works like this 21:40:43 ... On detecting packet loss, you increase latency a lot, up to 1 seocnd. If it continues like that, you an still understand what's said 21:40:52 ... If network conditions approve, go back to normal 21:41:21 Erik: Is the 20ms limit just with current implementation? 21:41:41 Paul: They say "currently", not clear in the blog post why it is 21:41:52 Erik: Typically you want long frame lengths 21:42:16 Eugene: Slide 36 says 2 new APIs needed. What are they? 21:42:45 Paul: One is indicating there was packet loss, but need something for where packet loss happened 21:43:00 Eugene: Feature detection 21:43:17 Paul: IF the decoder doesn't understand the additional info, it's skipped 21:43:32 ... If you change version, it won't break. That's designed into the bitstream 21:43:42 ... Enable in the encoder, with DRED=true 21:43:54 Eugene: Don't need a configuration parameter 21:44:09 Paul: Affects encoding schems 21:44:13 hta has joined #webrtc 21:44:26 q? 21:44:30 Bernard: Config parameters in the scheme, some might affect WebCodecs 21:44:37 -q 21:44:52 Topic: Improve spatial audio capabilities 21:45:27 Paul: Opus can now do new things. Opus is mono and stereo, them they tell you how to map multiple channels 21:46:04 ... If ithe bytestream has channel family 2 and 3, it's ambisonics. Use orientation and trigonometry maps you can reconstruct audio from different directions 21:46:10 ... Straightfoward to decode 21:46:20 ... Trig maps can be done by the browser at this point 21:46:30 ... Just need to what mapping familiy it is 21:47:03 ... 255 is interesting, can have up to 256 channels, you know what to do. Have an index, do custom proessing in WASM 21:47:13 ... App layer and the file need to udnerstand each other 21:47:25 ... Web uses a certain channel ordering, in Web Audio API 21:47:49 ... Propose remapping channels, so you have a consident mapping regardless of codec and container 21:47:55 It's now specced in Web Audio 21:48:08 s/It's/... It's/ 21:48:22 Paul: Proposal is to map everything to SMPTE. AAC would need remapping, but others not touched 21:48:40 ... With ambisonics, decode and the app does the rest 21:48:55 ... For decode and output, don't think the app should be doing that 21:49:33 ... Proposal is almost do nothing, just remap so considetnt between the APIs 21:49:33 ... Any concerns? 21:49:45 Harald: How to map multiple channels in RTP? 21:50:09 ... Need to tell which channels are coupled and which are mono 21:50:09 +q 21:50:19 ... Some implementations have something, not standardised 21:50:26 Harald: There are hacks, yes 21:50:52 Paul: So long as consistent with web platform, get channels in the index you expect, so don't have to tune the app code for different codecs 21:51:03 fluffy7 has joined #webrtc 21:51:07 Jan-Ivar: What about on encoding? Also use SMPTE there? 21:51:23 Paul: On the web it's supposed to be that audio 21:51:48 Jan-Ivar: If you want to play it, not all platforms will spport SMPTE playback 21:51:53 Paul: You'd remap the output 21:52:05 ack baboba 21:52:31 Bernard: In response to Harald, nothing in the formats from AOMedia. How to get it into WebRTC? 21:52:49 Paul: Thsi is about getting into WebCodecs, then figure out SVP 21:52:59 ... There are draft RFCs about it 21:53:13 Bernard: THere's no transport for the container 21:53:20 q? 21:53:49 Weiwei: There are sevaral spatial audio codec standards. Does it work with them? 21:54:13 Paul: All will be mapped to this order, but that's been the case for some time. Need to ensure all specs agree 21:54:36 Weiwei: In China, there's a spatial audio standard, will it work for them? 21:54:57 Paul: If you have channel info at the decoder level, you can remap and expose in the order you expect 21:55:10 Weiwei: We should look into it 21:55:14 q? 21:55:34 Topic: IAMF 21:55:47 Paul: IAMF and object based audio, how to deal with it on the web? 21:56:01 ... Web Codecs doesn't concern itself with the container 21:56:24 ... Do we feel that using low level APIs for decoding the streams is enough, then render in script? 21:56:39 ... DSP involved isn't complicated, just mixing, volume, panning 21:56:44 +q 21:57:27 Eugene: Agree this is an advanced feature, so leave to the app 21:57:46 Bernard: More complicated than that. Things like the Opus 1.5 extensions 21:58:29 ... IAMF can work with lots of codecs, but they want to do additional stuff 21:58:58 Paul: In that case, want to have WebCodecs work with it. Don't know if we want WebRTC WG do the work, more complications 21:59:04 ack b 21:59:21 Topic: Encoded Source 21:59:44 Guido: In WebRTC WG we want to support the ultra low latency broadcast with fanout use case 22:00:02 ... UA must be able to forward media from a peer to another peer 22:00:20 ... Timing and bandwidth estimates for congestion control 22:00:48 youenn has joined #webrtc 22:00:51 ... Specifically, we want to support this scenario, where you have a server that provides the media. Large number of consumers 22:01:15 ... Communication with server is expensive 22:01:39 ... Assume communication between nodes is cheaper than communication to server 22:01:46 ... Nodes can appear or disappear at any time 22:02:35 ... Example, Two peer connections receiving data from any peer in the network. 22:02:50 ... Use encoded transform to receive frames 22:03:11 ... Depending on network conditions, you might want to drop frames 22:03:14 xhwang has joined #webrtc 22:03:26 ... When app decides what to forward, sends frame to multiple output peer connections 22:03:41 ... Idea is you can fail over, be robust without requiring timeouts 22:03:51 ... So can provide a glitch-free forward 22:04:10 ... We made a proposal, patterned on RTCRtpEncodedTransform 22:04:47 ... This is similar to single-sided encoded transform 22:05:01 ... Got WG and dveeloper feedback tha we've incorporated 22:05:40 ... Allows more freedom than encoded transform. You can write any frames, so it's easier to make mistakes. Wuold be good to provide better error signals 22:06:06 ... It's less connected to internal control loops in WebRTC 22:06:21 ... In addition to raw error ahdling we need bandwidth estimates, etc 22:06:46 ... Basic example. We have a worker, a number of peer connections 22:07:01 ... Each has a sender. For each sender we call createEncodedSource() 22:07:16 ... This method is similar to replaceTrack() 22:07:19 q+ 22:07:34 ... On receiver connection, we use RTCRtpScriptTransform 22:08:15 ... On worker side, we receive everything, we use encoded sources. In the example, source has a writeable stream, a readable and a writeable 22:08:21 +q 22:08:41 ... For the receivers, can apply a transform 22:08:54 ... Write the frame to all the source writers 22:09:21 ... You might need to adjust the metadata 22:10:26 ... Errors and signals that developers say would be useful include keyframe requests, bandwidth estimates, congestion control, error handling for incorrect frames 22:10:42 ... e.g, timestamps going backwards 22:11:21 ... Other signals are a counter for frames dropped after being written that the sender decided to drop 22:11:36 handellm has joined #webrtc 22:11:42 ... Expected queue time once written 22:12:13 ... To handle keyframe requests, there's an eent 22:12:34 .. .writeable stream, and event handler for the keyframe request 22:13:00 ... For bandwidth we're proposing to use a previous proposal or congestion control from Harald 22:13:22 ... Recommended bitrate 22:13:37 ... Outgoing bitrate is already exposed in stats, convenient to have it here 22:13:52 ... Have an event that fires when there's a change in bandwidth info. 22:14:28 ... [shows BandwidthInfo API] 22:14:59 ... Use with dropped frames, after written, and expected send queue time 22:15:36 ... if allocated bitrate exceeds a threshold, add extra redundancy data for the frame 22:15:53 ... [Shows API shape] 22:17:24 ... Pros and cons. Similar pattern to encoded transform, simple to use and easy to understand 22:17:31 ... Good match for frame-centric operations 22:17:47 ... Allows zero timeout failover from redundant paths 22:18:04 ... Easy to adjust or drop frames due to bandwidth issues 22:18:17 ... It requires waiting for a full frame 22:18:38 ... In future could be ReceiverEncodedSource 22:18:41 q? 22:18:51 ... Have fan-in for all the receivers 22:19:17 Jan-Ivar: In general I agree this is a good API to solve the forwarding data use case 22:19:39 ... Seems to be a bit more than a source. Somehting you can assign to a sender in place of a track 22:20:11 ... Once you associate a sender with a source, that can't be broken again? 22:20:28 Guido: Yes. A handle-like object 22:21:12 ... I like it better with a method, can play track with a video element. But with this object there's nothing you can do with it 22:21:29 ... There isn't a lot of advantage to having this object, e.g., to send to another sender 22:21:55 q+ 22:22:00 ... We can iterate on the methods and finer details 22:22:11 ... I prefer methods, as they create the association immediately 22:22:32 ack jib 22:22:32 Jan-Ivar: That ends up being a permanent coupling 22:22:55 Guido Can create and replace an existing one 22:23:20 Jan-Ivar: The permanent coupling ... 22:23:30 Guido: It's just an immediate coupling 22:23:34 ... You can decouple it 22:24:09 ... Can do the same approach as encoded transform if we think that's better 22:24:11 q? 22:24:35 Youenn: Overall it's in the right direction. Similar feedback on the API shape, but we can converge 22:24:52 ... Not a small API. Good to have shared concepts 22:25:11 ... Encoded transform was very strict. Here we're opening the box. Have to be precise about error cases 22:25:42 ... We're opening the box in the middle. Need to be precise how it works with encoded transform 22:26:07 ... Improve the API shape and really describe the model and how it works. Implications for stats, encoded transform, etc. 22:26:25 ... I have other feedback, will put on GitHub 22:26:38 ... Let's go for it, but be careful about describing it precisely 22:26:39 q? 22:26:42 ack y 22:26:44 ack youenn 22:27:10 Guido: So we have agreement on the direction 22:27:35 ack h 22:27:35 Harald: encoded transport has bandwidth allocation. Should try to harmonise the other part 22:27:46 Topic: Timing Model 22:28:19 scribe+ hta 22:28:26 [slide 60] 22:30:48 i|[slide|Slideset: https://docs.google.com/presentation/d/1d5KdKhwd8PGkGJweJl0qDX9yy9s2GqDRH4EpQ0ccr5g/edit 22:31:05 q+ 22:31:14 q+ 22:31:40 ack hta 22:31:48 Harald: Recently developed we added stats countters to MediaStreamTrack, and should be reflected. Either that shouldn't exist or be consistent 22:32:12 youennf: we should be able to compute video playback counters based on the track 22:32:13 Youenn: We should define one from each other. Take the MST definition and define VideoPLaybackQUality in terms of that 22:32:14 baboba6 has joined #webrtc 22:32:37 handell: there's a different proposal that has total video frames in it. 22:32:39 Marcus: I have another prosal that would increment total video frames. So lean to proposal 2 22:32:44 ack h 22:32:49 +q 22:33:24 Harald; So sounds like we should spec the behaviour. We're trying to unify tha statess across sources 22:33:46 bernard: we should try to specify it. 22:33:47 Bernard: Suggest we do it more generally via tracks, which is more work 22:33:51 q+ 22:34:08 Chris: Where to spec? 22:34:14 cpn: Agreement to try to specify behavior - within Video Playback Quality? 22:34:35 youennf: each source should describe how they are creating video frame objects. 22:34:35 Youenn: You have different sources in differents specs, describe how they create video frame objects, and have defintions of countrs as well 22:35:20 dom: need a burndown list of fixing all the specs to supply that info. 22:35:59 jib: agree we need to define them for each mst source 22:36:26 Subtopic: -> https://github.com/w3c/mediacapture-transform/issues/87 What is the timestamp value of the VideoFrame/AudioData from a remote track? 22:36:43 scribenick: cpn? 22:36:59 scribenick: cpn 22:37:45 q+ 22:37:52 [slide 61] 22:37:53 Bernard: Timestamp is a capture timestamp, not a presentation timestamp. Shuold we change the definition in WebCodecs? Can we descirbe more clearly this and rVFC timestamp? 22:37:58 q- baboba 22:38:00 q- jib 22:38:10 Eugene: For video file, there's only presentation timetamp 22:38:42 q+ 22:38:47 ... For camera, it's capture timestamp by definitition. Needs to be source-specific 22:39:03 Bernard: Where would you put the definitions, media-capture-main? 22:39:07 Youenn: Yes 22:39:27 ack handellm 22:39:47 Marcus: In WebCodecs spec, there's no definition other than presentation timestamp. In chromium, starts at 0 end increments by frame duration 22:40:23 ... It's unspecified what it contains. We have a heuristic in Chromium that puts the capture timetamp 22:40:36 ... It's sourced up to rVFC 22:40:59 ... Shouldn't really be like that, it should be a presentation timetamp 22:41:09 q? 22:41:09 ack jesup 22:41:30 Randell: The use of the terms presentation and capture timestamp is a bit arbitrary 22:42:09 ... The fact it comes from a file and is a presentation timestamp, and from a capture is capture timstamp, isn't relevant. Just have a timetamp 22:42:17 Bernard: Want to move to next issue 22:42:54 [slide 66] 22:43:07 Subtopic: -> https://github.com/w3c/webcodecs/pull/813 Add captureTime, receiveTime and rtpTimestamp to VideoFrameMetadata 22:43:23 Marcus: Web apps that depend on the timestamp sequence, we want to expose capture time into VideoFrameMetadata 22:43:39 ... Why? Capture time is async, and enables end to end video delay measurements 22:43:51 ... In WebRTC, we prefer the ? timetamp 22:44:10 ... The capture time in this context is an absolute measure 22:44:23 ... Presentation timestamps not clear when were measured 22:44:35 .. Capture time can get the time from before the pipeline 22:45:00 ... There are higher quality timetamps from the Capture APIs, we want to expose them 22:45:14 ... PR 183 adds those,we refer to the rVFC text 22:45:27 ... People didn't like that. Now we have 5 PRs 22:45:55 ... We're trying to define this concept in media stream tracks. I'd place those in mediacapture-extensions 22:46:40 ... webrtc-extensions, and mediacapture-transform, then repurpose #813 to add fields to VideoFrameMetadata registry 22:46:59 ... That's the plan 22:47:04 q? 22:47:10 Eugene: The main problem was video/audio sync 22:47:20 fippo has joined #webrtc 22:47:26 +q 22:47:43 ... Audio frames captured from the mic had one timestamp, and video frames from the camera had different, and was confusing encoding configurations 22:48:18 ... Change made for videoframe timetamp to be capture timestamps is important change. Currently just Chomium behavior, want it to be specced behaviour 22:48:35 ... So you can do A/V sync, otherwise any kind of sync is impossible 22:49:24 Paul: Reclocking, skew, compensate for latency, so everything matches 22:49:43 Eugene:: Why not have the same clock in both places? 22:49:55 Paul: You take a latency hit as it involves resampling 22:50:19 Marcus: We don't have ?? 22:50:40 ?? = AudioTrackGenerator 22:50:40 Paul: There is reclocking happening, otherwise it falls apart 22:51:00 Eugene: Need example code to show how to do it correctly, for web developers 22:51:07 Paul: Sure 22:51:37 [slide 71] 22:51:45 Subtopic: -> https://github.com/w3c/mediacapture-transform/issues/96 What is the impact of timestamp for video frames enqueued in VideoTrackGenerator? 22:51:55 Youenn: VideoTrackGenerator timestamp model isn't defined 22:52:17 ... Not buffering anything. Each track source will define 22:52:29 ... Timetamp not used in any spec on the sync side 22:52:50 ... We define timestamp per track source 22:53:22 ... Video track sink, there's a diffenrce between webkit and Chromium on implementation 22:53:37 ... If spec says nothing, means we don't care about the timestamp 22:53:50 [slide 72] 22:54:08 Bernard: Are those statements about what happens true or not? 22:54:48 Harald: Video element has a jitter buffer 22:55:08 Bernard: So the staments seem accurate. 22:55:19 [slide 73] 22:55:44 Subtopic: -> https://github.com/w3c/mediacapture-transform/issues/80 Expectations/Requirements for VideoFrame and AudioData timestamps 22:55:51 Bernard: What if you append multiple VideoFrames with the same timetamp? Does VTG just pass it on, look for dupes? 22:56:05 Jan-Ivar: Yes, garbage-in, garbage-out 22:56:19 Youenn: It's the sink that cares about the timestamp 22:56:27 Bernard: Something to make clear in the PR 22:56:54 Jan-Ivar: Need to consider someone sending data over the channel 22:57:00 [slide 74] 22:57:24 Subtopic: -> https://github.com/w3c/mediacapture-transform/issues/86 Playback and sync of tracks created by VideoTrackGenerator 22:57:33 Bernard: HTMLVideoElement. no normative reuirement, might happen or ot 22:57:53 ... Describes issues with losing sync, need to delay one to get sync back, etc 22:58:28 ... Want to be more specific about this. A jitter buffer potentially in HTMLMediaElement. How does it work and what does it take into account? 22:58:38 i|[slide 60]|Subtopic: -> https://github.com/w3c/media-playback-quality/issues/19 Clarification needed for HTMLVideoElements that are playing a MediaStream 22:58:43 RRSAgent, draft minutes 22:58:45 I have made the request to generate https://www.w3.org/2024/09/26-webrtc-minutes.html dom 22:59:23 youenn has joined #webrtc 22:59:24 ... It's suggested it's more difficult for remote playout. In RTP, it's used to caluculat the sender/receiver offset 22:59:38 ... What's going on inside the black box? 22:59:57 [slide 75] 23:00:13 Youenn: Is it observable? With gUM, the tracks are synchronised. In other cases, we have separate tracks 23:01:18 Jan-Ivar: Depends where the source comes from. MediaSrram is a generic implementation for different sources 23:01:26 Jan-Ivar: Very old language on synchronization might be outdated. 23:01:40 Bernard: Thinking about remote audio and video. Need receive time and capture time from same source 23:02:11 Harald: Looked at this code recently. For a single AudioTrack and VideoTrack from same peer connecton with same clock source, WebRTC tries to synchronise 23:02:23 Marcus: MediaRecorder sorts samples 23:02:45 Paul: Similar in FF if you have multiple microphones, we consider it a high level API so it should work 23:03:04 Youenn: Spec should clarify there are some cases you should do it, other cases it's impossible 23:03:23 Bernard: If I'm writing a WebCodecs+WebTransport, is there someting i can do to make it work? 23:03:29 Paul: Implement jitter buffers 23:03:46 Marcus: If you have capture times from all streams, you can sort in JS 23:04:00 Youenn: make sure from same device 23:04:19 Jan-Ivar: If you have VTG, would it affect playback? 23:04:54 Bernard: You have capture time from sender 23:05:53 mjwilson has left #webrtc 23:06:02 RRSAgent, draft minutes 23:06:03 I have made the request to generate https://www.w3.org/2024/09/26-webrtc-minutes.html dom 23:06:09 Chris: Next steps, schedule more time for this discussion? 23:06:09 Bernard: Good idea, yes 23:06:26 Meeting: Joint Media/WebRTC WG meeting at TPAC 23:06:28 RRSAgent, draft minutes 23:06:29 I have made the request to generate https://www.w3.org/2024/09/26-webrtc-minutes.html dom 23:06:54 hta has joined #webrtc 23:29:12 markafoltz has joined #webrtc