14:55:19 RRSAgent has joined #sync-on-web 14:55:23 logging to https://www.w3.org/2024/09/25-sync-on-web-irc 14:55:23 RRSAgent, do not leave 14:55:24 RRSAgent, make logs public 14:55:25 Meeting: Sync on Web, now and next of realtime media services on web 14:55:25 Chair: KensakuKOMATSU 14:55:25 Agenda: https://github.com/w3c/tpac2024-breakouts/issues/54 14:55:25 Zakim has joined #sync-on-web 14:55:26 Zakim, clear agenda 14:55:26 agenda cleared 14:55:26 Zakim, agenda+ Pick a scribe 14:55:28 agendum 1 added 14:55:28 Zakim, agenda+ Reminders: code of conduct, health policies, recorded session policy 14:55:28 agendum 2 added 14:55:28 Zakim, agenda+ Goal of this session 14:55:29 agendum 3 added 14:55:29 Zakim, agenda+ Discussion 14:55:29 agendum 4 added 14:55:30 Zakim, agenda+ Next steps / where discussion continues 14:55:31 agendum 5 added 14:55:31 tpac-breakout-bot has left #sync-on-web 15:07:56 tidoust has joined #sync-on-web 20:07:34 ktoumura has joined #sync-on-web 20:07:56 ktoumura has left #sync-on-web 20:14:55 ohmata has joined #sync-on-web 20:16:58 handellm has joined #sync-on-web 20:17:57 ktoumura has joined #sync-on-web 20:18:25 hirata has joined #sync-on-web 20:19:08 kaz has joined #sync-on-web 20:19:22 scribenick: kaz 20:19:26 kzms2 has joined #sync-on-web 20:19:52 present+ Kaz_Ashimura 20:19:58 kota has joined #sync-on-web 20:20:10 chrisguttandin has joined #sync-on-web 20:20:16 scribenick: kota 20:20:18 sprang has joined #sync-on-web 20:20:20 present+ Kunihiko_Toumura 20:20:23 present+ Hisayuki_Ohmata 20:20:24 rahsin has joined #sync-on-web 20:20:53 cpn has joined #sync-on-web 20:21:01 present+ Chris_Needham 20:21:04 Time alignment for media synchronization will be discussed in this session 20:21:05 mjwilson has joined #sync-on-web 20:21:28 Sun has joined #sync-on-web 20:21:47 Kodajima4 has joined #sync-on-web 20:22:10 present+ Francois_Daoust 20:22:51 Kodajima4 has left #sync-on-web 20:23:01 Kodajima has joined #sync-on-web 20:23:21 Bernd has joined #sync-on-web 20:23:41 present+ 20:23:58 scribe+ cpn 20:24:20 Komatsu: Media over QUIC, no head of line blocking 20:24:31 .. In video cases, each frame is fransferred over an independent QUIC stream 20:24:49 ... That's the main difference compared to HLS or DASH 20:24:49 Media over QUIC is a relay protocol over QUIC or HTTP/3, often featured in the use of live streaming although the usecases are not limited to it. 20:25:09 ... CMAF is fragmented MP4. 50 seconds to 2 seconds latency, depending how you use it 20:25:21 song has joined #sync-on-web 20:25:32 ... WIth MoQ, per-frame transfer is used, so that each QUIC stream contains under 34 milliseconds duration of data 20:25:51 ... For realising low latency live stream services, this duration is important 20:26:09 ... You can get very low latency services over MoQ 20:26:37 ... HLS and DASH have flexibility, and similarly with MoQ, live and ondemand services 20:26:59 ... Synchronising A/V data and arbitrary data is interesting 20:27:18 ... Here's a demo to show low latency and sync 20:27:22 MoQT is flexible enough that developers could handle synchronization between multiple types of data such as audio and video 20:27:39 ... The sender sends the video and audio data and auxiliary data. Then it's transferred to a relay server 20:27:51 ... We have this in the cloud 20:28:01 ... We use moxygen, developed by Meta 20:28:34 ... [Shows demo with sender and receiver] 20:28:48 ... There's very low latency 20:29:02 .. Glass to glass delay is under about 100ms 20:29:21 ... Now I'll demo data synchronisation 20:29:49 ... [Demo shows real time face detection] 20:30:29 ... We can also send MIDI data 20:30:56 ... With this data we can provide live services 20:31:00 nigel has joined #sync-on-web 20:31:06 MoQT Synchronization Demo: face landmarks/avatar data and video synchronization is performed 20:31:26 ... [Demo shows a virtual avatar overlaid in the video image] 20:31:49 ... Just the data is transferred, and the avatar is rendered on the receiver side 20:31:59 present+ Nigel_Megitt 20:32:17 ... I think this is a fantastic feature of MoQ 20:32:19 Kodajima has joined #sync-on-web 20:32:41 ... Now I'll explain about the synchronisation. The diagram shows the sender side 20:32:46 sending point cloud data only enable developers to render the 3d avatar on the subscriber side with its own preferences 20:32:51 Niko has joined #sync-on-web 20:33:04 ... Video image is transferred to WASM 20:33:25 ... MIDI data will be transmitted to Web MIDI 20:33:34 Each AV and data are multiplexed into single MoQT session using multi tracks on the sender side 20:33:38 ... In the MoQ context, we can get capture timestamps 20:33:56 ... MOQT is the MoQ Transport protocol 20:34:03 ... We can send each data in a track: audio, video, data 20:34:22 ... Send over MOQT to the relay server 20:34:37 ... On the receiver side, the browser receives the MOQT 20:35:01 ... Get raw image data from each frame from the capture timestamp. and the MIDI data, and syncrhonise rendering 20:35:29 ... MIDI can be used with synthesizers etc 20:35:58 ... In live entertainment cases, you can show a live show on the screen, and with MIDI data, the piano sound can be enjoyed by the viewers 20:36:00 On the receiver side AV and other data are synchronized according to captured timestamp 20:36:17 ... Rendering not only to screen, but orchestrated to external devices 20:36:32 ... How do I synchronise the data? 20:37:03 ... On the sender side, the browser uses rAF, which clocks at 16ms (60 fps screen update) 20:37:41 ... Get the video image data, and data with the same timing. On the receiver side, get from each timeslot and render 20:38:10 Synchronizing face landmarks and AV is relatively easy as landmarks are sent at the same time as requestAnimationFrame function 20:38:13 ... With external midi devices the data is asynchronous. Inside the 16 ms slot, a MIDI event is fired in the browser 20:38:39 ... Playing to external midi devices on the receiver side. The video clock isn't enough, because there's a time interval 20:38:55 ... Web MIDI has a send() method, where you can indicate a time interval. MIDI works well on the receiver side 20:39:10 ... Concern is the time lag on the input side 20:39:57 ... In this model, once MIDI data is transferred to the browser, it goes to the event buffer, then an event is emitted 20:40:25 Concern about time lag of MIDIInput: is there a time lag between device driver and event emit? are melody and rhythm changes because of that? 20:40:27 ... With JavaScript we can the capture time at the time of event emission, not the time of input 20:40:48 ... Example of 120 bpm music, each note could be 62.5 ms apart 20:41:08 ... If the time lag is 3ms, it makes a 6% fluctuation 20:41:49 ... Other use cases beyond entertainment. It can apply in other cases: remote gaming, with the time lag of the GamePad API 20:42:23 ... Remote robot control over WebUSB. Is there a time interval argument to transferOut data, similar to WebMIDI? 20:42:29 There are other cases with the same kind of problems such as remote gaming, remote robot control and remote drawing 20:42:38 ... Similar problem in Smart City 20:43:36 Kaz: This would be useful for Smart City, as there are many components, devices, sensors to be connected with each other depending on the user need. So this can be an interesting mechanism 20:43:44 q+ 20:44:10 ... Given the potential time lag between client and server, would it help to a real time OS on each side to manage the time synchronisation? 20:44:22 question from kaz: would it be ideal for both publisher/subscriber to have some kinds of realtime operating systems? 20:44:26 Komatsu: A realtime OS could be onsidered 20:44:51 ... Jitter in the network itself, whether it works or not is a question 20:45:06 ... My idea is that event objects have a capture timestamp property 20:45:19 ... That would be enough for the internet cases I've seen 20:45:29 ... What accuracy is required depends on the use case 20:45:32 answer from komatsu: putting captured timestamp in objects might be enough for most usecases 20:46:04 ... Don't want to talk about details of API changes, but instead talk about whether this is a question or not 20:46:07 hta has joined #sync-on-web 20:46:09 baboba0 has joined #sync-on-web 20:46:17 +q 20:46:28 ... Is timeline alignment really a problem? What use cases should be considered? 20:46:39 ... Worth discussing? 20:46:47 q+ 20:46:53 ... Any other related topics to cover? 20:46:56 q? 20:47:22 ack song 20:47:27 q+ padenot 20:47:48 Song: Excellent presentation. I raised a similar topic in teh Web & Networks IG about cloud gaming 20:48:14 ... With cloud gaming, China Mobile launched last month, have millions of subscribers 20:48:29 ... We transmit the data with WebRTC, the time duration is 15 ms in the network across China 20:48:36 ... Rendering is 15-20ms, still acceptable 20:49:02 ... The biggest part for end to end commercial use is translation of mouse and keyboard events for games, this can cost 90 ms 20:49:13 hta has joined #sync-on-web 20:49:25 ... For every business case, e.g., digital twins, could be very different 20:49:42 ... With the data example I mentioned, we get complaints from game and web developers 20:49:48 song: in the new cloud gaming service from China Mobile, data are sent via WebRTC tkaing about 15ms, rendering takes 15-20ms and user events in games takes about 90ms 20:50:02 ... The infrastructure is based on IP network, which is best effort 20:50:04 q+ 20:50:22 Niko has joined #sync-on-web 20:50:41 ... The request we get from game companies is a deterministic network 20:51:06 ... The headache for us is breking the network conditions for milliions of users 20:51:42 ... In the Web & Networks IG, we have 2 solutions. One is MoQ. That's in 3GPP release 20, called 6G 20:52:19 ... That can change the synchronisation systematically, the router, switch, radio access, coordinate the MIDI with the time clock in the device. Long term solution 20:52:52 ... Second is use cloud edge client coordination. If I can't change the best effort IP network, this is why WNIG incubates the cloud edge 20:53:08 ... What do you think? 20:54:12 Komatsu: Delay would fluctuate, does that cause confusion for gaming services? 20:54:32 Song: Can follow up with you 20:54:36 q? 20:55:18 ack baboba 20:55:40 rrsagent, make log public 20:55:41 Bernard: There are several classes of use case: syncing media+events from a single particpants. Will discuss in Media WG tomorrow 20:55:45 rrsagent, draft minutes 20:55:46 I have made the request to generate https://www.w3.org/2024/09/25-sync-on-web-minutes.html kaz 20:56:10 ... Trickier is syncing from multiple clients. We found we need additional extensions, both to the network and the web 20:56:26 .... Web RTC WG is working on absolute capture timestamp, sync to the server's wallcock 20:56:28 Bernald: synchronizing between multiple participants would be way more difficult 20:56:35 s/wallcock/wall-clock/ 20:56:47 s/Bernald/Bernard 20:57:00 ... We're investigating the timing information necessary, then in the media, everything to be synchronised will need the capture timestamp 20:57:07 q? 20:57:42 Komatsu: I agree. NTP. Depends on the use case. The current WebRTC activity should be considered 20:57:58 Bernard: We want to make it general, not only WebRTC but MOQ and across the board 20:57:58 q+ 20:58:06 ack hta 20:58:18 Harald: A lot of these problems are familiar 20:58:33 ... To sync, you need to know the delay between the real world happening 20:58:41 ... Differs by device, which is awkward 20:59:11 ... Jitter in the network, difference in delays. That has to be handled with jitter buffers, which introduces delay 20:59:20 ... A too short jitter buffer means losing stuff 20:59:26 ... That's the trade-off 20:59:39 ... When we did WebRTC for stadia, we had a concept called ?? 21:00:00 ... You'd sync events that happened before, so the efects are visible. You wish for time travel! 21:00:17 ... Timestamping everything at the outset is a good starting point 21:00:38 Komatsu: MoQ we can manipulate the jitter buffer in the browser 21:00:39 q? 21:01:06 jya: We tried doing media sync with Media Session. Not to this level of synchronicity 21:01:24 Paul: Look at the Proceedings of the Web Audio conference over the years 21:01:45 ... We're able to do sub-millisecond sync of real time audio 21:02:00 .. In general, the question is tradeoff between latency and resilience 21:02:29 ... Need to consider clock domain crossing. Clocks on the sender side, different no the rx side. Need a source of truth, and re-clock and resample so there are no gaps 21:02:53 ... This means the relationship between the audio and midi events are preserved, then you offset that by the latencies (audio output, video, etc), reclock everything 21:03:04 .. Important to preserve the relationship between the two 21:03:36 .... Typically between two sound cards, can be 0.08% difference. If you watch a movie for 1 hour, it's skewed and broken, needs to be taken care of 21:04:12 ... Installation at WAC showed real time audio streams playing across different audio devices nicely. There is hope, but it's a clock thing. Delay vs resilience is the question 21:04:17 q? 21:04:22 ack pad 21:05:04 Jer: To add to jya's point, we were going for about a second 21:05:46 Michael: I'm in Audio WG, co-editor of Web MIDI. We have an issue about syncing MIDI with Web Audio on the same host 21:06:06 Niko has left #sync-on-web 21:06:11 ... IMO jitter in MIDI is more important than latency. Now is a good time to add things to the spec, if those are easy 21:06:14 ack mj 21:06:16 q? 21:07:12 Kaz: Given those potential and promising technologies, I wonder about what kind of mechanism would be preferred to handle sync of multiple threads? Interesting to think about the contorl mechanism 21:08:06 Paul: Web Audio gives you the system and audio clocks, so you can calculate the slope. rAF(), two timestamps, understand the slope and drift. With this, it's possible 21:08:07 q? 21:08:10 ack k 21:09:06 ... Real time OS might be overkill. We get good results with commercial OS's 21:09:30 ... If we're looking at variation about 1ms, a regular computer with proper scheduling classes will go far 21:09:42 q? 21:09:54 q? 21:10:10 Komatsu: To wrap up, I want to talk about next steps 21:10:12 q+ 21:10:29 ... Community group, or existing CG or IG? 21:11:00 ack k 21:11:45 okin has joined #sync-on-web 21:11:48 Chris: You're welcome to bring this to MEIG if you want to discuss more about use cases and requirements 21:12:16 Harald: Attend the Media WG / Web RTC meeting where we'll discuss sync 21:12:30 q? 21:12:40 Komatsu: Thank you all! 21:12:44 mjwilson has left #sync-on-web 21:12:45 [adjourned] 21:12:50 rrsagent, draft minutes 21:12:52 I have made the request to generate https://www.w3.org/2024/09/25-sync-on-web-minutes.html cpn 21:13:03 rrsagent, make log public 21:15:25 chrisguttandin has left #sync-on-web 21:17:13 ktoumura has left #sync-on-web 21:46:37 kaz has joined #sync-on-web 23:00:56 kaz has joined #sync-on-web