14:39:25 RRSAgent has joined #sync 14:39:25 logging to http://www.w3.org/2016/09/22-sync-irc 14:39:33 RRSAgent, make logs public 14:39:46 Meeting: Multi-Device Timing CG 14:39:50 Chair: Ingar 14:40:22 Agenda: https://lists.w3.org/Archives/Public/public-webtiming/2016Sep/0003.html 14:41:15 Present+ Ingar_Arntzen(MotionCorp), Francois_Daoust(W3C), Mike_Assenti(Dolby), Louay_Bassbouss(Fraunhofer), Chris_Needham(BBC) 14:41:37 Present+ Kumanan_Yogaratnam(Espial) 14:41:53 cpn_ has joined #sync 14:42:08 Present+ Toshihiko_Yamakami(Access) 14:42:27 RRSAgent, draft minutes 14:42:27 I have made the request to generate http://www.w3.org/2016/09/22-sync-minutes.html tidoust2 14:42:50 Present+ Anssi_Kostiainen(Intel) 14:43:13 yam has joined #sync 14:43:25 Louay has joined #sync 14:43:36 present+ Louay_Bassbouss 14:43:36 scribenick: cpn_ 14:43:46 scribe: Chris_Needham 14:44:28 kumanan has joined #sync 14:46:17 Ingar: This is the first F2F of the Multi-Device Timing CG 14:47:28 ... This work comes from the MediaScape EU project 14:47:58 ... The focus is on multi-device timing across web pages, and providing a programming model to deal with temporal aspects in web apps 14:48:12 ... This is a multi-device presentation 14:48:29 https://goo.gl/AlR9Nm 14:49:39 Present+ Shih-Chiang_Chien(Mozilla) 14:51:05 Ingar: I'd like to show a demo 14:51:16 http://mcorp.no/pres/vegas16/main.html 14:52:48 Ingar: [ demonstrates showing a video on several devices in sync ] 14:54:19 ... If you look at the edges between the browsers, this is where you'll see any sync errors 14:55:08 ... The synchronisation was between two browser windows on a single machine 14:55:25 ... You can achieve similar precision in the global scope, eg, between devices in this room 14:56:00 Toshihiko: What would happen when I return home to Japan? 14:56:31 Ingar: ... It would still play in sync 14:57:07 ... It's symmetric, in the sense that we all have control 14:57:32 ... [ demonstration of pausing the video, all devices pause ] 14:58:06 ... I think we're in the single-digit millisecond region 14:58:43 ... The play and pause control commands do have a day, may take 0.2 seconds or so 14:59:26 Mike: Are these commands also part of the spec? 14:59:29 Ingar: Yes 15:00:03 ... We also handle re-establishing sync on page reload 15:00:14 Topic: Introduction 15:00:29 kumanan has joined #sync 15:00:31 Ingar: Francois presented at TPAC last year 15:00:50 ... the draft Timing Object spec 15:01:17 ... using an online timing provider from Motion Corp 15:01:37 ... Since then we have an open source JavaScript implementation 15:02:07 ... We've demonstrated at NAB and IBC, and also published some papers 15:02:30 ... Media Element capabilities have also progressed 15:02:50 ... We were having trouble with Chrome on Android before, but this has become very good now 15:03:04 ... Safari has a bug 15:03:12 Anssi: Has that been reported? 15:03:21 Ingar: We'll be doing that, yes 15:04:21 Ingar: Timestamping on the client side, using timing info in getUserMedia 15:04:39 ... We've also reported bugs in Chromium, now fixed 15:05:07 ... Multi-device timing is about making stuff happen at the right time 15:05:46 ... It's not only about having a clock, but also interactive capabilities to control it 15:06:27 ... Timing scopes: page internal, within the browser context, and also cross-page 15:06:53 ... Even though the group's name is Multi-Device Timing, both of these scopes are important 15:07:31 ... The Timing Object is a client-side API 15:07:50 ... We don't prescribe a particular solution for the cross-page timing 15:07:57 ... It can be implemented in different ways 15:08:16 Topic: Timing Challenges 15:08:42 ... With the page internal timing, there could want to synchronise multiple videos 15:08:54 s/there could/you could/ 15:09:18 Ingar: Ad insertion can be done with MSE 15:09:34 Francois: There are some limitations with what you can do with MSE 15:10:01 Ingar: Our approach gives more flexibility 15:10:33 ... It works with different types and frameworks 15:10:57 ... iframes are important, eg for independently delivered adverts 15:11:24 ... You can split content across windows or frames 15:11:54 ... You can also sync between web pages and native apps 15:12:10 ... Collaborative applications 15:12:46 ... It can work across different types of network connection, eg, 3G or wifi 15:14:10 Media elements and animations have different APIs, making reusability and flexibility difficult 15:14:26 s/Media elements/... Media elements/ 15:14:55 Ingar: The Media element currentTime attribute can be quite coarse 15:15:14 Francois: The spec doesn't force the UA to apply a precise clock 15:15:54 Anssi: We specified an extension to Web Audio, the context time, which is when the audio hits the speakers 15:16:22 ... We'll be adding timing to sensors, etc, anywhere it makes sense 15:16:37 ... We're working on low-level primitives that make your use cases possible 15:17:30 ... An example is screen touch events, knowing when these happen 15:18:11 Ingar: It would be nice to write a sync wrapper for Web Audio 15:18:53 ... Non-determinism is a problem, especially in the distributed case 15:19:17 ... There's no support for cross-page timing at the moment 15:20:04 ... It's not a sound assumption to rely on NTP synchronised system clocks 15:20:53 ... Control and timing are typically dealt with separately, but we're trying to deal with these in a single abstraction 15:21:29 Francois: Have you thought about how Web Audio synchronisation might apply to the Presentation API? 15:21:38 Anssi: No, not yet. 15:23:12 ... [ some discussion of issues with Performance.now ] 15:23:48 Ingar: We get around that, using a Timing Object, which you can give to two processes in the same architecture 15:23:54 Topic: Goals for the group 15:24:14 Ingar: We can to simplify things for developers 15:24:53 ... Interoperability is important. With the web everything is composeable, but not timing 15:25:20 ... Also synchronising web and broadcast content 15:25:46 ... We have a concrete proposal, and our demos show that it's feasible 15:26:03 ... [ another demo ] 15:26:24 https://youtu.be/D6cBn65KXTk 15:26:44 Ingar: It shows how to create a web page that uses timing 15:28:55 Kumanan: There's an issue here with iOS, where it only allows one video playing at once 15:29:32 ... I have Blink in one and Firefox in the other 15:30:08 Topic: Proposal 15:31:14 Ingar: The timing object can play, pause, jump. But it's not connected to a media element 15:31:23 ... [ demo of local timing object ] 15:31:56 ... You can change position, velocity, acceleration 15:32:52 kumanan has joined #sync 15:33:10 ... through the API using a vector 15:33:54 Francois: There's the same kind of object being defined in WebVR, where a pause operation affects velocity 15:35:08 Chris: What's the motiviation for acceleration? Could you have easing functions for changing between one velocity and another 15:35:41 Ingar: Acceleration isn't a central thing in this. You could build such functions on top of the API 15:36:13 ... It's not critical, but could be useful 15:36:25 Kumanan: Does velocity have direction? 15:36:43 Ingar: Yes, so negative reverses the playback direction 15:37:04 Mike: Are the units arbitrary? 15:37:15 Ingar: Yes. 15:38:21 Mike: If we want to relate this to a media content pipeline, For example, if you want to double the rate, how do you know what the scale of the timeline is? 15:38:52 Francois: When you associate the Timing Object with a media element, position would be seconds, so the units are set there 15:39:22 ... We're also thinking about how to compose Timing Objects, different clocks at different rates 15:39:58 Ingar: There are different options. You could define a mapping, so we've defined a TimingConverter 15:40:08 ... You'd use this with media objects having different timelines 15:40:40 ... You could also change the data to fit the different timeline 15:41:36 Kumanan: When I change velocity, how do I know what my normal is? 15:42:00 Mike: I think the question is how the position and velocity relate to the timestamp value in the vector 15:43:30 Kumanan The timestamp comes from Performance.now, which is local 15:44:37 [ discussion of how timestamp relates to the vector ] 15:44:52 s/Kumanan The/Kumanan: The/ 15:46:46 Ingar; An application can remember what the normal is. The velocity doesn't jitter 15:46:55 s/Ingar; An/Ingar: An/ 15:48:03 Ingar: The TimingObject has change and timeupdate events 15:49:54 ... When someone changes the TimingObject, all others are updated 15:50:44 ... The TimingObject is in control, and it's the component's responsibility to respond 15:51:51 Toshihiko: Will the TimingObject cover other use cases, such as a carousel where you want to loop from the end to the start 15:52:10 Ingar: It's quite simple to add that on top of this 15:52:22 ... It's a matter of preference 15:53:02 Ingar: A hosted internet service can provide a timing service, so the local TimingObject becomes a proxy 15:53:35 ... There's latency but the state will be correct 15:54:52 ... One advantage of a hosted solution is availability 15:55:32 ... [ demo of online timing object ] 15:56:17 ... Doing this in user space JavaScript, you lose some precisiion 15:57:57 ... We have the TimingObject API for clients, but there's also an API for timing providers 15:58:27 ... This is simpler, as it just has to maintain two variables: the vector, and the skew 15:59:21 ... The main source of error is the step between the TimingObject and the media element. 15:59:48 Kumanan: If this becomes a proper Web API, would there be a timing provider mandated? 16:00:07 Francois: The default implementation would use Performance.now 16:01:30 ... [discussion of timing providers and skew ] 16:02:37 Ingar: The skew would be that between Performance.now and the underlying timing source 16:03:47 ... There are lots of applications: collaborative viewing, multi-screen displays, multi-device in the home, showing interactive content with a main video feed 16:04:45 ... Also remote control, music, multi-device capture, timestamping material according to an app-specific timeline 16:05:04 ... Which makes the content re-playable in a multi-device setup 16:06:09 Ingar: We've done some experiments on sync precision 16:06:37 ... We're measuring the difference between the TimingObject and the media position 16:06:53 ... When there are big differences, it's best to skip position 16:07:00 ... When it's closer, adjust the playbackRate 16:08:14 ... [ shows graphs of measurements taken on Android, Chrome ] 16:09:14 ... We'd like to be able to supply a timestamp to the media element's play command 16:11:11 ... [ graph shows problem with playbackRate on iPad ] 16:12:15 Topic: Standardisation 16:13:01 Ingar: Improvements needed with playBackRate, seek 16:13:45 Francois: We may also need to look at Web Animation, you may not know the latency, ie, when the animation will actually start 16:15:13 Ingar: Text tracks are limiting, we have a sequencer object that uses timing object 16:15:59 ... We use setTimeout in the sequencer, which gives precision of a few milliseconds, but delays if the browser is busy 16:16:35 Francois: Due to the event loop, but could use tasks or micro tasks instead 16:18:01 Kumanan: Wet set limits to the setTimeout interval 16:18:16 Topic: Related work 16:18:28 Ingar; There's the Second Screen WG 16:18:58 Louay: HbbTV 2.0 has sync, discovery, communication, app launch in one package 16:19:07 ... Similar for Google Cast 16:20:12 ... The protocol work in the Second Screen group will touch on synchronisation 16:20:21 ... Should we have one group at W3C looking at this? 16:21:11 Francois: I don't expect the Second Screen WG to take on this work 16:22:13 ... One thing in-scope for the Second Screen CG is reporting of currentTime for the Remote Playback API 16:22:53 Kumanan: That just needs to address reporting 16:24:04 Francois: I'm trying to gather interest in the multi-device timing CG 16:24:33 ... There have been improvements in the web platform, which make sync easier, but no specific work on sync itself 16:25:25 Kumanan: I suggest a separate timing spec is valuable more generally than just media applications 16:26:38 Francois: A native timing object isn't always needed, but it is needed where you want to connect to a media element 16:28:29 Kumanan: There are applications such as real time collaboration where this is valuable 16:29:32 Topic: Next steps 16:29:57 Ingar: We want to attract support to the CG 16:30:25 Francois: We could track progress in the existing specs that have timing involved 16:31:07 Kumanan: What's the blocker to moving this forward? 16:31:33 Francois: It needs someone willing to implement natively 16:32:00 ... [ discussion on differences between CGs, WGs, and IGs at W3C ] 16:34:47 tidoust has joined #sync 16:36:11 present+ Eric_Carlson(Apple) 16:36:34 Ingar: [ recaps issues with iOS timing measurements ] 16:41:05 Louay: Do you have any experience of syncing live streams, eg HLS? 16:41:18 Ingar: Yes, i have a demo 16:44:36 Louay: [ discussion of currentTime in live streams ] 16:44:47 Eric: startDate should work with HLS streams if the information is in the manifest 16:46:29 ... [ discussion of the play method that returns a Promise ] 16:49:17 RRSAgent, draft minutes 16:49:17 I have made the request to generate http://www.w3.org/2016/09/22-sync-minutes.html tidoust 16:49:50 Ingar: Thanks all for coming, please join the group, suggest use cases, point it out to others 16:49:56 [ adjourned ]