Multi-Device Timing CG F2F

22 Sep 2016


See also: IRC log


Ingar_Arntzen(MotionCorp), Francois_Daoust(W3C), Mike_Assenti(Dolby), Louay_Bassbouss(Fraunhofer), Chris_Needham(BBC), Kumanan_Yogaratnam(Espial), Toshihiko_Yamakami(Access), Anssi_Kostiainen(Intel), Louay_Bassbouss, Shih-Chiang_Chien(Mozilla), Eric_Carlson(Apple)


Ingar: This is the first F2F of the Multi-Device Timing CG
... This work comes from the MediaScape EU project
... The focus is on multi-device timing across web pages, and providing a programming model to deal with temporal aspects in web apps
... This is a multi-device presentation

-> Slideshow

Ingar: I'd like to show a demo
... [ demonstrates showing a video on several devices in sync ]
... If you look at the edges between the browsers, this is where you'll see any sync errors
... The synchronisation was between two browser windows on a single machine
... You can achieve similar precision in the global scope, eg, between devices in this room

Toshihiko: What would happen when I return home to Japan?

Ingar: ... It would still play in sync
... It's symmetric, in the sense that we all have control
... [ demonstration of pausing the video, all devices pause ]
... I think we're in the single-digit millisecond region
... The play and pause control commands do have a day, may take 0.2 seconds or so

Mike: Are these commands also part of the spec?

Ingar: Yes
... We also handle re-establishing sync on page reload


Ingar: Francois presented at TPAC last year
... the draft Timing Object spec
... using an online timing provider from Motion Corp
... Since then we have an open source JavaScript implementation
... We've demonstrated at NAB and IBC, and also published some papers
... Media Element capabilities have also progressed
... We were having trouble with Chrome on Android before, but this has become very good now
... Safari has a bug

Anssi: Has that been reported?

Ingar: We'll be doing that, yes
... Timestamping on the client side, using timing info in getUserMedia
... We've also reported bugs in Chromium, now fixed
... Multi-device timing is about making stuff happen at the right time
... It's not only about having a clock, but also interactive capabilities to control it
... Timing scopes: page internal, within the browser context, and also cross-page
... Even though the group's name is Multi-Device Timing, both of these scopes are important
... The Timing Object is a client-side API
... We don't prescribe a particular solution for the cross-page timing
... It can be implemented in different ways

Timing Challenges

Ingar: With the page internal timing, you could want to synchronise multiple videos
... Ad insertion can be done with MSE

Francois: There are some limitations with what you can do with MSE

Ingar: Our approach gives more flexibility
... It works with different types and frameworks
... iframes are important, eg for independently delivered adverts
... You can split content across windows or frames
... You can also sync between web pages and native apps
... Collaborative applications
... It can work across different types of network connection, eg, 3G or wifi
... Media elements and animations have different APIs, making reusability and flexibility difficult
... The Media element currentTime attribute can be quite coarse

Francois: The spec doesn't force the UA to apply a precise clock

Anssi: We specified an extension to Web Audio, the context time, which is when the audio hits the speakers
... We'll be adding timing to sensors, etc, anywhere it makes sense
... We're working on low-level primitives that make your use cases possible
... An example is screen touch events, knowing when these happen

Ingar: It would be nice to write a sync wrapper for Web Audio
... Non-determinism is a problem, especially in the distributed case
... There's no support for cross-page timing at the moment
... It's not a sound assumption to rely on NTP synchronised system clocks
... Control and timing are typically dealt with separately, but we're trying to deal with these in a single abstraction

Francois: Have you thought about how Web Audio synchronisation might apply to the Presentation API?

Anssi: No, not yet.
... [ some discussion of issues with Performance.now ]

Ingar: We get around that, using a Timing Object, which you can give to two processes in the same architecture

Goals for the group

Ingar: We can simplify things for developers
... Interoperability is important. With the web everything is composeable, but not timing
... Also synchronising web and broadcast content
... We have a concrete proposal, and our demos show that it's feasible
... [ another demo ]

Ingar: It shows how to create a web page that uses timing

Kumanan: There's an issue here with iOS, where it only allows one video playing at once
... I have Blink in one and Firefox in the other


Ingar: The timing object can play, pause, jump. But it's not connected to a media element
... [ demo of local timing object ]
... You can change position, velocity, acceleration
... through the API using a vector

Francois: There's the same kind of object being defined in WebVR, where a pause operation affects velocity

Chris: What's the motiviation for acceleration? Could you have easing functions for changing between one velocity and another

Ingar: Acceleration isn't a central thing in this. You could build such functions on top of the API
... It's not critical, but could be useful

Kumanan: Does velocity have direction?

Ingar: Yes, so negative reverses the playback direction

Mike: Are the units arbitrary?

Ingar: Yes.

Mike: If we want to relate this to a media content pipeline, For example, if you want to double the rate, how do you know what the scale of the timeline is?

Francois: When you associate the Timing Object with a media element, position would be seconds, so the units are set there
... We're also thinking about how to compose Timing Objects, different clocks at different rates

Ingar: There are different options. You could define a mapping, so we've defined a TimingConverter
... You'd use this with media objects having different timelines
... You could also change the data to fit the different timeline

Kumanan: When I change velocity, how do I know what my normal is?

Mike: I think the question is how the position and velocity relate to the timestamp value in the vector

Kumanan: The timestamp comes from Performance.now, which is local

[ discussion of how timestamp relates to the vector ]

Ingar: An application can remember what the normal is. The velocity doesn't jitter
... The TimingObject has change and timeupdate events
... When someone changes the TimingObject, all others are updated
... The TimingObject is in control, and it's the component's responsibility to respond

Toshihiko: Will the TimingObject cover other use cases, such as a carousel where you want to loop from the end to the start

Ingar: It's quite simple to add that on top of this
... It's a matter of preference
... A hosted internet service can provide a timing service, so the local TimingObject becomes a proxy
... There's latency but the state will be correct
... One advantage of a hosted solution is availability
... [ demo of online timing object ]
... Doing this in user space JavaScript, you lose some precisiion
... We have the TimingObject API for clients, but there's also an API for timing providers
... This is simpler, as it just has to maintain two variables: the vector, and the skew
... The main source of error is the step between the TimingObject and the media element.

Kumanan: If this becomes a proper Web API, would there be a timing provider mandated?

Francois: The default implementation would use Performance.now
... [discussion of timing providers and skew ]

Ingar: The skew would be that between Performance.now and the underlying timing source
... There are lots of applications: collaborative viewing, multi-screen displays, multi-device in the home, showing interactive content with a main video feed
... Also remote control, music, multi-device capture, timestamping material according to an app-specific timeline
... Which makes the content re-playable in a multi-device setup
... We've done some experiments on sync precision
... We're measuring the difference between the TimingObject and the media position
... When there are big differences, it's best to skip position
... When it's closer, adjust the playbackRate
... [ shows graphs of measurements taken on Android, Chrome ]
... We'd like to be able to supply a timestamp to the media element's play command
... [ graph shows problem with playbackRate on iPad ]


Ingar: Improvements needed with playBackRate, seek

Francois: We may also need to look at Web Animation, you may not know the latency, ie, when the animation will actually start

Ingar: Text tracks are limiting, we have a sequencer object that uses timing object
... We use setTimeout in the sequencer, which gives precision of a few milliseconds, but delays if the browser is busy

Francois: Due to the event loop, but could use tasks or micro tasks instead

Kumanan: Wet set limits to the setTimeout interval

Related work

Ingar: There's the Second Screen WG

Louay: HbbTV 2.0 has sync, discovery, communication, app launch in one package
... Similar for Google Cast
... The protocol work in the Second Screen group will touch on synchronisation
... Should we have one group at W3C looking at this?

Francois: I don't expect the Second Screen WG to take on this work
... One thing in-scope for the Second Screen CG is reporting of currentTime for the Remote Playback API

Kumanan: That just needs to address reporting

Francois: I'm trying to gather interest in the multi-device timing CG
... There have been improvements in the web platform, which make sync easier, but no specific work on sync itself

Kumanan: I suggest a separate timing spec is valuable more generally than just media applications

Francois: That's good feedback. I would have thought that browser vendors would not fancy supporting a timing object natively if it sits "on the side", meaning if it does not connect to e.g. media elements. That is the reason why we included the connection with media elements in the draft spec.

Kumanan: There are applications such as real time collaboration where this is valuable

Next steps

Ingar: We want to attract support to the CG

Francois: We could track progress in the existing specs that have timing involved

Kumanan: What's the blocker to moving this forward?

Francois: It needs someone willing to implement natively
... [ discussion on differences between CGs, WGs, and IGs at W3C ]

Ingar: [ recaps issues with iOS timing measurements ]

Louay: Do you have any experience of syncing live streams, eg HLS?

Ingar: Yes, i have a demo

Louay: [ discussion of currentTime in live streams ]

Eric: startDate should work with HLS streams if the information is in the manifest
... [ discussion of the play method that returns a Promise ]

[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.144 (CVS log)
$Date: 2016/09/26 08:44:50 $