W3C

Cross-device synchronization TPAC 2015 breakout

28 Oct 2015

See also: IRC log

Attendees

Present
About 20 persons
Chair
Francois
Scribe
Mohammed_Dadas

Summary

The session explored cross-device synchronization scenarios, including shared video viewing, lip-sync use cases, distributed music playback, video walls, cross-device animations, etc. These scenarios require mechanisms to (1) synchronize the local clock with an external one, (2) distribute a timeline, and (3) harness the media playback or animation. A pure JavaScript-based approach only works to some extent; proper cross-device sync needs native browser support. The Timing Object spec defines an API to expose cross-device sync mechanisms to Web applications. Discussion revealed that harnessing the media playback is not trivial as media rendering is often handled by the hardware itself, and that loose synchronization where e.g. a device temporarily runs out-of-sync could improve UI responsiveness in some cases. Work on the spec will continue in the Multi-Device Timing Community Group. Interested parties are invited to join the group!

Presentation

-> Presentation slides on cross-device synchronization

Discussion

tidoust: Fran├žois daoust W3C staff

Goal is to discuss ways to enable multi-device synchronisation,
... starting by a short presentation and a demo
... and then opening the floor for discussions.

The work started in Multi-Device Timing CG, following discussions in the Web and TV IG.
Use cases include: shared viewing, related companion content, Lip-sync, clean audio sound track, video wall, digital signage, silent disco, etc.
There are more UCs as cross device synchronisation can be applied more widely than to media context only

Some orders of magnitude for synch are given to explain the different situations. <10ms is probably the maximum that can be achieved within Web browsers, required for echoless audio across devices.

First thing needed to sync up content is to agree on a clock, that all devices refer to.
Clock synchronization can be done in JS (but native support coud be better)

Second thing needed is to share the timeline (distributed by the app) and to share updates to the reference timeline.
This also can be achieved in JS.

Third thing needed is a way to inform the UA to use the reference timeline and to follow the shared clock for a given media element to achieve synchronized media playback.

Mathieu hoffman: there is also an issue with the latency of the distribution of messages to all devices

Francois: Yes. Cross-device sync does not solve instantaneous propagation. In our proposal, the timing provider may ensure that all actions are taken at the precise same time by all devices by delaying the action to compensate the maximum latency. For instance, the provider may add a 500ms delay to all timestamps of messages it delivers to guarantee that all connected devices will have received the message before the update needs to take place.

Rus: there may not be any need to synchronize all the media at the same time. It may be preferable to have devices slightly out of sync to improve the application's responsiveness to user interactions.

Francois: Yes. There are probably different scenarios to consider, some where sync would be favored and others where responsiveness would take precedence.

Sychronized media playback can also be achieved in JS using skips and adjustemnets of playbackrate
... but that's a hack and only works to some extent.

Rus: in MSE, there is a discussion on insertion of alternative content which has close concerns

Francois: Yes, the notion of timing is needed in different works : MSE, web audio, HTML5 Media elements, CSS animations; hence the importance to have a common understanding and definition of it

[DEMO, see: http://kwz.me/MW]

The main idea of the TimingObject spec is to define timing as a web resource.
The timeline is defined as a state vector, called TimingStateVector.
The timing provider object makes the link to the timing provider and encapsulates the clock synchronization mechanism (at least the part that computes the clock's skew) and the definition of the timeline.

Mathieu: It seems it would be useful to split the TimingProvider interface into two, to expose the clock synchronization mechanism so that applications can reference usual ones such as NTP and another part that would handle the timeline.

Francois: Perhaps. What I gathered is that it may not be easy to mandate a particular (set of) synchronization mechanism(s) to be supported by user agents. Also, the risk is that this would require user agents to establish two communication channels (think two sockets), one for clock synchronization and one for timeline sharing, which does not strike me as such a good thing given that these channels should not be over-used: clock synchronization only requires a few packets now and then, timeline updates frequency can be very low as well depending on use case.
... Also, although not in the spec, one thing that will probably be needed is to have TimingProvider code run in isolated contexts for privacy and security reasons, so splitting would also require two isolated contexts instead of one.
... Now, that's a possibility, of course.

<padenot> b 4

The TimingObject object represents the whole notion of timing as seen by Web applications. The object is similar but different from a MediaController (now deprecated or about to be), as it acts as a real master for media elements (no automatic pausing when buffering issues occur). The timing object is associated with a media element through a proposed custom timingsrc attribute on MediaElement.

On top of this, the TimingTextTrack allows to sequence timed data independently of the presence of any media element, which HTML5 requires for TextTrack objects currently.

One question is: is this doable?
From discussion with several people here at TPAC, it seems that it is complicated, both from a spec persepective, as most media-related algorithms in HTML5 would need to be re-written, and from an implementation perspective, as media rendering may be handled by the hardward itself, using its own clock, or handled by third-party code not under the direct control of the browser vendor's code.

A few external organizations also work on the same topic (DVB, IEEE, IETF, MPEG, ....)

Roy Kawada: in digital signage, there is a need of synchronized screen (JR Shinagawa Station)

Francois: Yes. I note that the TimingObject spec lists a bunch of use cases (see draft). This could be completed as needed

Mark_Foltz: is this aiming to address multiple media on the same device also?

Francois: yes. Essentially, in the single-device case, this aims to replace MediaController. I'm not clear how to spec that better than it already is in HTML5, with the exception that the goal here would be to follow the clock exposed by Performance.now.

Mathieu: if you split the timing provider into two parts, one for the clock synchronization and the other for the timeline definition and propagation, you could associate the timing object with the local clock for the clock synchronization for the single device use case.

Francois: Right, but we don't need that. In the single device case, there is simply no need to associate the timing object with any timing provider. By default, it runs on the local clock and exposes a local timeline.

Mathieu: can you pass a specific timestamp when you send update commands?

Francois: Not with the current proposal, the time stamp is added by the online timing resource, and distributed to all devices. The devices only apply the update when they receive the message
... if you need to set a specific timestamp, you will need to create your own version of a timing provider object

Mark_Foltz: how do you deal with buffering? and in case of real time media?

Francois: That's a good question. For the live case, you need to ensure the UA will keep up with the timeline as much as it can (how is not defined)

Mathieu: it depends again on the UCs (buffering or not)

All participants are invited to join the community group and to contribute to the github repository!

[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.140 (CVS log)
$Date: 2015/11/11 12:50:29 $