Multi-Device Timing CG
TPAC 2015
Breakout session

Multi-Device Timing CG

Goal is to define a common, multi-device, timing mechanism and a practical programming model for time-sensitive, multi-device Web applications.

1. Created in February 2015 following discussions in Web and TV IG
2. Chaired by Ingar M. Arntzen, Motion Corporation
3. 28 participants (wanna join?)
4. Develops the Timing Object specification
5. Home page: https://www.w3.org/Community/webtiming

Table of Contents

1. Some use cases
2. How to sync content across devices
3. The Timing Object specification
4. Is it doable?
5. Next steps

Media use cases

• Shared viewing
• Related companion content
• Lip-sync
• Clean Audio soundtrack in theater
• Video wall
• Digital signage scenarios
• Silent disco
• Distributed music production and playback
  (Web audio, Web MIDI API)

Other use cases

• Online education
• Collaborative editing
• Multi-screen data visualisation
• Cross-device animations
• Evaluating distances or speeds
• Games

Orders of magnitude for sync (*)

> 100 ms

"Non-visual" or not closely related content
signing use cases should not require more precise sync

< 25 ms

Frame accuracy
"Lip sync", video on one device while audio plays on another device

< 10 ms

Echoless audio sync.
Needed for multiple devices to play audio or music together
Most likely the maximum accuracy that we may achieve in browsers

< 1 μs

GPS

(*) sync ≠ instantaneous propagation,
network latency still exists!

Clock synchronization

A Web app must be able to instruct the user agent to follow an external (*) clock.

(*) from a server, local peer or perhaps media transport stream

Clock synchronization

Clock synchronization

... can be achieved in JS (*)

(*) although native support would be better
(and required for UDP-based protocols)

Shared timeline

An app must be able to distribute the timeline (*):

    At 15:53:29.813 according to the sync clock,
        the position is 5.3s
            and the playback rate is 1.0

(*) and updates to that timeline, taking into account that
user agents can compute the position on their own:
pos(t) = pos₀ + playback_rate * (t - t₀)

Shared timeline

... can be achieved in JS (*)

(*) typically via a WebSocket connection

Synchronized media playback

A media resource has a media timeline that maps times (in seconds) to positions in the media resource [...]. The media timeline also has an associated clock. Which clock is used is user-agent defined, [...] but it should approximate the user's wall clock.

Need a way to tell the user agent to use our timeline, and in particular to follow our clock!

Synchronized media playback

... can be somewhat achieved in JS (*)

(*) through skips and continuous adjustments to playbackrate
but support depends on the user agent, codec and platform
and this is more a hack than anything else, right?

In short

Clock synchronization

Can be done in JS, restricted to protocols exposed to web apps

(but no UDP-based protocols, no access to internal media clock)

Shared timeline

Can be done in JS

Sync media playback

Harder to do in JS
playbackrate was not designed for that
Depends on the previous two bits
(cannot be standardised in isolation)

Timing as a Web resource

Timing State Vector

      interface TimingStateVector {
        readonly attribute double position;
        readonly attribute double velocity;
        readonly attribute double acceleration;
        readonly attribute double timestamp;
      };
    

pos(t) = pos₀ + velocity * (t - t₀) + 1/2 * acceleration * (t - t₀)²

Timing provider

Leave the door open to different clock synchronization mechanisms (*)

(*) Provider provides JS library to create TimingProvider objects.

Timing provider

      callback interface TimingProvider {
        readonly attribute TimingStateVector   vector;
        readonly attribute unrestricted double startPosition;
        readonly attribute unrestricted double endPosition;
        readonly attribute DOMString           readyState;
        readonly attribute unrestricted double skew;
        Promise update(TimingStateVectorUpdate newVector);
      };