Multi-Device Timing CG
Web and TV IG F2F
TPAC 2015

Multi-Device Timing CG

Goal is to define a common, multi-device, timing mechanism and a practical programming model for time-sensitive, multi-device Web applications.

1. Created in February 2015 following discussions in Web and TV IG
2. Chaired by Ingar M. Arntzen, Motion Corporation
3. 28 participants (wanna join?)
4. Develops the Timing Object specification
5. Home page: https://www.w3.org/Community/webtiming

Requirements for Home Networking Scenarios (*)

Time-synchronization

Conforming specifications should provide a means for applications to co-time the presentation of their own content (audio, video or other) with the timeline of a programme being played back on another device.

Accurate Time-synchronization

Conforming specifications should support any protocols or optional protocol features that enable applications to be aware of the progress (time within the programme) of the playback of a programme on a remote device to within frame accuracy (1/25th or 1/30th second) or better.

(*) published in 2011, good ol' times?

New (*) Web and TV IG use cases

UC2-3 Identical Media stream synchronization

The ability for two or more identical media streams to play in sync on separate devices.

UC2-4 Related Media Stream Synchronization

The ability for two or more related media streams to play in sync.

UC2-6 Clean Audio

[Bob] has discovered that Clean Audio is available for certain movies at the cinema and that he can select that alternative track for listening over headphones connected to his mobile phone using the same app that he uses at home.

(*) 2014

Other use cases

• Online education
• Collaborative editing
• Multi-screen data visualisation
• Distributed music production and playback
• Silent disco
• Cross-device animations
• Digital signage scenarios
• Evaluating distances or speeds
• Games

Orders of magnitude for sync (*)

> 100 ms

"Non-visual" or not closely related content
signing use cases should not require more precise sync

< 25 ms

Frame accuracy
"Lip sync", video on one device while audio plays on another device

< 10 ms

Echoless audio sync.
Needed for multiple devices to play audio or music together
Most likely the maximum accuracy that we may achieve in browsers

< 1 μs

GPS

(*) sync ≠ instantaneous propagation,
network latency still exists!

Clock synchronization

A Web app must be able to tell the user agent to sync with an external (*) clock.

(*) from a server, local peer or perhaps media transport stream

Clock synchronization

Clock synchronization

... can be achieved in JS (*)

(*) although native support would be better
(and required for UDP-based protocols)

Shared timeline

An app must be able to distribute the timeline (*):

    At 15:53:29.813 according to the sync clock,
        the position is 5.3s
            and the playback rate is 1.0

(*) and updates to that timeline, taking into account that
user agents can compute the position on their own:
pos(t) = pos₀ + playback_rate * (t - t₀)

Shared timeline

... can be achieved in JS (*)

(*) typically via a WebSocket connection

Synchronized media playback

A media resource has a media timeline that maps times (in seconds) to positions in the media resource [...]. The media timeline also has an associated clock. Which clock is used is user-agent defined, [...] but it should approximate the user's wall clock.

Need a way to tell the user agent to use our timeline, and in particular to follow our clock!

Synchronized media playback

... can be somewhat achieved in JS (*)

(*) through skips and continuous adjustments to playbackrate
but support depends on the user agent, codec and platform
and this is more a hack than anything else, right?

In short

Clock synchronization

Can be done in JS, restricted to protocols exposed to web apps

(no UDP-based protocols, no access to internal media clock)

Shared timeline

Can be done in JS

Sync media playback

Harder to do in JS
playbackrate was not designed for that
Depends on the previous two bits
(cannot be standardised in isolation)

Timing as a Web resource

Timing provider

Leave the door open to different clock synchronization mechanisms (*)

(*) Provider provides JS library to create TimingProvider objects.

Code example

      var provider = [provider specific];
      var timing = new TimingObject(provider);
      var video = document.getElementById('vid');
      video.timingsrc = timing;
      timing.update({ position: 5.0, velocity: 1.0 });
       
      var vec = timing.query();
      console.log("pos:" + vec.position + " vel:" + vec.velocity);
    

Media element extension

      partial interface HTMLMediaElement {
        attribute TimingObject timingsrc;
      };
    

When set, the media element would follow the media timeline of the TimingObject (*) and its associated clock.

(*) real master, no automatic pausing for buffering

Timed data

Timed data

      interface TimingTextTrack : TextTrack {
        attribute TimingObject timingsrc;
      };
    

Same as TextTrack but no dependency on MediaElement.

Example at: http://webtiming.github.io/sequencer/

Added benefits

• Exposes timing as a first class citizen on the Web
• Supports single-device cases (could replace MediaController?) (*)
• Leaves the door open to different clock synchronization mechanisms
• Not restricted to media
• Not restricted to read-only scenarios, clients may update timeline
• Can be implemented incrementally
• Alignment with Web Audio, Web Animations seems doable

(*) which may soon be deprecated

Next steps

• End of exploratory phase
• Need feedback from potential implementers
• Need feedback from all Web and TV IG participants!
• Discuss possible bug reports and improvements to playbackrate
• CG needs more contributions for spec to move forward (wanna join?)
• Spec would be in scope of Timed Media WG (*)

(*) provided the spec is mature enough and the WG created!