See also: IRC log
Ingar: This is the first F2F of
the Multi-Device Timing CG
... This work comes from the MediaScape EU project
... The focus is on multi-device timing across web pages, and
providing a programming model to deal with temporal aspects in
web apps
... This is a multi-device presentation
-> Slideshow
Ingar: I'd like to show a
demo
... [ demonstrates showing a
video on several devices in sync ]
... If you look at the edges between the browsers, this is
where you'll see any sync errors
... The synchronisation was between two browser windows on a
single machine
... You can achieve similar precision in the global scope, eg,
between devices in this room
Toshihiko: What would happen when I return home to Japan?
Ingar: ... It would still play in
sync
... It's symmetric, in the sense that we all have control
... [ demonstration of pausing the video, all devices pause
]
... I think we're in the single-digit millisecond region
... The play and pause control commands do have a day, may take
0.2 seconds or so
Mike: Are these commands also part of the spec?
Ingar: Yes
... We also handle re-establishing sync on page reload
Ingar: Francois presented at TPAC
last year
... the draft Timing Object spec
... using an online timing provider from Motion Corp
... Since then we have an open source JavaScript
implementation
... We've demonstrated at NAB and IBC, and also published some
papers
... Media Element capabilities have also progressed
... We were having trouble with Chrome on Android before, but
this has become very good now
... Safari has a bug
Anssi: Has that been reported?
Ingar: We'll be doing that,
yes
... Timestamping on the client side, using timing info in
getUserMedia
... We've also reported bugs in Chromium, now fixed
... Multi-device timing is about making stuff happen at the
right time
... It's not only about having a clock, but also interactive
capabilities to control it
... Timing scopes: page internal, within the browser context,
and also cross-page
... Even though the group's name is Multi-Device Timing, both
of these scopes are important
... The Timing Object is a client-side API
... We don't prescribe a particular solution for the cross-page
timing
... It can be implemented in different ways
Ingar: With the page
internal timing, you could want to synchronise multiple
videos
... Ad insertion can be done
with MSE
Francois: There are some limitations with what you can do with MSE
Ingar: Our approach gives more
flexibility
... It works with different types and frameworks
... iframes are important, eg for independently delivered
adverts
... You can split content across windows or frames
... You can also sync between web pages and native apps
... Collaborative applications
... It can work across different types of network connection,
eg, 3G or wifi
... Media elements and animations have different APIs, making
reusability and flexibility difficult
... The Media element currentTime attribute can be quite
coarse
Francois: The spec doesn't force the UA to apply a precise clock
Anssi: We specified an extension
to Web Audio, the context time, which is when the audio hits
the speakers
... We'll be adding timing to sensors, etc, anywhere it makes
sense
... We're working on low-level primitives that make your use
cases possible
... An example is screen touch events, knowing when these
happen
Ingar: It would be nice to write
a sync wrapper for Web Audio
... Non-determinism is a problem, especially in the distributed
case
... There's no support for cross-page timing at the
moment
... It's not a sound assumption to rely on NTP synchronised
system clocks
... Control and timing are typically dealt with separately, but
we're trying to deal with these in a single abstraction
Francois: Have you thought about how Web Audio synchronisation might apply to the Presentation API?
Anssi: No, not yet.
... [ some discussion of issues with Performance.now ]
Ingar: We get around that, using a Timing Object, which you can give to two processes in the same architecture
Ingar: We can simplify things
for developers
... Interoperability is important. With the web everything is
composeable, but not timing
... Also synchronising web and broadcast content
... We have a concrete proposal, and our demos show that it's
feasible
... [ another demo ]
Ingar: It shows how to create a web page that uses timing
Kumanan: There's an issue here
with iOS, where it only allows one video playing at once
... I have Blink in one and Firefox in the other
Ingar: The timing object can
play, pause, jump. But it's not connected to a media
element
... [ demo of local timing object ]
... You can change position, velocity, acceleration
... through the API using a vector
Francois: There's the same kind of object being defined in WebVR, where a pause operation affects velocity
Chris: What's the motiviation for acceleration? Could you have easing functions for changing between one velocity and another
Ingar: Acceleration isn't a
central thing in this. You could build such functions on top of
the API
... It's not critical, but could be useful
Kumanan: Does velocity have direction?
Ingar: Yes, so negative reverses the playback direction
Mike: Are the units arbitrary?
Ingar: Yes.
Mike: If we want to relate this to a media content pipeline, For example, if you want to double the rate, how do you know what the scale of the timeline is?
Francois: When you associate the
Timing Object with a media element, position would be seconds,
so the units are set there
... We're also thinking about how to compose Timing Objects,
different clocks at different rates
Ingar: There are different
options. You could define a mapping, so we've defined a
TimingConverter
... You'd use this with media objects having different
timelines
... You could also change the data to fit the different
timeline
Kumanan: When I change velocity, how do I know what my normal is?
Mike: I think the question is how the position and velocity relate to the timestamp value in the vector
Kumanan: The timestamp comes from Performance.now, which is local
[ discussion of how timestamp relates to the vector ]
Ingar: An application can
remember what the normal is. The velocity doesn't jitter
... The TimingObject has change and timeupdate events
... When someone changes the TimingObject, all others are
updated
... The TimingObject is in control, and it's the component's
responsibility to respond
Toshihiko: Will the TimingObject cover other use cases, such as a carousel where you want to loop from the end to the start
Ingar: It's quite simple to add
that on top of this
... It's a matter of preference
... A hosted internet service can provide a timing service, so
the local TimingObject becomes a proxy
... There's latency but the state will be correct
... One advantage of a hosted solution is availability
... [ demo of online timing object ]
... Doing this in user space JavaScript, you lose some
precisiion
... We have the TimingObject API for clients, but there's also
an API for timing providers
... This is simpler, as it just has to maintain two variables:
the vector, and the skew
... The main source of error is the step between the
TimingObject and the media element.
Kumanan: If this becomes a proper Web API, would there be a timing provider mandated?
Francois: The default
implementation would use Performance.now
... [discussion of timing providers and skew ]
Ingar: The skew would be that
between Performance.now and the underlying timing source
... There are lots of applications: collaborative viewing,
multi-screen displays, multi-device in the home, showing
interactive content with a main video feed
... Also remote control, music, multi-device capture,
timestamping material according to an app-specific
timeline
... Which makes the content re-playable in a multi-device
setup
... We've done some experiments on sync precision
... We're measuring the difference between the TimingObject and
the media position
... When there are big differences, it's best to skip
position
... When it's closer, adjust the playbackRate
... [ shows graphs of measurements taken on Android, Chrome
]
... We'd like to be able to supply a timestamp to the media
element's play command
... [ graph shows problem with playbackRate on iPad ]
Ingar: Improvements needed with playBackRate, seek
Francois: We may also need to look at Web Animation, you may not know the latency, ie, when the animation will actually start
Ingar: Text tracks are limiting,
we have a sequencer object that uses timing object
... We use setTimeout in the sequencer, which gives precision
of a few milliseconds, but delays if the browser is busy
Francois: Due to the event loop, but could use tasks or micro tasks instead
Kumanan: Wet set limits to the setTimeout interval
Ingar: There's the Second Screen WG
Louay: HbbTV 2.0 has sync,
discovery, communication, app launch in one package
... Similar for Google Cast
... The protocol work in the Second Screen group will touch on
synchronisation
... Should we have one group at W3C looking at this?
Francois: I don't expect the
Second Screen WG to take on this work
... One thing in-scope for the Second Screen CG is reporting of
currentTime for the Remote Playback API
Kumanan: That just needs to address reporting
Francois: I'm trying to gather
interest in the multi-device timing CG
... There have been improvements in the web platform, which
make sync easier, but no specific work on sync itself
Kumanan: I suggest a separate timing spec is valuable more generally than just media applications
Francois: That's good feedback. I would have thought that browser vendors would not fancy supporting a timing object natively if it sits "on the side", meaning if it does not connect to e.g. media elements. That is the reason why we included the connection with media elements in the draft spec.
Kumanan: There are applications such as real time collaboration where this is valuable
Ingar: We want to attract support to the CG
Francois: We could track progress in the existing specs that have timing involved
Kumanan: What's the blocker to moving this forward?
Francois: It needs someone
willing to implement natively
... [ discussion on differences between CGs, WGs, and IGs at
W3C ]
Ingar: [ recaps issues with iOS timing measurements ]
Louay: Do you have any experience of syncing live streams, eg HLS?
Ingar: Yes, i have a demo
Louay: [ discussion of currentTime in live streams ]
Eric: startDate should work with
HLS streams if the information is in the manifest
... [ discussion of the play method that returns a Promise
]