Frame accurate synchronization

François Daoust
@tidoust

Related GitHub issues

Frame accurate seeking of HTML5 MediaElement (#4)
Globally synchronized playback (#5)
Synchronization related use cases for Open Screen Protocol (#21)

Categories of use cases

Frame accurate seeking
Frame accurate rendering (during playback)

Main seeking use cases

Non-linear edition in a browser
Cloud-based media edition controlled from a browser
Collaborative review scenarios where users in multiple locations may annotate media or stop at a precise frame
Evidence playback
“ For the [...] evidence playback [of police body camera video] use cases our customers want the ability to step through a video frame-by-frame. ” – @boushley

Seeking gaps

currentTime is not precise enough to identify individual frames
No way to seek to the next/previous frame in the generic case

Main rendering use cases

Dynamic content insertion (splicing)
Video overlays (includes funny hats, annotations, player tracking, etc.)
Media playback synchronized with map animations
Subtitle rendering synchronized with frame changes
“ even a frame or two of subtitles hanging on the screen after a scene change happens is very much notable and ugly to look at during playback. ” – @Daiz
Synchronization between audio and timed text (e.g. Karaoke)
Synchronized playback across users/devices
Video playback in VR
video uploaded as WebGL texture

Some of them are described in Media Timed Events

Rendering gaps

currentTime is not precise enough to identify individual frames
timestampOffset is not precise enough to identify frame boundaries
Tracking the media timeline with sufficient precision (e.g. 20ms) is hard
Rendering of non media content can't be bound to the rendering of a frame
No way to relate the rendering of a video frame with the local wall clock
No way to tie the rendering of a video frame to the local wall clock
Unclear whether sync can work when playback is remoted (Remote Playback API, Picture-in-picture)

M&E IG Recommendations

“ In order to achieve greater synchronization accuracy between media playback and web content rendered by an application, the time marches on steps in [HTML] should be modified to allow delivery of media timed event start time and end time notifications within 20 milliseconds of their positions on the media timeline.
Additionally, to allow such synchronization to happen at frame boundaries, we recommend introducing a mechanism that would allow a web application to accurately predict, using the user's wall clock, when the next frame will be rendered (e.g., as done in the Web Audio API). ” – Media Timed Events

Rendering gaps that remain

currentTime is not precise enough to identify individual frames
timestampOffset is not precise enough to identify frame boundaries
~~Tracking the media timeline with sufficient precision (e.g. 20ms) is hard~~
~~Rendering of non media content can't be bound to the rendering of a frame~~
~~No way to relate the rendering of a video frame with the local wall clock~~
No way to tie the rendering of a video frame to the local wall clock
Unclear whether sync can work when playback is remoted (Remote Playback API, Picture-in-picture)

Seeking gaps that remain

currentTime is not precise enough to identify individual frames
No way to seek to the next/previous frame in the generic case

Next steps?

Follow up on Media Timed Events recommendations around synchronization? Who?
Write a use cases & requirements document on frame accurate synchronization? Who?
Feed needs back into WHATWG and Media WG? Who?
Provide inputs related to synchronization in upcoming breakout session on Efficient audio/video processing on Wednesday?

References

Thank you!

Questions?