Synchronization Approaches Discussion

Note: let's make this collaborative - please remove inaccuracies, add details, make new suggestions, point out problems, etc. :)

Introduction

This page contrasts and compares a number of approaches towards making sure animations are synchronizable.

Use Cases

1. Starting one or more animations in a script

(from Web Animations#Use cases number 3, in a bit more detail).

An existing timeline fires an event that calls a script. The script loops through a number of iterations, adding new animations. Those animations should be synchronized with the existing timeline.

There are two issues here: (1) some time elapses between the event occurring and the script executing (2) some time elapses between individual animations being added from within the script

Possible solutions

Developers Always Use Absolute Time

In this approach, we remove relative-time primitives like beginElement and allow only absolute-time primitives to be used. For example, in use case 1 above, each animation would need to be explicitly added with a start time that matched the event time extracted from the callback event.

Pause Playback on Event Callbacks

In this approach, playback of animations is paused at event generation time until the event has been handled by JavaScript. Relative primitives like beginElement then use the paused time as their reference time, and requests to get the absolute time also return the paused time.

Once execution of the JavaScript callback has returned, time is restored to its correct value and the timeline is fast-forwarded to ensure that any additional events are recovered.

One criticism of this approach is that it might cause jerkiness in playback. It's worth noting that this is the flash model, though, as jerky playback is not exactly characteristic of flash animations.

Brian's comments:

• One concern here is scalability. This model works fine for an isolated Flash animation. I wonder how it scales once those events join the queue with all the other events generated by a browser. I'm not sure if there's any facility for prioritizing events?
• It also possibly implies user agents know how many listeners they have for these events so they can avoid dispatching them when no-one's listening and avoid unnecessary jank. They possibly also need to know when all those listeners have finished executing so they can un-pause the time.
• In summary, I'm a little unsure about how this integrates with the event dispatch model currently implemented in browsers.
• Also, if you have an event that occurs in the past, do you seek backwards? I assume no. e.g.

<animate begin="b.begin-2s" onbegin="myfunc()" />

• Overall, I'm still interested in this approach. (It seems easy from an implementation point of view too—you don't even need to pause, just stop delivering clock ticks after firing the event, then start delivering them again after the handlers have finished and you automatically get the catch-up behaviour.)

2012-02-24 Brian:

I did some research and then contacted Mozilla's DOM events expert (Olli Pettay). I learnt that the DOM event model already allows for synchronous events such as MutationEvents. (I think Opera might already be dispatching animation events from SVG synchronously, or at least I'm pretty sure it used to.) Olli says:

Dispatching mutation events sync is isn't the problem, but the time some events should fire. If you're mutating DOM, mutation event listeners may change the tree while some mutations are still in process.

Is the same problem with animation? Would we dispatch the events at safe time? If so, dispatching events sync shouldn't have problems. But remember, event listeners can do anything. Rip out the whole DOM, close the window, move element in one document to another etc.

If we can do sync dispatching, it sounds like the easiest approach. The question is where in animation code and when would the events be dispatched.

Pause "Edit Cursor" on Event Callbacks

In this approach, playback of animations is not paused, but an edit cursor which normally tracks the playback cursor is. Relative primitives like beginElement use the position of the edit cursor as their reference time. When script execution terminates, the edit cursor is fast-forwarded back to the play cursor.

Events are generated off the play cursor (so that e.g. syncbase stuff works as expected), but events are not delivered to JavaScript until the edit cursor traverses the time at which the event was generated.

I'm pretty sure that this approach requires a "Timeline as Artifact" approach rather than a "Time Machine" approach to timelines (see below), as otherwise the play cursor could cause changes to the timeline while the script was trying to read or modify it.

One advantage of this approach is that it allows any timeline to be edited from any JavaScript event - the edit cursor simply needs to be frozen first (and this could happen automatically on the first access).

Brian's comments:

• I think maybe I haven't understood the behaviour of the edit cursor exactly. Also, I'm not sure if the "Timeline as Artifact" approach is the enabling factor here since in either model the play cursor can still mutate the timegraph, e.g.

<animate id="a" begin="0s; a.end" dur="0.01s"
  onend="if (i == 5) evt.target.setAttribute('dur', '2s'); else ++i;"/>

(Note the syntax there is almost definitely wrong)

• In the above case, you generate intervals by playing. So in-between firing the end event and running the event handler you'd generate all these 0.01s intervals (remember syncbase timing isn't tied to the event loop). However, when the event handler runs, you'd want to:
  1. act as if those intervals hadn't been generated (so that querying animated values, animation APIs etc. returned the expected results)—this may be the point where I've failed to understand the edit cursor behaviour properly
  2. delete those intervals because after updating the duration they should no longer exist.
• So, in fact, I think you want the Time Machine behaviour here because it will delete that intermediate state. Again, maybe I've misunderstood here.
• I think the Time Machine behaviour is ok so long as you can do backwards seeking in a way that isn't proportional to the elapsed time. I'm not certain yet how feasible that is—it may be that if we simplify syncbase timing it's ok?

Shane's response:

• At least in my head, what happens here is that the syncbase-generated stuff becomes a permanent part of the timeline, because it's declaratively described. So when the onend javascript executes, there would be an infinite number of 0.01 second intervals lined up head-to-tail.
  • yes, this is a problem. Not insurmountable, though - some kind of query-based access to the timeline could ensure we don't need to generate them all up front.
  • when evt.target.setAttribute('dur', '2s') executes then all of the post-target intervals are simply shifted by 1.99s.
• The problem with the Time Machine approach is that the play-head can cause modifications to the stuff that's being edited. With the artifact approach, this can't happen.

Brian:

• Ok, thanks, I think I understand this a bit better now. Maybe the first step in resolving this is to design our declarative synchronization behaviour. A lot hangs out that.
  • A sub-issue there is how to respond to changes to those synchronisation declarations.
  • Doing this may relate to how we approach time containers too—I once proposed time containers as a replacement for SVG's syncbase timing.
• Also, Rik points out that by allowing the play head to continue you may end up showing results you didn't intend to do (e.g. if you event handler deletes the animations or elements).
  • I wonder though, how important is this? It seems like the example Rik showed us should be possible without event handlers, i.e. declaratively. Then you avoid this issue altogether.

Timelines

I think the structure of timelines is pretty essential to the fine details of synchronization, so I want to explore a couple of possible models here.

The "Time Machine" model (SMIL)

A timeline in SMIL is a combination of scheduled animations and animations generated from interactivity (beginElement(), onClick, etc. etc.). When seeking, especially back in time, interactive animations with a start time after the seek point are removed. Brian likens this to a time machine - if you go back in time you can't change what's happened before you arrive, but anything after that is up for grabs.

One disadvantage of this approach is that timelines aren't deterministic - replaying them has different effects depending on seek point. I also had some trouble understanding why some, but not all, interactive animations were removed (until Brian started talking about time machines).

One advantage of this approach is that events generated from scripts don't cause duplicate animation created - the scripts will still fire on replay, but animations that are added by the first play-through will have been removed.

note<birtles>:

One example, I'd offer here for understanding this model is:

<animate id="a" begin="5s" ... />
<animate begin="a.begin" ... />
<animate begin="a.beginEvent" ... />

At t=6s, you seek to t=4s. At that point, if you update the begin time of 'a' to t=8s, both animations should start at t=8s. I know that example doesn't explain everything, but hopefully it helps make sense of why interactive times get cleared.

The "Timeline as Artifact" model

An alternative approach is to treat a timeline as an artifact that can't be changed, except by script. Seeking in a timeline doesn't modify the sequence of events at all, so if an event added an animation interactively to a certain point in the timeline then it would always be there until explicitly removed.

One disadvantage of this approach is that you need to be careful when seeking back in time to manually remove animations generated by scripts, or to write scripts to ensure that they don't add duplicate animations.

One major advantage (in my mind) is that this approach dramatically simplifies timelines - there's only one kind of animation in a timeline (unlike the time machine approach, which has two distinct classes of animation), a timeline always renders the same way for the same time ranges, timelines can be serialized, seeking within a timeline is simpler, etc.

A further consideration is that if we do decide to support a state-machine implementation in our API, and we decide that timelines (rather than elementary animations) should be the structure that attaches to edges in the state machine graph, then this approach is going to be a lot easier to reason about than the time machine approach.

Brian's comments:

• I think from an implementation point of view you still need to distinguish between types of animations because you can't record all times since otherwise you'd continue to consume memory over time. You need to work out which triggers can be re-generated and which need to be logged.
• I think the issue with scripts is worth considering. Arrangements like the following seem pretty common:

<animate ... onend="if (..some temporal condition..) animB.begin()"/>

• In the above arrangement, writing a script to: detect backwards seeks, look up animB, find its intervals and delete those in the appropriate range etc. seems non-trivial, like requiring the script author to implement the Time Machine?
• I think I understand the goal of this model, but I'm getting tripped up on some of the details. I feel like it doesn't actually make backwards seeking much easier, possibly harder for the author.
• I think forwards seeking remains the same. You still have to step through the timegraph because of syncbase arrangements that generate intervals indefinitely. You don't get random-access.
• I think serialization is not much different either. (In either case you can keep a record of all past intervals but you probably don't want to due to the concern of ever increasing memory usage. You can regenerate it though if you need to. And for future intervals you have to step through the timegraph anyway.)
• Also, I'm not sure how we can achieve backwards compatibility with SVG with this approach except by hoping the Time Machine approach is currently so poorly implemented that no one will notice :)

Looking back at my comments I've been too negative. This is so helpful. Thanks so much Shane. I also want to make things simpler. I think the main thing that I'm getting tripped up on is some of the details of syncbase timing, especially cycles of dependencies that generate intervals indefinitely. I think that feature changes a lot of things and it might be worth considering how useful it us.