Web Animations/Meetings/Meeting 8

Time, location: 1 Mar 16:00 PST / 2 Mar 11:00 AEDT / 2 Mar 09:00 JST, #webanimations

Agenda

1. Accessibility - Shane
2. Synchronization discussion
   • Synchronisation with script—Do we need synchronous events? How feasible are they? Do we need groups? etc.
     • Shane and Brian to look into feasibility / integration with DOM event model, RequestAnimationFrame etc.
   • Time Machine / Time as Artefact etc.—stepping back and designing declarative synchronisation first?
3. API / Object primitives
   • Brian to propose alternative API without stacking so we can compare

Actions

• Brian to ask what the basis was for including speed control in SVG Tiny 1.2
• Brian to add timed text track integration to the requirements doc
• Shane to ask about triggers and a11y
• Shane and Brian to find out about how rAF works and if it's possible to prioritise events
• Rik and Brian to comment on concerns with separation of play and edit heads

Log

<birtles> woah, it's web animations time!
<birtles> who's here?
<shans__> me!
<birtles> yay!
<birtles> cabanier, how about you?
<birtles> shans__, by the way, did you book Tokyo yet?
<shans__> birtles: not yet, but I'll be doing that pretty soon. Is there an event on or something?
<birtles> like a port festival or soccer final? (as per hamburg and bulgaria)
<shans__> yeah :)
<shans__> I often book my travel pretty late
<birtles> not that I'm aware of
<cabanier> I'm here!
<shans__> let's get started then
<birtles> I'm not sure if Tokyo ever really books out---just transport gets messy in golden week, new year etc.
<birtles> ok
<birtles> the agenda so far:
<birtles> 1) Accessibility
<birtles> 2) Synchronization discussion
<birtles> 3) API / Object primitives
<birtles> http://www.w3.org/Graphics/fx/wiki/Web_Animations/Meetings/Meeting_8
<birtles> any changes?
<cabanier> no
<shans__> looks good
<birtles> alright then, Shane, tell us about accessibility
<shans__> ok
<shans__> so what it all boils down to is that there's a single object in the DOM we need to keep in mind
<shans__> object type rather
<shans__> Timed Text or something like that
<shans__> so basically all the different subtitle / a11y text formats (and there are hundreds), if they're to be used in the browser, will map into Timed Text objects in the DOM
<shans__> and I think we need to consider them in 2 different ways
<shans__> (1) they're currently being used to carry video/audio-aligned snippets of data, not just for a11y, but also for animations / layout changes / etc.
<shans__> and
<shans__> (2) we should support TT timelines being aligned with our animations, which would then enable our animations to be made accessible
<birtles> sounds good
<shans__> so that's basically it :)
<shans__> any questions?
<birtles> does that cover the subtitling use case as well?
<shans__> yeah pretty much
<shans__> well, half of it
<shans__> if we've got an animation that needs subtitles
<shans__> then the Timed Text objects will carry the subtitle information
<cabanier> is this timed text object used by the accessiblity software?
<shans__> but we'd still need to work out a way to make it convenient for that information to be presented
<shans__> I believe that is the intention, yeah
<birtles> ok, sounds good
<birtles> I'll add that to the requirements list
<shans__> but I'm slightly worried that I've got the name wrong - I'll check again with Silvia
<shans__> great
<birtles> on that note, I added a few other minor ideas about accessibility and animation: http://www.w3.org/Graphics/fx/wiki/Web_Animations/Requirements#Accessibility
<birtles> stuff like role identifiers for animation triggers
<cabanier> so, every browser is exposing this interface somehow
<shans__> well, the interface is specified by the w3c
<shans__> (which isn't quite the same thing, I know)
<shans__> I'm not sure what browser support there is
<shans__> well, I think WebKit supports it
<birtles> is there a link for the DOM part of it?
<shans__> that's what I'm trying to find
<shans__> unfortunately Silvia's on her honeymoon this week
<birtles> I've found heaps of links for the markup (TTML) but not the interface
<shans__> but I can ask her next week
<cabanier> What about non-subtitle case?
<cabanier> I know for our software, people want to hover over part of a pdf or flash file and get a spoken description of what it is
<cabanier> or is that out of scope?
<shans__> right, but that's not animations
<cabanier> OK
<shans__> I think that's important, and I think there's a mechanism in HTML to allow that
<shans__> what animations brings that is new is the fact that the information being presented is time-dependent
<shans__> so the description needs to be time-dependent too
<birtles> so you might have a variation on subtitles that has descriptions along the way
<birtles> which would be covered by timed text
<shans__> yeah
<birtles> like on some DVDs they have the explanatory English track
<shans__> and like I mentioned before, people are using the timed text for all sorts of things
<birtles> which describes what's going on on the screen
<shans__> for example, CSS styles that should apply to the page at different points in the video
<cabanier> but you wouldn't do it per character
<shans__> (that needs JavaScript to work, of course, but the TT provides the alignment mechanism)
<cabanier> ie person a is playing, person b is now cheering, etc
<cabanier> OK
<birtles> ok, one other possible requirement...
<birtles> I think the SVG WG had a requirement for speed control on animations (and video/audio) due to accessibility
<birtles> for SVG Tiny 1.2
<shans__> there's a track on some video content specifically for the hard-of-hearing. Is that what you're talking about brian?
<birtles> shans__, no there's sometimes a track for visually impaired people
<shans__> it's like subtitles plus [loud explosion] .. [doorbell rings], etc.
<shans__> ah, right. At any rate, all of those things are meant to map back to this primitive
<birtles> it tells you "Now the logo for the production company is appearing on the screen"
<birtles> "Now descending on a 1920s bustling street"
<birtles> etc.
<birtles> so anyway, I think speed control might be another requirement but I'm not sure exactly who the customer is for that
<cabanier> if you want to slow things down, you probably want to slow everything down. So it could be a global browser switch and not something per animation
<shans__> I think it has to be a global browser thing
<birtles> perhaps I ought to just ask those involved
<shans__> otherwise complex animations would break badly
<birtles> in SVG Tiny 1.2 it's on a per <animation> level from memory
<cabanier> that makes it a lot easier
<cabanier> that wouldn't work I think
<birtles> ACTION Brian to ask what the basis was for including speed control in SVG Tiny 1.2
<shans__> http://www.whatwg.org/specs/web-apps/current-work/#timed-text-tracks
<shans__> that's a link to the web-facing API for Timed Text tracks
<shans__> plus other stuff
<birtles> ACTION Brian to add timed text track integration to the requirements doc
<birtles> ok, well that probably feeds into however we approach synchronisation
<birtles> where there any other suggestions for what accessibility features might be needed?
<shans__> I think it happens at a higher level, actually
<shans__> Silvia at least was reasonably dismissive of there being too many a11y concerns for animation
<shans__> but I think she was considering it from the CSS Transitions POV - i.e. really basic layout animations
<shans__> I need to talk to Alice too but I haven't had a chance to yet
<birtles> well, I think having declarative animation triggers is one thing that would aid accessibility
<shans__> dismissive is the wrong word
<birtles> so you can support alternative input methods
<shans__> interesting, yeah.
<birtles> there are a few other things
<birtles> like support role identifiers
<shans__> I've been thinking about the trigger area as being outside web animations largely though
<shans__> same with role identifiers
<shans__> like, if you've got a div that animates between different states in response to user input, then you should put aria roles on the div
<shans__> (but you should anyway, in the absence of animation)
<shans__> or am I missing something?
<birtles> I'm thinking about stuff like <animate begin="circle.click; accessKey('a')">
<shans__> well the thing is in that case the circle is a button
<shans__> and should be marked as such
<shans__> which means that the existing a11y mechanisms (tab order inclusion, trigger-via-enter, etc.) just work
<shans__> if you also want to support hotKeys, I think that's a pretty cool feature, but not an a11y one specifically
<birtles> there's already hotkey support in SVG--but one of the open requirements we have is about extending that
<birtles> Vincent was keen to see DOM3 key identifiers supported
<birtles> and in doing so we just need to be aware of making sure that information is available to accessibility agents
<birtles> anyway, we can talk about the details later
<birtles> perhaps we can move on to synchronisation
<shans__> yep. Actually I see what you mean now. If someone *does* specify 'a' as a trigger, then how can we let the a11y agents know that this trigger is available and (e.g.) speak it to the user
<birtles> yeah, or, if they are using an input device that doesn't have 'a', map it to something suitable
<shans__> right
<shans__> not sure if that's currently even possible, outside of animations
<shans__> I'll ask about it
<birtles> ok, cheers
<shans__> ACTION Shane to ask about triggers and a11y
<birtles> ok, synchronisation?
<shans__> yep
<birtles> oh, by the way I have to go in 30min
<shans__> I probably do too
<birtles> ok
<cabanier> OK
<birtles> I did a little research
<birtles> It turns out there are already synchronous events in some specs
<shans__> oh?
<birtles> e.g. MutationEvents (which everyone hates and is now deprecated)
<shans__> haha
<shans__> yes
<shans__> well, actually the thing is
<shans__> *all* events are synchronous by the time they get to JavaScript
<birtles> right
<birtles> but the dispatch is synchronous for MutationEvents
<shans__> so what you're talking about specifically is called-on-mutation rather than called-after-mutation, right?
<cabanier> :-)
<shans__> heh
<birtles> yeah, it's the "dispatch" that's synchronous
<shans__> ok great
<birtles> I spoke to our DOM events expert about using that kind of event for animation
<shans__> did they run away screaming?
<cabanier> you mean: you change something in the DOM in JS and this event is called immediatly?
<birtles> he seemed to think it's ok as long as you're careful about when you do the dispatch
<cabanier> that would be ugly
<shans__> oh that's a good sign :)
<birtles> (i.e. you make sure you're in a safe state and don't make any assumptions about the state of the world when each event handler returns)
<shans__> Rik - not sure about that
<birtles> I think Opera already does this for SVG animation events
<birtles> but I also spoke to David Baron about it and he was less enthusiastic
<shans__> so there are two reasons that everyone hates DOM MutationEvents
<shans__> (1) the programming model is hard for common tasks
<shans__> (2) including the events slowed down DOM manipulations by ~ 10x, even after optimisation
<birtles> yeah, the performance is terrible
<cabanier> that makes sense
<birtles> I think the Skype plugin was particularly infamous for this
<shans__> I think your DOM expert was saying that (1) probably wouldn't apply to the animations case, right?
<birtles> yeah, I don't think the programming model is hard, easier if anything
<birtles> (2) is really where David Baron was concerned
<birtles> can I give an example?
<cabanier> yes
<birtles> so if we have an animation that finishes at t=5s
<birtles> and our next sample is at t=5.5s
<birtles> in Gecko's implementation of SVG animation (but not CSS) we'll actually force a sample at t=5s
<birtles> a "sample" here just involves updating the animation model
<birtles> it doesn't render or anything like that
<birtles> it's cheap
<birtles> but it allows synchronisation to work as expected
<birtles> then we'll go on and do another sample at t=5.5s
<birtles> and *then* we'll render with the final values
<birtles> IF, however, we decide to synchronously dispatch an end event at t=5s...
<birtles> that handler could do anything
<birtles> including causing a reflow
<birtles> that's fine
<birtles> but if this sort of situation comes up a lot
<birtles> you could end up getting a lot of reflows and other expensive operations happenning at a frequency greater than the refresh rate
<birtles> (i.e. sample rate)
<birtles> and that's what David was concerned about
<cabanier> that is true
<birtles> poorly written web apps producing excessive reflows etc.
<cabanier> but considered 'bad programming'
<cabanier> I guess you could detect that and abort
<birtles> right, but there's a lot of bad programming on the web
<shans__> the thing about DOM layout was that the 10x slowdown occurs *even if you do nothing in the handler*
<birtles> cabanier, ok, that's an idea
<shans__> it's a consequence of just registering the handler
<birtles> shans__, yeah, that's right
<cabanier> the handlers should not execute that often
<cabanier> and doesn't requestanimationframe suffer the same issue?
<birtles> the mutation event stuff is expensive partly because you're supposed to supply "new" and "old" values
<birtles> (and event dispatch in Gecko is expensive, or at least it used to be)
<birtles> requestanimationframe only runs at the refresh rate
<shans__> well we should look at what would need to be provided in an animation event too then
<birtles> so you can only cause a reflow at the same frequency as the refresh rate
<birtles> not more frequently
<cabanier> true
<shans__> we were talking about whether requestAnimationFrame is more privileged than other event handlers the other day. Did anyone discover if it is?
<shans__> because I think if it isn't, that kinda sinks us being able to make animation events more privileged.
<birtles> what do you mean (/ "what did we mean") about more privileged?
<shans__> well, if I've got a bunch of outstanding events (e.g. DOM mutation events) and a rAF handler registered, is it guaranteed that the rAF handler gets called first?
<birtles> hmm, not sure
<cabanier> does it  matter?
<shans__> yeah
<birtles> if we go with "synchronous dispatch" then the handler *will* be called first
<shans__> if we don't do that for animation events then we have a situation where animation progress can be delayed by *other* event handlers too
<cabanier> you mean, someone presses a button at the same time and handler is invoked?
<shans__> Brian, that's not necessarily true.
<birtles> how so?
<shans__> ok, what happens in this situation:
<shans__> (1) a key press occurs, causing a KeyEvent to be enqueued. JS has not yet been called.
<shans__> (2) another key press occurs, causing another KeyEvent to be enqueued.
<shans__> (3) the first key press KeyEvent is delivered and JavaScript modifies the DOM. A DOMEvent is generated.
<shans__> which event is delivered first?
<birtles> I can't see the relevance?
<birtles> what is the final DOMEevent?
<birtles> s/DOMEevent/DOMEvent/
<cabanier> is the even in 3 one of the mutationevents?
<shans__> DOM Mutation of some kind
<birtles> then it will be delivered first
<cabanier> well, then 2) will be last
<birtles> right
<shans__> hmm
<shans__> does 1) finish executing before (2) is called?
<shans__> sorry
<shans__> does (1) finish executing before (3) is delivered?
<cabanier> (if they're synchronous like brian said)
<birtles> no
<shans__> then why is (3) delivered before (2)?
<birtles> it's synchronous, it's like a function call
<shans__> but it isn't
<shans__> the mutation occurred partway through (1)
<shans__> and it finishes executing
<shans__> so there is still a delay
<birtles> hang on, sorry, I misread your description
<birtles> (1) finished a long time ago
<birtles> so did (2)
<shans__> no
<shans__> the timeline is this:
<shans__> KeyPress (1) --- KeyPress (2) --- KeyPress Event handler (1) { ------ DOM Mutation *A* --------- } *B*
<shans__> now at *A* do we have DOM Mutation hander { ---- }, or nothing
<shans__> and at *B* do we have DOM Mutation handler { ---- } KeyPress Event handler (2) { ---- }
<shans__> or KeyPress Event handler (2) { ---- } DOM Mutation handler { ---- }
<shans__> or just KeyPress Event handler (2) { ---- }
<shans__> ?
<birtles> sorry, I don't quite understand the diagram
<cabanier> It sounds like the mutation events are generated AND dispatched when you change an attribute. You did't leave the keyevent handler
<shans__> time 0 is at the left
<shans__> things inside { } are JavaScript code executing
<shans__> yeah it's a lame diagram. Wish we had a whiteboard :)
<birtles> yeah, I was just thinking "can't wait until we meet face to face"!
<birtles> ok, can I try...
<birtles> 1. key press 1
<birtles> 2. key press 2
<birtles> 3. event handler from key press 1
<birtles> 4. event handler modifies something
<birtles> 5. mutation event handler is run
<birtles> 6. mutation event handler finishes
<birtles> 7. control returns to event handler from key press 1
<birtles> 8. event handler from key press 1 finishes
<birtles> 9.  event handler from key press 2 executes
<cabanier> yes
<shans__> ok
<cabanier> we definitely don't want this for the keyframe events
<shans__> agree
<birtles> I'm not saying this is great... but this is one path we "could" take
<birtles> originally this is how I implement the animations events for SVG
<shans__> it's very easy to end up in infinite modification loops with this approach
<cabanier> yes
<birtles> but you have to be really careful about the state of the world when you dispatch the events
<cabanier> and it's very confusing to the author
<birtles> so I changed it to async
<birtles> but at that time Opera was doing sync (from my testing)
<birtles> I thought this was closer to what Rik was looking for though?
<shans__> the new DOM Mutation model btw is to push DOM changes into a list
<shans__> then deliver the list to a listener some time later
<shans__> so you'll end up with
<shans__> 1. key press 1
<shans__> 2. key press 2
<shans__> 3. event handler from key press 1
<cabanier> You always want to return to the JS VM
<shans__> 4. event handler modifies something
<shans__> 5. event handler from key press 1 finishes
<shans__> 6. event handler from key press 2 executes
<shans__> 7. log of DOM changes delivered to mutation event handler
<cabanier> that is reasonable
<birtles> yeah
<birtles> hmm, I'm not sure we've solved this one yet, but at least we've got a better idea of the options and issues
<shans__> so if we do that approach for animation events
<shans__> then we definitely have a situation where an animation can be paused waiting on resolution of unrelated handlers
<shans__> (if we pause the animation on event generation)
<shans__> 1. keypress 1
<birtles> gotcha
<shans__> 2. keypress 2
<shans__> 3. animation event occurs. Animation paused
<shans__> 4. key press handler 1 executes
<shans__> 5. key press handler 2 executes
<shans__> 6. animation event handler executes. Animation unpaused
<cabanier> ah. 3 and 6 should happen at the same time though
<cabanier> there should be no delay
<cabanier> (or as short of a delay as possible)
<shans__> I don't think we can support that model. It's one of the reasons why DOM mutations were hated
<birtles> Shane's talking about if we don't have "synchronous dispatch" OR some sort of prioritisation for animation events
<shans__> yes
<birtles> so there's one option: async dispatch + pausing + prioritisation of animation events?
<birtles> if that's even possible
<birtles> as far as I know, there's no facility for prioritisation so far
<shans__> right, I think the only two viable options are that one
<shans__> and the edit / play head separation approach
<shans__> there might be others of course :)
<birtles> well, we've got to wrap it up soon
<cabanier> I don't think the seperation approach will work
<birtles> I'm inclined to agree although I'm not 100% sure the sync dispatch option is unworkable
<birtles> I'm also hesitant about the separation approach
<shans__> I think this is something we're going to end up talking about at our f2f
<cabanier> it seems that it could tie in with the requestanimaitonframe
<cabanier> yes
<birtles> yes, I agree
<shans__> it's not yet clear that requestAnimationFrame actually has priority
<birtles> maybe we can continue it then
<birtles> but in the meantime it would be worth researching if there is the possibility of prioritising events
<shans__> it's not even guaranteed on mobile firefox (e.g.) that rAF will complete before rendering of the associated frame occurs
<shans__> yeah I think we need to know what rAF actually does
<birtles> ok, well, Shane, perhaps we can ask our respective contacts?
<shans__> and I'd really like to see as many concrete objections to separation of play and edit heads as possible, written down somewhere so we capture them all
<shans__> birtles: sounds good
<birtles> ACTION Shane and Brian to find out about how rAF works and if it's possible to prioritise events
<birtles> ACTION Brian and Rik to comment on concerns with separation of play and edit heads
<birtles> next meeting?
<shans__> next Monday/Tuesday?
<cabanier> I might be able to do Monday
<cabanier> but I might be a bit late
<birtles> can do---not sure how much progress I'll make on my actions by then however (since I'm away on Monday)
<birtles> alright, should we delay it 30min?
<birtles> i.e. 4:30pm, 11.30am, 9.30am
<cabanier> OK!
<shans__> sounds good
<birtles> great, thanks guys!
<shans__> we can talk about the Animation API then
<birtles> ok, we can prioritise that if you like
<birtles> see you next week!
<birtles> have a good weekend!
<shans__> kk
<shans__> you too