This is an archived snapshot of W3C's public bugzilla bug tracker, decommissioned in April 2019. Please see the home page for more details.

Bug 12267 - <video> Make video state transitions happen in the same task as firing events
Summary: <video> Make video state transitions happen in the same task as firing events
Status: RESOLVED WONTFIX
Alias: None
Product: HTML WG
Classification: Unclassified
Component: LC1 HTML5 spec (show other bugs)
Version: unspecified
Hardware: PC All
: P2 normal
Target Milestone: ---
Assignee: Ian 'Hixie' Hickson
QA Contact: HTML WG Bugzilla archive list
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: 12175 12541 12556 12608
  Show dependency treegraph
 
Reported: 2011-03-08 15:46 UTC by Simon Pieters
Modified: 2011-09-21 21:18 UTC (History)
15 users (show)

See Also:


Attachments

Description Simon Pieters 2011-03-08 15:46:08 UTC
A major hurdle with the video spec is that state transitions (readyState/networkState) happen sync (and even while script is running, except in Firefox which keeps a snapshot of the states, which is better but still not perfect) while events are put on the event queue (i.e. get fired later). The WebSocket API spec OTOH keeps state transitions and events in sync by putting both on the event queue as a single task, e.g.

"When the WebSocket connection is established, the user agent must queue a task to first change the readyState attribute's value to OPEN (1); apply the cookies that were collected in the list of cookies when the connection was established; change the protocol attribute's value to the selected WebSocket subprotocol, if there is one; and then fire a simple event named open at the WebSocket object."
http://www.whatwg.org/specs/web-apps/current-work/complete/network.html#feedback-from-the-protocol

That example applies cookies etc as well in the same task.

The video spec should be overhauled so that state transitions and events happen at the same time, so that things are predictable, testable and easier to author against.

If we do this, we also need to freeze *all* state to keep things in sync. If other properties are still allowed to change while the script is running, weirdness ensues:

* ended == false even though readyState is still in HAVE_ENOUGH_DATA

* currentTime == duration even though readyState is still in HAVE_ENOUGH_DATA

* currentTime could increase beyond 0 while readyState is still HAVE_METADATA

* the two above three issues but observed though played

* buffered is changing even though networkState is NETWORK_IDLE

In short, if not all properties are frozen, they can become internally inconsistent as seen from scripts. Freezing one property would require freezing them all instantaneously, or they could stil become inconsistent. Freezing all of them means always calculating the buffered array even though it may not be used, which is a bit wasteful.
 
Overall, though, I think that we should move in the direction of freezing the properties for scripts, as what we have now is so full of race conditions it's not even funny.
Comment 1 Philip Jägenstedt 2011-03-08 16:53:22 UTC
(In reply to comment #0)
> * ended == false even though readyState is still in HAVE_ENOUGH_DATA

Oops, this should be ended == true
Comment 2 David Singer 2011-03-08 19:20:33 UTC
There seem to be two problems here:
(a) a script handling an event may be reacting to an 'old' state and (b) when testing, it's hard to verify behavior if it's time-dependent and not predictable.

We can't freeze the video while we process events; I assume that's not what is discussed here, but obviously actually pausing is not an option.

We could 'snapshot' the state as it was at the time of an event, but that just moves the problem down into the engine. That is, we've told the event/scripting layer a lie about the true current state, and so when it says "given that the state is X I want Y to happen" the state is no longer actually X, so should Y happen or not?

On testing, it may be that we can introduce some testing hooks to alleviate this problem, but I am not sure what they are. I don't think we should distort normal operation in order to make it testable, however.
Comment 3 Philip Jägenstedt 2011-03-08 20:11:12 UTC
(In reply to comment #2)
> There seem to be two problems here:
> (a) a script handling an event may be reacting to an 'old' state and

I don't consider this a problem, but if it were then making the suggested change would make things worse, as the event handler couldn't inspect readyState/networkState to learn the "real" state.

> (b) when
> testing, it's hard to verify behavior if it's time-dependent and not
> predictable.

Correct. More importantly, it's very easy to write code that is subject to race conditions if one is not very aware of the issues (and virtual no one except those who have implemented/done QA on the spec is).

> We can't freeze the video while we process events; I assume that's not what is
> discussed here, but obviously actually pausing is not an option.

That's not what's being suggested, no.

> We could 'snapshot' the state as it was at the time of an event, but that just
> moves the problem down into the engine. That is, we've told the event/scripting
> layer a lie about the true current state, and so when it says "given that the
> state is X I want Y to happen" the state is no longer actually X, so should Y
> happen or not?

This is already the case today, isn't it? When for example the loadstart event handler runs, it could be that readyState is already HAVE_FUTURE_DATA and videoWidth and videoHeight are known. However, this is a race condition, so sometimes it will be true and sometimes not.

Do you have any examples of state X and action Y which could be problematic?

> On testing, it may be that we can introduce some testing hooks to alleviate
> this problem, but I am not sure what they are. I don't think we should distort
> normal operation in order to make it testable, however.

IMO testing is the smaller issue here, the biggest problem is that web authors can easily write code that breaks randomly depending on network speed, how fast the decoding pipe line is set up, etc.
Comment 4 David Singer 2011-03-08 20:16:45 UTC
I think we need some examples of events and consequent actions, where the requested action is no longer possible or appropriate because of a race condition. Examples, anyone?
Comment 5 Philip Jägenstedt 2011-03-09 08:16:00 UTC
(In reply to comment #4)
> I think we need some examples of events and consequent actions, where the
> requested action is no longer possible or appropriate because of a race
> condition. Examples, anyone?

The problem isn't that the action isn't appropriate anymore, but that it depends on chance if it is taken or not. Take this code for example:

var v = document.createElement('video');
v.onloadstart = function() {
  v.currentTime = 10;
}
v.src = 'video.webm';

When the loadstart event is fired, it currently depends on the speed of the network and decoding if readyState has increased past HAVE_NOTHING or not. If it has, then the video will seek to offset 10 seconds. If it has not, an exception will be thrown.

People *will* come to depend on the most common outcome of race conditions like this and have their scripts fail randomly when the outcome is something else.
Comment 6 Philip Jägenstedt 2011-03-09 08:38:51 UTC
To give some background on the state of things, I argued for making abort, loadend and emptied async in <http://www.w3.org/Bugs/Public/show_bug.cgi?id=7843>, and got my way.

We absolutely do not want to go back to firing any events synchronously, but the model of the WebSockets spec where the state is also updated async is quite appealing to me.
Comment 7 Eric Carlson 2011-03-10 16:06:05 UTC
  Programming with media is hard, shrug, but I don't think pretending otherwise will help. There is no way to paper over the fact that state *does* change asynchronously so we should help developers understand this and deal with it. 

  An extreme example is that of a video which is playing when a modal dialog is shown, playback continues and events are queued but not delivered. Scripts that run when the dialog is dismissed will see the state of the movie at the time the dialog was shown - how can this be helpful?

> In short, if not all properties are frozen, they can become internally
> inconsistent as seen from scripts. Freezing one property would require freezing
> them all instantaneously, or they could stil become inconsistent. Freezing all
> of them means always calculating the buffered array even though it may not be
> used, which is a bit wasteful.
> 
  Do you propose to "freeze" the currentTime property too? This will lead to very strange behavior when the time jumps around relative to the wall clock, eg. it will be pinned when an event is pending and will instantaneously catch up the once the last event is posted. Imagine a plot of the delta between the wall clock and currentTime during normal playback.
Comment 8 Philip Jägenstedt 2011-03-10 16:38:21 UTC
(In reply to comment #7)
>   Programming with media is hard, shrug, but I don't think pretending otherwise
> will help. There is no way to paper over the fact that state *does* change
> asynchronously so we should help developers understand this and deal with it. 
> 
>   An extreme example is that of a video which is playing when a modal dialog is
> shown, playback continues and events are queued but not delivered. Scripts that
> run when the dialog is dismissed will see the state of the movie at the time
> the dialog was shown - how can this be helpful?

It's not meant to be helpful or solve a particular use case, the suggested change is to eliminate some race conditions to make scripting against <video> a bit more reliable. That doesn't really change things substantially from where we are today, as the state as seen from scripts can already become out of sync with reality the instance after it is checked.

To turn it around, how could it be helpful for scripts see the state change while they are running?

The fact that we have events for all interesting readyState transitions makes readyState rather uninteresting for scripts, what I hope to avoid is accidental dependency on de-facto implementation behavior for scripts that *do* check readyState or do things that indirectly check it, notably setting currentTime in the loadstart event handler.

> > In short, if not all properties are frozen, they can become internally
> > inconsistent as seen from scripts. Freezing one property would require freezing
> > them all instantaneously, or they could stil become inconsistent. Freezing all
> > of them means always calculating the buffered array even though it may not be
> > used, which is a bit wasteful.
> > 
>   Do you propose to "freeze" the currentTime property too? This will lead to
> very strange behavior when the time jumps around relative to the wall clock,
> eg. it will be pinned when an event is pending and will instantaneously catch
> up the once the last event is posted. Imagine a plot of the delta between the
> wall clock and currentTime during normal playback.

As I understand, Firefox already does this. I don't have a strong opinion about it, but there's certainly something to be said for internal consistency and freezing also currentTime. The accuracy of currentTime would be unchanged at the beginning of an event handler, so I'm not sure it matter that it would become a bigger and bigger lie the longer the script runs. Is there a use case for observing changes in currentTime while scripts are running?
Comment 9 David Singer 2011-03-10 20:17:57 UTC
On the example problem:
v.onloadstart = function() {
  v.currentTime = 10;
}
I would say that this is an *obvious* bug -- the action of a function should only depend on what's assured by the event that the function runs in response to. Loadstart does not assure seekability.

Now, if this were true, but somehow the state regressed between the event firing and the function running (perhaps something else changed the URL on the video element, for example), then that's more like the races that can happen.

Freezing is not appropriate, as we all recognize; and snapshot means that scripts are using state/values that are old, and then asking for action based on that. is currentTime one of the snapshotted values?  If so, is timeofday also snapshotted, so we can see how well we are playing in real-time? And so on...
Comment 10 Philip Jägenstedt 2011-03-10 22:12:04 UTC
(In reply to comment #9)
> On the example problem:
> v.onloadstart = function() {
>   v.currentTime = 10;
> }
> I would say that this is an *obvious* bug -- the action of a function should
> only depend on what's assured by the event that the function runs in response
> to. Loadstart does not assure seekability.

That it's an obvious bug doesn't change the fact that the behavior is unreliable. Unless the spec defines the behavior, web authors will inevitably come to depend on the most common behavior in the browsers they test in, because they develop by trial and error. Do you not think that this is a real risk?

> Now, if this were true, but somehow the state regressed between the event
> firing and the function running (perhaps something else changed the URL on the
> video element, for example), then that's more like the races that can happen.
> 
> Freezing is not appropriate, as we all recognize; and snapshot means that
> scripts are using state/values that are old, and then asking for action based
> on that. is currentTime one of the snapshotted values?  If so, is timeofday
> also snapshotted, so we can see how well we are playing in real-time? And so
> on...

The initial scope of this bug was to make the transitions of readyState and networkState happen in the same task that fires the related event. Do you find this objectionable? If so, what concrete problems would it cause?

The best behavior of currentTime isn't obvious, I'll agree thus far. Are there any use cases for scripts observing currentTime changes while they are running? If not, what concrete problems would freezing it (like Firefox, presumably) cause?
Comment 11 Eric Carlson 2011-03-10 22:55:23 UTC
(In reply to comment #10)
> The initial scope of this bug was to make the transitions of readyState and
> networkState happen in the same task that fires the related event. 
> 

Comment #0 says "*all*" properties should be frozen while events are pending:

    The video spec should be overhauled so that state transitions and events happen
    at the same time, so that things are predictable, testable and easier to author
    against.
    
    If we do this, we also need to freeze *all* state to keep things in sync. If
    other properties are still allowed to change while the script is running,
    weirdness ensues:
    
[SNIP]
    
    In short, if not all properties are frozen, they can become internally
    inconsistent as seen from scripts. Freezing one property would require freezing
    them all instantaneously, or they could stil become inconsistent. Freezing all
    of them means always calculating the buffered array even though it may not be
    used, which is a bit wasteful.
Comment 12 Philip Jägenstedt 2011-03-11 07:16:44 UTC
(In reply to comment #11)
> (In reply to comment #10)
> > The initial scope of this bug was to make the transitions of readyState and
> > networkState happen in the same task that fires the related event. 
> > 
> 
> Comment #0 says "*all*" properties should be frozen while events are pending:

Oops, I was reading the title of the bug when I wrote that. Still, I don't think it's a given that we must freeze everything, so let's discuss the actual pros and cons.

Starting at readyState and networkState, do you at all agree that what we have now is problematic? Do you think see a problem with making "video state transitions happen in the same task as firing events"?
Comment 13 Eric Carlson 2011-04-20 15:49:47 UTC
(In reply to comment #10)
> (In reply to comment #9)
> > Freezing is not appropriate, as we all recognize; and snapshot means that
> > scripts are using state/values that are old, and then asking for action based
> > on that. is currentTime one of the snapshotted values?  If so, is timeofday
> > also snapshotted, so we can see how well we are playing in real-time? And so
> > on...
> 
> The best behavior of currentTime isn't obvious, I'll agree thus far. Are there
> any use cases for scripts observing currentTime changes while they are running?

A script that displays captions, subtitles, slides, etc, in sync with a playing video? Keeping relatively accurate sync is hard enough, why make it worse?


> If not, what concrete problems would freezing it (like Firefox, presumably)
> cause?

What concrete problems would freezing currentTime solve?
Comment 14 Philip Jägenstedt 2011-04-20 16:15:03 UTC
(In reply to comment #13)
> (In reply to comment #10)
> > (In reply to comment #9)
> > > Freezing is not appropriate, as we all recognize; and snapshot means that
> > > scripts are using state/values that are old, and then asking for action based
> > > on that. is currentTime one of the snapshotted values?  If so, is timeofday
> > > also snapshotted, so we can see how well we are playing in real-time? And so
> > > on...
> > 
> > The best behavior of currentTime isn't obvious, I'll agree thus far. Are there
> > any use cases for scripts observing currentTime changes while they are running?
> 
> A script that displays captions, subtitles, slides, etc, in sync with a playing
> video? Keeping relatively accurate sync is hard enough, why make it worse?

Such scripts would have to "yield" by returning from their setTimeout or event handler to have any visible side-effects. You can't have a never-returning script observing changes in currentTime that doing anything useful, really. (AFAIK, only Opera actually has an interruptible script engine, so in any browser it would just be a broken script.)

Sure, if a script runs for a very long time before checking currentTime, then there would be sync issues. However, a browser could choose to freeze the HTMLMediaElement state when it is first accessed in each task, so I doubt it would be a practical problem. Also, with the addCue API, authors will be able to depend on the browser for timing instead.

> > If not, what concrete problems would freezing it (like Firefox, presumably)
> > cause?
> 
> What concrete problems would freezing currentTime solve?

It guarantees consistency between all the different states as observed by scripts. That makes writing test cases more reliable, but more importantly I assume we will see pages accidentally breaking because of this sort of thing if we allow things to be "random".

If we *don't* freeze currentTime, then the spec should really define what it is scripts would observe by repeatedly reading the property. How often should it be updated, at a minimum?

I'm not in love with the idea of giving myself more work by coercing the script-visible state in all kinds of ways, but the alternative seems worse, after all.
Comment 15 Silvia Pfeiffer 2011-04-21 02:21:32 UTC
As a developer I just know that when I take a peek at the readyState, it will be merely more than a Heisenberg measurement.

I am struggling to see the race conditions that are being used as a reason to make changes to state transitions here. Can you provide an example where the developer would get the wrong information? Or would get into an unsolvable situation?

Right now, I just make up for the uncertainty by allowing for time ranges rather than time points.
Comment 16 David Singer 2011-04-25 23:26:55 UTC
I really really think that trying to hide the fact that, when programming time-based media and time-based behavior, you are "standing on a moving conveyor", would be a mistake.  Highlight in the spec. that clients of the API should expect that things may be changing under their feet, sure.  As I said, you can't stop the world while the scripts think (e.g. freeze the video), and snapshotting the world is just telling you a truth about the past, not the present.  If you snapshot, you have to buffer all the actions a script takes until all the scripts stop running, for example, so that they don't affect the snapshot.  Now what do you do if the actions can't all be applied?  It's too late to give an error response. Ugh.
Comment 17 Philip Jägenstedt 2011-04-26 08:31:48 UTC
(In reply to comment #15)
> As a developer I just know that when I take a peek at the readyState, it will
> be merely more than a Heisenberg measurement.

I think you meant "as the author of The Definitive Guide to HTML5 Video" ;-) It's enough that 1% of web authors make a mistake and create scripts with race conditions for this to be a serious problem, causing browsers with slightly different timing to appear more broken than the ones that were tested.

> I am struggling to see the race conditions that are being used as a reason to
> make changes to state transitions here. Can you provide an example where the
> developer would get the wrong information? Or would get into an unsolvable
> situation?

It's almost always possible to work around the race conditions if you know they're there. I listed some examples of problems that less-than-perfect programmers could run into in <http://lists.w3.org/Archives/Public/public-html-a11y/2011Apr/0232.html>.
Comment 18 Philip Jägenstedt 2011-04-26 08:39:26 UTC
(In reply to comment #16)
> I really really think that trying to hide the fact that, when programming
> time-based media and time-based behavior, you are "standing on a moving
> conveyor", would be a mistake.  Highlight in the spec. that clients of the API
> should expect that things may be changing under their feet, sure.  As I said,
> you can't stop the world while the scripts think (e.g. freeze the video), and
> snapshotting the world is just telling you a truth about the past, not the
> present.

Is there any use case for observing these changes while the script is running? Firefox already does some kind of snapshotting, is there anything (reasonable) you cannot do in Firefox that works in e.g. Opera or Safari?

> If you snapshot, you have to buffer all the actions a script takes
> until all the scripts stop running, for example, so that they don't affect the
> snapshot.  Now what do you do if the actions can't all be applied?  It's too
> late to give an error response. Ugh.

It's perfectly possible to have some commands actually change the state of the snapshot, as long as it's well defined.

However, it seems to me that we already have exactly this situation. The script is never really in sync with reality, no matter how often it looks at readyState/currentTime, the state could always have changed by the time the script issues a command. Now what do you do if the action can't be applied?
Comment 19 Silvia Pfeiffer 2011-04-27 23:39:04 UTC
(In reply to comment #17)
> (In reply to comment #15)
> > As a developer I just know that when I take a peek at the readyState, it will
> > be merely more than a Heisenberg measurement.
> 
> I think you meant "as the author of The Definitive Guide to HTML5 Video" ;-)

I've also developed Websites that use HTML5 video, so I claim I also have a right to call myself a Web developer. :-) But indeed, I have come across inconsistencies between browsers when asking for the readyState value at certain points in JavaScript.


> It's enough that 1% of web authors make a mistake and create scripts with race
> conditions for this to be a serious problem, causing browsers with slightly
> different timing to appear more broken than the ones that were tested.

I'm happy if we can reduce the problem that different browsers are in different states when you probe them "at the same time", e.g. right after calling an event.

However, I have found the best approach is to discourage Web developers from using readyState on video. I've never needed to use it in any of my applications - there wasn't anything that I couldn't solve with events.

I am also shocked when I see people recommend the use of readyState for things where the event would be completely appropriate, such as in this stackoverflow bug http://stackoverflow.com/questions/2221029/problem-retrieving-html5-video-duration where the recommended solution uses readyState with a setInterval.

I feel very compelled to actually ask for removal of readyState, since it is causing more harm than it helps IM-NSH-O.

> > I am struggling to see the race conditions that are being used as a reason to
> > make changes to state transitions here. Can you provide an example where the
> > developer would get the wrong information? Or would get into an unsolvable
> > situation?
> 
> It's almost always possible to work around the race conditions if you know
> they're there. I listed some examples of problems that less-than-perfect
> programmers could run into in
> <http://lists.w3.org/Archives/Public/public-html-a11y/2011Apr/0232.html>.

I do agree that some things are underspecified.

Such as what is supposed to be the result of these actions that you listed:

* When removing the autoplay attribute in any of the loadstart,  
loadedmetadata, loadeddata or canplay event handlers, it's racy whether or  
not the media element will actually play.

* When registering a loadeddata event handler in the loadedmetadata event  
handler, it's racy whether or not that event handler will run. The same is  
true of any other pair of events that depend on readyState or networkState  
transitions. Having this not be racy would be useful when writing tests,  
e.g. to catch the first canplaythrough event after the seeked event.

* When currentTime is set in the loadstart event handler, it's racy  
whether or not it will actually seek.

The behaviour of browsers in these situations needs to be deterministic and so it probably needs to go into the spec. If freezing the changes on readyState and networkState to event changes helps achieve this, then this is good. We have to decouple the IDL attribute readyState from the browser-internal state machine of the media engine. I actually see this as a good thing since the Web developer really doesn't need to know internal states, but only needs to know when it is safe to do certain things.


As for this problem:

* For preload=metadata, when checking networkState in a progress event  
handler, it's racy between NETWORK_LOADING and NETWORK_IDLE. This makes it  
impossible to test if we follow the spec on fast networks.

The readyState and networkState are indeed much more part of the as-yet-to-be-developed statistics API than they are useful for basing script behaviour on.
Comment 20 Philip Jägenstedt 2011-04-28 08:26:14 UTC
(In reply to comment #19)
> (In reply to comment #17)
> > (In reply to comment #15)
> > > As a developer I just know that when I take a peek at the readyState, it will
> > > be merely more than a Heisenberg measurement.
> > 
> > I think you meant "as the author of The Definitive Guide to HTML5 Video" ;-)
> 
> I've also developed Websites that use HTML5 video, so I claim I also have a
> right to call myself a Web developer. :-) But indeed, I have come across
> inconsistencies between browsers when asking for the readyState value at
> certain points in JavaScript.

Sure, my point was only that you know a lot more than most about the idiosyncrasies of <audio> and <video>.

> > It's enough that 1% of web authors make a mistake and create scripts with race
> > conditions for this to be a serious problem, causing browsers with slightly
> > different timing to appear more broken than the ones that were tested.
> 
> I'm happy if we can reduce the problem that different browsers are in different
> states when you probe them "at the same time", e.g. right after calling an
> event.
> 
> However, I have found the best approach is to discourage Web developers from
> using readyState on video. I've never needed to use it in any of my
> applications - there wasn't anything that I couldn't solve with events.

I absolutely agree, but not everyone will take the advice.

> I am also shocked when I see people recommend the use of readyState for things
> where the event would be completely appropriate, such as in this stackoverflow
> bug
> http://stackoverflow.com/questions/2221029/problem-retrieving-html5-video-duration
> where the recommended solution uses readyState with a setInterval.

I voted against the accepted answer :) That's just the way the web works, people try "random" permutations of scripts and markup until it works. That's why we can be 100% sure that any race conditions in the platform will eventually be exposed.

> I feel very compelled to actually ask for removal of readyState, since it is
> causing more harm than it helps IM-NSH-O.

I would be tentatively supportive, if you file a bug we can see if the idea is shot down or not. It would break some existing content, that's for sure.

> > > I am struggling to see the race conditions that are being used as a reason to
> > > make changes to state transitions here. Can you provide an example where the
> > > developer would get the wrong information? Or would get into an unsolvable
> > > situation?
> > 
> > It's almost always possible to work around the race conditions if you know
> > they're there. I listed some examples of problems that less-than-perfect
> > programmers could run into in
> > <http://lists.w3.org/Archives/Public/public-html-a11y/2011Apr/0232.html>.
> 
> I do agree that some things are underspecified.
> 
> Such as what is supposed to be the result of these actions that you listed:
> 
> * When removing the autoplay attribute in any of the loadstart,  
> loadedmetadata, loadeddata or canplay event handlers, it's racy whether or  
> not the media element will actually play.
> 
> * When registering a loadeddata event handler in the loadedmetadata event  
> handler, it's racy whether or not that event handler will run. The same is  
> true of any other pair of events that depend on readyState or networkState  
> transitions. Having this not be racy would be useful when writing tests,  
> e.g. to catch the first canplaythrough event after the seeked event.
> 
> * When currentTime is set in the loadstart event handler, it's racy  
> whether or not it will actually seek.
> 
> The behaviour of browsers in these situations needs to be deterministic and so
> it probably needs to go into the spec. If freezing the changes on readyState
> and networkState to event changes helps achieve this, then this is good. We
> have to decouple the IDL attribute readyState from the browser-internal state
> machine of the media engine. I actually see this as a good thing since the Web
> developer really doesn't need to know internal states, but only needs to know
> when it is safe to do certain things.

Indeed, what I've implemented now is a kind of filtering state machine that has an actual state and a script-seen state. Ugly, but better than unreliable/untestable behavior.

> As for this problem:
> 
> * For preload=metadata, when checking networkState in a progress event  
> handler, it's racy between NETWORK_LOADING and NETWORK_IDLE. This makes it  
> impossible to test if we follow the spec on fast networks.
> 
> The readyState and networkState are indeed much more part of the
> as-yet-to-be-developed statistics API than they are useful for basing script
> behaviour on.

I'm not sure either will ever be truly useful, but I'm glad to see you're an optimist :)
Comment 21 Simon Pieters 2011-05-16 22:04:16 UTC
(In reply to comment #4)
> Examples, anyone?

http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2011-May/031652.html
Comment 22 Robert O'Callahan (Mozilla) 2011-05-17 04:28:07 UTC
I support the changes mentioned in this bug. As noted, what Simon has proposed is pretty close to (but a definite improvement on) what Firefox already does.

> As I said, you can't stop the world while the scripts think (e.g. freeze the
> video), and snapshotting the world is just telling you a truth about the past,
> not the present.

Every browser is taking snapshots and returning "truths about the past". Most browsers take a snapshot of each individual property, and take a new snapshot every time each property is accessed. Firefox takes one snapshot of the entire state (conceptually; the implementation is smarter than that) and uses that snapshot for the duration of the HTML5 task. The advantage of our approach is that it guarantees the script sees a view of the state that's consistent across properties and across the duration of the HTML5 task.

For example, the following code snippets are guaranteed to be equivalent in Firefox, but not in other browsers:
  a.time = video.currentTime;
  b.time = video.currentTime;
and
  a.time = b.time = video.currentTime;
I think preserving "obvious" equivalences like that is worth having, meets author expectations and will avoid script bugs.

Another example: the following code is guaranteed to work in Firefox (at least not throw due to t not being seekable), but not in other browsers:
  var timeRanges = video.seekable;
  if (isInTimeRanges(t, timeRanges)) {
    video.currentTime = t;
  }
(We achieve this by guaranteeing not to evict data from the media cache during the execution of an HTML5 task.)

(In reply to comment #13)
> (In reply to comment #10)
> > The best behavior of currentTime isn't obvious, I'll agree thus far. Are there
> > any use cases for scripts observing currentTime changes while they are running?
> 
> A script that displays captions, subtitles, slides, etc, in sync with a playing
> video? Keeping relatively accurate sync is hard enough, why make it worse?

Please forgive me for repeating Philip, but Web apps with long-running HTML5 tasks don't work. Even in an aggressive multiprocess, multithreaded browser implementation, such apps can't respond to user input or update any of their rendering apart from video (since browsers guarantee that only the state of the page between HTML5 tasks is rendered). The only way for an app to be responsive and stay in sync with other activity is to issue a series of very short-lived HTML5 tasks. That's why using one snapshot per task works well in practice.

Echoing what Philip said in comment #18, state changes performed by the script should certainly be reflected immediately in the script's view of the video state.

[Ideally I think we would buffer the side effects and make sure they're applied atomically at the end of the HTML5 task. Consider the following code:
  <audio id="audio1" ... muted></audio>
  <audio id="audio2" ...></audio>
  ...
  audio2.muted = true;
  audio1.muted = false;
Authors would expect this to switch audio streams with no detectable overlap or gap. We should provide that. However, that goes way beyond the scope of this bug, so pretend I didn't mention it :-).]
Comment 23 Philip Jägenstedt 2011-05-17 08:19:31 UTC
(In reply to comment #22)
> [Ideally I think we would buffer the side effects and make sure they're applied
> atomically at the end of the HTML5 task. Consider the following code:
>   <audio id="audio1" ... muted></audio>
>   <audio id="audio2" ...></audio>
>   ...
>   audio2.muted = true;
>   audio1.muted = false;
> Authors would expect this to switch audio streams with no detectable overlap or
> gap. We should provide that. However, that goes way beyond the scope of this
> bug, so pretend I didn't mention it :-).]

[Sounds like a great idea to me. Will you file a bug?]
Comment 24 David Singer 2011-05-17 09:37:26 UTC
(In reply to comment #22)
> Please forgive me for repeating Philip, but Web apps with long-running HTML5
> tasks don't work. Even in an aggressive multiprocess, multithreaded browser
> implementation, such apps can't respond to user input or update any of their
> rendering apart from video (since browsers guarantee that only the state of the
> page between HTML5 tasks is rendered). The only way for an app to be responsive
> and stay in sync with other activity is to issue a series of very short-lived
> HTML5 tasks. That's why using one snapshot per task works well in practice.


You've just described a way to write buggy scripts (long-running tasks) that is just as subtle as the original problem this thread started with (scripts that don't base their actions on what the event indicated was true) or that came up in discussion (scripts that ask for an event notification after it has happened).

Time-based programming is not easy, and there are limited ways it can be made so.
Comment 25 Philip Jägenstedt 2011-05-17 10:40:25 UTC
(In reply to comment #24)

> Time-based programming is not easy, and there are limited ways it can be made
> so.

I have a hard time decoding what it is you're arguing, if anything. Are you saying that we shouldn't try to eliminate/reduce race conditions because it won't solve all problems? That it isn't worth the implementation effort? Something else?
Comment 26 David Singer 2011-05-17 10:50:46 UTC
(In reply to comment #25)
> (In reply to comment #24)
> 
> > Time-based programming is not easy, and there are limited ways it can be made
> > so.
> 
> I have a hard time decoding what it is you're arguing, if anything. Are you
> saying that we shouldn't try to eliminate/reduce race conditions because it
> won't solve all problems? That it isn't worth the implementation effort?
> Something else?

 I am arguing against the idea that somehow we should try to make buggy scripts 'work'.  I have no problem with the idea that we should strive to make buggy scripts less likely in the first place, that writing naturally is less likely to be buggy.

In particular, bugs of the kind "I wrote this assuming that the world was standing still" when it clearly is not, are going to be hard to make into not-bugs.
Comment 27 Robert O'Callahan (Mozilla) 2011-05-17 10:56:08 UTC
When we guarantee that an HTML5 task sees a snapshot of media element state that is consistent and stable over its lifetime, that is not "making buggy scripts work". It is providing more predictable semantics that make it much easier to write simple and correct code.
Comment 28 Ian 'Hixie' Hickson 2011-06-02 23:00:40 UTC
Doing what comment 0 suggests doesn't fix the code in comment 5:

> var v = document.createElement('video');
> v.onloadstart = function() {
>   v.currentTime = 10;
> }
> v.src = 'video.webm';

...as there's no state being read here.

We can't possibly make the API work according to frozen state rather than current state, imagine what would happen if there was a task that alert()ed, then the video ended, then the script called play()... do you want to have to remember that the video was playing, so play() should do nothing, even though now the video has finished, so really it should restart? What about if the script got the "buffered" TimeRange before the alert(), then seeked into one after the alert(), but in the meantime the browser had to dispose of the particular part of the buffer? Do you just seek to the given time anyway and play elevator music while you desperately try to rebuffer the video without letting the script know that the data it got from the API was actually out of date? That way lies even more madness than having race conditions.

If there are specific high-risk cases where we can make things more stable, I'm certainly open to ideas, but I don't think generally freezing everything is a viable solution.


EDITOR'S RESPONSE: This is an Editor's Response to your comment. If you are satisfied with this response, please change the state of this bug to CLOSED. If you have additional information and would like the editor to reconsider, please reopen this bug. If you would like to escalate the issue to the full HTML Working Group, please add the TrackerRequest keyword to this bug, and suggest title and text for the tracker issue; or you may create a tracker issue yourself, if you are able to do so. For more details, see this document:
   http://dev.w3.org/html5/decision-policy/decision-policy.html

Status: Rejected
Change Description: no spec change
Rationale: see above
Comment 29 Robert O'Callahan (Mozilla) 2011-06-02 23:11:42 UTC
(In reply to comment #28)
> We can't possibly make the API work according to frozen state rather than
> current state, imagine what would happen if there was a task that alert()ed,
> then the video ended, then the script called play()... do you want to have to
> remember that the video was playing, so play() should do nothing, even though
> now the video has finished, so really it should restart?

alert() needs special treatment; for our purposes here, it needs to be treated as ending the current task and starting a new one, so fresh video state is available after alert() returns. This is very much like how alert() gets special treatment to drop the storage mutex etc. Same deal for sync XHR and the other usual suspects.

Keep in mind that we have implemented most of this in Gecko. We have alert() etc spin a restricted nested event loop; we update the script-visible media element state in events that can be processed in such a nested event loop.

I think alert() and its evil siblings don't have to block us from making the programming model saner here. Please reconsider.
Comment 30 Ian 'Hixie' Hickson 2011-06-10 19:40:13 UTC
How does Gecko fix the race in comment 5?

If it doesn't: what is the exact set of problems we are trying to solve here? This bug lists a whole series of problems, some of which would get fixed by pinning certain attributes, others of which would get exacerbated by pinning attributes, and others of which are unaffected. It's unclear to me which you want me to fix.
Comment 31 Robert O'Callahan (Mozilla) 2011-06-10 23:55:28 UTC
That is something we don't handle yet.

The best description of what we want is still comment #0, although it's not a complete proposal. Do you want one? Normally we just give you an idea and you run with it :-).

The way to address comment #5 would be to apply paragraph 4 of comment #0 so that the task that fires loadstart is responsible for setting the networkState to NETWORK_LOADING first, and the task that fires loadedmetadata sets the readyState to HAVE_METADATA. This would guarantee that the code in comment #5 always throws an exception.
Comment 32 Ian 'Hixie' Hickson 2011-07-14 23:59:45 UTC
What I need to know is what specific problems we're trying to fix. It's not at all clear to me.

Should we be pinning "new Date()" as well?
Comment 33 Ian 'Hixie' Hickson 2011-07-15 00:03:24 UTC
Similarly, are we trying to fix the issue in bug 12556 comment 2?

I'm just trying to work out where we're drawing the line, so that, if I spec this, I know what result I'm trying to spec.

Personally I'm not at all convinced this is a good idea. Video playback changes in realtime. I don't think hiding this is helpful.

Maybe it would be helpful to know what kind of code it is that we expect people to write that would be badly affected by this? Does anyone have any concrete examples?
Comment 34 Robert O'Callahan (Mozilla) 2011-07-15 00:07:55 UTC
(In reply to comment #32)
> What I need to know is what specific problems we're trying to fix. It's not at
> all clear to me.

The problem that when an event fires, there are no guarantees about what state the element is actually in, so if Web authors assume that e.g. an event handler for "loadedmetadata" means the readyState is HAVE_METADATA, they will usually be right but sometimes be wrong. We should be able to make it so they're always right or always wrong.

> Should we be pinning "new Date()" as well?

We can't for compatibility reasons (script timing etc), and anyway it's much less of an issue since authors don't have expectations that Date() will have a specific value when it's called in a specific context.
Comment 35 Ian 'Hixie' Hickson 2011-07-15 22:00:17 UTC
(In reply to comment #34)
> The problem that when an event fires, there are no guarantees about what state
> the element is actually in

There is no way we can fix this in a non-confusing way. For example, say that a 'timeupdate' event fires and currentTime is t=5. The script sets a cue with pauseOnExit set for a time of t=6. But really, the time when the script does this is t=7. Are you saying we should somehow warp the video back to t=6 and pause?

What about a script that tries to fast-forward by 5 each time it is invoked. The script is invoked at t=1, but by the time the script is ready to do the fast-forward, it's actually at t=7, such that the user actually sees the script go _backwards_ rather than jumping forwards.

I don't understand why authors would find it intuitive that new Date() is consistent with reality but video.currentTime or video.readyState is not.


Could you enumerate exactly which features you think should be pinned?
Comment 36 Robert O'Callahan (Mozilla) 2011-07-15 22:48:33 UTC
(In reply to comment #35)
> (In reply to comment #34)
> > The problem that when an event fires, there are no guarantees about what state
> > the element is actually in
> 
> There is no way we can fix this in a non-confusing way. For example, say that a
> 'timeupdate' event fires and currentTime is t=5. The script sets a cue with
> pauseOnExit set for a time of t=6. But really, the time when the script does
> this is t=7. Are you saying we should somehow warp the video back to t=6 and
> pause?

In Gecko, after the cue has been set up, an internal task updates currentTime from 5 to 7, and the cue handler would run and pause the media when the real playback time is something greater than 7.

Of course, the same result is possibly with pretty much any implementation. For example, the script might call currentTime and get t=5, sets a cue for t=6, the cue handler runs, but the handler's pause call doesn't happen until the real playback time is greater than 7. So Gecko's behavior is certainly no worse here.

However, in some implementations it's entirely possible that once in a while the script sees currentTime=5, sets a cue for t=6 after the currentTime has advanced to 7, and then the cue never fires. So Gecko guarantees that with this code, the cue always fires, but with other implementations we could easily have a situation where the cue almost always fires but once in a while it doesn't. I think it's clear that this is bad. I think Gecko's behavior is clearly better here.

More coming...
Comment 37 Robert O'Callahan (Mozilla) 2011-07-15 23:10:15 UTC
(In reply to comment #35)
> What about a script that tries to fast-forward by 5 each time it is invoked.
> The script is invoked at t=1, but by the time the script is ready to do the
> fast-forward, it's actually at t=7, such that the user actually sees the script
> go _backwards_ rather than jumping forwards.

This can happen with pretty much any implementation. For example, after reading currentTime perhaps the script is stalled for some reason (GC or whatever) while the video continues to play. Then the script adds 5 and gets a number smaller than the current playback position.

If we want to advance the playback position by exactly 5 in a reliable manner, we need new API.

> I don't understand why authors would find it intuitive that new Date() is
> consistent with reality but video.currentTime or video.readyState is not.

None of these features are guaranteed to be consistent with reality in any implementation. In any implementation, by the time you've finished fetching the value of currentTime or readyState, it may no longer reflect reality.
Comment 38 Robert O'Callahan (Mozilla) 2011-07-16 10:34:05 UTC
(In reply to comment #35)
> Could you enumerate exactly which features you think should be pinned?

All the media element state, at least.

This bug is asking that readyState and networkState be changed by the HTML task that fires the event that signals the state change (before the event fires, of course).

We can do something similar for the rest of the script-visible media element state. Some of the HTML tasks that change media element state would not fire DOM events of course, for example not every change to currentTime fires timeupdate.

Beyond media elements, I would like to be able to say that in general, changes to script-visible state other than those triggered by the script execution itself are not observable during script execution, unless we have specific reasons to make an exception. Usage of "new Date()" for timing script activity, especially legacy usage, is a reasonable exception. New timing APIs under the web-perf group may be another exception. There may be others.
Comment 39 Ian 'Hixie' Hickson 2011-07-26 17:44:58 UTC
I really couldn't disagree more. If we're going to get confusing behaviour anyway, let's not have both confusing behaviour *and* make the APIs lie all the time.

I'm fine with making APIs appear stable during script execution if doing so actually makes the APIs stable. So for instance, it makes perfect sense for something like XHR to be stable, because the user agent can fake it convincingly. But with <video> this doesn't work. You're just exchanging one set of weird behaviour for another set. At least when the APIs reflect reality there's no difficulty in working out why things are acting as they are. If the API is lying on top of it, debugging this stuff will become nightmarish, with the API saying one thing and the browser showing another.


Much of the spec machinery here depends on the variosu states. For example, what events fires when you call play() depends on whether it's able to play or not. Are you suggesting that things like that be based on the state when the script task started? Should doing something like calling load() reset the state somehow? If the script calls pause(), should currentTime get updated to the actual time it was paused at?

If I'm to spec this, I need a comprehensive list of what should happen. I can't work it out myself; every attempt I have made at working out how to do this has resulted in me concluding that it's a bad idea and that nothing should be frozen here.


(In reply to comment #36)
> 
> In Gecko, after the cue has been set up, an internal task updates currentTime
> from 5 to 7, and the cue handler would run and pause the media when the real
> playback time is something greater than 7.

IMHO this is a complete failure. The whole point of pauseOnExit is to pause at exactly the given time. Not pausing makes sense (because the time was missed), but pausing at the wrong time is inexcusably buggy and breaks the use case entirely.
Comment 40 Robert O'Callahan (Mozilla) 2011-08-01 00:00:20 UTC
(In reply to comment #39)
> I'm fine with making APIs appear stable during script execution if doing so
> actually makes the APIs stable. So for instance, it makes perfect sense for
> something like XHR to be stable, because the user agent can fake it
> convincingly. But with <video> this doesn't work. You're just exchanging one
> set of weird behaviour for another set.

I claim that we can actually reduce the set of weird behaviors. The weird behaviors with my proposal that you've raised so far can all happen with the current spec and other browser implementations as well.

> At least when the APIs reflect reality
> there's no difficulty in working out why things are acting as they are. If the
> API is lying on top of it, debugging this stuff will become nightmarish, with
> the API saying one thing and the browser showing another.

I don't understand why you say that a value that's X > 0 microseconds stale is a "lie" but a value that's Y > 0 microseconds stale is not, just because X happens to be greater than Y sometimes. They're either both lies or they both aren't.

Debugging is not going to affected if you only use logging and your event handlers don't take long to run. If you pause a running script in the debugger and then repeatedly inspect media element state in the debugger, perhaps by calling DOM APIs, then the author might be confused by those APIs not reflecting what's on the screen, I agree. On the other hand, breakpoints and single-step debugging are often quite useless for finding and fixing bugs in code dealing with asynchronous state changes; a more snapshotting approach can actually *aid* breakpoint-based debugging because if you hit a breakpoint, execution will continue after the breakpoint the same as if you hadn't stopped in the breakpoint. Changing behavior when you add breakpoints sucks! Also, I believe reducing the incidence of very hard-to-reproduce bugs would more than compensate for any confusion here.

> Much of the spec machinery here depends on the variosu states. For example,
> what events fires when you call play() depends on whether it's able to play or
> not. Are you suggesting that things like that be based on the state when the
> script task started?

Yes.

> Should doing something like calling load() reset the
> state somehow?

Yes.

> If the script calls pause(), should currentTime get updated to
> the actual time it was paused at?

That sounds like a good idea although I don't think it's absolutely necessary.

> If I'm to spec this, I need a comprehensive list of what should happen. I
> can't work it out myself; every attempt I have made at working out how to do
> this has resulted in me concluding that it's a bad idea and that nothing
> should be frozen here.

Fair enough. I don't know when I'll have time to come up with that :-(.

> (In reply to comment #36)
> > 
> > In Gecko, after the cue has been set up, an internal task updates currentTime
> > from 5 to 7, and the cue handler would run and pause the media when the real
> > playback time is something greater than 7.
> 
> IMHO this is a complete failure. The whole point of pauseOnExit is to pause at
> exactly the given time. Not pausing makes sense (because the time was missed),
> but pausing at the wrong time is inexcusably buggy and breaks the use case
> entirely.

Sorry, I misread your original mention of pauseOnExit. I agree that when adding a cue, the best behavior would be to simply ignore the cue if the true current time has already advanced past the end of the cue. My general suggestion would be to make a task's MutableTextTrack changes all take effect atomically after the task has completed.
Comment 41 Robert O'Callahan (Mozilla) 2011-08-01 00:10:50 UTC
(In reply to comment #40)
> On the other hand, breakpoints and
> single-step debugging are often quite useless for finding and fixing bugs in
> code dealing with asynchronous state changes; a more snapshotting approach can
> actually *aid* breakpoint-based debugging because if you hit a breakpoint,
> execution will continue after the breakpoint the same as if you hadn't stopped
> in the breakpoint. Changing behavior when you add breakpoints sucks!

A concrete example of this would be Philip's example in comment #5:
v.onloadstart = function() {
  v.currentTime = 10;
}
With the current model, this would often work but sometimes fail. Let's optimistically assume that the developer can reproduce the bug. So they set a breakpoint on "v.currentTime = 10". The breakpoint gets hit. The developer single-steps. Now the assignment to v.currentTime succeeds because the metadata was loaded during the significant delay between the breakpoint being hit and the developer hitting the single-step button.
Comment 42 David Singer 2011-08-01 14:40:45 UTC
(In reply to comment #41)
> (In reply to comment #40)
> 
> A concrete example of this would be Philip's example in comment #5:
> v.onloadstart = function() {
>   v.currentTime = 10;
> }

I think that we should be careful to make sure that program snippets such as this stay unreliable; at a glance, it is obviously wrong, as previously discussed. I think if you write obviously wrong code, you deserve unpredictable behavior!

Overall, I agree with Ian; we need to make sure we have a 'cure' that is actually better than the 'disease'.  Every time I look at the comments here, and think about it, I end up concluding that freezing some values on script entry merely moves the difficulties around - that it doesn't seem to reduce the number of difficulties at all.

I'm open to hearing concrete suggestions, as I am all in favor of APIs where 'naive' simple usage is actually also the least likely to be buggy, but I am having a hard time seeing what the consistent change is that actually leads to improvement.
Comment 43 Ian 'Hixie' Hickson 2011-08-01 20:47:54 UTC
I'm with David on this.

We can make things like currentTime=10 being set before the UA knows how long the resource is work (for example by having currentTime mutations before the metadata is loaded be saved and used when we know the duration, the same way the fragment identifier is used). (In fact that seems like a good idea. Filed bug 13503 on that.) But freezing the state before running script wouldn't affect this case, as far as I can tell.
Comment 44 Robert O'Callahan (Mozilla) 2011-08-01 21:14:24 UTC
(In reply to comment #42)
> (In reply to comment #41)
> > (In reply to comment #40)
> > 
> > A concrete example of this would be Philip's example in comment #5:
> > v.onloadstart = function() {
> >   v.currentTime = 10;
> > }
> 
> I think that we should be careful to make sure that program snippets such as
> this stay unreliable; at a glance, it is obviously wrong, as previously
> discussed. I think if you write obviously wrong code, you deserve unpredictable
> behavior!

No, you deserve predictable behavior --- failure. See the last paragraph of comment #31.
Comment 45 Ian 'Hixie' Hickson 2011-08-01 21:36:19 UTC
Preventing the video from playing until the events have been processed seems like building latency into the design. Event dispatch can sometimes be delayed quite a lot, e.g. if there's a lot of activity going on. Are we sure we want to be introducing this kind of latency?

The spec does currently delay a number of transitions until stable states (i.e. til the end of the current event loop task), e.g. .seeking only changes in this way, and most networkState transitions are done in stable states. I could go through the spec and see if there's any other transitions that could be made more stable. I don't know that readyStates would make sense done this way though. There's no point readyState saying there's FUTURE_DATA when the video has actually ended. Similarly, it's not like you can actually prevent the video from ending before it's ended. The video ends when you run out of material, regardless of how busy the browser is.
Comment 46 Robert O'Callahan (Mozilla) 2011-08-01 22:40:27 UTC
(In reply to comment #45)
> Preventing the video from playing until the events have been processed seems
> like building latency into the design. Event dispatch can sometimes be delayed
> quite a lot, e.g. if there's a lot of activity going on. Are we sure we want to
> be introducing this kind of latency?

I don't think we need to prevent the video from playing. It's only the script-observable values of readyState/networkState that are not updated until the relevant events have fired.

> I don't know that readyStates would make sense done this way
> though. There's no point readyState saying there's FUTURE_DATA when the video
> has actually ended.

I think there is. I think it will confuse authors to find that sometimes when canplay fires, readyState is HAVE_CURRENT_DATA (or in general, that the preconditions listed in 4.8.10.16 usually hold when the event fires, but occasionally don't). Setting readyState in the task that dispatches the event fixes that problem.

Sure, that could mean readyState returns HAVE_FUTURE_DATA when the video has already ended, but that's effectively possible today when you allow readyState to change asynchronously and the video ends the moment after readyState was fetched. E.g.
  if (video.readyState >= video.HAVE_FUTURE_DATA) {
    // video happens to end here
    doSomethingAssumingVideoHasNotEnded();
  }
There is no way to prevent this problem in anyone's model.

> Similarly, it's not like you can actually prevent the video
> from ending before it's ended. The video ends when you run out of material,
> regardless of how busy the browser is.

Sure.

I'll be honest and admit that without having done all the spec work myself, I can't be 100% sure that the spec can be made to work this way across the board without creating new problems. I'm pretty sure the general approach of only updating script-observable state during stable states can work because Firefox is implemented that way and it's working well, but I'm less sure about this bug (delaying readyState/networkState updates until the event fires).

For example, I think to make this work we'd have to sometimes cancel events in response to scripted state changes. E.g. if the readyState is HAVE_CURRENT_DATA, the video loads some future data, and a task is dispatched to set readyState to HAVE_FUTURE_DATA and fire the "canplay" event, but some script does "currentTime = 0;" before that task runs, we will probably want to prevent readyState from temporarily flipping to HAVE_FUTURE_DATA and back after the seek, and prevent "canplay" from firing. I hope that can be worked out, but maybe there are some bad implications.
Comment 47 Ian 'Hixie' Hickson 2011-08-02 06:52:04 UTC
If the readyState change is delayed, the video can't start playing (unless we are decoupling the video logic from the readyState, which I thought you said in comment 40 we should not do). The video only starts playing once it reaches the readyState that has data.
Comment 48 Robert O'Callahan (Mozilla) 2011-08-02 09:23:23 UTC
(In reply to comment #47)
> If the readyState change is delayed, the video can't start playing (unless we
> are decoupling the video logic from the readyState, which I thought you said in
> comment 40 we should not do).

I didn't mean to imply that. In Firefox the playback engine can advance through its internal states freely while the script-visible readyState is only updated between tasks.
Comment 49 Philip Jägenstedt 2011-08-02 09:59:11 UTC
(In reply to comment #48)
> (In reply to comment #47)
> > If the readyState change is delayed, the video can't start playing (unless we
> > are decoupling the video logic from the readyState, which I thought you said in
> > comment 40 we should not do).
> 
> I didn't mean to imply that. In Firefox the playback engine can advance through
> its internal states freely while the script-visible readyState is only updated
> between tasks.

This sounds similar to Opera, where the state of the media pipeline is most definitely not coupled synchronously to readyState. Decoding happens on separate thread while readyState is updated asynchronously on the main thread.
Comment 50 Michael[tm] Smith 2011-08-04 05:17:39 UTC
mass-move component to LC1
Comment 51 Ian 'Hixie' Hickson 2011-08-19 22:32:42 UTC
(Another example of this kind of thing is cues: when a cue is activated or deactivated, it queues an event. By the time the event is fired, lots of cues might have come and gone. The "activeCues" list doesn't change while the script is running, but the cue in question might no longer be active. I'm pretty sure we don't want to delay the cues appearing and disappearing until the events have fired, that would make the cues laggy. We could keep have the list of activeCues be maintained by the events, but as with other things, I really think that having the API lie is worse than having the events be delayed — what if one of these events pauses the video, for example: is currentTime the time at the time the cue event was queued, or at the time the event fired, or the time the script called pause()? Here the lie becomes really clear, since the user can easily compare the actually visible cues to the cues in the scripts. Once things are paused, it's no longer lag, it's just plain wrong. I really don't see any way to keep this API sane while making the scripts run in entirely predictable environments.)

I think this is WONTFIX for the general case.

There may be specific cases that make sense. I recommend bringing the very specific cases up (either here on in bugs marked as being blocked by this one), so I can look at them individually.
Comment 52 Robert O'Callahan (Mozilla) 2011-08-20 11:54:03 UTC
I would say that pause() takes effect asynchronously, so after pause() you can still get pending events for state changes that were triggered before the pause(), including cue changes. (This behavior can also occur with the current spec, where state changes can be triggered by media playback just before the script invokes pause().) Then even if activeCues is only updated by the cue events firing, it will quickly converge to the visible cues after a pause().
Comment 53 Ian 'Hixie' Hickson 2011-08-29 07:25:15 UTC
Asynchronous APIs are a huge pain to work with. Trading a volatile API (changes as the script is running based on the latest state; events may execute after the state they are notifying about is no longer relevant) for an inaccurate one (the APIs don't actually represent the current state, they represent in some cases the state at the time the script started and in other cases the state as it was when an event was fired, even though the state the event is notifying about is no longer relevant) is at least just trading one bad API for another. Trading a synchronous API for an asynchronous one just to make the inaccurate API less obviously inaccurate is a much worse trade, IMHO.

Once an author calls pause(), the video should IMHO pause and the API should IMHO immediately show a stable state. Having .paused return false after a call to pause() would just make authors think the API was broken.
Comment 54 Ian 'Hixie' Hickson 2011-08-29 07:33:12 UTC
As part of bug 12541 I have made currentTime freeze at script start and when it's set or the video is paused. As noted in comment 51, if there are other specific cases that you think we should do, I'm happy to consider them on a case by case basis, but I do not intend to try to do this for the entire API.
Comment 55 Robert O'Callahan (Mozilla) 2011-08-29 09:31:35 UTC
(In reply to comment #54)
> As part of bug 12541 I have made currentTime freeze at script start and when
> it's set or the video is paused.

Thanks for doing that.
Comment 56 Ian 'Hixie' Hickson 2011-09-21 21:18:26 UTC
EDITOR'S RESPONSE: This is an Editor's Response to your comment. If you are satisfied with this response, please change the state of this bug to CLOSED. If you have additional information and would like the editor to reconsider, please reopen this bug. If you would like to escalate the issue to the full HTML Working Group, please add the TrackerRequest keyword to this bug, and suggest title and text for the tracker issue; or you may create a tracker issue yourself, if you are able to do so. For more details, see this document:
   http://dev.w3.org/html5/decision-policy/decision-policy.html

Status: Rejected
Change Description: no spec change
Rationale: I'm willing to consider concrete focused suggestions for making specific changes to this API to make it more predictable, but at a high level I think I've done pretty much as much as can be done.