27342 2014-11-17 06:37:21 +0000 Clarify if the source buffer is still valid when appendBuffer/appendStream meets 'decode' error 2014-12-03 06:39:45 +0000 1 1 1 Unclassified HTML WG Media Source Extensions unspecified PC All RESOLVED FIXED P2 normal CR 1 jiajia.qin acolwell acolwell cyril.concolato jiajia.qin mike philipj public-html-media public-html-bugzilla oldest_to_newest 114984 0 jiajia.qin 2014-11-17 06:37:21 +0000 Currently, when SourceBuffer.appendBuffer/appendStream fails in segment parser loop algorithm, it will run the end of stream algorithm with the error parameter set to "decode". However, in end of stream algorithm, 'If error is set to "decode". If the HTMLMediaElement.readyState attribute is greater than HAVE_NOTHING. Run the media data is corrupted steps of the resource fetch algorithm' will result aborting the overall resource selection algorithm and the media player pipeline will be shutdown. So, if we execute SourceBuffer.appendBuffer/appendStream again, the media data will be ignored since the demuxer has been shutdown. What's more, if the MediaSource's readyState is set to 'closed'. Then, that SourceBuffer will be cleared. But if we only look at MSE spec, when executing end of stream algorithn, it changes the readyState attribute value to "ended" not "closed". It seems that we can still use the current MediaSource and SourceBuffer which called the end of stream algorithm. But it's kind of conflict with above paragraph. 115019 1 acolwell 2014-11-17 17:31:29 +0000 If you look at the "If the media data is corrupted" step in the HTML5 spec (http://www.w3.org/TR/html5/embedded-content-0.html#fatal-decode-error) you will see that there is a step that causes networkState to get set to NETWORK_EMPTY. This causes the "Detaching from a media element" algorithm (https://dvcs.w3.org/hg/html-media/raw-file/default/media-source/media-source.html#mediasource-detach) to run. That is why there is a transition to "closed". Based on this, I don't believe there is a conflict in the MSE spec. A compliant implementation should transition readyState to "ended" as a result of running the end of stream algorithm, and then transition readyState to "closed" as a result of the detaching from a media element algorithm being run because networkState got transitioned to NETWORK_EMPTY. I can add a note under the "Run the media data is corrupt steps of the resource fetch algorithm." saying something like "This step can cause the 'detaching from a media element algorithm' to run." 115042 2 jiajia.qin 2014-11-18 07:19:07 +0000 (In reply to Aaron Colwell from comment #1) > If you look at the "If the media data is corrupted" step in the HTML5 spec > (http://www.w3.org/TR/html5/embedded-content-0.html#fatal-decode-error) you > will see that there is a step that causes networkState to get set to > NETWORK_EMPTY. This causes the "Detaching from a media element" algorithm > (https://dvcs.w3.org/hg/html-media/raw-file/default/media-source/media- > source.html#mediasource-detach) to run. That is why there is a transition to > "closed". You are right if media element is going to transition to NETWORK_EMPTY. But, if it changes to NETWORK_IDLE, what will happen to MediaSource? Does it still work? > Based on this, I don't believe there is a conflict in the MSE spec. A > compliant implementation should transition readyState to "ended" as a result > of running the end of stream algorithm, and then transition readyState to > "closed" as a result of the detaching from a media element algorithm being > run because networkState got transitioned to NETWORK_EMPTY. > > I can add a note under the "Run the media data is corrupt steps of the > resource fetch algorithm." saying something like "This step can cause the > 'detaching from a media element algorithm' to run." Only transition to NETWORK_EMPTY will call "Detaching from a media element" or NETWORK_EMPTY and NETWORK_IDLE both can call this algorithm? 115071 3 acolwell 2014-11-18 19:14:56 +0000 (In reply to Qin Jiajia from comment #2) > (In reply to Aaron Colwell from comment #1) > > If you look at the "If the media data is corrupted" step in the HTML5 spec > > (http://www.w3.org/TR/html5/embedded-content-0.html#fatal-decode-error) you > > will see that there is a step that causes networkState to get set to > > NETWORK_EMPTY. This causes the "Detaching from a media element" algorithm > > (https://dvcs.w3.org/hg/html-media/raw-file/default/media-source/media- > > source.html#mediasource-detach) to run. That is why there is a transition to > > "closed". > You are right if media element is going to transition to NETWORK_EMPTY. But, > if it changes to NETWORK_IDLE, what will happen to MediaSource? Does it > still work? Ok. Now I see what you are saying. My eyes were playing tricks on me and I was actually reading step 4 of the "If the media data is corrupted" algorithm in the opposite sense. Based on my current understanding, the condition in the "end of stream algorithm" actually guarantees that networkState will always transition to NETWORK_IDLE instead of NETWORK_EMPTY so I see your point now. I'll need to think about this a little more because I had always intended for this path to result in the MediaSource closing. If we decide to change this to allow further appending, at a minimum we'll need to call the "reset parser state" algorithm to recover from the corrupted data and put the parser in a known state. I'll need to look at the HTML5 spec a little closer to get a sense of what is still allowed if the "resource selection algorithm" is aborted. It isn't clear to me right now whether allowing appends to proceed after this makes sense or is consistent with "normal non-MSE" playback. In the non-MSE case, I believe the UA doesn't try to download any more data once the "resource selection algorithm" gets aborted. This seems like it would be equivalent to preventing further appends until the "resource selection algorithm" is initiated again. Perhaps the MediaSource shouldn't close, but instead appendBuffer/appendStream should throw an InvalidStateException if HTMLMediaElement.error is non-null? What do you think? > > > Based on this, I don't believe there is a conflict in the MSE spec. A > > compliant implementation should transition readyState to "ended" as a result > > of running the end of stream algorithm, and then transition readyState to > > "closed" as a result of the detaching from a media element algorithm being > > run because networkState got transitioned to NETWORK_EMPTY. > > > > I can add a note under the "Run the media data is corrupt steps of the > > resource fetch algorithm." saying something like "This step can cause the > > 'detaching from a media element algorithm' to run." > Only transition to NETWORK_EMPTY will call "Detaching from a media element" > or NETWORK_EMPTY and NETWORK_IDLE both can call this algorithm? As specified right now only the transition to NETWORK_EMPTY. I would not be surprised if Chrome currently doesn't make this distinction and always closes the MediaSource. If this is happening, this is technically a bug in Chrome right now. 115112 4 jiajia.qin 2014-11-19 10:38:10 +0000 (In reply to Aaron Colwell from comment #3) > (In reply to Qin Jiajia from comment #2) > > (In reply to Aaron Colwell from comment #1) > > > If you look at the "If the media data is corrupted" step in the HTML5 spec > > > (http://www.w3.org/TR/html5/embedded-content-0.html#fatal-decode-error) you > > > will see that there is a step that causes networkState to get set to > > > NETWORK_EMPTY. This causes the "Detaching from a media element" algorithm > > > (https://dvcs.w3.org/hg/html-media/raw-file/default/media-source/media- > > > source.html#mediasource-detach) to run. That is why there is a transition to > > > "closed". > > You are right if media element is going to transition to NETWORK_EMPTY. But, > > if it changes to NETWORK_IDLE, what will happen to MediaSource? Does it > > still work? > > Ok. Now I see what you are saying. My eyes were playing tricks on me and I > was actually reading step 4 of the "If the media data is corrupted" > algorithm in the opposite sense. Based on my current understanding, the > condition in the "end of stream algorithm" actually guarantees that > networkState will always transition to NETWORK_IDLE instead of NETWORK_EMPTY > so I see your point now. > > I'll need to think about this a little more because I had always intended > for this path to result in the MediaSource closing. If we decide to change > this to allow further appending, at a minimum we'll need to call the "reset > parser state" algorithm to recover from the corrupted data and put the > parser in a known state. I'll need to look at the HTML5 spec a little closer > to get a sense of what is still allowed if the "resource selection > algorithm" is aborted. It isn't clear to me right now whether allowing > appends to proceed after this makes sense or is consistent with "normal > non-MSE" playback. > > In the non-MSE case, I believe the UA doesn't try to download any more data > once the "resource selection algorithm" gets aborted. This seems like it > would be equivalent to preventing further appends until the "resource > selection algorithm" is initiated again. Perhaps the MediaSource shouldn't > close, but instead appendBuffer/appendStream should throw an > InvalidStateException if HTMLMediaElement.error is non-null? What do you > think? I think if MediaSource doesn't close, it means this MediaSource is still workable. Assume that let appendBuffer/appendStream throws an InvalidStateException when HTMLMediaElement.error is non-null. So, in what kind of situation, the SourceBuffer can apppendBuffer/appendstream again when having met the 'decode' error? It seems there is no such kind of situation. Then, it becomes meaningless not to close MediaSource when there is 'decode' error. But in another side, if we meet 'decode' error, maybe we only need to call ' reset parser state algorithm', wait for new data and don't need to notify HTMLMediaElement. So the media pipeline is still active. Then, the UA can use appendBuffer/appendStream for new data. In sum, We need to figure out whether MediaSource/SourceBuffer can still use when appendBuffer/appendStream faild (via end of stream algorithm with parameter 'decode'). 115134 5 acolwell 2014-11-19 17:16:51 +0000 (In reply to Qin Jiajia from comment #4) > (In reply to Aaron Colwell from comment #3) > > (In reply to Qin Jiajia from comment #2) > > > (In reply to Aaron Colwell from comment #1) > > > > If you look at the "If the media data is corrupted" step in the HTML5 spec > > > > (http://www.w3.org/TR/html5/embedded-content-0.html#fatal-decode-error) you > > > > will see that there is a step that causes networkState to get set to > > > > NETWORK_EMPTY. This causes the "Detaching from a media element" algorithm > > > > (https://dvcs.w3.org/hg/html-media/raw-file/default/media-source/media- > > > > source.html#mediasource-detach) to run. That is why there is a transition to > > > > "closed". > > > You are right if media element is going to transition to NETWORK_EMPTY. But, > > > if it changes to NETWORK_IDLE, what will happen to MediaSource? Does it > > > still work? > > > > Ok. Now I see what you are saying. My eyes were playing tricks on me and I > > was actually reading step 4 of the "If the media data is corrupted" > > algorithm in the opposite sense. Based on my current understanding, the > > condition in the "end of stream algorithm" actually guarantees that > > networkState will always transition to NETWORK_IDLE instead of NETWORK_EMPTY > > so I see your point now. > > > > I'll need to think about this a little more because I had always intended > > for this path to result in the MediaSource closing. If we decide to change > > this to allow further appending, at a minimum we'll need to call the "reset > > parser state" algorithm to recover from the corrupted data and put the > > parser in a known state. I'll need to look at the HTML5 spec a little closer > > to get a sense of what is still allowed if the "resource selection > > algorithm" is aborted. It isn't clear to me right now whether allowing > > appends to proceed after this makes sense or is consistent with "normal > > non-MSE" playback. > > > > In the non-MSE case, I believe the UA doesn't try to download any more data > > once the "resource selection algorithm" gets aborted. This seems like it > > would be equivalent to preventing further appends until the "resource > > selection algorithm" is initiated again. Perhaps the MediaSource shouldn't > > close, but instead appendBuffer/appendStream should throw an > > InvalidStateException if HTMLMediaElement.error is non-null? What do you > > think? > I think if MediaSource doesn't close, it means this MediaSource is still > workable. Assume that let appendBuffer/appendStream throws an > InvalidStateException when HTMLMediaElement.error is non-null. So, in what > kind of situation, the SourceBuffer can apppendBuffer/appendstream again > when having met the 'decode' error? It seems there is no such kind of > situation. Then, it becomes meaningless not to close MediaSource when there > is 'decode' error. I think we should make the MediaSource case consistent with what would happen in the non-MediaSource case. I believe in the non-MediaSource case playback of the buffered data and seeking is allowed to continue eventhough more data is not fetched anymore. This is why I was suggesting simply throwing an exception when error is non-null instead of closing the MediaSource. It keeps the buffered data around, but doesn't allow any more to be added. > > But in another side, if we meet 'decode' error, maybe we only need to call ' > reset parser state algorithm', wait for new data and don't need to notify > HTMLMediaElement. So the media pipeline is still active. Then, the UA can > use appendBuffer/appendStream for new data. This feels less consistent with the non-MediaSource case to me. This is essentially equivalent to the "resource fetch algorithm" continuing to run even though step 1 of the "If the media data is corrupted" sequence explicitly says "The user agent should cancel the fetching process." Since the UA isn't in direct control of fetching in the MediaSource case, it seems like preventing appends, and possibly removes, achieves a good approximation of that intent. > > In sum, We need to figure out whether MediaSource/SourceBuffer can still use > when appendBuffer/appendStream faild (via end of stream algorithm with > parameter 'decode'). At this point, I believe the answer is no. Appends should not be allowed. 115188 6 jiajia.qin 2014-11-20 07:01:59 +0000 (In reply to Aaron Colwell from comment #5) > (In reply to Qin Jiajia from comment #4) > > (In reply to Aaron Colwell from comment #3) > > > (In reply to Qin Jiajia from comment #2) > > > > (In reply to Aaron Colwell from comment #1) > > > > > If you look at the "If the media data is corrupted" step in the HTML5 spec > > > > > (http://www.w3.org/TR/html5/embedded-content-0.html#fatal-decode-error) you > > > > > will see that there is a step that causes networkState to get set to > > > > > NETWORK_EMPTY. This causes the "Detaching from a media element" algorithm > > > > > (https://dvcs.w3.org/hg/html-media/raw-file/default/media-source/media- > > > > > source.html#mediasource-detach) to run. That is why there is a transition to > > > > > "closed". > > > > You are right if media element is going to transition to NETWORK_EMPTY. But, > > > > if it changes to NETWORK_IDLE, what will happen to MediaSource? Does it > > > > still work? > > > > > > Ok. Now I see what you are saying. My eyes were playing tricks on me and I > > > was actually reading step 4 of the "If the media data is corrupted" > > > algorithm in the opposite sense. Based on my current understanding, the > > > condition in the "end of stream algorithm" actually guarantees that > > > networkState will always transition to NETWORK_IDLE instead of NETWORK_EMPTY > > > so I see your point now. > > > > > > I'll need to think about this a little more because I had always intended > > > for this path to result in the MediaSource closing. If we decide to change > > > this to allow further appending, at a minimum we'll need to call the "reset > > > parser state" algorithm to recover from the corrupted data and put the > > > parser in a known state. I'll need to look at the HTML5 spec a little closer > > > to get a sense of what is still allowed if the "resource selection > > > algorithm" is aborted. It isn't clear to me right now whether allowing > > > appends to proceed after this makes sense or is consistent with "normal > > > non-MSE" playback. > > > > > > In the non-MSE case, I believe the UA doesn't try to download any more data > > > once the "resource selection algorithm" gets aborted. This seems like it > > > would be equivalent to preventing further appends until the "resource > > > selection algorithm" is initiated again. Perhaps the MediaSource shouldn't > > > close, but instead appendBuffer/appendStream should throw an > > > InvalidStateException if HTMLMediaElement.error is non-null? What do you > > > think? > > I think if MediaSource doesn't close, it means this MediaSource is still > > workable. Assume that let appendBuffer/appendStream throws an > > InvalidStateException when HTMLMediaElement.error is non-null. So, in what > > kind of situation, the SourceBuffer can apppendBuffer/appendstream again > > when having met the 'decode' error? It seems there is no such kind of > > situation. Then, it becomes meaningless not to close MediaSource when there > > is 'decode' error. > > I think we should make the MediaSource case consistent with what would > happen in the non-MediaSource case. I believe in the non-MediaSource case > playback of the buffered data and seeking is allowed to continue eventhough > more data is not fetched anymore. This is why I was suggesting simply > throwing an exception when error is non-null instead of closing the > MediaSource. It keeps the buffered data around, but doesn't allow any more > to be added. > Agree. Thanks for your explanation. 115821 7 acolwell 2014-12-03 06:39:45 +0000 Change committed... https://github.com/w3c/media-source/commit/0c052b1ac9841aedc0c378b7eda447dbc95c5034