This is an archived snapshot of W3C's public bugzilla bug tracker, decommissioned in April 2019. Please see the home page for more details.
We have partner feedback indicating that jitter is very useful as a feedback mechanism for video quality. It allows detection of situations where no frames are dropped, but jitter is sufficient to be visible to users. Restoring it would mean changing the video quality metrics to this, based on the currently adopted metrics: interface MediaPlaybackQuality { readonly attribute DOMHighResTimeStamp creationTime; readonly attribute unsigned long totalVideoFrames; readonly attribute unsigned long droppedVideoFrames; readonly attribute double playbackJitter; }; The "playbackJitter" attribute represents the sum of all duration errors for frames intended to be presented to the user, where: Ei = Desired duration of frame i spent on the screen (to nearest microsecond) Ai = Actual duration frame i spent on the screen (if the frame is never presented to the user, then Ai == 0). then: playbackJitter = sum(abs(Ei - Ai))
Marking all pre-Last Call bugs
Changes committed https://dvcs.w3.org/hg/html-media/rev/63675668846c
VideoPlaybackQuality playbackJitter was recently incorporated as a video quality metric. I believe it was still being discussed with others in the working group, and that unresolved issues were raised about it on the last MSE call. We believe a revision may be needed to accommodate feedback from other members. This feedback agreed that jitter would be a useful measure of video rendering stress, but wasn't ideal because it is an instantaneous measurement that does not reflect AV sync issues. Persistent video async might show up as a jitter spike, but then show no more indication of a problem on the current metric. This makes it difficult for an app to detect accumulated latency that could eventually lead to dropped frames. A preferred measure would represent the accumulated total frame delays, so that this persistent latency condition would be more directly detectable. In response to this feedback, we are proposing VideoPlaybackQuality totalFrameDelay as an improved jitter metric, one that reflects both instantaneous jitter and accumulated latency. It would continue to accumulate total delays for the duration of a media session. The app would be responsible for keeping a record of its value when the video representation is shifted, so that total delays for the currently rendering content quality could be calculated. We believe this revised measure addresses the feedback we've received on the previous metric. interface VideoPlaybackQuality { readonly attribute DOMHighResTimeStamp creationTime; readonly attribute unsigned long totalVideoFrames; readonly attribute unsigned long droppedVideoFrames; readonly attribute unsigned long corruptedVideoFrames; readonly attribute double totalFrameDelay; // =====> to replace PlaybackJitter }; DEFINITION - Given: Li represents the lateness of the frame i. The value is positive if the frame is late. The value is set to “0” if the frame is (available from decoder) ahead of rendering time. Then: totalFrameDelay = sum( Li )
I have a few questions about the new metric. 1. Is this only supposed to apply to displayed frames or does it include dropped frames? 2. If it includes dropped frames how should pre-decode dropping be represented? 3. How is this supposed to allow the application to detect A/V sync issues? I wouldn't expect the delays to accumulate in a way to cause A/V sync problems. It seems like if frames are too late they should just be dropped instead of displayed extremely late.
(In reply to comment #4) > I have a few questions about the new metric. > > 1. Is this only supposed to apply to displayed frames or does it include > dropped frames? I understood it would only apply to rendered frames. > 2. If it includes dropped frames how should pre-decode dropping be > represented? > 3. How is this supposed to allow the application to detect A/V sync issues? > I wouldn't expect the delays to accumulate in a way to cause A/V sync > problems. It seems like if frames are too late they should just be dropped > instead of displayed extremely late. Certainly frames *should* be dropped if they are extremely late. However, the existing playbackJitter does not allow detection of a scenario where the A/V gets out of sync, even temporarily. For example if a whole string of frames are all rendered half a frame interval late, the playback jitter will increase as you enter this state but then remain constant as long as you stay in that out-of-sync state and then increase again when you exit that state (because of the abs() then frame durations being less than normal - as needed to bring you back into sync - show up as an increase in playbackJitter as well.) So, when you see an increase in the playback jitter value you don't know if there was a move out-of-sync and then back (i.e. just one frame was late) or just a move into a long-running out-of-sync state. The proposed totalFrameDelay allows detection of this as well as detection of all the events that playbackJitter could have detected. Even if players "should not" do this it's good to be able to detect what is actually happening as often all bets are off for actual behavior when you're in a resource-constrained state.
Changes committed. https://dvcs.w3.org/hg/html-media/rev/2b2d8865de83 Removed playbackJitter and added totalFrameDelay.