This was was cloned from bug 14851 as part of operation convergence.
Originally filed: 2011-11-17 10:25:00 +0000

================================================================================
 #0   contributor@whatwg.org                          2011-11-17 10:25:54 +0000 
--------------------------------------------------------------------------------
Specification: http://www.whatwg.org/specs/web-apps/current-work/multipage/the-video-element.html
Multipage: http://www.whatwg.org/C#the-video-element
Complete: http://www.whatwg.org/c#the-video-element

Comment:
Add seek() function for non-exact (fast) seeking

Posted from: 2001:4c28:a032:32:72f3:95ff:fe02:1442 by philipj@opera.com
User agent: Opera/9.80 (X11; Linux x86_64; U; Edition Next; en) Presto/2.9.220 Version/12.00
================================================================================
 #1   Philip J                                        2011-11-17 10:40:01 +0000 
--------------------------------------------------------------------------------
http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2011-January/029806.html
http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2011-November/033845.html

Background:

Compressed video streams are seldom truly random access, as most frames depend on the preceding or following frames. Seeking to a key frame is always faster, as any other frame requires decoding more than one frame. How slow depends on the distance between key frames. (Somewhat simplified. Frames with multiple reference frames and rolling intra without any key frames also exist, but is not really relevant here.)

Use cases:

* In scripted controls, while dragging the seek head ("scrubbing") you want to show a frame as fast as possible, not necessarily one at the exact time the seek head is at.

* Seeking relative to the current position should guarantee that the direction of seeking is correct even when using non-exact seeking. You need this for fast forward/backwards skip.

Proposed solution:

Introduce a seek() function with two parameters:

* a reference point: initialTime or currentTime

* an accuracy: exact or fast
================================================================================
 #2   Ian 'Hixie' Hickson                             2011-12-07 23:28:15 +0000 
--------------------------------------------------------------------------------
Is this urgent?
================================================================================
 #3   Philip J                                        2011-12-09 10:00:21 +0000 
--------------------------------------------------------------------------------
It doesn't need to be done today, but I don't think we should postpone the issue for a year, if that's what you mean.
================================================================================
 #4   Ian 'Hixie' Hickson                             2011-12-09 23:04:03 +0000 
--------------------------------------------------------------------------------
Roger.
================================================================================
 #5   Ian 'Hixie' Hickson                             2012-04-30 22:54:25 +0000 
--------------------------------------------------------------------------------
Surely you need three arguments to do what is described in comment 1: a reference time, a flag saying whether to be precise or fast, and most importantly, a target time. Presumably, the reference time only matters if you're not being precise.

   media.seek(targetTime, {
     referenceTime: media.currentTime,
     mode: 'fast', // or 'precise'
   });

Also presumably:

   media.currentTime = targetTime;

...assumes the 'fast' mode with a reference time equal to the previous currentTime.

Alternatively:

   media.currentTime = targetTime; // fast; relative to previous currentTime
   media.seekFast(targetTime, referenceTime);
   media.seekPrecise(targetTime);

Then again, why would you ever want the reference time to be something different than the current time? Maybe we should just always assume that the currentTime is the reference time, and then we only need to add:

   media.preciseSeek(targetTime);

...since assigning to currentTime already handles the fast version.
================================================================================
 #6   Ian 'Hixie' Hickson                             2012-04-30 22:56:05 +0000 
--------------------------------------------------------------------------------
On Thu, 13 Jan 2011, Robert O'Callahan wrote:
> 
> I think setting currentTime should be exact, since a) exact seeking is
> simpler from the author's point of view and b) it would be unfortunate   
> to set currentTime to value T and then discover that getting currentTime   
> gives you a value other than T (assuming you're paused).
================================================================================
 #7   Ian 'Hixie' Hickson                             2012-04-30 22:57:53 +0000 
--------------------------------------------------------------------------------
On Wed, 12 Jan 2011, Eric Carlson wrote:
> > I agree that precise seeking follows the principle of least surprise,
> based partly on the bugs files against the <video> element on iOS where
> this hasn't always been the behavior.

On Thu, 13 Jan 2011, Philip Jägenstedt wrote:> 
> Changing the default at this point wouldn't really hurt since not all
> browsers are doing exact seeking anyway, right? I think that keyframe
> seeking is more often what you want and it'd be better to let the few
> who want frame-exact seeking jump through hoops than having everyone who
> makes a simple seeking UI discover that seeking is a bit slow and 20% of
> them figuring out that it can be fixed by using seek().

On Fri, 21 Jan 2011, Michael Dale wrote:
> 
> Is accurate seeking *that* much slower than keyframe seeking? I would
> discourage changing the accurate seek default. If supported at all you
> should opt into fuzzy seeking.
> 
> Assuming the client can buffer fast enough for real-time playback, a
> seek to keyframe + time in the worst case would would take time offset.
> But in most cases the data to time ratio after the keyframe is much more
> sparse then the keyframe itself. So the ranged request for key + data to
> time will normally not be a lot more than keyframe + data for future
> playback for seeking to a keyframe. You can see seeking is pretty fast
> in the indexed seek firefox demos.

On Fri, 21 Jan 2011, Gregory Maxwell wrote:
> 
> It's usually the decoding, not the file access that kill you.  Firefox
> seeking on low resolution clips is very snappy index or not. But try a
> 1080p clip encoded with a 10 second maximum keyframe interval...
> 
> If your client is just capable of realtime playback then you can be
> waiting as much as ten seconds for an exact seek. Even if it's 2x
> realtime (a pretty reasonable multiple for 1080p) you're still taking  
> about 5 seconds in the worst case.
> 
> Basically, as the decoding speed approaches realtime the seeking time
> approaches what you'd get by seeking to the prior keyframe and playing
> up to the current point, except with the exact seeking you sit around
> twiddling your thumbs while the client looks broken.
> 
> I'm personally a fan of a heuristic approach where the first seek from a 
> user goes to a keyframe while subsequent seeks soon after the first are
> all exact.  But this is a user interface thing, and not really a
> question of what the API should provide.
================================================================================
 #8   Ian 'Hixie' Hickson                             2012-04-30 23:02:03 +0000 
--------------------------------------------------------------------------------
On Mon, 24 Jan 2011, Robert O'Callahan wrote:
>  
> Interop problems are going to arise with approximate seeking no matter 
> what we do, which is why it shouldn't be the default.
> 
> You don't want seekApproximate(T) to always land on the same T'. For
> example, if a player wants to "seek forward approximately N seconds" via
> seekApproximate(currentTime + N), but the approximation to currentTime +  
> N is fixed to be some T' less than or equal to currentTime, that would    
> be broken.
> 
> I would say that seekApproximate(T) should be specified to aim as close   
> as possible to T while being "fast", but the only guarantee is that it    
> seeks somewhere after currentTime if T > currentTime, or somewhere 
> before currentTime if T < currentTime.

(This seems to agree with my earlier comments regarding whether we actually need a reference point or whether to always use currentTime for it.)
================================================================================
 #9   Ian 'Hixie' Hickson                             2012-04-30 23:06:36 +0000 
--------------------------------------------------------------------------------
Based on that commentary, maybe currentTime shouldn't be the approximate version. So:

   media.currentTime = targetTime; // precise, may take seconds to complete
   media.fastSeek(targetTime); // fast, could be inaccurate

fastSeek() would also additionally ensure that if targetTime is before currentTime, the new currentTime is also before the old currentTime, and vice versa.

I like this proposal best of the ones so far because it is more intuitive, API-wise. setting currentTime always sets it, it doesn't end up with some random other value, and the operation that sets it but does so potentially inaccurately is a method.
================================================================================
 #10  Ian 'Hixie' Hickson                             2012-04-30 23:18:57 +0000 
--------------------------------------------------------------------------------
On Fri, 21 Jan 2011, Glenn Maynard wrote use cases!:
> 
> Seek bars should use "fast", particularly for longer videos and videos
> with keyframes spaced far apart.  Videos encoded to allow quick seeking
> by using frequent keyframes, on the other hand, might use "accurate"
> even for seek bars.
> 
> Chapter seeking should use "accurate".  If a chapter is at 1h45m12.2s, a
> fast seek would end up seeking into the previous scene, or into the   
> middle of a fade-out.
> 
> User-specified seeks, like YouTube's "tm" URL parameter, should be
> "accurate", for the same reason as chapters.
> 
> A "seek ahead by N seconds" button should use "accurate".
================================================================================
 #11  Ian 'Hixie' Hickson                             2012-04-30 23:21:36 +0000 
--------------------------------------------------------------------------------
On Sat, 22 Jan 2011, Michael Dale wrote:
>
> It seems the needs for a seek bar are different from the api usage of
> setting video.currentTime. As people mention rules of 'least surprise'
> are important. If you set currentTime to 3.453 you expect it to return
> something very close to 3.453 ..
> 
> Why not have a separate api for "seek bars" that just seek to a
> percentage of the video stream. Letting implementations do what's
> fastest and consistent, but without guarantees for a given play time
> being set. This way the different needs are addressed and we don't  
> conflate the currentTime api with parameters for accuracy.
> 
> In terms of real world use cases, we use accurate seeks in an html5
> video editor. Its especially important when flattening with firefogg
> that as we can accurately set a given time in source videos per our 
> output frame interval.
>
> http://commons.wikimedia.org/wiki/Commons:Sequencer
================================================================================
 #12  Philip J                                        2012-05-02 09:00:06 +0000 
--------------------------------------------------------------------------------
(In reply to comment #9)
> Based on that commentary, maybe currentTime shouldn't be the approximate
> version. So:
> 
>    media.currentTime = targetTime; // precise, may take seconds to complete
>    media.fastSeek(targetTime); // fast, could be inaccurate
> 
> fastSeek() would also additionally ensure that if targetTime is before
> currentTime, the new currentTime is also before the old currentTime, and vice
> versa.
> 
> I like this proposal best of the ones so far because it is more intuitive,
> API-wise. setting currentTime always sets it, it doesn't end up with some
> random other value, and the operation that sets it but does so potentially
> inaccurately is a method.

I think this makes sense now that that currentTime is stable during script execution. (When I wrote "a reference point: initialTime or currentTime" I think I meant "absolute search" vs "relative search", but that can be achieved without a flag as long as currentTime is stable.)
================================================================================