Bug 17273 - <track> Introduce an API to occupy parts of the video rendering area to push away cues
<track> Introduce an API to occupy parts of the video rendering area to push ...
Status: ASSIGNED
Product: WHATWG
Classification: Unclassified
Component: HTML
unspecified
All All
: P3 enhancement
: Needs Impl Interest
Assigned To: Ian 'Hixie' Hickson
contributor
:
Depends on: 22075
Blocks:
  Show dependency treegraph
 
Reported: 2012-05-31 15:21 UTC by Simon Pieters
Modified: 2014-07-21 20:48 UTC (History)
8 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Simon Pieters 2012-05-31 15:21:40 UTC
Currently if you set line:-2 and the line wraps, the -1 line gets filled with text. If you wanted to have the -1 line "free", that seems annoying, especially since you can't know if the line will wrap or not.

I can think of two ways to solve that.
(1) introduce a setting to make horizontal lines grow upwards instead of downwards.
(2) introduce a way to make a "hole" in the rendering area that won't be filled by cues, kind of like the native controls makes a hole covering up part of the rendering area. Such a hole probably shouldn't move when the controls are shown, unlike normal cues.
Comment 1 Simon Pieters 2012-05-31 16:09:50 UTC
The <video> rendering area might not be the same as the rendered video. If you want to make sure part of the rendered video is not overlapped by cues, then (2) but using the rendered video's area instead of the full <video> area seems best.
Comment 2 Simon Pieters 2012-05-31 16:10:28 UTC
(The rendered video's area can also be changed with CSS object-fit)
Comment 3 Simon Pieters 2012-06-11 14:18:10 UTC
A variant of (2) would also be useful when you use native cues but custom controls -- you don't want your controls to overlap the cues, you want the cues to move, just like with the native controls, but there's currently no way to do that. There should be an API to say "now my controls are visible and take up this space" and "now they're hidden".
Comment 4 Ian 'Hixie' Hickson 2012-07-24 16:06:50 UTC
We really need more data to see how useful this would be, I think.
Comment 5 Ian 'Hixie' Hickson 2012-08-07 23:28:21 UTC
Things that would be helpful here:

- the use case (what is the hole for? is it just overlapping scripted controls?)
- how common is this? (how common is it in cases with automatic wrapping, as opposed to manual wrapping where you can just move the text up a line?)

In particular, if the use case is just scripted controls it might make sense to have an explicit way to either squeeze the lines area (see the overscan bug) or have an explicit way to say "don't overlap over this area", rather than a way to say "overflow up". This is because in general you might not have control over the subtitles in the scripted controls case, so changing the settings isn't a great solution.
Comment 6 Simon Pieters 2012-08-13 10:57:22 UTC
I can think of two use cases:

A. custom controls that want to push away subtitles just like the native controls.
B. authoring subtitles and wanting to avoid overlapping a particular part of the video, for instance a sign.

Use case A can't be a WebVTT setting, as you say. For use case B, what I recall from TV is that the subtitles are usually positioned at the top. That might be fine, but a `line:-2 grow:upwards` is maybe less distracting. I don't have data, but I'd say use case A is common for anyone wanting to use custom controls and subtitles. Anecdote: I recall when Firefox 3.5 was released their home page had a <video> with custom controls and custom subtitles, and the subtitles moved when the controls became visible.
Comment 7 Silvia Pfeiffer 2012-08-23 05:12:20 UTC
YouTube does that with subtitles, e.g. http://www.youtube.com/watch?v=QRS8MkLhQmM . See how they move down a bit when the transport bar minimizes? Also, when you go fullscreen and wait until the controls disappear, the subtitles move down, too.

Similarly when there are ads, e.g. http://www.youtube.com/watch?v=2bgk0dnNjaY&cc=1 .

Neither of these examples are automatically line-wrapped, though, but they support the use case of making a "no-go" zone on the video.
Comment 8 Simon Pieters 2012-08-23 07:33:28 UTC
(In reply to comment #6)
> A. custom controls that want to push away subtitles just like the native
> controls.

So for this use case I propose the following API:

partial interface HTMLVideoElement {
  readonly attribute VideoViewport viewport;
};

interface VideoViewport {
  void occupy(double x1, double y1, double x2, double y2);
  void clear();
};

video.viewport.occupy(x1, y1, x2, y2) occupies the area specified by the coordinates so that cues do not overlap with this area, and reruns the rules for updating the text track rendering. The coordinates are in percentages of the video's rendering area. If any argument is less than 0 or greater than 100, throws IndexSizeError.

video.viewport.clear() removes all occupied areas added by the occupy() method and reruns the rules for updating the text track rendering.

Example: video.viewport.occupy(0, 90, 100, 100) occupies the bottom 10% of the video.
Comment 9 Philip Jägenstedt 2012-08-23 08:15:41 UTC
Wouldn't a simpler solution to the original line:-2 problem be to simply change the algorithm to initially position the last line at -2 and using line -1 only if no position in the other direction (up) was found?

As for a VideoViewport API, something like that would indeed be handy for scripted controls. However, I think that such an API should also make it possible for multiple clients to not interfere with each other, so that native controls implemented using JavaScript would not clobber areas added by on-page scripts, or vice versa. Similar to onclick vs add/removeEventListener('click', ...).
Comment 10 Simon Pieters 2012-08-23 09:04:54 UTC
(In reply to comment #9)
> Wouldn't a simpler solution to the original line:-2 problem be to simply change
> the algorithm to initially position the last line at -2 and using line -1 only
> if no position in the other direction (up) was found?

That doesn't help if the sign you don't want to cover is above the cues instead of below.

> As for a VideoViewport API, something like that would indeed be handy for
> scripted controls. However, I think that such an API should also make it
> possible for multiple clients to not interfere with each other, so that native
> controls implemented using JavaScript would not clobber areas added by on-page
> scripts, or vice versa. Similar to onclick vs add/removeEventListener('click',
> ...).

That's a good point. Maybe something like setTimeout's design, where an id is returned that you can pass as argument to clear()?

interface VideoViewport {
  long occupy(double x1, double y1, double x2, double y2);
  void clear(long handle);
};
Comment 11 Simon Pieters 2012-08-24 11:01:00 UTC
(In reply to comment #10)
>   void clear(long handle);

  void clear(optional long handle);

No argument means everything is cleared.

Also, when these methods are called before playback has started (i.e. if the poster would show), the track rendering rules should not be run.
Comment 12 Ian 'Hixie' Hickson 2012-10-30 00:37:35 UTC
Proposal:

  partial interface HTMLVideoElement {
    readonly attribute VideoViewport viewport;
  };
  
  interface VideoViewport {
    VideoViewportArea add(double x, double y, double width, double height);
    void clear();
  };
  
  interface VideoViewportArea {
    void remove();
  };

(I changed x2/y2 to width/height for more consistency with the bulk of APIs on the platform, but if there's a strong reason to prefer two coordinates instead, I'm happy to do that too.)
Comment 13 Philip Jägenstedt 2012-10-30 08:48:25 UTC
I think this looks reasonable for the use cases in Comment 6. Do we also want the native controls to use this API so that scripts doing stuff with metadata tracks can avoid overlapping them? Given that native controls can be implemented using HTML+JavaScript+CSS internally, it's certainly an attractive option. However, should scripts be allowed to clear() the areas from the native controls, or should those areas have a magic bit that prevents that?
Comment 14 Silvia Pfeiffer 2012-10-30 10:42:38 UTC
So to achieve the coverage of the bottom area as in Simon's example, we would write: video.viewport.occupy(0, 90, 100, 10) ?

SGTM
Comment 15 Ian 'Hixie' Hickson 2012-10-30 17:50:18 UTC
foolip: Ah, interesting idea. I hadn't considered script-positioned cues. For that, we'd have to add some logic that lets scripts determine what areas are covered or not, somehow. Maybe we could even include areas covered by cues, or something. What would be most useful, would it be a way to get a list of covering areas, or a way to check whether an area is covered or not?

This seems simplest:

  interface VideoViewport {
    VideoViewportArea add(double x, double y, double width, double height);
    void clear(); // clears only author-added areas
    boolean isCovered(double x, double y);
    boolean isCovered(double x, double y, double width, double height);
  };

(Or maybe just the one method, and require width=0 height=0 to check a point?)

Any opinions on how to deal with negative numbers in width and height?
Should out of range values just clamp to the viewport?
Comment 16 Philip Jägenstedt 2012-10-31 08:46:37 UTC
An isCovered function could be a nice utility function, but I assume that we would also expose a list of the covered areas? Otherwise, it seems like one has to write a non-trivial algorithm to find an area that is not covered, right?
Comment 17 Ian 'Hixie' Hickson 2012-10-31 20:00:51 UTC
How about some method to find the nearest space of a particular dimension, then? Would that be more useful?
Comment 18 Philip Jägenstedt 2012-11-01 08:35:21 UTC
I suppose it would, but wouldn't the simplest solution for the first iteration be to just expose a (possible merged) list of covered areas? Is there some reason to avoid that?
Comment 19 Ian 'Hixie' Hickson 2012-11-01 22:27:25 UTC
Exposing a list of covered areas doesn't seem especially useful, and has a much bigger API surface. Hence my question about what the use cases are.
Comment 20 Philip Jägenstedt 2012-11-02 08:19:30 UTC
The use cases in Comment 6 are:

> A. custom controls that want to push away subtitles just like the native controls.

This doesn't require an API for exposing the covered areas at all, they only need to be known internally when doing WebVTT layout.

> B. authoring subtitles and wanting to avoid overlapping a particular part of the video, for instance a sign.

This seems like it should be part of the WebVTT syntax, but also doesn't require exposing the covered area.

In addition to this, we have the use case of scripted cues (likely using metadata tracks) wanting to avoid overlapping native controls and native cues. I have myself written demos where I wanted to use the native controls but put other stuff on top of them. However, I am not convinced that this is common enough outside of demos to warrant a solution at this point.

So, not exposing the covered areas to scripts at all might be an OK first step...
Comment 21 Philip Jägenstedt 2012-11-02 08:21:16 UTC
And I guess another use case is to allow User Agents to avoid putting cues in the overscan area, which is already solved in the spec but could potentially be simplified if handled together with controls and cues overlap handling.
Comment 22 Ian 'Hixie' Hickson 2012-11-02 21:15:50 UTC
B is kind of solved by just putting an empty cue over the area of the sign, at least if the styles don't give a background.

If we don't need to solve the C problem (scripted cues), then are we good with comment 12's proposal?
Comment 23 Philip Jägenstedt 2012-11-05 10:29:33 UTC
Yeah, I think comment 12 is a good start. Simon, Silvia?
Comment 24 Simon Pieters 2012-11-05 10:57:15 UTC
Looks good.
Comment 25 Silvia Pfeiffer 2012-11-05 11:22:39 UTC
I think it's good.
But how about comment 15?
Comment 26 Philip Jägenstedt 2012-11-05 12:45:09 UTC
(In reply to comment #25)
> But how about comment 15?

The question is if it's important enough to add an API for just yet. At least I hadn't thought about avoiding overlap with scripted cues before this bug, so I self-centeredly guess that it's not a huge problem, yet.
Comment 27 Ian 'Hixie' Hickson 2012-11-05 20:03:01 UTC
Ok, I'll do comment 12 for v2. For v3 we can revisit this; if you want this to be examined regularly then file a bug and mark it LATER; otherwise, we'll just revisit this if people actually run into it.
Comment 28 Ian 'Hixie' Hickson 2013-03-15 22:02:53 UTC
Silvia, this will require a way for the WebVTT spec to receive reserved regions from the HTML spec. Once I've done the change in HTML, I'll hand it over to you.

Thinking about the API in comment 12, video.viewport.add(...) is unclear about what it's adding. We could rename "add()" to "reserve()"?
Comment 29 Ian 'Hixie' Hickson 2013-03-15 22:05:59 UTC
*** Bug 20349 has been marked as a duplicate of this bug. ***
Comment 30 Silvia Pfeiffer 2013-03-16 01:58:25 UTC
I agree, reserve() would be a better name for the function.

I'll have to think in more detail about the interaction between TextTrackRegion and Viewport and normal cues.
Comment 31 Ian 'Hixie' Hickson 2013-04-15 23:08:03 UTC
Is there (still) implementation interest for this? (Just checking given recent browser engine changes.)
Comment 32 Silvia Pfeiffer 2013-05-24 10:00:59 UTC
Please consider alternative proposal in bug 22075