User:Spfeiffe/TextTrackChange

Text Track Cue API Changes

Recently, the Text Track Cue API has changed from:

enum AutoKeyword { "auto" };
[Constructor(double startTime, double endTime, DOMString text)]
interface TextTrackCue : EventTarget {
  readonly attribute TextTrack? track;

           attribute DOMString id;
           attribute double startTime;
           attribute double endTime;
           attribute boolean pauseOnExit;
           attribute DOMString vertical;
           attribute boolean snapToLines;
           attribute (long or AutoKeyword) line;
           attribute long position;
           attribute long size;
           attribute DOMString align;
           attribute DOMString text;
  DocumentFragment getCueAsHTML();

           attribute EventHandler onenter;
           attribute EventHandler onexit;
};

to

interface TextTrackCue : EventTarget {
  readonly attribute TextTrack? track;

           attribute DOMString id;
           attribute double startTime;
           attribute double endTime;
           attribute boolean pauseOnExit;

           attribute EventHandler onenter;
           attribute EventHandler onexit;
};

with the addition of a new WebVTT specific text track cue API:

enum AutoKeyword { "auto" };
[Constructor(double startTime, double endTime, DOMString text)]
interface VTTCue : TextTrackCue {
           attribute DOMString vertical;
           attribute boolean snapToLines;
           attribute (long or AutoKeyword) line;
           attribute long position;
           attribute long size;
           attribute DOMString align;
           attribute DOMString text;
  DocumentFragment getCueAsHTML();
};

Relevant patches:

• Split TextTrackCue into an abstract TextTrackCue interface and a WebVTT-specific interface VTTCue.
• Split more WebVTT-specific things into the WebVTT spec.

The aim of these patches:

• Remove WebVTT-specifics from the generic TextTrackCue object
• Create an abstract TextTrackCue API that can be extended to other file formats such as TTML,

Related bugs:

• addTextTrack incorrectly assumes WebVTT content type
• TextTrackCue() constructor incorrectly assumes WebVTT content type
• revert removal of text and getCueAsHTML members; address constructor changes

Discussion issues

(1) Removal of .text from TextTrackCue

Argument to re-introduce:

• every TextTrackCue from every format that has been implemented by UAs has a content - this content should be exposed by the same IDL attribute in all TextTrackCue objects
• examples are the VTTCue API, TTML support in IE10, and MPEG-2 to HTML mapping
• even binary timed data has this far been delivered as text, e.g. data urls for timed thumbnails in WebVTT
• the @inBandMetadataTrackDispatchType on the TextTrack object interprets the content of a generic TextTrackCue - yet, there is now no attribute that contains that content
• we are not solving hypothetical use cases of potential binary cue content that nobody is currently intending to implement

Argument against:

• there may be timed cue formats that don't expose text, e.g. images, bitmaps or URLs

Proposal:

• re-introduce the .text attribute

(2) Removal of getCueAsHTML() from TextTrackCue

Argument to re-introduce:

• text track cues generally need to be rendered and getCueAsHTML() will provide a document fragment that the UA can render
• examples are WebVTT cue rendering, TTML rendering, and MPEG-2 to HTML making use of getCueAsHTML()

Argument against:

• there will be many timed cue formats with binary content that will not be converted to a HTML fragment for rendering, e.g. images, bitmaps, data URLs
• every new format that a UA supports needs an extensive algorithm to convert a cue's content to HTML - this needs to be specified and implemented in the UA and discoverable that it's available - own Object definitions make sense, e.g. TTMLCue API, CEA708CueAPI
• the MPEG-2 to HTML document defines getCueAsHTML() always as null except where the UA knows how to convert a caption cue to HTML - it makes sense that the UA returns a more appropriate object in these cases, such as VTTCue or TTMLCue, such that it is discoverable what type the UA identified

Proposal:

• no change

(3) Removal of TextTrackCue constructor

Argument to re-introduce:

• TextTrack.addCue() requires a cue to be constructed in HTML; by removing the constructor, we can only create new cues of a particular type (VTTCue only at this point in time)
• it makes more sense for JS-created, non-rendered kind=metadata cues that don't use most of the VTTCue attributes and not its rendering algorithm to be a TextTrackCue object
• previously it was possible to construct a cue of the WebVTT type and append it to a track with interpretation of cue content done by JavaScript, e.g.

cue = new TextTrackCue(0,2,'<region xml:id="r1" tts:origin="10px 100px" tts:extent="300px 96px" /> <body xml:id="b1"> <div xml:id="d1"> <p xml:id="p1">Text 1</p> </div> </body> </region>'); that's now possible only using new VTTCue and thus can break existing JavaScript implementations

• if the browser knows how to parse a text track (either from <track> or from an in-band track), but doesn't know how to parse the cue content, it has no way to expose them to JS right now
• spec has been changed before and looking at https://github.com/search?l=javascript&q=TextTrackCue&ref=searchresults&type=Code , there are already several incompatible uses of TextTrackCue constructor out there

Argument against:

• TextTrackCues that can be constructed but not rendered make sense only for @kind=metadata tracks
• breaks backwards compatibility without people noticing the change

Proposal:

• re-add constructor to TextTrackCue:

[Constructor(double startTime, double endTime, DOMString text)]
interface TextTrackCue : EventTarget {
 ...
};

Potential further improvement:

• introduce a cueFormatHint attribute for TextTrack, replacing the existing inBandMetadataTrackDispatchType attribute and providing an optional cue format hint for any TextTrack
• cueFormatHint is by default "plaintext"
• for WebVTT files parsed from <track>, the cueFormatHint would be "webvtt" for cues that are supposed to be parsed with the VTTCue, or "plaintext" otherwise
• however: browser should create VTTCue instead of TextTrackCue with "webvtt" as the hint and JS devs should use instanceof to find out what object type the browser created for in-band tracks