Meeting minutes
Chris: Welcome everyone, this is the WICG DataCue meeting
TextTrackCue constructor
Slideset: https://
Rob: Thank you for joining today
… I'm leading development of Web Video Map Tracks, synchronising location and video
… Made some good progress this year, so want to give an update and discuss next steps
… TextTrackCue provides a common interface, common attritubes, startTime and endTime, and a pauseOnExit flag
… There's a cue content interface, the abstract component
… VTTCue is for timed text, takes a string, and DataCue which takes type and value
… DataCue has only one browser implementation, so removed from the HTML spec
… Currently the constructor isn't exposed, and this prevents custom cues
… I was puzzled, but now I know why. Browser subclassses werent' possible when TextTrackCue was first designed in 2014
… TextTrackCue has a constructor, and exposing this would be a small change, easy to implement
… Constructor access shouldn't be confused with instantiation
… The instance is returned by the end of the constructor. An abstract class can be implemented in WebIDL with suitable constructor steps. Errors can be raised during instantiation, as a way to implement an abstract class
… The parent class is an input to the constructor. I've discussed with Alicia, foolip, and he gave it a thumbs up
… Abstract classes and objects shouldn't be confused with abstract functions. TextTrackCue doesn't have abstract functions
… What does the constructor look like? startTime and endTime are required. HTML gives defaults to the remaining attributes
… pauseOnExit is false, identifier is empty string
… That is in the TextTrack section of the HTML spec
… There could be a second form of the constructor that includes an identifier. That might be unnecessary as the id can be set after construction
… The proposal here is designed to be as simple as possible
… I'm aware there's a proposed WebKit enhancement, from 2020 or 2021, to make TextTrackCue a concrete class. Not sure why that is.
… This is equivalent to DataCue type = undefined, which I feel impedes access to the content
… I think it was proposed for the same reason as I propose access to the construtor, to enable subclassing
Eric: It was proposed as a way to handle the situation where a site doesn't use VTT, and creates DOM nodes from some other text track cue format
… That's a challenge for accessibility when it's possible for the user to define their preferred cue styling at the system level.
… If the browser renders cues, it can apply the user's custom styles
… If a page converts from a non-browser supported cue format, and inserting DOM nodes into the page, there's no way for the browser to know they're cues, and it can't apply the user's preferred format
… Our proposed extension to TextTrackCue takes a document fragment, and some additional markup that allows the page to identify the cue and the background
… WIth that the browser can apply the user's preferred style, and put the node into an isolated shadow DOM that the page can't access, and so use the user's preferred styling as a fingerprinting vector
… A large complication with WebKit with your proposal is that we have an experimental version of the API in WebKit
… It has a constructor that has startTime and endTime, and requires a document fragment
Rob: I've seen that. Can it be done with a subclass. In issue 35 I propose a subclass that sits between TextTrackCue and VTTCue, so there's a FragmentCue subclass that are then inherited by VTTCue. Functionally equivalent
Eric: It would be a breaking change to WebKit. You propose moving the implementation from TextTrackCue to a new FragmentCue class
Rob: Yes
… The proposal to exposing the TextTrackCue constructor would allow anyone to make subclasses
… The other problem is, the changes are specific to timed text, and inherited for every other sort of cue, e.g., timed metadata, where they don't make sense
Alicia: Eric, couldn't we still make that work. I can think of two possibilities
… Use the extra argument, and return a new FragmentCue. I assume it's possible as the extra arg is a node. I haven't studied in detail though
… Other way is having a class in the other direction, instead of a child of TextTrackCue, maybe something like TimeCue, not specific to text, just timed metadata. It makes some sense as a name, and could be possible vector for extension
Eric: If that can be done without breaking existing DataCue, which is used extremely widely
… It's used for ad insertion and ad metadata
… Also used for IDv3 metadata in shoutcast streams for radio stations, for example
cpn: If you were designing this from scratch, TimedCue would be the parent, with TextTrackCue and then timed metadata branch where DataCue lives
… How to determine compatibility on that?
Alicia: Changing DataCue, which inherits TextTrackCue, could be breaking
Eric: Possible to have multiple constructors, e.g., MediaStream has 3 forms of constructor. If we make a TextTrackCue constructor that takes start, endtime and id,
… Alicia's first proposal would work. It would be an implementation detail in WebKit, but not a breaking change
Rob: There's currently no constructor, and given WebKit's implemntation takes document fragment, it could distinguish [missed]
Eric: Did you suggest having an identifier as the third argument?
Rob: I did, but not necessary, as we can distinguish a cue fragment and a string as third argument
Eric: That wouldn't matter
Rob: I'm trying to make the minimum change, assuming that's better
Eric: If it makes sense to have the id in the constructor, then have that. The amount of work involved shouldn't be the criteria
Rob: My motivation was the reason it was dropped from HTML
Eric: It can be defined in IDL by setting the default value
Rob: So I think stick with just start and end time
Chris: Not sure I understand the first option
Rob: Arguments are passed as an array, they're not explicitly typed, so in the constructor you'd have to count them.
Chris: So would DataCue get a CueFragment that it doesn't use
Eric: I think it makes sense to have the ID, as it's an attribute in TextTrackCue. Marking it in IDL that it's empty when not supplied, takes care of it
… When script wants to initialise the ID, script doesn't need an additional step
… to set the id
Rob: To confirm, we just have the second constructor (with startTime, endTime, id)
… I'm happy with that
Chris: Need to have other browser interest to add to the spec
Rob: With the constructor, developers can write polyfill
… If not native to the browser you can delcare it as a class easily. That would encourage adoption by other browsers
Rob: We have a proposed way forward
… While I was looking at all this, I looked at the WPTs. I noticed that polyfills weren't behaving as expected
… There are some WPT failures, onenter and onexit. Those fail because of dispatchEvent, which should inherit from EventTarget, but wasn't doing anything
… It should call synchronously but doesn't
… It works in Firefox. I raised issues in Chrome and Safarai
Alicia: WebKit/
Rob: Fixed in Chrome, and there's a PR in WebKit
… It affects how TextTrackCue can be used, with WebVMT for example we dispatch events directly on the cue, and bypassing TextTrackCue as time marches on doesn't meet our requirements as it's stateful
… WPTs provide useful insight
… Tests still fail on unbounded cues
… [Summarises test result numbers]
… Exposing the constructor means we can streamline the WPTs by removing the VTTCue dependency from the tests, which made tracking down bugs difficult
… Community Benefits. Allowing JS access allows support to progress by the community, reducing burden on browser developers
… It also reduces browser overhead, by streamlining the tests
… There are working polyfills. In the Sync on the Web CG live streaming use cases, there are polyfills that work on the 3 main browsers. This includes demos to explore design features, such as streaming, multiple tracks and handlers, duplicate cue types. I'll add a custom scheduler demo, as TextTrackCue can be used on its own without TextTrack
… The Live Screening case, e.g., a music or theatre event, it's streamed live to remote venues
… The use case is to stream the lighting effects so that lighting can be re-created in the remote venue
… It includes a DataCue polyfill, and highlights some interesting requirements, e..g, duplicate cue types, many concurrently active cues, and concurrent cue changes
… It's quite different from timed text
Alicia: It depends, some timed text, e.g., WebVTT allows overlapping cues. Support is a bit flaky though
Eric: How does the lighting control go from data to controlling the lighting?
Ken: For DMX lighting device, it's controlled by USB, for example using WebUSB API
Eric: So it's only working in Chrome
Ken: Yes
Chris: Should we propose spec text change, and if so, where?
Rob: I've been thinking about that. Changes are needed are minimal
… Philip suggested raising it as an issue in HTML so he can comment directly
… I think we resolved the conflict between this proposal and WebKit's approach, so no problem raising an HTML spec issue
Eric: I agree. DataCue is orthogonal, but I think it would be extremely useful to have a proposal to reintroduce DataCue, as it's widely used, and it is useful
Chris: Is it the surfacing of media in-band events you're thinking about?
Eric: Yes
Rob: Would it be possible to develop a polyfill, using TextTrackCue?
Eric: There's huge utility in being able to embed metadata in the media file, so you have one single file and the delivery of the metadata is timed with the audio and video samples
Alicia: In the case of MSE, there's lots of JS code parsing the binary data. This constructor proposal allows you to write a DataCue polyfill to handle those, put them in a new track
… It could motivate other browsers to implement, as it would be a performance improvement
Eric: Our existing DataCue constructor is used with libraries that parse containers for use with MSE
Alicia: Do they work with other browsers?
Eric: I don't know that they do
Alicia: It creates a chance of getting adoption, and enables something that can't be done already
Chris: I'm happy to support work on DataCue through chairing, if there's a willing community
Rob: I was thinking about a breakout at TPAC
Rob: I can move forward, and very happy to have had the discussion
… Demos, the OGC Testbed, the DMX lighting demo from Ken
Chris: Happy to coordinate on a TPAC breakout, if you want to. I think the goal there would be to encourage other implementations, based on the design we've arrived at today
[adjourned]