Media Timed Events / DataCue API -- 20 Apr 2020

<kaz> Meeting: Media and Entertainment IG - Media Timed Events TF

<scribe> scribe: tidoust

Agenda

<inserted> https://docs.google.com/presentation/d/1KPX8OYQzKbNXG_AqpDgktMkS9aKwei_ffj_3G0E3ygk/edit Slides

cpn: Two calls in one. IG Note, the document is, I believe, complete. Needs for improved synchronization capabilities were pursued in WHATWG, and then datacue for need to expose media timed events (in-band or not).
... Question that remains is whether we believe there is more for the TF to do.
... Second topic is the Datacue incubation itself.
... I'd like feedback to guide the specification.

Media Timed Events IG Note

cpn: [overview of spec sections]
... In the last few weeks, we've done some work to address remaining issues. Everything's merged, no open issue at the moment.
... Do we think from our TF perspective that this document is complete?
... I'd like us to focus our attention on the DataCue API itself as next stage.
... Assuming you agree, I'd like to send a Call for Consensus to the IG, which we need to publish an updated IG Note.
... Peter, it may be worth hearing about your use case, as it may be something that needs to be added to the document.
... Do others who contributed to the document think that it is complete?

RobSmith: I think that it is very complete.

[No other comment]

cpn: OK, let's go through your use case then, Peter.

PeterThoPesch_IRT_: I don't think that there may be additional requirements derived from this use case.
... The use case is about graphical elements that we would like to transport separately from the video. Things like TV graphics where the name of a guest is added to the picture, or sports graphics.
... For the time being, these graphics are integrated in the video and streamed to all devices.
... But video ratio may depend on the device (landscape, portrait, 1:1). We'd like to make the video experience more responsive by separating the video and the graphics which could then be adapted to the video ratio and positioned correctly.
... We didn't find any proper solution except to use text tracks to connect that with the video.

cpn: How do you expect to deliver the images?

PeterThoPesch_IRT_: That's a different problem. We could imagine having an HTML/CSS animation on top of the video, or some (proprietary) JSON format to describe the graphic.
... There's one company I know who does that with a proprietary JSON format to render HTML content.
... Also example of body moving in Adobe.

cpn: I was wondering whether you were considering web resources tracks that go inside your MP4 container.

PeterThoPesch_IRT_: That could be a solution. And then it would be synchronized by DASH streams.
... One problem is that the player may not be Web-based. It may be a native player.
... There may be multiple delivery channels.

cpn: It sounds like DataCue as we're discussing here could be a solution to this, as a way to associate metadata with a position in the media timeline.
... The other part of the solution would be the actual rendering which is something that we touch on in this document in the gap analysis around synchronized rendering.
... The "requestVideoFrameCallback" proposal would help.

PeterThoPesch_IRT_: Synchronization needs are probably stronger for graphical elements than regular text captioning. There may be vector graphic animations for instance.

Description of responsive graphics synced with video use case

PeterThoPesch_IRT_: I don't know if we should add the use case to the document.

cpn: The requirements are covered although use cases are slightly different.
... Part of me wants to conclude work on this document.
... If you feel strongly that this should be included, we can.

PeterThoPesch_IRT_: I trust that the requirements are covered. I could perhaps draft a section by tomorrow noon?
... If it's going to slow things down, I would say no.

cpn: It doesn't slow things down by a lot. Not a big problem, really.
... The main thing is that I want to capture the correct recommendations. Please check the part on rendering.

PeterThoPesch_IRT_: OK, I'll look into it by noon tomorrow.

<kaz> https://www.w3.org/TR/2011/NOTE-hnreq-20111201/

tidoust: +1 to including the use case, it seems more mainstream than the use case that's in the document that covers the same needs.

kaz: OK to include, wondering about the title of the document.
... E.g. "Requirements for Media Timed Events"
... Also consider a shortname update, maybe?

tidoust: doable but a slight pain given that document has already been published to /TR/

rob: Another way to capture the use case would also be to open an issue in the DataCue repo

cpn: Right, I agree with including the use case in the IG Note. We need to be a little bit careful about the explainer of DataCue

DataCue open issues

cpn: We haven't started to draft a spec yet. Focus is on the explainer. In the next week or so, I hope to have a first draft of a spec.
... We have a good collaboration starting now with the DASH-IF.
... [showing an architecture diagram from DASH-IF]
... You can see how the DataCue API interfaces with the rest, with timed event buffers, the manifest parser.
... The DASH player can subscribe to receiving specific timed events. And then we may have events that are specific to a Web application. The Web application can subscribe to specific media timed events.
... "onreceive" lets you know that an event has been received in advance, allowing to do things such as fetching external resources before the event takes place.
... "onstart" is triggered when the event fires.
... Also worth looking at a "Type 1" player, which is not standardized at W3C, but is common. Any API that we come up with, I'd like us to support both Type 3 and Type 1, although Type 1 is more the focus.
... Anyone that wants Type 1 integration would likely have to specify it outside of W3C.
... That's the architectural overview of where I think the DataCue API fits.

-> Updating cue time attributes can generate unexpected events (#19)

<kaz> Issue 19

cpn: Regarding issue #19, there is this potential case where, if you change the timing of a cue, the platform may trigger duplicate "enter" and "exit" events.
... The current outcome is that maybe we don't need to propose a change to the HTML spec for now, but need to be something that we keep in our heads when we specify the API.

RobSmith: That's correct.

cpn: Thanks a lot for the description. That crystallizes things quite nicely.

DataCue or a more specific API for DASH and emsg events

cpn: Introduce TextTrackCue.text, DataCue blob data, and then thinking a DASH events in particular, should we have a dedicated emsg cue interface?
... TextTrackCue does not have a constructor, and there is no way to attach the actual data payload.
... VTTCue has a text string which you could use to carry the data, and a constructor. But it also has a bunch or positioning properties that we don't need for DataCue. Not a very good fit for the requirements that we have.
... What remains is the DataCue interface.
... As specified in HTML5, it would have a "data" attribute, and then extensions implemented in WebKit: constructor, the actual payload in a "value" attribute and the type of payload that is being carried in a "type" attribute.
... It would be interesting to see how the mapping between "type" and "value" works in practice. ID3 tags are supported in WebKit, I believe, for instance.
... Another option would be to create an EmsgCue interface.
... Similar to DataCue, you'd have a scheme ID URI to identify the type of value.
... I don't think that we need a specific EmsgCue. The DataCue API, as specified, would provide the right abstraction provided that we define the "type" mapping properly.

RobSmith: I have a limited understanding of the emsg interface, but my feeling is that it's a good example of actual data, so including that in the DataCue design seems important.

cpn: In the explainer document, there is a section about mapping between emsg events and DataCue.

<kaz> DataCue Explainer

<PeterThoPesch_IRT_> How can I put myself on the speaker queue?

cpn: I think this can work, and I know that the DASH-IF group has some very useful work that we can point at that talks about timing scales (timing is represented differently in emsg events)

<PeterThoPesch_IRT_> ok, thanks :-)

cpn: Can we align closely with Webkit implementation.
... Or should we depart. Feedback from Webkit developers would be welcome.

RobSmith: Also a good way to testing out their design. I'm wary of reinventing something that's already been done. If we can reuse, that would be great.
... Also, application-generated vs. user-agent generated cues is going to be an interested use case.

cpn: Yes, I'm seeing the application generated use case, as something that would be correctly supported by the DataCue API as designed here.
... The "any value" would cover all possible needs.
... The emsg boxes and in-band events are where I see complexity in defining the API and processing rules.
... The "data" and "value" attributes are redundant and possibly confusing. There may be a need to preserve that. Conversation to be had with implementers.
... I encourage you to look at open issues

<PeterThoPesch_IRT_> Just a thought: Might there be an event for seeking? Like onTimelineDiscontinued? If so, would that affect the solution?

cpn: A note to say that W3C members are welcome to DASH-IF calls every second Friday.

Ali: Just to confirm you're welcome. We'll also probably have updates based on ongoing MPEG discussions.

Next meeting

cpn: May, 18th 2020.
... One call per month. We can revise that if that feels needed.

- DRAFT -

Media Timed Events / DataCue API

20 Apr 2020

Attendees

Contents

Agenda

Media Timed Events IG Note

DataCue open issues

Next meeting

Summary of Action Items

Summary of Resolutions

Scribe.perl diagnostic output