14:54:54 <RRSAgent> RRSAgent has joined #me
14:54:54 <RRSAgent> logging to https://www.w3.org/2022/08/15-me-irc
14:54:58 <Zakim> Zakim has joined #me
15:00:04 <cpn> Meeting: Media Timed Events TF
15:00:16 <cpn> Present: Chris_Needham, Yuhao_Fu
15:00:23 <cpn> Chair: Chris_Needham
15:00:29 <cpn> scribe: cpn
15:00:43 <cpn> present+ Kazuyuki_Ashimura
15:00:54 <xfq> xfq has joined #me
15:01:19 <cpn> present+ Fuqiao_Xue
15:03:29 <cpn> Topic: Introduction
15:03:42 <cpn> ChrisN: Main topic is preparations for upcoming TPAC meeting
15:04:31 <cpn> ... Agenda for Media WG and MEIG is in planning, we could include an update on the work here
15:05:19 <cpn> Topic: TPAC plans
15:05:20 <RobSmith> RobSmith has joined #me
15:05:32 <cpn> ChrisN: I've started preparing some slides
15:06:03 <Yuhao> Yuhao has joined #me
15:06:08 <cpn> ... We have open questions, particularly about SEI events
15:06:29 <cpn> ... It's not clear that DataCue is the right solution in all cases
15:06:39 <kaz> agenda: https://lists.w3.org/Archives/Public/public-web-and-tv/2022Aug/0002.html
15:07:11 <RobSmith> Could someone post a link to join the meeting please.
15:08:33 <cpn> ... For example, if we have metadata on every video frame, then DataCue not the right solution
15:09:07 <cpn> ... Need something more like the proposal from Yuhao
15:09:51 <cpn> Yuhao: Do we need additional description for this scenario?
15:10:09 <cpn> ChrisN: I think so, yes. Is that a use case you are interested in?
15:11:43 <cpn> Yuhao: Yes, sometimes, but I don't think it's a good case. If video is 60FPS, each frame takes 16ms to trigger the metadata, frequency is a problem
15:12:20 <kaz> q+
15:13:35 <cpn> Kaz: Wondering about TPAC agenda
15:14:11 <cpn> ... Should we concentrate on SEI event and clarify use cases, we can look into some expected template for that description
15:14:33 <cpn> ... For device and network settings, expected behaviour and pain points, e.g., time resolution
15:14:50 <kaz> ack k
15:14:55 <kaz> present+ Rob_Smith
15:15:18 <RobSmith> Thanks for the link
15:15:21 <cpn> ChrisN: We do need a clear written description for what we propose for SEI event, linked to application use cases
15:17:34 <cpn> ChrisN: For TPAC, what do we want to say? What do we want input on, what do we want from browser vendors, etc?
15:18:03 <cpn> ... There's interest here in adding support for SEI events
15:18:31 <cpn> ... We've discussed, separately from these meetings, about possible alignment with DataCue proposal
15:18:46 <cpn> ... But it's not clear DataCue is the right solution
15:28:25 <cpn> ChrisN: [talks through draft slides]
15:29:08 <cpn> ... Overall, I think we can separate the DataCue API part from the in-band event surfacing parts
15:29:36 <cpn> ... There's interest from DASH-IF on MSE and DASH emsg events and MPEG timed metadata tracks
15:30:04 <cpn> ... But that needs further exploration, and that may depend on how much implementer interest this has
15:30:52 <cpn> ... Do we do more analysis work on figuring out the MSE implementation algorithms? Or do we seek indications of level of interest before going ahead with that?
15:31:14 <cpn> ... MSE issue: https://github.com/w3c/media-source/issues/189
15:31:49 <cpn> ... Question about the SEI events, and what do we want to report there?
15:32:34 <cpn> ... Needs its own consideration, may be different solution proposals than DataCue?
15:32:41 <cpn> ... https://github.com/leonardoFu/video-sei-event/blob/main/explainer.md
15:33:26 <cpn> ... I need your help to drive that forward
15:34:46 <cpn> ... Link to draft slides: https://docs.google.com/presentation/d/1Xy6nb2RIxMCdbiO6rxRYbmjFug1cvESjWdVdXhiujkY/edit
15:35:05 <cpn> ... What additional information to include for SEI event support?
15:36:14 <cpn> ... Options: We could have a fully developed proposal, go into the detail of the problem space and requirements
15:36:57 <cpn> ... Or we can mention briefly but say that this is ongoing work and we would develop a proposal in the future
15:37:23 <cpn> ... Or we present the existing proposal
15:37:28 <RobSmith> q+
15:38:25 <cpn> RobSmith: So I think that's a good approach. The two parts, the API (accessing the data), and in-band handling (whichi is processing the data)
15:40:07 <cpn> ChrisN: The browser would only parse to extract the events, then present them to the JS layer
15:40:18 <cpn> ... Otherwise the browser would need to understand the data format in the in-band event
15:40:32 <cpn> RobSmith: There's a type field
15:40:45 <cpn> ... So it can present the bytes and the type?
15:41:10 <kaz> s/whichi is/which is/
15:41:32 <cpn> ChrisN: Yes. In the emsg case, the type field could say "this is a DASH emsg" event, and here are the bytes.
15:41:42 <kaz> q+
15:41:46 <kaz> ack r
15:41:55 <cpn> ... But within the emg there's its own type infomation
15:42:13 <cpn> RobSmith: Nested type information
15:42:37 <cpn> ... But if it's exposed as the top-level type, with the data, it's useful, and as a first step to explore how to process further
15:43:24 <cpn> ... If we want the UA to do something as a result, could handle in some automatic way. It would allow developers and browsers to explore further on what's needed
15:44:01 <cpn> ... On VTTCue vs DataCue, it'll work, but there's at least two overheads: stringifying data, and the unnecessary rendering attributes
15:44:57 <cpn> ChrisN: There are different kinds of tracks, metadata. Does using a metadata track prevent rendering of a VTTCue?
15:45:47 <cpn> RobSmith: If you have a VTTCue on a metadata track, it's still valid. Does it see the text and apply region and other rendering attributes? It could be an overhead
15:45:56 <cpn> ... DataCue is a cut down version of VTTCue
15:47:37 <cpn> ... Another thought, if we expose data through the API, we'd quickly find out the response issues of processing overhead of VTTCues and their attributes (processing time, memory, latency)
15:48:36 <cpn> ChrisN: That would be part of the further exploration, e.g., around MSE integration
15:49:07 <cpn> RobSmith: May help gauge interest and use cases
15:50:09 <cpn> Kaz: I think the two-step proposal might be OK. We might want to check the existing possible solutions and workarounds used by vendors and broadcasters, as basis for further discussion
15:50:25 <cpn> ... What has been done to cope with the existing problems?
15:51:13 <cpn> ChrisN: Many media players have this
15:51:50 <cpn> ... Ready to present progress?
15:52:28 <RobSmith> q+
15:52:32 <kaz> ack k
15:54:30 <cpn> RobSmith: Ready to present a first draft, we're fairly happy that that's well designed. That would be a talking point for feedback on the next stage
15:54:54 <kaz> ack r
15:55:40 <cpn> ChrisN: I agree, the second stage, MSE integration for DASH emsg, is a lot of work. It seems dependent on whether browser vendors are interested, in principle, in a solution. Then, it's dependent on the group coming up with a proposed solution
15:55:51 <cpn> ... I see it similarly with SEI events
15:56:28 <cpn> ... Potential interest in WebCodecs
15:57:10 <cpn> ... But that's only a small part of a solution. More generally, it's also MSE playback but also HLS playback on devices that don't support MSE
15:57:44 <kaz> q+
15:57:47 <cpn> ... I don't think there's time to document it all ahead of TPAC
15:58:20 <cpn> Kaz: I'm ok with starting with the initial draft proposal, and have more discussion at TPAC
15:59:19 <cpn> ChrisN: I was thinking of presenting this at the Media WG meeting. We may only have 30 minutes, not sure we'll have a full hour
15:59:39 <cpn> ... WG agenda should be finalised during this week
16:00:16 <cpn> ... ChrisN: At this stage, I'd like to include the key points for SEI events in the presentation
16:00:25 <cpn> s/... ChrisN/ChrisN/
16:00:38 <cpn> Yuhao: What's the most important thing to explain?
16:01:46 <cpn> ChrisN: We currently have two possible proposals. One is to use DataCue. The other is to define an event listener on the <video> element.
16:02:19 <cpn> ... Do we need both? Which is most useful?
16:03:15 <cpn> ... Or would an integration with requestVideoFrameCallback be more useful (e.g., if the metadata is per-video frame)
16:04:00 <cpn> Yuhao: The per-frame question, it needs to be limited. I think DataCue isn't suitable for this case
16:04:47 <RobSmith> q+
16:05:35 <kaz> ack k
16:06:05 <cpn> ... The listener can trigger fast. How to limit? The listener function would be run at a high frequency, it's harmful for the application
16:06:26 <cpn> ... The SEI proposal is not for this use case, it's not a high accuracy event
16:07:46 <cpn> ... On the question of DataCue, except for per-frame, is it suitable?
16:09:40 <cpn> ChrisN: That may depend on the latency we can achieve. Ideally you need see the event some time before the playback reaches the time it needs to trigger the event
16:10:21 <cpn> Yuhao: Is it not designed for low latency scenarios?
16:11:46 <cpn> ChrisN: It may not be. Because the SEI event is instantaneous, it has zero duration. In that case it may not be visible from a cuechange handler
16:12:36 <cpn> ... The only way to handle instantaneous DataCues is to use the onenter event, and you can only use that if the event has been created already
16:13:21 <cpn> ... If web app creates the DataCue, it's simple. The web app can assign the onenter handler at the same it creates the cue, and it will be triggered at the right time.
16:13:56 <cpn> ... If the UA creates cue, how does the web app see the cue before it must be triggered, so that it can assign the onenter handler?
16:14:24 <cpn> ... This is the potential difficulty I see
16:18:09 <cpn> Yuhao: For zero duration events, are onenter and onexit suitable?
16:19:13 <cpn> ChrisN: I recommend we test in browsers. But my understanding is that browsers should trigger the onenter and onexit events, in the right order
16:19:38 <cpn> ... But you not see the event if you use cuechange event, because of delay
16:20:35 <cpn> ChrisN: The delay comes from the firing of the cuechange event. When you handle cuechange, the event does not include the list of cues that triggered the event.
16:20:50 <cpn> ... Instead, your application has to query the textTrack.activeCues list
16:21:18 <cpn> ... activeCues gives the list of cues that overlap the current playback position
16:21:37 <cpn> Yuhao: So zero-duration cues will exist in activeCues?
16:21:53 <cpn> ChrisN: We should test this, because they may not
16:23:56 <cpn> Yuhao: If we add zero duration cue support, and the cuechange can be triggered, what's the problem?
16:25:28 <cpn> ChrisN: That could be a potential solution, but I think activeCues is intended for non-zero duration, so doesn't meet definition of an activeCue
16:25:40 <cpn> Yuhao: They're not in an active status
16:26:22 <cpn> ChrisN: Yes. If we made SEI duration match the frame duration, e.g., 33ms for 30FPS
16:26:50 <cpn> s/Yes. If we made SEI duration match the frame duration, e.g., 33ms for 30FPS/Yes/
16:26:58 <cpn> Yuhao: If we made SEI duration match the frame duration, e.g., 33ms for 30FPS?
16:27:04 <cpn> ... I think that's fair
16:27:37 <cpn> Chris: The other consideration is how often the web app can query activeCues
16:27:56 <cpn> ... If that frequency is low, you may miss the cue, because the duration is short
16:28:15 <cpn> ... How important is it that the web app sees every cue?
16:29:00 <cpn> RobSmith: Two different use cases. If you use cuechange, you assume there's no latency between handling the event and the activeCue list
16:29:36 <cpn> ... If you want to know if a cue has fired, onenter and onexit must be fired, so you see the history
16:31:26 <cpn> ChrisN: TextTrackCue designed for subtitles, much longer duration, so unlikely to miss
16:31:50 <cpn> ... Making it handle short duration cues is different
16:33:03 <cpn> RobSmith: For WebVMT, reporting temperature data synchronised to video, it's only measured at discrete times, but you want a continuous measurement
16:33:27 <cpn> ... Draw a line between the measurements. Or other examples, like counting number of people in an image
16:33:57 <cpn> ... That would be an instantaneous example, associated with the frame, but interpolation isn't appropriate
16:34:37 <cpn> ... There's a way to do it in WebVMT, use an instantaneous cue to interrupt an unbounded cue, which represents an ongoing measurement
16:35:45 <cpn> ... Could publish guidance on how the tools are expected to be used
16:36:35 <cpn> ChrisN: For SEI, does it matter if the application misses a single event?
16:37:29 <cpn> Yuhao: Sometimes, it's important. For different use cases, the importance is not the same. For example, using SEI to trigger opening something. Here it's important
16:38:01 <cpn> ... Sometimes the payload is just for sync between video stream and position information for rendering something
16:38:16 <cpn> ... In this case, if we lost one or two frames, it doesn't matter
16:38:52 <cpn> ... So it's important to be able to support both kinds of scenario
16:39:34 <cpn> RobSmith: Are particular kinds of SEI events always important, and others always not important?
16:40:34 <cpn> Yuhao: So users define the meaning of the SEI, if it's important or not
16:40:58 <cpn> ChrisN: Is that definition intrinsic to the SEI event data itself, or is it defined by the application?
16:41:52 <cpn> Yuhao: So have a spec that SEI information with some indication so the browser knows how to handle it, it's a good idea
16:43:02 <cpn> ... A configuration API for users. Another question is about the production, to generate SEI with this signalling
16:43:19 <cpn> ... It's the provider's problem, not the consumer's problem
16:44:27 <cpn> ... Provider is producing content not only for web app, also apps, so need to modify the SEI structure. Depends on how we define the SEI data structure to not affect other applications
16:44:45 <cpn> ... It's a good point, for the design
16:44:56 <cpn> ChrisN: Don't need to solve right now
16:45:08 <cpn> Topic: Next steps
16:45:58 <cpn> ChrisN: For SEI I think we need a more detailed description, that captures the considerations we discussed
16:46:45 <cpn> Yuhao: I'm wondering if we need a design, define API and behaviour?
16:47:34 <cpn> ChrisN: Not at this stage, I would focus on describing the user scenario and the limitations of the browsers that exist today
16:48:02 <cpn> ... Maybe also consider alternative approaches
16:48:07 <kaz> q+
16:48:28 <xfq> xfq has joined #me
16:49:09 <cpn> Kaz: I agree. Yuhao, from your viewpoint, most of the points are clear, but for W3C standardisation activity, we need to describe the detail so that we can explain ideas to others, and get good feedback
16:49:17 <kaz> ack k
16:50:46 <cpn> ChrisN: My recommendation is to take the notes from this meeting, and the notes from our previous discussion. Everything is there, but it's not in the explainer document so far
16:51:31 <cpn> Yuhao: I updated the explainer, just to extend use cases. But it doesn't include the problems we discussed
16:51:43 <cpn> ... Which part of the problems?
16:52:15 <cpn> ChrisN: Describe the current solution you have
16:52:48 <cpn> ... Is this something you currently cannot do at all? Or is it something you can do, but with expensive workarounds?
16:54:15 <cpn> ... Also, look at the existing API support in browsers (in web specifications), and describe their limitations
16:54:39 <cpn> ... For example, the TextTrackCue limitation that we discussed today
16:55:06 <cpn> Yuhao: Include DataCue in that?
16:55:20 <cpn> ChrisN: Yes
16:56:20 <cpn> Yuhao: So I could extend the explainer to cover solutions, WebCodecs, DataCue, and then include limitations and use cases
16:57:00 <cpn> Chris: We also have this report, we already published: https://www.w3.org/TR/media-timed-events/
16:57:34 <cpn> ... So you could refer to this, where you see an issue in common
16:57:48 <cpn> ... It's possible you don't need to write from scratch
16:58:17 <cpn> ... This report does not cover your use case exactly, and it does not cover SEI technology
16:58:32 <cpn> ... But hopefully it's helpful
16:58:38 <cpn> s/Chris/ChrisN/
17:01:28 <cpn> ChrisN: I'll continue to prepare presenation, and I'll report back on TPAC outcomes
17:01:37 <cpn> ... I'm hoping you can also join too
17:01:50 <cpn> ... I'll share the meeting details later
17:03:05 <cpn> ... Date for next meeting, to be confirmed
17:04:34 <cpn> [adjourned]
17:04:41 <cpn> rrsagent, draft minutes
17:04:41 <RRSAgent> I have made the request to generate https://www.w3.org/2022/08/15-me-minutes.html cpn
17:04:45 <cpn> rrsagent, make log public
18:31:56 <Karen> Karen has joined #ME
19:33:39 <Zakim> Zakim has left #me