Meeting minutes
Agenda
Chris: Thank you for joining this MEIG monthly meeting
… Two main topics.
… Media Timed Events: Most of the meeting - would like a sense of what to do next.
… Rob will give an update on the related work he's been doing on WebVMT.
… Application Development, what we do next there.
Media Timed Events
Chris: Sub-topics:
… DataCue API proposal
… Browser API for surfacing DASH emsg events via DataCue
… Support for H.264 / H.265 SEI events
… WebVMT
DASH emsg events
Chris: Discussed at TPAC in September this year
<cpn> Slides
Chris: We discussed allowing DataCue to surface some in-band events to the browser.
… Matt Wolenetz who works on Chrome fed back that there is some potential interest in the use cases
… but some technical details that needed to be figured out.
… He was looking for more specific proposals.
… Detail in MSE#189 that captures his particular questions.
MSE #189
… Implementing this is non-trivial at this stage from their perspective.
… At the same time, we've been collaborating with the DASH-IF
… who are developing interop guidelines for DASH generally, including event handling.
… It seems that where they've got to is figuring out how interop should work with EMSG events.
… For them, it feels too early to propose a browser implementation given practical details
… of interop for how events are delivered in the media and acted on in the player.
… They suggested for now, leave this to player libraries rather than baking into the browser.
… Establish support first, e.g. in dash.js, then potentially in the future review browser integration.
… Representatives from the CTA WAVE project said they are in support of something,
… there's a question about whether they are still interested in supporting and pursuing that,
… because it needs somebody to actively contribute to move that forward.
… The proposal I'm sharing with you is: let's stop activity on this particular aspect
… unless we get a strong indication from members of our group, or WAVE, who we can liaise with,
… to indicate that this is important enough to invest more time in, to develop proposals.
… Any thoughts or suggestions?
…
… I will contact my contacts in the WAVE project and explain that's where we've got to.
… That's the next step for us on this.
DataCue API
Chris: This is valuable even without EMSG support because it supports a wider set of use cases.
… If we move the parsing of the messages from the media into javascript then there still needs
… to be an API for scheduling and triggering events at the right time.
… The current API for that is VTTCue which is more oriented towards caption rendering.
… Having something more targeted towards timed metadata has value still.
… Since the VTTCue API exists and can be used it may not be such a compelling reason
… for introducing a new API.
… I'm proposing here that we reframe the DataCue proposal independent of any specific kind of
… timed metadata.
… This requires some work to update the explainer, look at the draft spec.
… I'm asking for indications that this is a useful thing to do, and is worth spending time on.
… It has been moving fairly slowly and may or may not be worth spending time,
… depending on the interest in using it.
… If you're interested, please let me know and get in touch because that helps make the case
… for doing more work.
… If you'd like to contribute towards it then even better.
… [wonders if colleagues from ByteDance are on the call]
… Their use case is surfacing H.265/H.264 SEI events as timed metadata because
… they have a production pipeline that encodes their timed metadata in this way.
… They want accurate rendering so using video callback is useful.
… The lack of SEI support on iPhone devices, with browser native HLS playback, is an issue for them.
… I'd like to ask them: if we move towards MSE implementation on those devices does that change
… their need for API support for events, e.g. if we can move the application handling into javascript.
WebVMT
RobSmith: I've been working on two strands.
… First, OGC Testbed. OGC looks after location geospatial standards.
… I'm involved in moving features, speed, direction etc.
… 3rd year doing this. Last year, tracking cyclist using a Dashcam from a moving vehicle.
… Not trivial. This builds on that.
… This year we are interested in aggregating data from multiple sources. Two use cases from road networks people
… (UK authorities). They are the most mundane!
… Identifying traffic travelling in the wrong direction,
… Litter monitoring, the buildup of foreign objects - paper cups, bottles etc, bits of tyre or other debris,
… traffic cones where they should not be, and other maintenance issues like potholes,
… growth of vegetation covering signs etc.
… These are two sides of the same coin.
… Identifying static objects from moving vehicles or moving vehicles from static locations.
… Geotagged video can help model when things are building up and intervention is needed.
… The work done is identifying what the issues are in determining orientation.
… Hot off the press! Just at the start of this month, new standard: GeoPose, combination of location and
… orientation, which way you're pointing in some sense.
… I built a 3D compass demo, in conjunction with the GeoAlignment activity in W3C.
… First in a web page, then in an app for experimentation.
… The 3D part is because it floats like a ship's compass. If you tilt/pitch/roll the compass stays in the
… horizontal plane so you get an accurate heading.
… The engineering report is being reviewed, due for publication in January 2023.
… A free of charge app will accompany it.
… The other thing is getting WebVMT from an ED to publication.
… Spatial data on the web are offering to publish it.
… I've been adding new features including data synchronisation, altitude (which is contentious
… due to the use of WGS84, height above the ellipsoid), and CSS selector system support,
… so the different paths and zones can be styled in different ways by CSS.
… Spatial Data on the Web are reviewing prior to publication early next year.
… Issue: the HTML interface for timed metadata and how to represent it.
… In WebVTT the timed text format, metadata is handled as JSON objects, structured in some way.
… The question is how to make that available in HTML and whether there's any precedence for creating
… an interface of that sort.
Nigel: Could you explain what you mean about making the data available in HTML?
Rob: From the WebVTT, timed metadata is a text encoding of a JSON object
… The references in that document refer to HTMLTextNode, which doesn't seem appropriate for timed metadata, as it has fort, colour, display region, etc
… so how to make data available in the HTML interface. One way could be plain text rather than HTML text, then parse as JSON
… But JSON is a structured data format, and there doesn't seem to be anything to handle that
Nigel: What would a representation of the data look like ideally?
… To an end user, I mean
Rob: It depends on the designer of the page, the metadata could represent anything
… How might you represent temperature visually, there are different ways. But you still need to pass the data through, with a timestamp attached
Nigel: So a cue handler?
Rob: Yes, you know it's a JSON object. DataCue has an associated type, so you could have a JSON schema to describe the structure
… How it's displayed is an implementation issue
Piers: There's a geolocation API already, how is it different?
Rob: That API describes your current location, but with geo-tagged video there are multiple locations, based on the video
… Commercially available devices, smartphones and dashcams, these are location aware and record video
… So having a way to identify those locations and things appearing in the video is useful
… Relate objects in the frame to location information, e.g., inferred from imagery
Chris: Anything written down on this?
RobSmith: I wanted to consult here first because the broadcast perspective is different to the
… geospatial perspective.
Chris: Yes. I don't know that we'd put the data into the HTML,
… you'd fetch a separate resource rather than having it inside the HTML.
RobSmith: I mean something like that, where the WebVMT resource would be equivalent to the WebVTT
… resource, and would contain content timed synchronous with some timeline like media for example.
… It's how to access that, get the data from the WebVMT file to the JS interface or some API that the
… programmer can access to read, process and display it.
Chris: Typically in entertainment use cases you have a JS media player library that's typically using
… MSE to fetch the media segments, and it could also be responsible for fetching other kinds of document,
… then there's custom JS code to parse the document, extract the object and then apply whatever
… logic is needed to turn it into a visual representation.
… I think there's a question that I don't think we'll have time to answer today, but we can follow up
… in a future meeting.
Rob: Yes, the key to this is how to represent generic data with some arbitrary data and identify it
… on the other side and know what it means and get what you're expecting from it. That's the trick.
Kaz: From the M&E viewpoint, a possible use case might be metaverse, with 3D video and user avatars,
… and you need to identify the user's position, direction and time, for overlaying on video content.
… Another possible use case is some kind of robots are getting popular in smart cities and buildings,
… and identifying those automatically driven vehicles or robots might be a use case. Those robots
… can also provide detailed signage too.
Rob: Agree. AR is another potential use case, overlaying virtual objects on a video feed, that are location-specific.
Chris: This is a whole thing in itself. Interested in connections to other W3C groups?
Kaz: Yes, I started discussions with some at TPAC and those use cases are also for web-based digital twins
Chris: We'll follow up on these ideas.
… and the HTML integration parts.
<cpn> Nigel: About understanding what's in the JSON, it sounds like you want to define a vocabulary, which has been done before. Could look at things like RDF
<cpn> Chris: or JSON-LD
Application Development for Consumer Products
Chris: We've had a few meetings on this.
… Started with Chris Lorenzo (ComCast) talking about lightning-JS to allow devs
… to provide more performant UI on TVs. We had a series of meetings talking about
… different perspectives of the experience of building TV applications, and the performance issues
… because of memory or CPU limitations on different devices.
… We heard input from NHK and BBC in previous meetings.
… What I wanted to think about next is what we do with this information and what might our next steps be.
… Perhaps it would be useful to consolidate this information into a single place
… as a way to capture everything.
… I started a draft document.
<cpn> https://
Chris: This is not complete. There are lots of gaps.
… It might be a useful starting point to capture everything and
… for each issue raised, to figure out what practical steps we could take.
… For example, some may point to a new standardisation requirement or a change to a standard.
… Some might be about implementations or adoption.
… Some might be testing related.
… As a way to try to guide us to what to do next, writing this stuff down and doing analysis
… to figure out what's needed could be helpful.
… [shows document on screen share]
<kaz> Draft document on Application development for consumer devices
Chris: Performance issues: HTML and CSS based rendering challenges:
… use of animations and transitions in particular, where it's not possible to know if an effect
… will apply smoothly from a visual perspective.
… The CSS Will Change module has been proposed as a hint to the UA, to pre-render animations.
… There may not be a specification requirement here, but a testing requirement to discover which
… devices work well or less well with certain animation effects.
… There's an open question about standardising the testing approach.
… For Canvas based rendering I'm hoping that Chris will write something here.
… Potential opportunities like with the lightningJS framework. We've mentioned an accessibility concern
… with that, writing directly to a canvas.
… Then maybe its an implementation issue.
… I'm hoping to identify for each problem area what standardisation needs there are, if it is an implementation issue etc.
… This goes on, e.g. memory allocation, pointing to existing specs where they exist.
… Spatial nav and voice control.
… Application distribution and installation.
… Development and debugging.
… Application launching
… Stream switching with minimal latency.
… I invite your suggestions and feedback as where we should go next with this.
… Is this a useful exercise?
… Worth capturing more detail?
… Turn it into a useful set of recommendations?
… I'd like to invite your collaboration to help capture this, if it is useful.
… I need your help!
… Turning this information into something actionable and useful from a standardisation perspective.
… The document is here, you are welcome to add your comments to it.
… If there is any particular topic here that interests you I would encourage you to add details
… where they are missing, and then we can review it in a future meeting, to see if we
… have identified gaps that need to be addressed or potential liaisons with other groups.
… Any thoughts or questions? Is this useful? Is it the right direction?
Kaz: I think the topic and approach is good. There are several technical topics mixed up here.
… We should be careful to describe what is the pain point from each viewpoint, expected use cases from services, etc
… Each company or industry is expected to describe their initial idea, then we can look at that and look at a more detailed template for requirements
Chris: We could make this into a template if useful
… Better to consolidate into a document or consider separately. For example, we could have separate GitHub issues for each item
… We can ask input from people with interest
Chris: Please make suggestions in the document on topics you're interested in
AOB
Chris: Anything else?
[nothing]