14:53:40 <RRSAgent> RRSAgent has joined #me
14:53:45 <RRSAgent> logging to https://www.w3.org/2024/02/06-me-irc
14:53:45 <Zakim> Zakim has joined #me
14:53:58 <cpn> Meeting: MEIG meeting
14:54:18 <cpn> Chair: Chris_Needham
14:55:09 <cpn> Agenda: https://www.w3.org/events/meetings/689f94d6-7a6a-4fd3-9723-2ad37b09bb64
14:59:40 <tidoust> tidoust has joined #me
14:59:40 <ryo> ryo has joined #me
14:59:41 <nigel> nigel has joined #me
15:01:12 <kaz> present+ Kaz_Ashimura, Chris_Needham, Tatsuya_Igarashi, John_Riviello, Kinji_Matsumura, Ohmata, Ryo_Yasuoka
15:01:30 <cpn> Chair+ Tatsuya_Igarashi
15:01:45 <nigel> Present+ Nigel_Megitt
15:02:02 <kaz> present+ Francois_Daoust
15:02:28 <kaz> present+ Hisayuki_Ohmata
15:02:36 <kaz> present- Ohmata
15:02:38 <kaz> rrsagent, make log public
15:03:16 <kaz> agenda: https://lists.w3.org/Archives/Public/public-web-and-tv/2024Jan/0001.html
15:03:23 <ohmata> ohmata has joined #me
15:03:43 <JohnRiv> JohnRiv has joined #me
15:03:44 <cpn> scribe+ cpn
15:03:47 <nigel> scribe+ nigel
15:04:40 <cpn> slideset: https://github.com/w3c/media-and-entertainment/files/14183262/2024-02-06-W3C-MEIG-Meeting-Media-Capabilities.pdf
15:05:14 <nigel> [slide 1]
15:05:27 <nigel> Topic: Intro and agenda
15:05:35 <nigel> Chris: Welcome to the first MEIG of the year.
15:05:54 <nigel> .. Following up on discussions in the MediaWG about media capabilities.
15:06:01 <nigel> .. They've done some triage and prioritisation.
15:06:19 <nigel> .. I saw that there were a number of issues where the WG could use wider industry input,
15:06:23 <nigel> .. which is what today is about.
15:06:41 <nigel> .. There are some proposed features raised in issues that the WG has not prioritised.
15:06:41 <kaz> s/MediaWG/Media WG/
15:06:59 <nigel> .. We can re-evaluate those and discuss if they continue to be useful, and provide advice into the Media WG
15:07:08 <nigel> .. to help with that prioritisation.
15:07:13 <nigel> [slide 2]
15:07:17 <nigel> [slide 3]
15:07:24 <nigel> Chris: Recap:
15:07:33 <nigel> .. Media Capabilities API is a browser API
15:07:49 <nigel> .. Provides info to the page about the browser's ability to decode and play various media format.
15:08:01 <nigel> .. Also for encoding and transmission, applies more in a WebRTC context.
15:08:18 <nigel> .. This one API works in both a streaming media context and also in a WebRTC context.
15:08:38 <nigel> .. One interesting aspect of the design is that the Media Capabilities API is intended to focus on
15:08:49 <nigel> .. the decoding and encoding capabilities, and the ability to render the decoded media
15:08:59 <nigel> .. is not really in scope except for a couple of exceptions.
15:09:04 <nigel> .. It's a design choice.
15:09:13 <nigel> .. e.g. things to do with the properties of the display is excluded.
15:09:34 <nigel> .. It works by providing a MIME type, if it's HDR content, use parameters to describe the use of HDR metadata
15:09:37 <nigel> .. and colour spaces.
15:09:54 <nigel> .. The information you get back indicates if the format can be decoded, and if so, if playback is expected to be smooth.
15:10:03 <Louay> Louay has joined #me
15:10:07 <nigel> .. This is in some implementations dependent on real time feedback based on previous experience
15:10:14 <Louay> present+ Louay_Bassbouss
15:10:14 <nigel> .. that the browser might accumulate.
15:10:33 <JohnRiv> present+
15:10:34 <nigel> .. A flag tells you if playback is power efficient, which could be due to hardware acceleration.
15:10:44 <nigel> .. In some cases software decoding could be as efficient.
15:10:59 <nigel> .. I'm focusing on MSE decoding and playback for today, rather than encoding or WebRTC,
15:11:06 <nigel> .. but if you're interested in those we can talk about them.
15:11:16 <nigel> .. This is intended to get your feedback on these open questions.
15:11:28 <nigel> [slide 4]
15:11:46 <igarashi> igarashi has joined #me
15:11:59 <nigel> Chris: Implementation status. There are differences in how up to date the implementations are
15:12:05 <nigel> .. with the draft specification.
15:12:10 <nigel> .. Particular items for today:
15:12:15 <nigel> .. Text Track Capabilities
15:12:24 <nigel> .. Ability to decode multiple streams simultaneously.
15:12:28 <nigel> .. Transitions (e.g. for ad insertion)
15:12:38 <nigel> .. Rendering and Decoding capabilities.
15:12:56 <nigel> .. Open question: are there other requirements that should be prioritised beyond the ones I chose today?
15:13:03 <nigel> .. Any thoughts or initial questions?
15:13:23 <nigel> no group questions
15:13:35 <nigel> Topic: Text track capabilities
15:13:36 <nigel> [slide 5]
15:13:43 <nigel> Chris: 3 GitHub issues from review.
15:13:59 <nigel> .. Accessibility HR highlighted that audio and video media are often accompanied by text tracks,
15:14:14 <nigel> .. either embedded or separate. They asked if media capabilities' scope would include that.
15:14:35 <nigel> .. w3c/media-capabilities#157 is "text tracks not supported"
15:14:51 <nigel> .. Raised by Mike Dolan who pointed to a general need that detection for TTML. IMSC, WebVTT is out of scope.
15:14:56 <nigel> .. There's some discussion in the thread.
15:15:10 <nigel> .. The other issue, 99 is more specific.
15:15:30 <nigel> .. w3c/media-capabilities#99 is about SEI messages in the video stream carrying 608 or 708 captions
15:15:46 <nigel> .. As I understand it, of the major browser engines, only Safari has support for embedded timed text media.
15:15:52 <nigel> .. Chrome and Firefox don't do that.
15:16:11 <cpn> Nigel: Can you explain embedded timed text media?
15:16:30 <cpn> Chris: I was avoiding using the term "in-band"
15:17:01 <cpn> Nigel: Is it something in the manifest, or something multiplexed with the video itself? Or a HTML track element?
15:17:27 <nigel> Chris: I see the HTML <track> element as a separate thing. Text Track Cues can be programmatically
15:17:33 <nigel> .. added to a track object.
15:17:39 <nigel> .. This depends on how you're providing the media.
15:17:55 <nigel> .. For example, if you were to give an HTML <video> element a DASH or HLS manifest and it can
15:18:09 <nigel> .. play whatever is in that manifest then maybe there's a case for knowing whether it can play it.
15:18:19 <nigel> .. For example if it can play TTML subtitles delivered as separate files.
15:18:27 <nigel> .. You'd need to know if the UA can handle and render those.
15:18:41 <nigel> .. I was also thinking about MSE-based playback, where the browser doesn't know about manifests,
15:18:56 <nigel> .. at that point the player that's running in the browser would be fetching the timed text content
15:19:06 <nigel> .. and then using the Text Track API to render.
15:19:27 <nigel> .. Then there's the case where the multiplexed video includes other content, e.g. SEI messages,
15:19:45 <nigel> .. VTT or TTML multiplexed in, e.g. to an MPEG2 TS.
15:20:08 <nigel> .. I assume MP4 and CMAF has similar capabilities.
15:20:29 <cpn> Nigel: MP4 is predicated generally on having separated components, but that's not essential
15:20:50 <cpn> ... CMAF has a hard requirement that each media type is in a separate MP4 sample
15:21:30 <cpn> ... So the CMAF profile says you must provide IMSC and may have WebVTT, and each is in a separate resource whose URL can be derived from the manifeset
15:21:45 <cpn> .... So although CMAF might have provision for the embedding, I think it's not encouraged
15:22:10 <nigel> Chris: That's interesting because that suggests to me that this multiplexing and embedding use case
15:22:16 <nigel> .. is maybe not one that needs to be prioritised.
15:23:05 <cpn> Nigel: Interested in other's views, but thinking about BBC's approach I'd agree. But I suspect there are others with different needs, please speak up
15:23:15 <kaz> rrsagent, make log public
15:23:19 <kaz> rrsagent, draft minutes
15:23:20 <RRSAgent> I have made the request to generate https://www.w3.org/2024/02/06-me-minutes.html kaz
15:23:55 <nigel> Chris: Certainly very interested to know from anyone else who has a view on this.
15:25:05 <cpn> Nigel: Some of the US based folks may be interested in 608/708, possibly translated into TTML, so separate resources might be how things are expected to go
15:25:31 <nigel> Chris: You're saying timed text content would be in a separate resource, fetched by the player and
15:25:45 <nigel> .. then rendered either in the browser or in custom script via the text track api?
15:25:50 <nigel> Nigel: I think so, yes
15:26:07 <nigel> Chris: In that sense, I'm not hearing a strong requirement that media capabilities needs to do anything at all
15:26:22 <nigel> .. with respect to text track cues or formats, because it would all be handled through script.
15:26:44 <nigel> .. For those browsers that have VTT support you can know from the user agent that you're on, e.g.
15:27:05 <nigel> .. a Safari browser would give enough information, rather than needing to extend this API.
15:27:55 <cpn> Nigel: There's something here we shouldn't skip past. The architectural model for presenting captions and subtitles, is that the page doesn't know about the user's choice
15:28:09 <cpn> ... Either the choice to fetch and render the captions, or any customisation options
15:28:33 <cpn> ... If you're using MSE, and you're fetching caption resources, you already have script that knows it's fetching timed text
15:29:04 <cpn> ... But with presentation, and the additional layering in of customisation for user preferences, if you do it all in script, you have to have your own model for the user preferences
15:29:27 <cpn> ... These may be different from what's set in the browser. And those can't be queried in the web context
15:29:38 <cpn> ... Or you need to pass the content to the browser and the browser applies the preferences
15:30:10 <cpn> ... In principle Safari allows for IMSC to be used directly, so you can have IMSC in an HLS manifest, and Apple's documentation says it will play back
15:30:42 <cpn> ... If you're doing that, the browser controls the rendering and appearance. What could be useful is for the web app to know if the UA can handle that presentation
15:31:07 <kaz> q+
15:31:11 <cpn> ... Or if you need to do that yourself, and if you do, do you need to handle personalisation, so not use the user's browser level preferences
15:31:34 <nigel> Kaz: I would agree with Nigel's view and I think there are two points here.
15:31:44 <nigel> .. how to deal with the capability for timed text handling itself,
15:31:52 <nigel> .. and how to deal with embedded data by the API.
15:32:09 <nigel> .. For both purposes, thinking about more use cases would be helpful, for examples.
15:32:22 <nigel> .. When I watch Netflix at home I prefer English captions but my wife prefers Japanese captions.
15:32:32 <nigel> .. It would be useful for us if we could see both at the same time.
15:32:50 <nigel> .. We ourselves don't care how the timed text is provided, embedded or separate.
15:33:07 <nigel> .. Do we really need embedded captions or not? I personally am okay with separate files for each language
15:33:13 <nigel> .. and allowing the browser to iterate all of them.
15:33:26 <nigel> .. If someone needs embedded style that should be clarified as a concrete use case.
15:33:27 <kaz> ack k
15:33:51 <nigel> Nigel: Style? Most formats support styling.
15:34:12 <nigel> Kaz: I mean in-band in the MP4 rather than separate TTML. If we really need that option we should clarify the use case.
15:34:20 <nigel> .. Using CMAF is fine, but...
15:34:47 <nigel> .. I mean where to have the caption information, rather than the appearance of the captions themselves.
15:34:56 <nigel> .. Data structure, carriage etc.
15:35:11 <nigel> Chris: This is a conversation we've had a few times, this idea of the application of user preferences
15:35:16 <nigel> .. and the privacy issue and so on.
15:35:30 <nigel> .. I'm wondering to what extent that has an impact on the Media Capabilities API.
15:35:44 <nigel> .. In what Nigel was saying there was a suggestion that that in itself could be something that's useful
15:35:49 <nigel> .. to query.
15:35:59 <JohnRiv> q+
15:35:59 <kaz> rrsagent, draft minutes
15:36:00 <RRSAgent> I have made the request to generate https://www.w3.org/2024/02/06-me-minutes.html kaz
15:36:22 <nigel> JohnRiv: Along those lines, it's been a while since I discussed it at work, but in the past we talked about
15:36:39 <nigel> .. if the user sets preferences in one device, and wants to carry them through to another device, that's
15:36:43 <nigel> .. a use case.
15:37:00 <JohnRiv> q-
15:37:01 <nigel> .. It might be interesting to be able to query a binary "does the user have presentation preferences or not"
15:37:13 <kaz> present+ Javier_Arellano
15:37:21 <kaz> present+ Ray_Holton
15:37:33 <cpn> Nigel: I'd be interested to know how you'd use that information
15:37:40 <kaz> present- JohnRiv
15:38:06 <kaz> rrsagent, draft minutes
15:38:07 <RRSAgent> I have made the request to generate https://www.w3.org/2024/02/06-me-minutes.html kaz
15:38:44 <cpn> JohnRiv: To say, use the browser based preferences or the app-level preferences. I'm not sure
15:39:08 <cpn> Nigel: I'm concerned that the model doesn't really work. Every page you visit collects data about usage, e.g., for doing A/B tests
15:39:42 <cpn> ... But the web mechanism here directly pushes against that. It says the needs of the user in terms of privacy are more important than the needs of the user for improved product
15:40:21 <cpn> ... That makes me unformfortable. A mechanism for using that would be helpful. But by not providing any data for improving players and products, it acts against the community whose privacy we're protecting
15:40:25 <cpn> ... So I kind of want both
15:40:54 <cpn> ... This may be away from the core question, i.e, querying the UA for ability to play timed text
15:41:30 <cpn> ... As a player developer, if you're in a browser that says it will take your subtitle format and present it, that would really help some implementers
15:41:48 <cpn> ... Also helps users as there's a unified world for how things work in every page for timed text support
15:42:22 <nigel> Chris: What do we do next with this?
15:42:36 <nigel> .. I've seen various proposals, particularly from Apple, on how to provide support for custom formats.
15:42:45 <nigel> .. When they say that they mean anything not VTT.
15:42:55 <nigel> s/not/that is not
15:43:20 <nigel> .. I'm conscious that turning this into something we can propose, or next step...
15:44:03 <cpn> Nigel: I think the idea of having a canonical transfer format for text tracks that need to be rendered, probably based on HTML and DOM elements, is a good one
15:45:19 <cpn> ... It separates the decoding of your format. But in that case the query is simply: which types of texttrackcue do you implement: only a VTTCue or an HTML Cue (or whatever we call it)
15:45:51 <cpn> Chris: Makes sense to me
15:46:21 <cpn> Nigel: Then, the follow on is, why need a separate API? Because you could test for the HTML Cue constructor, so there'd be nothing to do in MC API
15:46:51 <nigel> Chris: Right, there's a detection mechanism already in a more relevant place
15:47:01 <nigel> .. I came to this from the idea that embedded support might be needed,
15:47:17 <nigel> .. but if that's less critical then these issues that have been raised like SEI events with 608 and 708,
15:47:35 <nigel> .. if the browser is not going to do anything with those captions, i.e. no browser,
15:47:41 <nigel> .. then there's no need to query a capability for them.
15:47:50 <nigel> .. The expectation is you present them as a separate resource.
15:48:06 <nigel> .. Ultimately the Media Capabilities API is about providing choices for which encoding of media to deliver.
15:48:10 <nigel> .. I think we.
15:48:15 <nigel> s/e./e
15:48:29 <nigel> .. have come to a reasonable place on this, unless there's more input from anybody on this.
15:48:57 <nigel> Topic: Transitions
15:49:00 <nigel> [slide 10]
15:49:10 <nigel> .. There's been some discussions on this but we haven't specified anything in the WG yet.
15:49:25 <nigel> .. In MSE v2 there's the source buffer change type method which allows you to tell MSE that
15:49:31 <nigel> .. the format of the media you're passing in is changing.
15:49:41 <nigel> .. This allows you to do codec or media container transitions within a source buffer.
15:49:59 <nigel> .. The use case that's driving the need for that API is advertising content that may be encoded differently
15:50:08 <nigel> .. or have different encryption from the primary content.
15:50:25 <nigel> .. There may be implementations that support different kinds of transitions between each,
15:50:38 <nigel> .. depending on the decoding mechanism, if it can be seamless or requires reinitialisation etc.
15:50:59 <nigel> .. There are issues in the GitHub repo to talk about how this might look in terms of an API shape.
15:51:07 <nigel> .. We don't need to go through all the proposals,
15:51:19 <nigel> .. but one is a transition method where you query for one set of capabilities and then
15:51:31 <nigel> .. make another call to say given that state, can you transition to this other one?
15:51:46 <nigel> .. Another proposal is, given two encodings, can the transition be supported,
15:52:00 <nigel> .. and it answers to say supported, smooth, power efficient etc as applicable.
15:52:15 <nigel> .. A third is to use the API as it stands with an additional flag reported in the information you get back
15:52:30 <nigel> .. so it's not just supported and power efficient, but also what codec switching is supported.
15:52:46 <nigel> .. If both encodings say they support transition then that tells the web app that smooth transitions are supported.
15:53:01 <nigel> .. There are these different proposals. The one we're seeing now is the current one.
15:53:14 <nigel> .. I think there's a pull request but it hasn't been merged yet or strongly prioritised.
15:53:24 <nigel> .. The question here is if this is something that is important and needs to be solved?
15:53:37 <nigel> .. Are there other ways of doing this that mean it's not a high priority for websites or players?
15:53:56 <nigel> .. I'm looking for input and feedback from anybody here as to the immediate need for this and
15:54:05 <nigel> .. whether the WG should prioritise specifying this.
15:54:33 <nigel> .. If you have thoughts about the shape of the API that's important.
15:54:48 <nigel> .. The issue numbers are #102 for the ergonomics. If you don't want to say anything in this meeting
15:54:59 <nigel> .. that's okay, you're welcome to leave comments in the GitHub issue as well.
15:55:09 <nigel> [slide 13]
15:55:33 <nigel> Topic: Decoding and Rendering
15:55:37 <nigel> [slide 15]
15:55:53 <nigel> Chris: The Media Capabilities API is really about querying the browser's ability to decode.
15:56:06 <nigel> .. We have a separate set of feature proposals that focus on the display capabilities for the rendering side.
15:56:24 <nigel> .. CSS Media Queries let you query for the dynamic range and color gamut support.
15:56:41 <nigel> .. We've proposed additions to these video- prefixed features that allow you to query the video plane
15:56:54 <nigel> .. on the TV device which might be different to the graphics plane.
15:57:06 <nigel> .. These may have separate capabilities (on each plane).
15:57:23 <nigel> .. The idea is to find out if the TV supports 4K and HDR even if the graphics plane is only sRGB, say.
15:57:50 <nigel> .. Similar question around the height and width of the video plane, to query for its resolution.
15:58:03 <nigel> .. We have a proposal in the linked issues, which has a lot of conversation.
15:58:24 <nigel> .. We ended up exposing a pixel ratio, as a numeric value, but not through a media query which is a yes/no
15:58:45 <nigel> .. only, but a number is more useful here than a series of yes/no queries for each resolution.
15:58:59 <nigel> .. This needs to be taken back to the CSS WG to progress it. Last discussed a few years ago.
15:59:10 <nigel> .. The question here is if these are useful features for us to progress?
15:59:27 <nigel> .. They're not really in the Media WG, they're CSS WG, so we shouldn't necessarily look to the WG to push them.
15:59:44 <nigel> .. This is where the IG can have a role and make recommendations into CSS WG to move them forwards.
16:00:04 <nigel> .. Anything you can do to help raise the priority, e.g. commenting on the issue, or commenting here would help.
16:00:17 <nigel> .. The other concern is that as CSS progresses through the Rec track they will need to see implementations
16:00:37 <nigel> .. of these features. It would be helpful for us to point to devices that include these features (TVs),
16:00:49 <nigel> .. otherwise there's a risk of the features being removed potentially.
16:00:57 <nigel> .. Because of a lack of uptake.
16:01:22 <nigel> .. We're out of time, any more thoughts from members now?
16:01:27 <kaz> q+
16:01:48 <kaz> ack k
16:02:02 <kaz> scribe+
16:02:16 <kaz> Kaz: do you want to discuss the remaining 3 slides as well?
16:02:36 <kaz> ... maybe you can ask people for feedback by email?
16:02:44 <kaz> ... or talk about them as well again during the next call
16:03:04 <kaz> rrsagent, draft minutes
16:03:05 <RRSAgent> I have made the request to generate https://www.w3.org/2024/02/06-me-minutes.html kaz
16:18:56 <cpn> rrsagent, draft minutes
16:18:57 <RRSAgent> I have made the request to generate https://www.w3.org/2024/02/06-me-minutes.html cpn