Meeting minutes
Add 'enterfullscreen' action
Frank: From Picture-in-Picture you may want to enter fullscreen, but there's no way.
… Other scenarios where it's not possible.
… Adding it to Media Session would make that possible.
cpn: From a BBC perspective, this seems pretty useful.
… If there's a UA-level interface for this, we would want the ability to choose which element goes fullscreen instead of a default behavior that may not align with the site's UX.
Tommy: Just to note that we're not planning on adding an exitfullscreen. Similar to enterpictureinpicture. The UA can close this without requiring help from the web site.
cpn: How would a user end fullscreen, then?
Frank: Escape, for example.
… The UA can exit fullscreen and provide a user interface for that.
cpn: Got it, no need to involve the site in the interaction.
cpn: When we're looking to add new features, we want to hear multiple expressions of support from implementers. That's what I'm looking for here.
Marcos: It would be great to provide more details in here.
… There's a new UI affordance within the PiP window. Is it supported in the PiP window already or in the browser?
… How does that interface with requestFulscreen? Is there any potential for user annoyance at the same time?
… Those are my initial reactions.
Frank: I'll update the request with more information.
Marcos: How could that be potentially be abused. E.g., with third-party ads.
… And then some expected usage examples as well.
Frank: That makes sense. I'll add the information.
Mark_Foltz: More of a comment. In the context of PiP, I know that depending on the platform, PiP and fullscreen might not be mutually exclusive.
… You might end up both having a PiP and a fullscreen element available at the same time to the user.
… Need to take that into account when specifying things out.
Frank: Two steps here. 1. As a standalone thing, the action handler could be straightforward. If we're going to mention use cases, I definitely agree we should not spec that behavior.
… But there may be cases where it's difficult for the site to know what to be doing.
… I want to untangle things a bit, just to make sure it doesn't break anything.
cpn: Do we describe the behavior when there's no action handler registered in Media Session?
Tommy: I'm pretty sure that there are cases mentioned in the spec about default handler for some of the actions.
Jer: The action about muting is a good example.
… Use case is when the page wants to take another element to fullscreen rather than the video that is playing in PiP, right?
cpn: Yes.
Frank: a declarative alternative would not be as easy.
Jer: It occurs to me that the same argument could be made for PiP. Is that coming next?
… There's not enterpip action for now.
… I suspect that if we expose an enterfullscreen, we may follow that with enterpip.
Frank: It already exists, actually.
Jer: OK, I think there's a relatively low bar to adding new actions.
Tommy: today, try/catch can be used to detect whether an action is supported.
Jer: That seems more like a side effect of an implementation does not support the full spec.
… The hard part is ensuring that multiple implementations of the action handle things in a consistent manner.
cpn: I'm interpreting this as "let's try to progress this".
Marcos: Formalize it a bit more would still be good. To record the history, and provide a more complete overview of what it is meant to provide.
Add site-provided transcripts to media session metadata
Jiren: [presents slides]
… Brief overview of the proposal. We're proposing to add a transcripts attribute.
… Before I dig into details, why not using TextTrack.
… Many sites prefer customized captions displays and do not use TextTrack.
… Browsers may need transcript for improved accessibility features.
… This would provide contextual information.
… There are challenges with reusing TextTrack transcripts.
… Some sites user WebVTT files for other purposes, for example to pass coordinates.
… Example of sites using WebVTT to pass coordinates in JPG files.
… Also, there's a lack of maintenance by most browsers.
… This has been lingering for years.
… Also WebVTT transcripts may contain HTML and CSS which are only suitable for web display.
… Some benefits of a dedicated transcripts API include centralizing all media playback information.
… Also, background playback could be supported as transcripts would be independent from the DOM.
… Also, prioritizing transcripts in case multiple videos are playing.
… We won't user to interact with the video to get the transcripts.
… Also more robust.
… Requirements include multi-language support, classification, speaker identification, timestamps and extensibility.
… Goal is to add a transcripts attribute to MediaMetadata.
… The Transcripts will have a language attribute and a MediaTranscript.
… MediaTranscript would have a type and text.
cpn: Thanks for presenting this.
Jer: Comparison with WebVTT. Why not use TextTrack. I did not see a good reason not to use TextTrack except that WebVTT is not good for the scenario you have.
… That's fine, we have DataCue, and other such things that could cater for this.
… Regarding maintenance, well, that seems like something that would benefit from more love.
… The WebVTT spec is an exploration in Interop 2026. We should fix problems instead of creating new interfaces.
… Web sites do use WebVTT for a number of things. There is a way to tell the difference, through categorization. It should have been categorized as a metadata track.
… If there's something you want to WebVTT, adding a new type seems easier.
… My broad reaction to the proposal is that this is the wrong venue. The Timed Text WG would be more appropriate.
… Media Session does not have a notion of time. This would add a lot of complexity to something that didn't have this need before.
… We should put more resources into fixing existing problems. Interop 2026 seems well positioned here, there's an opportunity to not give up on TextTrack. Web sites don't use TextTrack because of interoperability issues. Let's fix those!
Marcos: To echo from a different perspective, when I joined Apple, we tried to "break" WebVTT to propose more customization, for the exact same reasons. We documented all of thoses with a similar solution.
… Feedback we got was that we were just bypassing the problem.
… I understand why you propose this. I sympathesize.
… There's a comprehensive test suite that we can leverage.
… I wanted to share my personal experience here. I'm hoping that you'll reach the same conclusion.
… Multiple years that we're trying to bring WebVTT to Interop.
… This should get us some resources to look into it.
… We should be able to get back to it once that exploration and work is done.
cpn: Chair hat off, I agree with Jer: do we need to reinvent the same thing in a new place?
… Also, DataCue exists as a dormant WICG incubation. We would love to see a general timed metadata carrier to support a variety of use cases.
… As a generic capability, it opens up application usages. I would be delighted if we could revive that particular incubation.
… There is something that I like about the proposal: the uncoupling with WebVTT. Web sites could provide the transcripts in the format they use rather than in the format that's supported.
… From a web site point of view, I'm supportive to adding this capability. Not convinced that putting it in Media Session is the right shape that it should take.
Jer: We did have a proposal for solving the styling problem that Eric Carlson has been working on.
… Linked to web sites that wanted to support additional captions format.
… One thing that I like about this proposal, there are times when the UA wants to have an indication of available captions on a page. That's definitely something that is missing at the moment.
… There is no way for the UA to tell the web site that the user is willing to display captions.
… An action that goes in that direction seems good.
<Zakim> cpn, you wanted to comment on VTT and DataCue
Jer: Injecting transcripts into Media Session seems to be taking things too far.
Frank: Lots of good points. One of the things that wasn't addressed is the coupling between text tracks and display.
… If you wanted a way for the user to display captions in the place they want to display captions is hard.
… There is a valid use case for separating the content from the display.
Jer: I am 100% behind finding a solution to custom captions rendered on top of page content and accessibility impact.
… Various use cases that lose captions, including in Vision Pro.
… There are also platform features that get lost.
… I think that web sites will still want to provide their own styling for captions.
… That's why my weight remains with TextTrack.
… The thing you mention about taking captioning to other windows, etc., I see those flowing through TextTracks as well. I agree that they are important.
… If WebVTT is insufficient for some cases, e.g., for speaker identification, I agree it's hard right now with WebVTT.
… IMSC has support for such advanced features.
… If that's a goal, the Timed Text Working Group is the right place to discuss this.
… If I missed any use case, I'm happy to hear it.
Frank: The right thing to reflect on the opinions and think about what may be left over afterwards, and come back to it.
Jer: The multiple video playing case, I see that as a web app problem. We can do more work here to make the timelines more exposed. The synchronization of media elements would make a lot of these things easier.
… Synthesizing a timeline, I'm all for having a discussion in the Media WG if there's interest.
… The Web platform used to have a MediaController.
Alicia: Re. speaker identification in WebVTT, there's specific text in the spec for specifying the speaker name. That's in the first example.
Jer: That's true. Voice. Some substring that can be added. I don't know if it is addressing the needs, but that's a call out.
Alicia: Another concern I have with trying to put the transcript into Media Session is that right now, this proposal addresses what seems to be the obvious cases. But that will then escalate. How would the current proposal address live content, for example.
… I'm also of the opinion that we should better fix TextTrack.
Marcos: I can try to do some archeology to find our own proposal, so that we can compare and learn from our mistakes.
… This might save you some time.
cpn: Independence of DOM elements, maybe this is not something that TextTrack solves very well. I'd be interested to dig into this.
… Which is the active media session if there are multiple playing sources.
… I feel that this would warrant an explainer.
… In general, chair hat off, I'm very supportive of the motivation behind this.
… Chair hat back on, an explainer may be the next useful thing to do and we can reconvene. And perhaps do a gap analysis of TextTrack.
Media Playback Quality
Sangwhan: corruptedFrames seems to be only reporting 0 in Firefox and Safari. I'd like to propose that we deprecate the feature if that is the case.
Jer: I don't know if corruptedFrames is one of the stats that gets reported by the underlying media framework.
… If folks aren't using it, we should deprecate it, especially if that is not supported in Chrome and Firefox.
… The biggest consumer is probably Netflix.
… Or Youtube?
Sangwhan: Youtube does not do anything with this particular measure. Hard to do something when the value is always 0.
… Mozilla unshipped it.
Jer: We do have the ability to return corrupted frames from the unerlying decoder.
… That's also the case for other information. It does not necessarily imply that we should expose that to the web.
… We can probably get rid of it.
Sangwhan: I'll ping you on the PR.
Jer: We would love to get more help triaging issues and test cases as part of the Interop 2026 effort from Google folks.