Meeting minutes
Media Session API
Issue #377 Media session should have a generic call-to-action handler
Dana: I added a link to an explainer in the issue
… As brief background, this proposal is motivated by requests from web developers to provide a way for end users to interact with an ad while in PIP
… That's currently not possible, the proposal is for a generic button to be added to PiP, as a media session action
… Call setActionHandler() with a callback that opens the advertiser's website
… It's a generic name, as we imagine the label being customisable, with an enum of strings
… To be discussed which strings to allow
… We have ideas how a site indicates what label they want
… One idea is a new attribute on MediaMetadata
… Another idea is an optional dictionary passed to setActionHandler()
… For both ideas, if the site author sets a callback but doesn't provide a label, we woudln't show the button at all, or provide a default label
Tommy: With the strings that are allowed, how does it work with different languages?
Jer: The reason for a limited set of verbs is that the UA handles localisation. We'd use the right word based on the user's system-level language settings
Tommy: Could the website set any label, what's the need for restriction?
Jer: We don't want an opportunity to create a marquee with rotating text or other weird behaviours.
… How do you ensure the label fits in the space available? We can replace it with a suitable icon for example
… We propose this for PiP, but not limited to that
… Don't want it to limited by requiring display of words. Phone lock screen. Great to have an icon you can click
… There's no way to intuit that if it's an open string
Tommy: Does it make sense to have those as new actions, e.g., for Download and Learn More
Jer: Could do that, yes. I think adding new actions for each type is more overhead than updating an enumeration
… We'd need boilerplate spec text for each of those actions
… Another reason is it doesn't make sense conceptually having both the Download and Install actions enabled at the same time. Difficult to spec the behaviour
… The action itself is an enumeration, so the discussion is around where it should live. I can see both sides of the argument
Mark: Media Session already allows sites to set arbitrary strings for media metadata elsewhere. What's different about the call to action? There might be important info that might be elided if the display is shortened. Tooltips might mitigate that, but all the important info in the call to action should be rendered, hence have an enum
… How is this related to user activation? A user could do a lot of things, that might be useful or that might be surprising
Jer: I don't think MediaSession does anything with user activation state when a handler is called
… A user gesture for audio playback woudn't work if it didn't set the activation state
Mark: Do implementations bypass activation, or set the activation state?
Jer: In WebKit we set the activation state. Conceptually there was an activation, even if it doesn't come from a user gesture or keypress
… I don't think this is different. We don't constrain the actions in the play handler to things affecting playback. They could popup a window or do other things
Tommy: Same in Chrome
Mark: Activation behaviour should be defined
Jer: Good point, the spec might be underspecified, so should write down for interoperability
Alastor: How to define relationship between Media Session and the window. If the main frame has video in PiP, how to ensure the button won't incorreclty be displayed on the main frame's PiP. What if there are multiple PiP windows?
Jer: Great question. Original sin of Media Session is it's global state rather than assinged to a particular media element
… Answer is heuristics
… I think this problem already exists for other action handlers. I'd love to scope MediaSession to a particular playback or media element. Until then, it's a problem
… There are relatively few websites with multiple video presentations in the same frame. Might be a problem that doesn't regularly occur where there's this kind of conflict
Alastor: From Firefox perspective, the proposal is good. Just want to sure we can address the edge cases in implementation
Jer: Agree. We should have an issue open with the global Media Session and multiple media elements. Similar things happen with WebRTC.
Chris: Is this intended to be ad specific calls to action? I can imagine calls to action in our programmes, e.g, vote for your favourite contestant in a show, or donate to charity. Does anything constrain these handlers to when an add is playing, e.g., so the controls are only shown at the right time?
Jer: Now playing, thumbs up, thumbs down. So there are non ad related action handlers. Nothing in the proposal limits it to only working with ads
… So if sites like BBC have use cases for non ad related use cases, this is the time to discuss
… May new verbs could be allowed. Donate could be reasonable
… One of the reasons for this proposal is to forestall the requirement for Document PiP
… There are places where we can't support a fully interactive PiP window, but want to enable the things we can specify
Chris: Any strong feelings either way?
Tommy: Seems reasonable proposal to me. Not sure on API shape, but seems reasonable to add
Jer: Regarding how a UA knows whether the content is program or add, we could expand on chapters. If it's marked as an ad, the UA could do something smart with the UA, e.g., show progress countdown in the UI
… Could be alongside chapters, but seemed too complex at the time
… Something the web lacks right now is the concept of a playlist. Could be added pretty easily, given websites provide a "move to next" UI, so elevating that, so we show in a button
Jer: We have big websites interested. Do we have implementer interest?
Alastor: In this proposal, it's a new button in the PiP window. Any spec for what a PiP window shuold contain? It's very implementation defined
Jer: There's no web spec for that right now, for good reason. It's a space that's opaque to websites now
… We've heard requests to show different controls depending on what action handlers are installed, e.g., next and previous in vertical video. UA can do that now based on what's installed, but I'd hate to try to specify that and mandate the interaction model
… If you don't put a button tin the PiP window, you get the status quo. It should be feature detectable
… I wouldn't want to standardise the behaviour outside the viewport, from an implementer perspective
Alastor: I feel we need feature detection
Tommy: We didn't land on a good solution regarding feature detection
Chris: We should revisit that. I can imagine that similar feature detection considerations could come up again with the call to action label enumeration
Chris: Hearing support. We can discuss which enumeration values make sense.
Jer: Good way to start the conversation
Chris: There's a related issue regarding skip (#212)
Jer: Skip has semantics that match the call to action problem. You can skip to end, or the ad, or other things
… We don't have the ability to specify the thing being skipped. To make it more generic. If the alternative we consider is adopted, it might be relevant to skip handler in future.
… This is where Chapters came in, as if the UA knows you're in an ad it can do something smarter
Chris: So could revisit that, having made more progress on this
Issue #376 MediaSession Actions create a backdoor for nonstandard features
<cpn4> w3c/
Alastor: Concern here is that Chrome has a behaviour that PiP can automatically appear, Chrome specific, make it hard to interoperate. Chrome has Media Experience Index
… His concern is about the auto-PiP, not the enterpictureinpicture itself
Tommy: Is the request to specify auto-PiP behaviour more?
… Ben has been looking into this
… I can point him to the issue to see if he has comments
Chris: Jan-Ivar also mentions play. Should we constrain what sites could do?
Tommy: Not sure what else we would do. The spec talks about the intent
Jer: The default behaviour, e,g., for hangup action, if the site doesn't install an action handler
… Or if the UA doesn't perceive the appopriate action is taken, it could apply a default
… That heuristic could go very wrong
Chris: Do we spec defaults if no action handler installed?
Tommy: No spec that the UA has do to some default thing
Jer: What I took from the issue is not a conflict between the action name and what the handler actually does. It seems more about when the UA would issue certain actions
… Maybe some UAs will send the action handler always when playback starts
… I'm sensitive to that, as Safari does things, e.g., a call coming in during playback, we send a pause action
… That's not something specified, and on the grey edges of the spec
… This maybe more the concern expressed
Alastor: We'd want to add something to Audio Session
Jer: Weird behaviour from sites, when the paused state doesn't match, move to the have future data state. You're right, Audio Session is the more fitting vehicle for that, but its less adopted by sites
Jer: It's underspecified when the UA would send enterpictureinpicture
… on iPhone, dismissing fullscreen fast enough, we'lll enter PiP. The spec is silent on that
… That might be appropriate to be silent, we don't want to specify all interctions between webviews
Tommy: I'm cautious about over-specifying
Mark: I find it hard to be too presecriptibe because of the variety of ways input can flow into the Media Session. Different stacks can interpret signals in different ways
… Could have language that changes to the MS state reflects user interaction
Jer: I wonder if Jan-Ivar would be mollified if we added some non-normative notes
Mark: I think that would be fine, but needs to be adaptive
Jer: Question for him, does he want normative spec language or informative guidance.
Future meetings
Chris: We haven't discussed Media Session in a while. There's a lot of open issues. Happy to schedule more meetings as needed
… Alastor and I can discuss
[adjourned]