Meeting minutes
Slideset: https://
Agenda bashing
cpn: [reviewing the agenda]
… Re. WebCodecs, some follow-up to yesterday's discussion with WebRTC WG.
… Then Autoplay Policy Detection, Media session issues and session capture and an introduction on Document Picture in Picture API.
… Suggestions to changes/additions?
[some discussion on switching agenda topics to accommodate early flights back]
Media WG Introduction
cpn: Mission is to develop media specifications to improve media support on the web.
… Looking at an overview of the specs as they stand.
… MSE v2, we're at WD. Question is then: what do we think we need to publish a CR?
… For EME v2, we made a number of updates, but they are in an ED. I'd like to get a sense of where we are towards publishing a FPWD.
… Media Capabilities is pretty mature. It is not on the agenda but it would be interested to understand the blockers towards publication as CR.
cpn: Media Playback Quality is complete but we agreed to integrate that into HTML and we need someone who can help do that.
… Autoplay Policy Detection was published as FPWD. We probably want to initiate horizontal reviews.
… WebCodecs, some question on publishing the registry as a real W3C Registry now that there is such a track, instead of as a draft Note.
… For both Media Session and Picture-in-Picture, some of these are implemented and deployed. What are the steps to progress these to CR?
… Picture-in-Picture is more complete in its scope, Media Session more open following recent discussions on presentation actions.
Media Source Extensions v2
Matt_Wolenetz: Approximately 20 issues to go through but we won't have time to go through all of them.
Matt_Wolenetz: Two goals here today: present the current status. 1) What changed since the Recommendation. 2) Get some feedback on priorities for v2.
… If you're interested in any of these features, please reach out directly to me or through the GitHub repository, I'm interested!
<Matt_Wolenetz> https://
<ghurlbot> Issue 100 Have appendBuffer and remove return promise. (jyavenard) TPAC-2022-discussion
Matt_Wolenetz: Since Rec, we landed one major feature: changeType. It's been used for a lot of different things. AV1/VP9 switching for instance.
… Without user visible pauses.
… Second feature: MSE in workers is in the specification. One question to answer today.
… We had to unship the feature due to backward incompatibility issues with other MSE features.
… The issue tracker lists a number of features that could be considered for v2.
Issue #100
Matt_Wolenetz: Back to issue 100.
… Have appendBuffer and remove return promise. That seems pretty obvious to me.
… If you see blockers, speak up now or comment on the issue tracker.
Issue #232
Matt_Wolenetz: Next one is #232
Matt_Wolenetz: What a conforming implementation might do with regards to picking up ranges for buffer eviction.
… To my knowledge, the only remaining technical blocker for getting MSE implemented on small device such as iOS is pre-emptive buffer eviction mechanism.
… ManagedMediaSource and ManagedSourceBuffer could perhaps be used to distinguish cases with pre-emptive eviction from those without.
jernoble: I was thinking along the same lines, not necessarily about new objects but some configuration objects to switch to a different mode.
… Whether it's a completely new object or some mode, we can bikeshed a lot but that's eventually the same.
… There is a little more SF concept. On iOS, they can mark certain memory as purgeable. We certainly don't want to expose memory purging to JS.
… But we need to do something about it.
Matt_Wolenetz: To be clear, that's not only about iOS, Chrome implementation also could use some low-memory signals we get from the OS.
… Look forward to working with you on that. That would be super for the ecosystem.
cpn: What needs to happen next?
Matt_Wolenetz: A PR would be a great starting point for a discussion on whether to use new objects or a new mode.
… External contributions absolutely welcome on this.
Issue #189
Matt_Wolenetz: Discussed extensively at last TPAC. Add support for DataCue exposure of in-band metadata in emsg boxes.
cpn: I think it was TPAC 2019.
… Since then, we've been very slowly working on the DataCue API.
… Two things: the DataCue API could be entirely separated from MSE. The current approach is to use a VTTCue but it would be good to be able to do that with any kind of cue.
… Looking at the DataCue implementation that Webkit has, the proposal is that we standardize on that for the API shape.
… Really, I'm looking for explicit support from implementers.
<dom> DataCue explainer
cpn: Separated from that, there's the question of in-band events.
… The browser could extract those and surface them to the application through DataCue.
… There has been some collaboration with DASH-IF.
… Constrained devices don't want to have to touch the media twice. The argument I hear is that parsing the container for metadata is not computationally expensive, so could be done through scripts.
… For more constrained devices, the question still stands.
… Parsing and exposing through MSE could be a direction. Here as well, I'm looking for explicit support for that.
… We don't want to figure out the details of how everything works without some initial expression of support.
Matt_Wolenetz: The Chromium implementation for even in-band text has been removed from MSE.
… If we're going to expose in-band metadata somehow, it would certainly be some non-trivial impementation work.
… Interactions with timestamps could perhaps trigger weird behavior that need to be looked into if we integrate that in MSE.
cpn: Should I take that as a semi-positive feedback?
Matt_Wolenetz: Sure, it would be good to have external input. It would help speed up specification work.
… External PRs would be helpful.
cpn: That's where we struggle perhaps. People in DataCue are more from the media container angle, not experts in MSE.
Matt_Wolenetz: If there's a bytestream format that this would be particularly tied with, people could consider looking into that first and pointing out where precisely things need to happen.
MSE v2 feature prioritisation and developer input
Matt_Wolenetz: One issue wasn't caught that since the main thread implementation throws when handle is not available, the MSE in worker feature introduces a backward incompatibility that affects existing products.
Gary: Old version of video.js typically
Matt_Wolenetz: 2 options: restrict that getter visibility to only the dedicated Worker context.
… Even there we would change it to never throw an exception.
… The alternative is to return null if it's not supported instead of throwing an exception.
… We'd have to specify that through text as WebIDL does not allow that to be expressed directly.
jernoble: Do we have an articulated use to be able to access the handle from the main thread?
Matt_Wolenetz: For now, no. But there is something on that table.
<dom> +1 to visibility instead of null
jernoble: My vote would be to do the hidden option. If we do have that use case, we can extend it afterwards.
Matt_Wolenetz: That also includes not throwing an exception
Matt_Wolenetz: Thanks. That helps.
<Matt_Wolenetz> Next, group of issues: (#259, #209, #172, #40, #35) relating to "introspection API".
<ghurlbot> Issue 35 Report buffer changes in update events (dmlap) feature request, TPAC-2022-discussion
<ghurlbot> Issue 209 Expose SourceBuffer as an ArrayBuffer (guest271314) feature request, TPAC-2022-discussion
Matt_Wolenetz: A group of 5 issues to come next. First on exposing SourceBuffer as an ArrayBuffer.
… Both are related. Getting metadata.
… Better visibility about what is still buffered.
… These have all received mixed expression of interest. I would like to get better and more explicit expressions of support and details on how these features would be used.
<Matt_Wolenetz> #185, #14
<Github> https://
Matt_Wolenetz: Moving to 14, which was cut from the Rec because implementations had not caught up with the spec.
… Maybe add a writable endpoint to a SourceBuffer so that you can fetch and pipe to it.
cpn: I think that was one that Zach was particularly interested into.
… We can probably get some input from him.
jernoble: You said this could help with fixing low-memory conditions. Could you expand on them?
Matt_Wolenetz: Trying to recall. Buffer eviction. Previous prototype was around cases where we provide P-frames very infrequently
jernoble: Interesting that we're contemplating giving back control to the user agent whereas the whole motivation for MSE was to take it away from it.
Matt_Wolenetz: OK, I'll follow-up with Zach from Disney on this one
<dom> #184
Matt_Wolenetz: Next one is linked to buffering WebCodecs encoded chunks for playback with MSE.
<dom> #302
<ghurlbot> Pull Request 302 MSE-for-WebCodecs draft feature specification (wolenetz)
Matt_Wolenetz: MSE for WebCodecs was in origin trial in Chromium but interest seems to be relatively low.
… If you are very interested by this issue, please speak up, otherwise we'll just remove the feature.
cpn: From WebCodecs, you can receive and decode. If you want to render, you need to integrate a lot of concepts such as buffering. With this feature, would that give a way to hand things over to the user agent?
Matt_Wolenetz: I think HLS.js does transmuxing from MPEG-TS to MP4 and feeds that into MSE. That could simplify things.
… I didn't get enough support to create a full pipeline and see how that would improve things.
Bernard_Aboba: I actually know some customers of HLS.js that are very interesting in low latency. Would it be useful to reach out to them?
Matt_Wolenetz: Yes, it would be useful to have more active participation.
cpn: We're reaching the end of the allocated time.
Matt_Wolenetz: Yes, there are a bunch of them. Audio PCM support. Main thread SourceObject attachment. Long been asked for.
jernoble: Don't we already have it? I'm pretty sure we have it in Safari.
Matt_Wolenetz: You probably do, and Firefox as well I think, but it's not in the spec.
… And then a bunch of interoperability issues, mostly around timing.
… These are lower-complexity issues.
<dom> [MediaSource as a valid value for srcObject has been in HTML for quite some time IIRC https://
cpn: Live stream and playback through unbuffered ranges is something we're interested in at BBC.
… For those interoperability, we can schedule a future call and go through those.
Media Session
Media Session & WebRTC
<ghurlbot> Issue 274 [closed] Should we add slide presentation specific actions? (youennf)
<ghurlbot> Pull Request 284 [closed] Add presenting slides actions (beaufortfrancois)
cpn: We had a call with Media Session editors and WebRTC folks to add slide presentation specific actions. We got agreement to add support for them in Media Session.
youenn: PR merged this morning.
<dom> The Capture-Handle Actions Mechanism
youenn: Next step is implementation. Other part is reusing in Media Capture Actions.
… This work should probably be tackled within the WebRTC WG since it owns the Media Capture Handle Actions.
cpn: Media Session only defines the action ids?
youenn: yes.
elad: More things will be needed. Right now, if you get the event, you don't know whether it got delivered by the user or by a presentation.
… You don't know what the user intended.
… That's ok provided the application that receives these events is willing to treat them regardless of where they were issued.
… I think something needs to be added.
youenn: Yes, but that's still very early and indeed the security model needs to be defined for media capture actions first.
<ghurlbot> Issue 285 The active media session might not be the only one being notified of actions (youennf)
youenn: This discussion led us to next issue. There are cases where the media session is not active but you still want to send the actions.
… A page still needs to be notified.
youenn: If the source is not tied to a particular media session, we may want to associate it with an active one,
… The PR is available. I don't think it has been reviewed yet.
Tommy_Steimel: Just to be clear, it does not change the actual API, right?
youenn: That's correct.
… No implementation change.
Status of AudioFocus API #277
<ghurlbot> Issue 277 Status of AudioFocus API (chrisn)
cpn: I noticed that if we define more precisely AudioFocus, we can define more precisely what happens in media session.
youenn: It's a good idea. We should spend time on that. I know that we have some user experiences which are not great. E.g. in iOS when you're mixing audio and microphone for instance.
… AudioFocus would allow to tune the user experience. I'm hoping that I'm not alone here and that other implementers are interested.
<dom> AudioFocus API Explainer
Tommy_Steimel: Specifically with Media Session or AudioFocus as standalone?
youenn: The fact that you're suspended and when it happens in iOS would be very cool. It could be tied to Media Session. But just on its own, it's good as well.
… On iOS, only one application is able to run audio. If a page plays audio and you e.g. switch to Youtube, audio will stop. But the app does not know it got suspended.
… So it won't know whether it can resume either.
… It would also allow you to specify applications that can blend well with other applications.
eric-carlson: An example. "I play audio that's fine to mix with others". If you play file based audio, we assume that it cannot be mixed with others.
… Whereas you may have an app that plays an MP3 audio file when an event occurs and it will interrupt the audio.
padenot: Are we going to send notifications to say something else is attempting to play? Or are we going to fire an event to say you've been interrupted?
eric-carlson: That happens at a lower level, we cannot interrupt that.
padenot: I'm trying to remember why previous proposal stalled.
… Regardless of the technicalities, I would also like to have something like this on the platform.
youenn: Also a problem with audio level on iOS. Mike level is different from audio level. No way for the browser to know which level to use.
elad: Is this purely specific to iOS?
youenn: Also on MacOS.
elad: What about Windows?
Paul: there is something equivalent on android and windows
Tommy_Steimel: We do some amount of something similar to the AudioFocus API internally.
… I don't know of definitive use cases in Chrome that would make use of it.
cpn: It seems that we're looking for someone who could pick up the explainer and progress that.
youenn: I have limited time but willing to look into it. At some point, it was part of the Media Session specification.
… I can understand why it was there in the first place.
<padenot> e.g. Windows, https://
youenn: If it's less premature, it could perhaps be worth re-introducing it in the spec.
<dom> Potential Normative Specifications, includes "Audio Focus API"
[group looking at charter, audio focus is a "potential normative specification" so no problem adopting it in the group if it is believed it is sufficiently mature]
Media Session and Web Audio
<ghurlbot> Issue 244 Work without an <audio> element (AshotN)
<ghurlbot> Issue 213 Does media session work with WebAudio? (mmontag)
cpn: Does that suggest that Audio Focus is the next piece before we dive into relationship with Web Audio API?
jernoble: I agree with Mounir. If you begin playback Web Audio, it does not interrupt other audio. It was meant for game sounds, etc.
… But common ways that user interacts with the device are not properly routed. We need some ways to indicate that the AudioContext should be considered and be able to receive controls.
youenn: Yes, I agree, also for video conferencing.
jernoble: This is a good example of an interaction where the presence of Audio Focus would make the experience better.
padenot: We really need that. It's been the case for super long.
Tommy_Steimel: Proposed solution would be to have the Audio Focus and web apps could then use the API to tell the OS about the type of audio mode they're in and get feedback from the OS?
jernoble: That's correct.
Encrypted Media Extensions
cpn: the charter has limited scope for additions to EME
… that includes HDCP Policy Detection, for which there is a WICG incubation
… the TAG review refers to implementations in FF and in Chrome
… has it been folded in EME at this point?
Joey: the EME spec hasn't changed in the past 12 months
… no update on this in particular
… I can go through the issues offline
… it is implemented and in use
cpn: that would be good
… our charter renewal is coming up in May next year
… it would be good to show some progress as we go through rechartering
joey: sounds reasonable; I'll try to free time in the next couple of quarters for this
… I'm no longer part of the Chrome org at Google
cpn: are you still able to do this or should we find someone else?
Joey: I'm committed to do this; also happy to get help (and have been getting help)
… my new team understands that I4m committed to that
Greg: I can also take the lead on merging HDCP from WICG to W3C
Joey: that'd be fantastic
Greg: my question to the chairs - I can create a PR for that, for which there will be comments
… but what about moving the new version to FPWD
… or should we wait?
cpn: what's in the current draft?
Joey: the encryption scheme capability has been added to the draft and is implemented in Chrome
cpn: FPWD triggers horizontal review
… having more of the features in helps for that
Dom: FPWD also triggers a call for exclusions for IPR
cpn: so we could land these 2 features and use this as a criteria to launch our FPWD
cpn: third issue is avoiding duplication sessions within the same origin
joey: you don't necessarily want to do the work of parsing the init data for EME to figure out what's in it
… when you're going to hand it off to a more efficient c++ system
… but you do want to avoid a bunch of duplicate sessions
… e.g. in widevine, unless you parse the data, the player can't know this will end up creation sessions with the same keys
… #467 provided a solution to ask a browser for potential key ids
<ghurlbot> Issue 467 API to find existing sessions (mounirlamouri) feature request
joey: there are situations with legacy cases where you wouldn't be able to get that data
… I'm not sure what would be available from playready and fairplay, but assumes it would be available
… Another approach would for the app to provide its data and ask if there is an existing matching session
… we went back and forth between the two approaches
… the last discussion we had about this was that "find a session by init data" was distasteful to some
… but looking for confirmation / new input on this
… The explainer has also additional alternatives
Greg: another id would to add key ids to the encrypted event
joey: that wouldn't work in case you get a PSSH box out of a @@@1 file
Xiaohan: not trivial to figure key ids out of init data
Finding Existing Sessions in EME Explainer
joey: I think the data is there now; it seems like something the CDM should provide info for
… better than a PSSH parser in JS
Greg: going back to media-encrypted event exposing key ids
… media status exposing key ids feels odd
… the media-encrypted event could be correlated with key status
Joey: if it's available in the PSSH, it could be exposed through a method or through the mediaencrypted event
… but when you get a license back, you definitely have @@@2
Joey: they come back in key status because that's the time they're guaranteed to be available
… it may not be guaranteed in init data, but it's likely they would be which would help players (which could still fallback to current workflowà
Xiaohan: one issue is that these init data are proprietary
… but they would need to be parsed by the CDM
… the encrypted eveent is fired by the demuxer
… this would require hooking them to the CDM
Joey: in any case, the event approach wouldn't be sufficient given how often you get the PSSH data out of band
… any blocker on the widevine side to add a method to the mediakey to list key ids based on init data
… or what sessions match that init data
Xiaohan: the former is probably easier
… for widevine there is also the legacy contentId
Joey: I would be happy to have a method that might return "no info available" in some situations
… that's the existing situation
Xiaohan_Wang: so you really need a PSSH parser
Joey: yes to return the list of key ids if they're available
Xiaohan_Wang: the implementation would be easy, but how does it solve the problem?
… e.g. how to deal with situations where expiration times are different
Joey: nothing is stopping the player to have overlapping sessions
… it allows avoiding them if they wish
cpn: are we saying that the explainer example is close to the direction we want to go?
joey: the two different options are returning a session, or return key ids and let the player figure out the right approach
… I hear key ids is a bit simpler, and would suffice for my use case
… that would mean dropping this explainer and proposing something else closer to #467
<ghurlbot> Issue 467 API to find existing sessions (mounirlamouri) feature request
Bernard: I can find out re MS impl
Jer: likewise for Apple
Joey: widevine still uses contentid but also lists key ids
Xiaohan_Wang: the CDM is required to periodically check the status of the key
… even when the CDM is not in use (e.g. tab is in background, not playing)
… it's bad for complexity and for user's battery life
… they're 2 cases that the key status can change:
… - from the CDM itself - e.G. a policy that the key expires after an hour
… - the key cannot be used e.G. outside of HDCP - outside of the control of the CDM
… The first one can be done through a timer without regular polling
… For the latter, if there is no activity, there probably isn't much value in polling
… while it remains useful when decrypting
… THis is related to the HDCP detection policy, the JS could use that API to poll
Joey: it seems to me that nobody is married to having a timer per se so long as updates are sent whenever they're useful to the app
… maybe we could define the events that one would care about, then we no longer need to require regular polling
Greg: the initialization phase is really important, and I wouldn't want that to depend on timers
… beyond that, I think it would be displayoutputchange and expiration timer
… any other event that may affect the key status?
Xiaohan_Wang: essentially yes
Xiaohan_Wang: in Chromium, the CDM is running in a sandbox with limited capabilities
… I'm not sure it can detect the output change
… the browser could detect it, notify the CDM which could then do that
… maybe doable then, but not trivial
Greg: let's you're plugged into a display, there is an HDCP change, do you change the key status?
Xiaohan_Wang: we would not change the key status
… but when the user hits play, the CDM will check everything include HDCP status
… at that time, it may update the key status
… it could also refuse to play if the HDCP is not good enough
… Would that be too late a signal?
Greg: when there are display changes, we pause the video element automatically
… checking the key status at that time would be good
Joey: but Xiaohan_Wang you're saying that may not be feasible from the sandbox?
Xiaohan_Wang: yes
… the chromium browser definitely knows about display changes
… but not sure if it would work in a sandbox context
… and whether it would work across platforms (since these detections tend to be platform specific)
cpn: so I'm hearing we want to do this, but we're not sure about feasibility which requires a bit of investigation to be done
Xiaohan_Wang: +1 - we can do some investigation to come up with a more realistic proposal
cpn: if we can get some of this specified to get a FPWD ahead of our rechartering, that would be helpeful
joey: I'll make that a goal
Greg: so will I
WebCodecs
cpn: did we cover everything during the joint meeting with WebRTC yesterday?
cpn: re #270, you were suggesting a per-codec approach?
<ghurlbot> Issue 270 Support per-frame QP configuration by VideoEncoder (daijh) extension, TPAC 2022
eugene: yes, QP are codec specific, and not all encoders provide this feature
… we would like to start with a prototype to check that this can provide useful control
… until then, we can discuss the API shape, but it doesn't seem critical
… I would like to start with prototypes for AVC and AV1
Bernard: it might be useful to go over the resolutions in the issue themselves
… QP having different ranges across codecs
… there were other questions that emerged in the API design
eugene: we're not settled on all of them
… if we do it by codec in the register, if it's per-frame, we provide an extra argument to the encoder encode call
… how would API users that they can provide this
… isConfigSupported is only useful for testing configs
… this wouldn't be via config, but an extra argument to the encode call
… we haven't looked at that yet
… the VideoEncoderConfig could add a special mode
… First priority is prototyping
tguilbert: there is a PR in progress for #317
<ghurlbot> Pull Request 559 Introduce VideoFrame metadata (youennf)
<ghurlbot> Issue 371 AudioEncoderConfig.latencyMode (or similar) (chcunningham) extension, TPAC 2022
<ghurlbot> Pull Request 551 Add frameDuration attribute to OpusEncoderConfig (bdrtc)
Bernard: for Opus, we would do complexity, dtx and fec
… this would be essentially the ptime
… ptime is usually expressed in ms
<dom> [re milliseconds, see also 8.3. Use milliseconds for time measurement ]
registry for webcodecs
<ghurlbot> Issue 398 Moving codec registry to "Registry Track" (chcunningham) editorial
cpn: with the registry track now available, what should be our steps in terms of moving to the registry track?
francois: we would need a resolution from the group to do that; it's a good fit for the document we have
… the Director recommended we look into that when we published a recent registration
francois: this would be for the registry document itself
… the registration themselves would remain in draft notes
… this would have much impact on the document
… it mostly adds a review step to a process
cpn: we're already running CfC to add registrations
… does that impose additional requirements beyond that?
francois: the registry document needs to document the process to add registration
… our document is already aligned with these provisions
… I can double check tbis is so
cpn: so it's only a mechanical step
… should we run a CfC for this?
francois: yes
PROPOSED_RESOLUTION: The WG will move the WebCodecs registry to the W3C Registry Track
RESOLUTION: The WG will move the WebCodecs registry to the W3C Registry Track
cpn: in terms of doing the update - will you take care of this francois?
francois: I will
… and get back to group as needed
status check
cpn: where are we? what are the points that you are seeking input on?
danS: Chrome shipped webcodecs to production
… we've been waiting for new browser to catch up, with a few additions in the meantime (svc, new codecs)
… next thing is metadata for color gamut
… see #384
<ghurlbot> Issue 384 HDR support (ZmGorynych) extension, p1
danS: most of the questions are about integration
… e.g. with WebGPU for zero-copy
… HDR & integration with CSS Color 4 and HDR Canvas
tguilbert: there were talks about Web Containers, everything mux/demux is done in JS
… we could look into this
DanS: that's indeed one of the most frequest requests
… this is showing a gap - the library ecosystem hasn't provided a popular solution to this yet
cpn: is this something you're waiting for other impl to catch up before doing that?
DanS: our real hope is for this to be done via JS libraries
… but if event with these libraries the demand keep up, adding something outside of WebCOdecs may make sense
Bernard_Aboba: in terms of WebCodecs challenges, a big one is config (we talked about Opus)
… another one is for Reference frames (e.g. in H264)
… configuration e.g. for screensharing
Dom: are these codec specific or systematic issues?
Bernard: content-configuration is a pretty big deal for screen sharing, or for gaming
… or you're not going to get the compression you want
… Long term references is for robustness
cpn: In terms of work mode on this. WebCodecs has developed quite rapidly, and pace has now slowed down.
… I very much welcome integrating WebCodecs in the regular cadence of the working group.
… If you're open to doing that.
DanS: Good idea to do that, editors meetings have not been regularly held recently.
cpn: New editor now that ChrisC is gone?
DanS: No update for now.
cpn: WebCodecs is really an exciting spec.
eugene: Previously, the spec was driven by Chromium, but now that we have a spec that is useful by itself, we're trying to add things that only make sense. We don't see major new features to add.
… So pace has normally slowed down.
Autoplay Policy Detection
cpn: One main issue was relationship with Permission Policy.
alwu: I wanted to start with a brief update on what the spec is.
… This API provides some way for a web site to detect if a media can be allowed to autoplay or not.
… Works with media element and Web Audio. It may return allow, allow muted, or disallow.
… We're now focusing on horizontal reviews. Went through self-review questionnaires for i18n, privacy and security.
<ghurlbot> Issue 27 Relationship to Permissions Policy (tidoust)
alwu: Re. link with Permission API. With Autoplay, you can request status for the whole window and for an element. So the scope is different.
… Permissions Policy is about whether user will be prompted. Autoplay is different in that regard as well.
… Also we have more than allowed/disallowed in Autoplay.
… There are similarities, but we don't need to integrate this.
… Also willing to know if others are interested in implementing the spec.
jernoble: with my implementer hat on, Safari is interested.
… One of the key parts of the Permissions API is about actively asking the user. I don't think we want to do that with autoplay.
cpn: I note issue #13 about requesting permission.
<ghurlbot> Issue 13 Media autoplay permission (Fyrd)
cpn: I'm wondering whether there's a need for a declarative approach. I'm thinking about a video element with multiple sources. No way to declaratively switch to alternate content if the sources are not allowed to autoplay.
jernoble: Media queries are being re-introduced in HTML for source selection. That was removed.
… If autoplay is denied, we may want to change the background color, etc. so media queries could be used to detect that. Without creating a new API.
… I don't know if you can author a media query to react on a specific element.
… Both approaches are necessary but they would complement each other.
cpn: I will try to dig up issues relevant to media queries if there are any.
Document Picture-in-Picture
cpn: Proposal spec that is not in this WG but it relates to the Picture-in-Picture API.
Tommy_Steimel: Just to give some background, currently there is the Picture-in-Picture API that works with a video element to create a PiP window.
… One limitation is that it only shows the video and the only interactions that can happen are through controls that user agents provide, essentially media actions. Pretty limited.
… Goal is to have a richer PiP experience.
… This has been attempted before, a few years ago.
… I don't know exactly why that stalled, we're trying to handle it carefully.
… Essentially, the web app requests PiP and receives a handle to a new window and they can use that to put content in it.
… The website cannot set the position, cannot navigate, resize, etc.
… As far as status goes, I do have a draft spec that have not been shared yet. That should be released fairly soon.
… We need to move the explainer and draft to an incubation.
… Our goal is to get an origin trial soonish, within a quarter or two.
cpn: This suggests a concern around malicious use.
Tommy_Steimel: we've had many talks with security folks internally.
… Essentially, we need to show the origin at all times, and implement a number of constraints.
… We have had a number of feedback and eventually approval on our approach.
cpn: One of the use cases that I would have for this is essentially a custom video player.
… Still video playing.
… Is there a way to maintain continuity of video playback between PiP and non PiP.
Tommy_Steimel: Essentially, you move the video element to the new window.
cpn: As simple as that?
Tommy_Steimel: Yes. At least in Chrome, that should work.
joey_parrish: Are you talking about removing a video element from a document and appending it to another document through this? Does that work?
Tommy_Steimel: Yes.
joey_parrish: I would have assumed that removing a video from a document would tear down everything.
jernoble: You can do plenty of things with a video element in the background, e.g. to render the video in a canvas. Nothing says that the video element needs to be attached with a document.
… There's nothing that prevents things from working to transition between two documents from the same origin.
Tommy_Steimel: Relative links are parsed according to the originating tab.
jernoble: Curious about the requirement that PiP window should never outlive the opening window. If the opening window navigates, does the PiP close?
Tommy_Steimel: Yes.
jernoble: That was a shortcoming with the PiP document. I'm surprised that this requirement gets copied here.
… From a user perspective, there is not a direct connection between the navigation state and the fact that they're watching something in PiP.
Tommy_Steimel: I think the requirement was mostly to follow the behavior of the original PiP.
jernoble: Twitch has its own PiP behavior. Through a second player. They would probably want to use that API. They can survive the closure of PiP. I would look at how Twitch would make use of the API.
gkatsev: Does the PiP window have any kind of chrome? Or does it have like nothing?
Tommy_Steimel: It does have something that makes it clear that it's a Chrome window, for anti-spoofing reasons.
gkatsev: If you want to have a custom player, that's going to end up being not as nice as the native PiP player, but don't know if there's a way to do that while preserving security.
jernoble: I can imagine other problems like with touch-start actions in non-chrome parts of the window.
… In the existing video PiP use case. When the user drags the window by any point that is not a control. If you want to define controls yourself, you may end up in a situation where the page itself blocks effective dragging of the window.
… Some pitfalls that we should probably consider.
Tommy_Steimel: It definitely works better in a desktop environment. Not planning on shipping on Android for the time being.
jernoble: PiP is usually pretty small. And you'll find people willing to make this as unostrusive as possible.
cpn: Have you thought about a more tightly scoped API for video players with custom controls and styling
Tommy_Steimel: That should be fairly easy, main use case.
jernoble: Have you considered restricting user interactivity to a small subset of the window. In the inset (safe area), things are interactive.
… Outside, things are not.
… Some way to make full "interactive" PiP difficult to end up with
<Zakim> gkatsev, you wanted to ask whether this new pip have any UI elements built-in
cpn: You mentioned taking the spec to an incubation venue.
Tommy_Steimel: Yes, I need to run a TAG review.
cpn: Yes, and typically incubation goes through the WICG.
… Thank you for the introduction!
AOB
jernoble: One thing that someone brought to me a couple of days ago. We added a flag in media capabilities around spatial audio. Right now, if you plug an headphone, that changes the value and there's no event that's fired.
… Something to consider in discussion with the Audio Working Group.
… If I was there at TPAC, I would have brought that to the Audio Working Group.
cpn: Focusing Media Capabilities on decode rather than rendering. I agree.
jernoble: Exactly.
cpn: New Community Group proposed by Dolby on A/V formats for web browsers. They cannot tell whether the content will be rendered as-is or down-sampled somehow.
jernoble: Same problem we had with HDR.
… We solved that with CSS for HDR. But we don't have media queries for audio, so we need another mechanism.
… This is related to the question I had. Not directly from them, but also about Dolby content.
cpn: Thank you all for coming or being online.