Meeting minutes
<markafoltz> +Present Mark_Foltz
Bernard Aboba
cpn: Sad news. Bernard made an enormous contribution to the group and W3C at large. It seems difficult to talk about some of the practicalities. He was an editor of WebCodecs and some of the registrations.
… From a publication perspective, we will have to update the list of editors in those documents.
… Wondering whether anybody in the group would like to become a co-editor on the registration documents.
Jean-Yves: Do we need two editors?
cpn: Nothing mandates it, we do it conventionally.
Eugene: I'm happy to be listed as editor of all the registries, as editor of the WebCodecs specification as well.
cpn: That would be very welcome from my point of view.
cpn: Would you find it appropriate to add a dedication in the WebCodecs specification? We have an acknowledgment section.
… Has this come up in the WebRTC WG? I'm open to opinions as to whether you think this is good to do.
Eugene: Yes, definitely. Without Bernard, there would be no WebCodecs. It seems right and proper to do that.
Jan-Ivar: WebRTC has not met yet. For WebTransport, we will move Bernard to the list of former editors, we haven't discussed things beyond that for now.
… It sounds nice to add a dedication.
cpn: OK, let's do that.
WebCodecs - VideoFrameMetadata
Jan-Ivar: There's some WebIDL with a dictionary that points to the registry. Dictionary members are all optional by default. No way to say which members are mandatory, which is a good thing in practice. I'm assuming we want all of them to be optional but it would be good to be explicit about it.
… Second point is that, if you add a new metadata type, should it appear in the browser regardless of source, or is it source specific? Example of backgroundBlur.
… It makes sense that source fills out the metadata that is relevant to that source.
Eugene: I agree. I thought it was kind of obvious. For example, a frame from canvas has no idea whether background is blurred. We probably need to spell it out.
… Most metadata in entries will not be populated by many sources. At the same time, any source can potentially populate any metadata.
… All the metadata fields need to be optional and we don't expect that all sources will populate them.
Jan-Ivar: I agree with that. This came up when we were reviewing a patch in WebRTC which assumed that the metadata was always there.
Eugene: Could you prepare a PR with appropriate wording for the spec?
Jan-Ivar: Happy to do that.
cpn: Will there ever be a case when there will be a required member?
… Also, do we expect all implementations to set the metadata for similar sources?
Jan-Ivar: The difference between absence and false is that false tells you that background blur was not enabled.
… Consistency between implementations will likely happen naturally.
cpn: I'm just wondering if there's something in Media Capture Extensions that says that the field must be provided.
Jan-Ivar: I don't think it says so explicitly. The assumption is that browsers should support this for frames that come from a camera.
cpn: So, proposed change would be additional wording around the dictionary in WebCodecs.
WebCodecs - Guidance for user-defined VideoFrameMetadata entries
Jan-Ivar: I noticed the discussion on supporting custom metadata. IDL does not support unknown members at compile time. Supporting custom metadata in the constructor requires additional work. WebIDL cannot preserve whatever member the constructor receives.
cpn: The issue was around naming collision.
Jan-Ivar: I don't know that, if they call the constructor with members that are unknown to the user agent, the browser will just strip them.
cpn: Oh, so maybe there's nothing to be done here.
… Perhaps closing it with no further action is the right way to go.
… I'll review it and close it unless something further needs attention.
Media Capabilities - next steps after merging
cpn: Thanks, Mark for the work on the MIME type.
… Two issues left in spec, one around single codec and a more general question about the WebRTC MIME type where things seem more open ended.
… Did you have an opportunity to look at the test cases?
markafoltz: No, not specifically for this. I thought there were tests that check for single codec and they all pass.
… Some GitHub commenter asked why we reject when there are multiple codecs, so I raised an issue.
… The way the spec was written before, the single codec only applied to the file case. The second question is thus whether that also applies to WebRTC.
… While there are tests for the single codecs, I don't think there are tests yet for WebRTC.
cpn: Yes, I may have added the file tests some time ago.
markafoltz: The way the spec is written, we just reject whenever there are multiple codecs.
… The why that restriction exists predates my work on this spec. If we allow multiple codecs, we'll have to define what that needs.
??1: For audio codecs, it's not clear for example.
markafoltz: Does it mean that streams could use two codecs as once? Etc.
Jan-Ivar: I'm assuming the context is querying Media Capabilities.
… Is it about reading information out of getCapabilities?
… WebRTC can negotiate a number of codecs, JavaScript can be more specific about the specific codec it wants.
… I don't know that we need entries in Media Capabitilies with multiple codecs.
markafoltz: My understanding is, from a WebRTC perspective, understand whether my hardware supports, e.g., spatial scalability or temporal scalability.
… You can make multiple queries in a loop if needed.
cpn: It would be interesting to understand what current implementations do when given complete WebRTC codec strings.
Jan-Ivar: The benefit of multiple codecs might be that you could make one call, distinguishing you from a bad actor as multiple requests could end up being rejected.
??1: Some might want to know precisely what specific codec they should send.
Jean-Yves: You can't specify in either OPUS or Vorbis. I don't believe that the return data contains the MIME type of the data.
markafoltz: Indeed, you wouldn't have information on which codec is supported.
Jean-Yves: Then there is nothing to choose from. If the browser chooses the best codec, there will be no way for the site to know which codec has been used. You end up with a useless blob.
??1: Yes, I think you're right. I just wanted to explain what Netflix and Youtube might be willing to do.
Jean-Yves: My understanding is that Youtube asks specific codec strings with specific width and height that are not per codecs.
cpn: It feels like we need to look at what implementations are currently doing with WebRTC codec strings.
markafoltz: Yes, there's a gap in our test coverage.
Audio Session - Set Default Audio Output Device API
cpn: I think we picked this up at TPAC.
Sunggook: The UA uses the system default audio outpout as a deault output device.
… A frame can control audio output with setSinkId.
… But the host name cannot control sub frame's audio outpout. For example some communication application.
… The solution we propose is to provide a new API that affects all the frames for the web application.
… Permission-wise, same as setSinkId.
… Chromium supports this new API behind a feature flag.
… Some interest from consumers.
… Question is whether the API can be part of Audio Session?
… The proposed API is really just an extension of setSinkId. Audio Session is more a global thing.
cpn: When we discussed this with Youenn at TPAC, we were thinking that Audio Session would be an appropriate home for this.
… I'm not sure I understand the rationale for why it should live somewhere else.
Sunggook: Audio Session give applications some authority to the user agent. With this API, the application decides which specific audio output.
… it uses.
Jean-Yves: The proposal you make is very similar to the one that Youenn made a couple of years ago. It was setDefaultSinkId.
Jan-Ivar: I would love to hear Youenn's rationale for why it fits in Audio Session.
Jean-Yves: Audio Session issue #6 has the history and rationale.
SteveBeckerMicrosoft: Having an API at the Audio Session level on top of the interface. The way it's prototyped currently is for cross-origin iframes. I think there's currently an issue with Audio Session about that. If that was possible to use the same Audio Session, then potentially that could work.
cpn: I think we need Youenn for this discussion and probably Tommy as well.
… Can we schedule a follow-up? If we can get agreement on the scenario you just describe where Audio Session supports cross-origin iframes, that seems better to having multiple places where a sink gets defined.
Sunggook: Sounds good.
media-playback-while-not-visible
Gabriel: I work for Microsoft too.
… I'd like to get feedback and gauge interest from other folks, and maybe find a home for it.
… It's a permission policy that will allow the user agent to pause media playback on iframe that are not being rendered on the screen at certain moments.
… Iframes can be resource heavy.
… Sometimes there may be a need to hide the iframe. Pausing a media may affect the user experience and user agent may need to re-create the iframe from scratch.
… Which can be resource heavy.
… Our proposal is to add this permission policy.
… Application would set it to "none" or "iframe". In these situations, media playback would pause for iframes when hidden.
… This proposal interacts with media elements, Web Audio and also the Web Speech API.
… We've made some progress, implemented in Chromium behind a flag.
… Integrated with media elements. The integration with Web Audio was more laborious due to the need to integrate with the spec and the state machine.
… We have a PR ready.
… Good input in the WHATWG thread.
… A PR ready to add the interrupted state for AudioContext
… This is what we have done so far.
… Our next step for Chromium is to make the feature available through an origin trial.
… And find a home for this.
… The problem spans several media APIs.
… We may need to start with WICG.
Jan-Ivar: The permission would be an escape hatch?
Gabriel: If it does not have the allow, it will just work as today. If you set the allow, there is another specifier, then you active the behavior.
Jean-Yves: Webkit will not allow any element to play in an iframe that is not visible. Primarily, we found that it was a performance improvement. Allowing something audible to play, that kind of defeat the purpose of having a user gesture to have audio play.
Gabriel: I think in this case the behavior would be the same. You owuld not have no user gesture.
… If you have an audio element in an iframe that gets hidden and showed again, the media can resume. If the iframe was hidden from the start, it would not have had permission to autoplay.
Jean-Yves: If you call play() without user gesture, the Promise returned will be rejected.
Jan-Ivar: I think you're trying to add a restriction. But through a permission policy that looks permissive by default, which seems counter-intuitive.
… Sandboxing options tend to be restrictive.
… There might be confusion on how it gets expressed.
Marcos: Yes, there are different sets of options. It doesn't quite fit anywhere. It sort of fits the definition of powerful feature.
… A few more eyes on this would be great. It is a valid use case, but feels strange.
cpn: The WHATWG thread seems to raise the same concerns.
… The venue question is a good one. It spans a number of things.
… I'm not sure about taking this to WICG because you've already identified the right places that need updates.
… Are you looking for a place to hold the discussion?
Gabriel: I was also looking for a spec that I could add this definition to. I don't know if I need to write my own spec, or if there's one I could add this to.
Marcos: HTML seems the right eventual place for where this could land.
… Something's not right about the definition of this thing. It does feel that this goes into HTML. At the same time, it could be its own indenpendant little spec, in which case WICG could be a good home.
… We have Autoplay Policy Detection. That's a bit related.
cpn: It makes me wonder whether there is a more general Media Policy that Autoplay Policy would be a part of.
Gabriel: Do you think that this would make more sense if it was an iframe policy?
marcos: We could perhaps have a media policy.
… Sandbox could also be a good target for this as well.
… My suggestion is to look at all the options and list pros and cons, and that might help answer these questions.
tidoust: Media WG charter will need to recharter soon. So if that feels like a good idea, this could be added to the scope and list of deliverables of the Media WG.
cpn: Once you've done the evaluation, we could revisit in this group where to take this.