Media WG

Media Capabilities

Audio channel configuration issue

Better define channels in AudioConfiguration (#73)

mark: I don't have a concrete proposal yet. Two questions: I looked a little bit at Web Audio. When Media Capabilities queries are made, there is a property that is a string that describes the audio channel configuration that the user agent can decode or encode. Issue is around what goes into that string.
… Web Audio allows to remix channels. Web Audio uses an enum to describe the number of channels or the layout.
… We could deprecate the old property and add a new one that reuses that, but that would be a breaking change to the API.
… To what extent should the Media Capabilities API support describing audio channels? Some of the capabilities may be at the OS level, e.g., different capabilities to support different speaker layouts.
… So, scope is one question. The second question is whether to borrow from the Web Audio API.
… I didn't look at WebCodecs, I don't know if WebCodecs allows specifiying certain types of audio layouts.

Eugene: For WebCodecs, there is the notion of codec strings but they are separate from codec configs.

cpn: There's the AudioData interface.
… Uses an unsigned long number of channels.
… Which is the part of Web Audio you were talking about?

Mark: I just added a comment to the issue on GitHub.

cpn: I think the scope question is key to it. If you look at Jon Piesing's comment, if they know that the implementation is going to do some sort of downmix, like from 5.1 to stereo, they might want to provide a stereo stream instead.
… There may be something about decoding capabilities and rendering capabilities that may be different.
… WebRTC has this setSinkId that lets you enumerate audio output devices, but I don't know that that has any speaker type configuration to it.

Mark: That's probably permission guarded as well to enumerate output devices.

Marcos: I would look at what the options are without looking too much into the actual design. It's good to look at what's already in use.

Mark: I think Mounir made a comment about looking at DASH. They may have something related.

cpn: My guess is that DASH may have a string representation for audio channels. That's a thing that we could use rather than mint our own.

Wolfgang: DASH has a combination of schema URI and values. Something that is common in any case is referencing a table and values in a spec.

Mark: Getting the syntax and doing the validation are the two gaps to fill.
… The downmix capabilities, I'm not sure that's in scope, but to be thought about some more.

cpn: The other place that is weird in Media Capabilities is the Spatial Rendering where we're mixing decoding and rendering already.
… How strict do we need to be about being only about the decoder and not rendering capabilites.

Mark: I also do not know whether we have WPT tests for that, but if they're missing, we should look into adding those, and see if browsers accept some subset of the same strings.
… There's more to do here. I may have more to share next time we come to it.

cpn: Would it be useful for someone to look into DASH?

Mark: Yes, that would be useful.

Wolfgang: I may provide some pointers.

Markus: I did run this discussion through an LLM. It says that DASH descriptors are very precise, with an XML schema and such.
… What would a number mean? Like 6? Is it 5.1? 6.0?

Wolfgang: The table in DASH describes a well-known set of audio configurations, 6 would mean whatever the table says it is.

Mark: Next steps: Look at DASH, check WPT, come back with more ideas.

Media Playback Quality

cpn: 4 issues on the agenda for this. #28 is the key one. Which is about how we handle changes to the spec.
… For a very long time, this spec has been dormant in the WG.
… It's just a simple spec, with one interface to return a set of counters about the video decoder, like corrupted frames.
… In 2021, since it's an extension to HTMLMediaElement, we said we would upstream the spec to the HTML spec. That's reflected in the MoU W3C signed with WHATWG.
… At TPAC, there's been expression of interest to do more work on the spec.
… If media element is connected to a WebRTC context, we need to have behavior specified in the document.
… This year, Markus presented a number of additional counters that are useful in a WebRTC context that you'd like to add to the specification.
… This begs the question on how the group wants to work on this.
… Is upstreaming to HTML still the right thing to do?
… If it is, what would we upstream? The spec as it stands?
… Or should we hang on to the whole spec, do the work, and then upstream the whole thing?
… Or pursue the spec on the Recommendation track at W3C?

sangwhan: I looked at implementation, and some counters are supported everywhere and could be upstreamed.
… Interoperability of corruptedFrames is a bit more sad.
… Looked at implementation in Firefox in particular.
… I would propose to drop corruptedFrames, regardless of how we address the spec.

Marcos: With my Webkit hat on, I'll check internally. If we're shipping it and we find that it's being used, we would be opposed to removing it.
… In terms of the Process, we should make a group decision. There is value to taking it to REC, from an IPR perspective. Getting the licensing on the spec before we hand it over would be a good thing.
… Instead of monkey patching in our spec, the HTML spec could call into our spec.

Sangwhan: For what's been implemented, codedFrames and droppedFrames are fairly easy to upstream, no need to add callbacks. For other stuff, that's more complicated.
… If the group wants to go to REC, I could volunteer to be a temporary editor.

Sangwhan: I did check that there was one shared library that was making use of corruptedFrames. Didn't seem to break.

cpn: Relatively few JS library, such as dash.js, etc.

cpn: So follow up action is for Marcos to check with the Webkit team.
… The question about upstreaming is still open, I think.
… I don't have a strong opinion.
… Getting IPR commitments seems valuable, certainly for the new stuff that we're adding.
… For the existing stuff, I'm less certain about.
… That ends up in a weird position as to how would the v2 spec look like.

Sangwhan: The new things, it would be a while before they become useful to developers.
… If there is a strong appetite to do REC.
… What's already been implemented for years should have a slot in HTML.
… It's based on the fact that it's stable.

cpn: I have the feeling that we already made a decision in the group and I don't want to change that. The stable argument does not really sway things. It could be stable in a W3C spec.

Sangwhan: Upstreaming has been delayed for a long time.

Marcos: It's also tedious work that no one is willing to do.
… With the integration in HTML, they need to know all the places that needs to be updated. Only a few people have that knowledge.

Francois: The MOU says we'd transfer the development of the spec, so I'm all for the WG doing the work to take this to Rec, but we should mention the plan to the WHATWG before enacting it

Marcos: We could perhaps return 0 for corruptedFrames once we check that there's no significant breakage. Removing it entirely might be more problematic.

Sangwhan: Firefox has removed it for 5 years and counting, I'd be surprised if removal created issues.
… We could update the spec to say "return 0"

cpn: And remove the language about this being deprecated. Turn it into a note or something.

Markus: My question is: what are the next steps to get the new counters into the spec?

cpn: I'd like to see another implementer express interest in some of it.
… There were some discussions about formulas that you've been floating around, but I'm not clear that we got a yes from other implementers for now.
… E.g., Mozilla or Webkit. That's the criteria I would suggest.

Markus: The harmonic counters are already exposed and used in WebRTC in Chromium. I don't know if other browsers are doing it.
… Would support from Edge work?

Marcos: We should definitely push to have multi-implementer commitment, at least one willing to do it, and another having expressed interest.
… Usually Chromium and Edge are not considered different implementations. But if it's implemented with two different OS libraries, that may still count as two. That's a case by case analysis.
… We'd have to evaluate whether they're both using the same code.
… I can take it back to the media team. The best way to get feedback is by requesting a standard position.

Markus: That seems a good way forward.

Marcos: Reviewers would be the Webkit media team.

cpn: The issue that you raised talks about WebRTC getStats().

Markus: What you get from requestVideoFrameCallback() is not accurate. It's an estimated timestamp. This would be the perfect place to place this.
… requestVideoFrameCallback() can be used to compute things, but we get power regression whenever we try to do things with it on frame by frame basis.

Markus: So next step is getting a standard position.

cpn: Back to the upstreaming question, I'd like a more concrete step.

Marcos: I think we could issue a Call for Consensus, and based on that, we can move forward.

cpn: Agreed, let's do that.
… I think we can do that based on issue #28, turning that text into a call for consensus.

Marcos: Call to action would be for engineers in the group to take that back to their legal teams and get a response from them. It may be "we don't care", and that would be fine.