Meeting minutes
WebCodecs - Information on encoder/decoder performance
aboba: Some feedback from developers getting different results when using "prefer-hardware" and "no-preference". With hardware acceleration, you may get an error. With "no-preference", the implementation may fallback to software, hiding the error.
… Developers were confused.
… One option is to give them the error from the ground up, instead of having a fallback mechanism in place that may hide it.
… For a high level API, we would keep things under the cover. For a low-level API, it may make more sense to expose the error directly.
jernoble: So, no "no-preference"? If developers don't care, why would they care about getting the error?
aboba: An alternative is to use an event. But it seems preferable to surface the error. Also, the spec does not say anything about fallback. Question is: if you select "no-preference", do you get the error?
… Unusual to hide things in an encoding API.
jernoble: Is there any guarantee in the spec that "prefer-hardware" will give you hardware?
aboba: You may get an error back. If you don't get an error back, in Chrome you will not have a software fallback from under you, although that's not explicit in the spec.
jernoble: We don't have an API that requires hardware, so I'm confused.
aboba: In practice, that's not a hint in Chromium. If you don't get hardware, you get an error.
jernoble: We went through this in the previous round of discussions on these hints.
… Now, we're coming back saying that Chromium implemented the hint as requirement and thus we need an event or error. That seems backwards.
… In this particular use case, they should be calling "prefer-hardware".
aboba: Treating this setting as a hint does not work well enough.
… If it's not a hint, you get the error immediately. That's more simple.
jernoble: A lot of things are easier if you don't try to protect privacy.
aboba: No privacy either way, just look at performances.
… FPS will drop significantly with software encoding.
jernoble: We have to go through the Privacy Interest Group when we develop the spec.
aboba: What we're talking about here is the meaning of "no-preference", and about fallback.
… This is not about performance information
… Because of the fallback, hardware errors are not being surfaced.
cpn: The PR #645 has been overtaken by more recent discussions.
aboba: Right.
eugene: From an implementation point of view, do you plan to change the spec to prevent browsers from switching when "no-preference" is set?
aboba: No, it would be clearer that "no-preference" is not the right setting if you're willing to be notified about hardware errors.
… The question becomes when should the error be fired.
… That's a separate question. I can work with the developers to understand what they really want.
… If you provided the error, they can probably call configure() again. If that works for them, event may be the solution.
… If hardware does get taken away, there should be a notification.
cpn: Proposed resolution is to add a note to the spec about "no-preference", and surfacing the error as event.
hta: In WebRTC, this is useful to tell peer that another encoding/decoding config needs to be used. In the WebCodecs, is it really easy to switch between codecs?
aboba: That's what I'm trying to figure out. If it's a serious GPU issue, then the error may not be recoverable. I don't really know.
… Otherwise configure() may work.
… We may not want to expose the fact that you crashed your GPU for instance.
cpn: Has that come up in WebGPU for instance?
aboba: That's a good question.
… Resolution is to add a note that "no-preference" means no preference, and create an issue to check whether and how to surface the error.
cpn: Implication is that "no-preference" would not trigger an error.
Where to put per-frame QP in VideoEncoderEncodeOption
Issue #270
eugene: Advanced feature that allows constant quality encoding, and a few advanced use cases. Two ways to do this.
… First is to have these QP parameters inside a generic quantizer field to VideoEncoderEncodeOptions. Second one is to have codec specific quantizer field to VideoEncoderEncodeOptions through sub dictionaries.
… The second option is more complicated.
… Different parameters per codec.
… There may be codecs in the future that don't have these parameters at all.
… If we put the parameters in a generic field, we still need to explain the ranges per codec.
tess: Why not do both?
eugene: The argument against having both is that this may be misleading for users who might not know where to put these parameters in the end.
cpn: In the discussion, it appeared that these QP parameters could have different values per channel, e.g. Y, U, V.
eugene: Yes, codec-specific would be more flexible to accommodate such issues in the future.
… We only need to provide supported ranges in the registry. They're not going to scale. WebCodecs is low-level API, so we don't want to introduce more complexity.
cpn: The decision is that there will be something per codec that defines the ranges no matter what, and then the question is whether to put the parameters in a generic field of codec specific field.
jernoble: With my engineering hat on, we should reserve generic settings to settings that apply across all codecs. It's not obvious to me that these settings are generic enough, e.g. for old codecs?
eugene: I'm more concerned about future codecs than about old ones.
… Codecs like this would reject QP parameters if they don't support them.
hta: If the application knows the difference between codecs, it knows where to put codec-specific info. If it does not know, it won't know what value ranges to use.
eugene: That's a good argument. Based on this feedback, codec specific subdictionary seems a preferred option.
cpn: I think we have a consensus on this one.
eugene: Thanks, I'll write down the position of the Media WG and merge the relevant PR.
hta: On a different topic, I'm wondering about the privacy implications of revealing the use of a hardware codec. Discussed in WebRTC as well.
cpn: That may be worth doing as a joint discussion.
<hta> w3c/
hta: We've been asked to provide a sensible algorithm to reveal that hardware codec is used. The feeling of WebRTC stat folks is that WebCodecs would be the right place for it.
TPAC 2023
cpn: W3C Team is starting to look into TPAC, and asking groups to reserve slots. Do we want to meet? Do we want to have joint meetings?
… Deadline around May, we still have a bit of time.