Meeting minutes
WebCodecs issue #940
Youenn: There's inconsistency between implementations on colorspace when configurin the video decoder
… When a page provides a colorspace where some members are undefined, e.g., primaries or transfer function, what happens?
… Safari tries to merge. If unavailable, it looks to the bitstream and merges the properties
… Chrome doesn't do that, I think, but if the config isn't fully define, it's ignored somehow. Not sure about Firefox
… First, let's validate where implemantations are doing, then we can converge
Dale: Chromium isn't overriding based on the config colorspace. One wrinke I realised, that on many platforms we have do decode things not the same colorspace
… In those cases maybe want to do the same as with the null format, i.e, drop the colorspace entirely on the emitted video frame
… I'm happy to converge on behaviour
… We tried merging the dictionaries. Some are partially in the header, some in the bitstream, so not consistent
… For media we've tried that before, and it failed ... you end up with two sets of problems
Youenn: If the primaries and colorspace don't match, what should we do? expose to the web, or only apply at rendering time?
… Merging, you might have a fully defined colorspace but it's invalid
Dale: If you merge, you have two valid colorspaces: one in the bitstream, one in config. Which to choose?
Youenn: In webkit we don't try to detect that, just merging
… If invalid and difficult it's to handle at rendering
… The config primes what's in the bitstream
Paul: Re what firefox does, it's sketchy, esp with HDR
… When you test every platform, there are complications. In general I care more about it being well define in the standard
… If you are doing override, maybe you're controlling the bitstream
… In professional applications, I mean, where you might do somethign less standard, for good reasons
Dale: My view is similar, want it to be well defined. WebCodecs always tries to follow the principle of least surprise. Merging is a bit surprising
… So when there's something partially valid, it would follow to treat as invalid
Paul: There are a few places in the standard, there are ways to fall back to colorspaces that are well defined. I think that's sensible. E.g., BT709, would be a well defined guess that's unsurprising
… If more debatable, we should avoid
Youenn: On the encoder side, you could always provide a fully valid colorspace. I don't think we require that yet?
Dale: None might be given to us, that we'd pass through
… In Chrome only a fully specified colorspace is valid
Youenn: So we could validate it in the spec, as mandatory
… For decoding, there are ways to fall back to well defined colorspaces. What standard are you referring to?
Paul: Valid output config after encode is already in the spec
Dale: All codec strings have defaults of BT7.09. HEVC should have that do, but not sure what the spec says
Paul: In the Peak colorspace algorithm, it says "otherwise return Rec709". Sensible, if you find RGB pixels in the buffer
… When we say that, we mention the set of primaries and transfer, and fall back all four items at once
Dale: Do you oppose language such as: if the config colorspace is always fully specified use it. [...] Or fall back to the [...]
Youenn: But that doesn't solve the case that it's fully specified but invalid colorspace. Fall back to ignore it?
… So you specify all members, but they're not compatible with each other. For example, BT709 defines a transfer function, but some might not be compatible with it
Paul: That's not very common
Dale: Might be, could be BT709 with BT601
… I think it's a special case of the decoder choosing some thing else: Use fully specified codec string if you can, otherwise look at the bitstream, otherwise choose a special colorspace string if your decoder has paarticular requirements
Youenn: So it's discoverable by the web app. Don't know if there are interop issues. But we can discover those as we go
Dale: [...] swapping out the colorspace
Youenn: So ideally, we'd need to reject if the colorspace isn't fully defined but it's there. But as we've shipped it, to avoid a breaking change, ignore it
… All members are optional, currently
Dale: Changing the IDL probably not possible at this point. I thought we want if specified, you include all the fields, but that might not be possible
… I can write up next steps, what I've just described. SOunds reasonable?
Youenn: Yes, seems ok. File a PR and we can test. And we'd have to change the Webkit implementation, shuoldn't be too difficult.
Dale: I'll aim to write up today for review.
AudioDecoderConfig with Opus, Issue 826
<tidoust> Require AudioDecoderConfig.description for channels > 2 with Opus?
Thomas: This issue came up through a Chromium bug report
… It doesn't configure, check isConfigSupported first
… Opus with more than 2 channels requires a description, and FLAC requires a description
… Core question is: what is a supported but non-configurable codec? How much validation to do in isConfigSupported()
… I initially thought to pre-validate as much as we can in isConfigSupported()
… Then I looked again yesterday, reevaluating, I guess what Firefox is doing seems OK
… The thing I thought was not ergonomic if we try to pre-validate as much as possible, and force descriptions to be there for isConfigSupported to return "supported", users might have to create a false description just to know they can play FLAC or > 2 channel Opus
… Chrome and Safari are out of sync with the current tests. The current tests say you don't need a description. I think that's OK
Youenn: Safari doesn't use the description. Are you saying Safari isn't aligned with Firefox?
Thomas: I haven't looked at how Safari fails the tests
Youenn: I think we're OK not requiring a description in isConfigSupported(). Will check with Jean-Yves
Dale: The FLAC case is still busted with a description
… There's a bug i the tests, where there are tests for the opposite things. What you're proposing is OK.
Thomas: This mirrors what we do on the video side. So can clarify the spec language and tests
… I can take an action to do that
Eugene: It's a bit deeper for AVC, if you configure as AVCC you're supposed to have a description. So the precedence is that isConfigSupported() doesn't always require a valid description even if configure() does
Timestamp handling, Issue 944
Paul: Also w3c/
Dale: Looking at 944, many audio codecs have priming information stripped away during decode
… Question is what we do with these samples. In WebAudio and media playback these are usually discarded and timestamps adjusted
… In WebCodecs it impacts the first time stamp coming from the decoder, e.g., a -10,000 timestamp, but there are 15,000 samples of padding
… So your first timestamp would be +5000
… Options: Never discard anything, and leave to the user. Or, make it configurable (ffmpeg and other libraries do this). Or, discard and shift the timestamps based on the initial timestamp given to us
… Following principle of least surprise I think we shuold do 2 or 3. I think some people will always want it done
Paul: Agree. Firefox implements option 3, always remove
Dale: Is that true? There's a WPT that FF is passing
Paul: We have internal tests for that, not in WebCodecs
… We do use negative timestamps internally to signal samples to discard, but that might not be reflected in the WebCodecs API surface
… Timestamp 0 should be the first audible sample decoded
Dale: That's largely true. So you think option 3 and there's a bug in firefox?
Paul: Yes, I'm sure there are reasons not to want to do that. I care that the content round-trips nicely, you get what you expect, files should start at the same time
… One complication is for some AAC files where the right info is in the container and not the packets
Dale: Also in AAC it's very deep in the codec and have to inspect it
… For those cases I assume the packets come from the container with negative timestamps, and the site could discard everthing before zero.
… Or have some mechanism to tell us what to discard
… If we have the priming information, there could be a double discarding problem, e.g., in Opus
Paul: I care most about round tripping, and consistence. Not sure how to reconcile that with AAC edge cases
Dale: The user has a workaround, they can handle if in the container. We can handle the bitstream ourselves
Paul: And then the encoding side
Dale: We could emit what the platform does in some cases, but in many cases we're guessing based on OS version
… Apple might be able to tell us what the true pre-skip is
Dale: So let's close 944, if you think it's a bug in the test right now. I'll update the tests and leave a note in the issue
Dale: On issue 626, not sure yet.
Paul: Roundtripping is important. Amount of work needed isn't a lot
Dale: It's labelled CR blocking. We can export information about what the codec is doing, what the site would need. I'd support emitting number of priming frames on the encoding side
… We could emit negative timestamps for priming frames, might be confusing even though it's symmetric
Paul: I implemented a primingFrames attribute that would discard a number of frames. I think it worked nicely
Dale: More concerned about the decoding side. How to reconcile the priming frames with information already available. LEss conentious on the encoding side
Paul: With a regular MP3 you have to find the decoder delay, you pass the number, pass it to WebCodecs, then WebCodecs returns the first real audio frame intended to be heard
Dale: In cases where the numbers match, that's easy to do. If they're slightly off, which do you prefer?
Paul: That needs to be defined. We have a corpus of encoders in Firefox, all encoders and various we found
… I think it's workable. Our tests are sample accurate. But it does need to be well defined if the two disagree
Dale: You have a prototype for encoding and decoding?
Paul: The decoder side wasn't landed, it needed IDL changes
… I implemented the priming frame one, was simple
… We do have decode tests that are very precise. We could port that from our internal tests to WPT
Dale: No strong opinion, just want it to be well define
… Can you update the tests and propose a resolution to the issue?
Paul: Yes
Dale: Do you have a proposal for padding also?
Paul: I'll propose something
Media Session
Jer: The purpose of exposing these things is that Media Session has a skip action for website that have interstitial or pre-flight ads, to indicate to the UA that the user can skip the ad
… We did this based on feedback from sites. But insufficient for some big sites, as they want the UA to present a button for the user to indicate interest in the ad
… E.g., a Buy or Install button
… We didn't want an open-ended, unlocalized text field, so we tried to come up with an initial set of verbs to indicate the user action
… This isn't a replacement for 'skip'.
… Personally, I'm not sure I'd use it, but some big video sites see this as critical.
… The proposal would be to have some metadata that the site could say the current action type, and a new "calltoaction" action
… Happy to get feedback on it. We're interest to implement, and we see site interest, and we'd want it to be interoperable and widely supported
Alastor: If we declare a fixed set, what's the difference from adding a new action?
Jer: We could add a set of new actions. Seems like an unnecessary explosion in the number of action types
… Expanding the list would mean expanding the action list vs the call to action types. The downside of having an enumerated set of types. Making it a single piece of metadata prevents websites shooting themselves in the foot, and if all those actions are supported simultaneously, which one wins?
… The request from the streaming sites, is the button has a label. We can install the callback, but if the label is added later. There's friction between those two calls. But I'd still prefer a single action with metadata than separate actions
Alastor: Separate metadata sounds good to me. Then we can discuss what detail to add in the metadata
Chris: Happy to get back to it in an upcoming Media WG meeting, typically next week. I'll email everybody.