Meeting minutes
Recording: https://
Slideset: https://
Screen Capture 🎞︎
Issue #276: Handling of contradictory hints 🎞︎
[+1s from fippo, bernard, harald]
Jan-Ivar: this shows maybe we went a bit fast with these hints; the UA is free not to react to them
… I would just ignore them
Bernard: I've noticed this pattern with hints in other places
… it's problematic - this feels like bad config
… maybe they should not be hints
elad: certain constraints can be self-contradictory leading to overconstrained error
Timp: what's the goal here? notify the developer they set an incompatible set of options? protecting the user from something bad?
Elad: what error should be thrown interoperably?
Timp: but what would happen with the error thrown?
Elad: the developer should fix their code
TimP: you want it to fail then?
… this is probably the right thing to do
Elad: having fail it the same across browsers would be good
Harald: +1 for specifying something; I don't it matters so much what is specified - ignoring may be OK in some cases
Elad: +1 to focus on interop first and foremost
bernard: some tricky aspects: given the goal is to guide developers, how would they be guided toward their mistake?
… if you ignore or reject, it needs to be clear to the developer what was ignored/rejected, and what remains in the end
Elad: the error could point to the list of things allowed/disallowed
… ignoring the hints makes it probably trickier to avoid unexpected results
Bernard: can you retrieve what was eventually applied?
Elad: if you reject, nothing is applied
Bernard: but what if you ignore?
Elad: hence why I think rejecting is the right thing
Youenn: normally we try to minimize API to avoid contradictory choices - we should avoid that moving forward
… here, rejecting to get interop is OK
Jan-Ivar: we have to take into account display switching (as we will discuss)
… looking at a PR will help
Elad: the PR for display switching has some discussion on how it is impacted by constraints
RESOLUTION: Consensus to specify an interoperable behavior and iterate initially on a pull request to be proposed by Elad
Issue #281: Distinguish cancellations from absent OS permissions 🎞︎
Elad: Jan-Ivar mentioned that permission.query might serve this purpose; I wonder if it introduces a fingerprinting concern
Jan-Ivar: you're right that it would, but the mid-term solution proposed in the issue would match mitigation for permission.query, so I think we could make that work
Elad: would that work with Safari in embargo mode?
… when the user cancels 3 times a call
Youenn: this is under the control of the UA; it could report "blocked" under these circumstances
Elad: but would there be a way to distinguish OS-blocked vs user-blocked?
Harald: if we want to have distinguishable situations, there needs to be matching API surface
Youenn: I'm not sure embargo-mode or not is relevant
… what Elad is asking for is to tell the user that "something happened"
… if we have more than OS-rejected vs user-reject, an enum is needed
… otherwise, the permission API may be enough
… it would still need to be specified given that permission.query is still a bit fuzzy
… do you foresee more values in the enum?
Elad: another problematic scenario
… an app being relaunched may not be able to determine if it's blocked because of the user vs OS without calling getDisplayMedia
Youenn: that's true of your proposal as well
… so for the enum, do you see more values?
Elad: a boolean may be OK but harder to extend
… the new API I propose would work better e.g. if a browser in the future decides to embargo permission after a single call
Jan-Ivar: I would object going further to permission.query - we shouldn't reveal an OS level decision, the user agent is the party
Elad: how about boolean "user-rejected" yes or no?
Jan-Ivar: that's equivalent to permission.query
Elad: the user won't know what they can do to solve the situation
Jan-Ivar: that's up to the UA to guide them
Elad: I think we should empower the app to help the user instead of always relying on the UA
Youenn: this isn't specific to getDisplayMedia - this would probably apply to mic and cameras
… how are we dealing with this?
… if it's a problem worth solving, it should be solved across sources
Elad: this solution could be extended to getUserMedia
Youenn: I think we should explore permission.query - if it works for gDM, it would work for gUM
Elad: I'll explore this, although I think the embargo issue will be problem
harald: the user operates with the UA & OS separately, and the UA and OS aren't friends - there needs to be a way to guide the user toward the OS
… a query based system can only tell you about the situation as it is now; the error feels like a superior situation
Elad: permission.query is async - so indeed the answer may no longer the right one
[skipping issue 219]
Solve user agent camera/microphone double-mute (mediacapture-extensions) 🎞︎
[jan-ivar points to w3c/
[jan-ivar points to https://
[hta, aboba +1 MuteReason on chat]
jan-ivar: the 1st thing we should is to make it clear where the discussion is happening on github
… I didn't feel like the slides were representative of the discussion on github
… I see the same issue with MuteReason as what I described earlier
… I don't think we should expose an OS setting
… there are cases where the UA might be muting - which is why I opened an issue to explore that question in w3c/
Elad: I want to impress on everybody that this is an important issue for users and developers, and brought critique on alternative solutions that had been suggested
Youenn: there is an existing solution with the MediaSession API that is shipping in Chrome
… we need to discuss with them whether this will solve this issue or not, and if not, we need to understand the differences
… it may be that we could fix or extend the mediasession API
… we need to understand the relationship between the two no matter what
… I also agree with Jan-Ivar we need to clarify the meaning of mute vs end - this is hurting developers
… this feels as important as this issue
… there are interoperability issues in end vs mute - they're all different across browsers
… I would like to start a discussion with the Media WG to understand the interactions between track.muted and MediaSession
Elad: we've looked into this and it didn't look like a compelling solution
Bernard: when the app controls mute, it can inform the user you're speaking while muted
… but it can't do that when the OS is in control; what would you do with MuteReason?
Youenn: the track would be muted, but you would still be able through a separate event to notify the app that the user is speaking
Elad: but if the user is not trying to speak, the user would still not know they're OS-muted in the app UI
Bernard: how would you surface this?
Elad: the app could reflect that via multiple UI states, or reflect the latest detected change, or give a bit more context in the UI
HTA: I tried in Chrome: setMicrophoneActive: true doesn't unmute, setMicrophoneActive: false doesn't mute
… re end vs mute - if they're diverging behavior, we should fix that - in implementations or specs, as needed
… but this is orthogonal
Jan-Ivar: the app receives enabled/muted already - MuteReason doesn't address that
… the MediaSession API provides an API for applications that have mute toggles to expand those to keyboard, hardware, lock screen, etc
… there is no muting in MediaSession at the moment
… the UA is already allowed to mute at any point
Youenn: we do need to talk about the intents of the MediaSession API
Elad: we've shown a PR of what MuteReason could do; we haven't heard why we shouldn't expose an OS setting
… our own privacy review has qualified this as benign
… I suspect solving this with MediaSession will be complex, but happy to be proved wrong
… Hope we can make progress on this before the next meeting
Dynamic Switching in Captured Surfaces 🎞︎
jan-ivar: good summary of the discussion, thanks! looking at slide 40, would sourceswitch fire if there is no opt-in?
Trove: I think we could
jan-ivar: would the event come with the same stream in that situation?
Trove: it will be a new stream
Jan-Ivar: I'm in favor of the late decision model: you get the event regardless you opted in to assisted switching
… having the injection model as the default, and preventing it with preventDefault() is aligned with well-known Web platform patterns
Elad: looking at the code slide 42, I find it hard to understand what preventDefault() does
… and if there is not preventDefault(), it's even less clear that something default is happening
… this feels foot-gunny
… what purpose does this serve? What app needs to make a late decision? it feels like app would always want one or the other model
Youenn: I wasn't sure whether surface switching with injection model was good or not
… I think it's good to support surface type change - we have the configuration event for that
… allowing apps to optimize it is good; late-switching is a nice-to-have, but not required if it's particularly difficult to implement
… but having web developers flexibility is nice
Elad: you've often mentioned that some complexity is to the detriment of the developer, which seems to be the case here
Jan-Ivar: the complexity comes from the fact that this is changing an existing shipped behavior - we can't avoid it
… no matter what solution we choose, the default behavior won't be obvious
… I think it's better to fall back on the injection model
Trove: it's hard to know *when* you need to switch tracks
… it affects capabilities, it may affect which methods can be called
jan-ivar: the question is when the decision happens - the app chooses
… a late decision can allow to minimize the glitch by detecting what kind of replacement is happening
Elad: shouldn't the UA fix that?
jan-ivar: the UA can't fix this - e.g. with replaceTrack because of downstream consequences e.g. in MediaRecorder
Elad: could we demonstrate a path forward for a later addition of late decision?
Jan-Ivar: -1
RtpSender Encoded Source 🎞︎
Slide [47]
Slide [48]
Slide [49]
Slide [50]
Slide [51]
Youenn: in general, we want to design the API to allow plugging an encoder
… then we add specific constraints for cases where sources and encoders are combined
… that would be my general advice
… I would be in favor to have immutable objects in general
Guido: so constructor over setMetadata?
Youenn: yes - that has been the approach with WebCodecs
Guido: I can agree with that
… For the forwarding use case, you already have frames that you are forwarding
Harald: we have an agreed upon use case, the fan out, but we don't have one for encoders
… I kind of agree treating frames as immutable is better if we can solve the @@@ issue
… but for this use case, handling RTP-relevant data is what matters, so I don't see the need for a new object
Jan-Ivar: we've heard many proposals for ways to expose the media pipeline to JS
… via transforms
Guido: this is not for encoding, but forwarding already encoded frames
Jan-Ivar: but does this not also allow JS to get frames from anywhere?
Guido: from another PC?
Jan-Ivar: Youenn's proposal allowed to get frames from WebCodecs
Guido: yes, but in that case they don't have the RTP metadata that would allow forwarding
Jan-Ivar: but I'm not sure there is consensus on Youenn's proposal
… re [slide 48], where would be RTCRtpSenderEncodedSource exposed?
Guido: I think Youenn meant to expose on Worker only, with the handle being transferable
Jan-Ivar: happy to hear that
[TimP, Harald gives +1 to the approach]
Bernard: the RTCRTpSenderEncodedSource - do we really need two enqueue methods?
Guido: we only need one for the forwarding use case
Bernard: can we convert between audiochunk and and rtpchunk?
… this could simplify the API
Guido: there isn't such a way at the moment, but we could look into one
Harald: a constructor with a chunk as input could do this, and avoids touching rtpsenderencodedsource
Jan-Ivar: I think it may be a good direction, but I would love to see JS code that shows where the encoded frames are coming from
Guido: [shows slide 47]
Jan-Ivar: is there a receiver equivalent to this?
Guido: that would be the logical follow up to this
… e.g. to turn multiple input into a more reliable input for playback
RESOLUTION: seems like a promising direction for which to see a more complete proposal
Keyframe API 🎞︎
Harald: #215 is merged - we now have a spec for a keyframe event
Jan-Ivar: +1 - hoping we can present API proposals next time around