Meeting minutes
Slideset: https://
WebRTC Extensions
PR #148 Delete Removed Features Section
Bernard: brought by Florent - features removed from webrtc-pc that are now causing errors in the extensions doc
Florent: it's easy to add them back if/when needed anyway
RESOLUTION: Remove removed features from webrtc-extensions (section 13)
PR #147 Add RTCRtpEncodingParameters.codec to change the active codec
Henrik: at TPAC and at the January meeting, we discussed adding a codec setting to RTCParameters to allow mixed codec use cases and allow to change codecs without round-trip time if previously negotiated
… and one API to specific codec and scalabilityModel
… #147 is providing that
Florent: the pull request also describes error conditions, similar to SVC
… please bring feedback on the pull request
[support from fippo and harald]
WebRTC Stats
Issue #742 Assorted comments on RTCAudioPlayoutStats
Paul: I discovered new stats landed in the stats repo; I and my co-editor were unsure about they were and initially misunderstood them
… now with greater clarity, we're unsure why they're needed
Paul: I understand that Chromium has multiple paths to play audio out depending on where you put the mediastream
… in case of the media element, a mixer gets into play with a ringer buffer that sometimes cause issues
… the Stats API allows to detect issues that happen at that particular boundary
Henrik: this covers the path from webrtc to the playout device
… metrics in webaudio measure what's happening in the graph, but it cannot detect the path after that to the device
… this metrics is also added when web audio isn't used
… it also measures a webrtc-specific thing, the synthesized sample
… it's the symetric of the media-source stat
Paul: the Web Audio quality API can tell you everything that happens - I can't imagine anything more reported
Henrik: how would you get a playout delay or a synthetized sample?
Paul: the jitter buffer has multiple purposes: conceal packet loss and @@@
… the issue is that are two crossing of domain clocks - the mixer and the connection to the audio device
… this feels like an internal implementation concern - e.g. in firefox we don't have that at all
… we don't buffer and we avoid clock crossings since that require long convergence
… pulling directly from the audio device to the jitter buffer solves everything
… this is addressed by stats one level higher in the WebRTC stack
… from the point of view of the playout system, inbound streams, we have stats already
henrik: we considered adding something to the HTML media element, but we got push back on the challenge this would be
… also considered adding metrics to the mediastreamtrack - this was added recently there, so some of this could be moved there
… but webrtc has value for metrics beyond what's available
… we don't really care about where we get them, but metrics on synthetized samples are webrtc specific
jan-ivar: this addition was merged in the spec by editors, but without consensus from the WG; I think it should be removed
Henrik: I disagree
Harald: let's continue discussion on the issue
Issue #730 The HW exposure check does not solve Cloud Gaming use cases
SunShin: follow up to discussions from last month
SunShin: two proposals were brought forward, but we couldn't get consensus on how to expose hardware capability
… the discussion on github focused on the option in PR #725, exposing a boolean metrics on decoder fallback
… it only turns to true when the software fallback is actually triggered, so doesn't expose user agent capbilitilies
Bernard: can this go back and forth quickly? will this be poll based?
… without events, you wouldn't know when or why it happens
SunShin: correct
Bernard: what would the app do with this?
SunShin: when we detect fallback, we try to recover to return back to hardware e.g. by reducing resolution or reinitiating the session
SunShin: any feedback on moving forward with this?
youenn: I think it creates a side-information channel - it would be good to prevent this
youenn: the goal is to solve a performance issue e.g. with a reduced resolution - maybe that's what the UA could expose (e.g. "you're using too much energy")
… that might be better to expose switches from hardware<->software
Bernard: yes - fallback information on its own isn't very actionable
SunShin: right now, the only way to detect is to measure the decode time
… but that comes with heuristics errors
… e.g. due to network glitches
… having an explict flag would be an improvement
Henrik: it's performance you want to act on, maybe this is about power efficiency rather than hardware/software
… it looks like we know what we want to have but it exposes information
… this PR allows to avoid hardware info
Bernard: my overall question is about the exact purpose - the Media WG was about exposing actionable information for the app
… do we know what that is? or is it just for stat?
henrik: I imagine it's about changing resolution or scalability
dom: feels like relying stats as a way to trigger a change feels like an anti-pattern
… also, the issue youenn was raising is not fingerprinting, but side channel communication where site A & B could detect when they're used together in the same browser
henrik: re stats - this could be moved to an event, that would be fine
… re side channel, maybe we could explicitly exclude the situation where the error is due by multiple usage across site?
… or in general, exclude side channel situations?
harald: how would the side channel work? one site would get it and the other not
henrik: not sure - maybe due to changes over time
youenn: if you start with hw decoder and another site tries to grab it, its UA dependent whether the new one gets it or not
… I like the description of the use case you had - it would be great to describe it on the github issue, it had been difficult so far what exposing the flag would solve
… having events that are actionable by the app feels much more convincing than stats which aren't expected to bring direct user benefits
Henrik: so reconvene with an event proposal that avoids side-channel communications
Jan-ivar: the issue is cross-site channel correlation, but also use grabbing/releasing hardware to message cross-origins
WebRTC-pc
Issue #2820 / PR #2829 setParameters/insertDtmf/replaceTrack should reject on [[Stopping]] as well as [[Stopped]]?
jan-ivar: transceiver.stop() puts the transceiver in stopping state
… WPT and Chromium haven't caught up with that change in the spec
… we're proposing to align with these implementations, and align with the precedent we have for setting direction
Harald: in all cases, this isn't reversible, right?
henrik: yes, it's irreversible
harald: so +1
RESOLUTION: no objection to PR #2829
Issue #2827 / PR #2828: Hard to tell if there are state gaps in connectionState algorithm
jan-ivar: this is a bit editorial - to make clear the tables capture all states
youenn: +1
RESOLUTION: PR #2828 can be merged
Issue #2835: Section 4.4.2: createOffer() and setLocalDescription() resource handling
Bernard: 4.4.2 claims that resources are reserved during the setLocalDescription promise
… it appears this hasn't been implemented that way - resources aren't really acquired
henrik: it's accurate - we delay creating encoders and decoders till the last minute for performance
… so handling fallback later sounds better
Bernard: imagine a codec only available in hw and the hardware is no longer available after the negotiation resolves?
henrik: right, but the alternative is hogging on all resources
Bernard: so yes, we should change the text, and it may be connected to the software fallback situation
henrik: and another motivation to add the event
Bernard: so let's try to make a matching PR
RESOLUTION: #2835 is ready for pull request
WebRTC Encoded Transform: Negotiating Custom Codecs
Harald: when we first designed encoder transform, it was clear I hadn't completed the job - one of the dangling pieces we had found a way to change frames format on fly which meant they no longer matched what had been negotiated
… that's OK when the endpoints can agree out of bands, but not when parties rely on SDP for this, the exact motivation for SDP
Harald: there are elements on the way (SFU, packetizers) that kind of expect the negotiation not to lie about this
… e.g. if you encrypt the first bytes of an H264 buffer, the packetizer can't recognize it
… the right approach is to negotiate what you send and send what you negotiate
Harald: a browser has certain capabilities that the SDP negotiation module exposes and agrees with in the negotiation, before informating the packetizer and the encoder
Harald: we need to expose control to the app to say what to SDP negotiate, including what transform it is applying so that the other party can detect whether or not it supports it
… and likewise, to inform the encoder / packetizer to adapt how it works
… it wasn't obvious how to expose this in SDP via a JS API
Harald: I now have a proposed minimal API to achieve this
… one new parameter in codecCapbility - the packetization mode
… to say what we're sending and how to packetize it
… this matches closely what Florent is proposing for layer encodings - we should align with what gets merged for that
Harald: not ready for adoption, but I intend to gather feedback and experiment with implementation to check if it achieves its goals
… I'll iterate in a spec branch on my repo, and hope to come back to the WG in a month or two
<TimPanton> I am on a train - but If someone could voice this for me...
<TimPanton> I really don't like embedding SDP in javascript objects.
<TimPanton> undefined sdp too.
jan-ivar: I like the problem being solved; have a few nits on the API, but will need to check with my SDP guy
… but it looks helpful
<TimPanton> The problem needs solving of course.
Harald: if Byron comes to IETF, tell him to come say hi
Dom: [voicing TimPanton above^]
<TimPanton> Thanks.
Bernard: this looks like how SKEEP (sp?) work which will be discussed at IETF next week
Harald: I have thought using it for the packetization mode indeed - they have packetization rules of some kind
Youenn: we'll try to make progress on encrypted packetization in AVTCore
… I wonder once supported if we need to make any SDP change
… This API can be shimmed by doing SDP munging - except that depacketizers won't be able to inspect buffers
… but that feels like encrypted packetization is addressing this
Harald: I thought about drafting a proposal for AVTCore focused only on packetization
… it only needs to figure out to frames into packets
Youenn: the generic packetizer Sergio and Youenn proposed is close to that
… I have an upcoming proposal in that space as well
Harald: looking forward to reviewing these drafts
… Hear support for the problem and experimenting, and coming back after more experience
Ice Controller API/
Samir: there are a few WebRTC NV use cases for which this proposal is relevant
Samir: the proposal is an incremental API - the browser does as usual until the app asks to intervene by using the preventDefault() method on events
… it sounds like a good pattern to override the default behavior
… it limits what the app has to implement in terms of the ICE logic, relying on the browser for most of the work
… what the app can do is exposed via methods on the interface - more methods can be added as needed over time
… e.g. gathering is not part of the initial proposal, but could be added later
Peter: some of the feedback last time was that the API shape could be improved
… also tried to think about how to integrate more of the NV use cases, incl forking and gathering
… leading to a proposal I call "WebICE"
… but first explaining the difference between web and native in this space
Peter: most browsers (except maybe FF) have additional ICE optimizations to speed things up
… closely related to what's desired in WebRTC NV Use casees
Peter: there is a gap between web & native; developers have asked for more capabilities for years
… can we close this gap?
Peter: yes - there is a wide spectrum of possibilities
… FlexICE was such a proposal; ICE controller bring even more flexibility
… WebICE would close the gap completely - optional A & B are two versions of that, more or less manual
(slide 54]
Peter: key different between Option A & B is that A is manual - the app has responsibility to send the checks
… the app needs to decide which candidate pairs to send checks on, and which is better
… this is currently built-in logic in the browser - here it would be required for the app to implement this
Peter: similar to what was suggested for Ice Controller - Option B allows to opt-in to manual controls (defaulting to automatic behaviors)
Peter: this simplifies simple examples quite a bit - you only have code for what you want to customize
Peter: for controlling gathering, WebICE allows to exclude a number of candidates during gathering
… candidates can also be pruned after gathering
Peter: there has not been proposals for ICE forking so far - but WebICE supports this
Peter: selecting candidate pair is made very easy with this
Peter: networkCost can be used to select candidates with WebICE
Henrik: in terms of API shape, the main difference is that you have a cake with pre-defined slices you can ask for, whereas ICE controller lets you override specific events
Peter: two difference: capabilities (e.g. forking and gathering - they could be added to ICE Controller but aren't part of the initial proposal)
… and a difference of style - WebICE gives full control (either fully manual or semi-manual)
Henrik: I do see more features; but I'm concerned it may be challenging to iterate on adding features over time with WebICE - it's clearer how to feature-detect with ICE controller
Samir: one example - an app opts out to do something, but suddenly a new slice is brought in - does that break the app?
Henrik: also, how do you detect components defined e.G. in dictionaries?
Peter: that probably matters mostly for option A; one solution might be that the default is automatic; if you opt-out from automatic, the responsibility is clear; if more behaviors can be opted out over time, then it's clear where the responsibility is
<TimPanton> {sorry I'm losing quite a bit of audio here - unsurprisingly but here are some thoughts}
<TimPanton> I have a problem with Peter’s option B - it requires an exhaustive list of things you might want to do - which can be derived from nv-usecases but seems to me not to allow for innovation. I am warming to Samir’s cancelable events - except that they too require us to know which events might be cancellable (and implicitly defines what you are cancelling - which means we are documenting _all_ the ice stacks.) Also some of this looks [CUT]
Henrik: but given the large API surface, it would be nice to be able to ship things incrementally
Harald: [looking at option B example on slide 56]
Harald: splitting constructing an IceTransport without a peer connection and feeding it afterwards feels complicated
Peter: we had a chromium implementation of that years ago; but there are other ways to combine the two objects - the goal was to avoid having the PC create another ICE transport by doing it ahead of time
Jan-Ivar: my feedback is on scope - I'm worried about feature creep
… we already use RTCIceTransport on PeerConnection
… for all intent and purposes, it's the ICE Agent
… my proposal was that the ICE Controller didn't need to exist separately from the PC
… that would be my advice for ICE Controller
… for this new proposal... would IceTransport be really RTCIceTransport?
Peter: in terms of scope, this is about addressing NV Use cases
… it's hard to see how to support forking / gathering while relying the PC Ice Agent
Jan-Ivar: but only forking requires a separater gatherer, right?
Peter: without forking, you may be able to combine gatherer and transport
Jan-Ivar: in this API, you don't use event.preventDefault() approach used in ICEController
Peter: right - we could sitll make the default be automatic in this proposal though
… you have to be explicit about opting in in automatic
Jan-Ivar: I would suggest scoping down
Peter: but this is scoped to WebRTC NV and real-world applications that improve on ICE
… there can be discussions on shape & scope
… on scope, the question is which part of the gap we're not going to support
Jan-Ivar: is there a way to converge on a single proposal - is there alignment between proposers?
Samir: two kind of differences: capabilities and shape
… except for ICE forking, I think both proposals can evolve to have the same scope
… ICE forking require a separate ICE gatherer
… stylistically, I think asking the app to take fully over the ICE agent seems like a big ask
… probalby not so far between Option B & ICE Controller
youenn: +1 to henrik on iterative shipping - figuring out the MVP with an API that can evolve towards an end goal (e.g. create an RTCIceTransport without a PC)
… which of these 2 APIs are closest to the native implementation (libwebrtc of FF's)
Peter: I like the idea to have a roadmap for incremental shipping
… in terms of implementation - I'm familiar with the ICE implementation in libwebrtc - either of these are implementable; there is a slight asterisk for that with regard to ICE forking, but I think this would be doable in libwebrtc with fairly small changes based on past implementation experience
Samir: there is quite a bit that was added in libwebrtc and chromium to prototype ICEController
… the ICEController interface in libwebrtc would work for this; would probably also work for WebICE - some implementation is already there
Bernard: coming back to the use cases (slide 39) - which ones are the most important for the group to consider?
… N14 for instance comes from IoT due to the cost on battery performance
… some were linked to mobile networks performance
… N15 had come up in a number of services wrt reliability
… The forking use case came up in situations of gaming IIRC
… any sense of prioritization in terms of importance?
<TimPanton> I'm keen on N14 !
Peter: the ones I hear people ask about are mostly N01 and N04 - control which local candidates are used and when, which things are paired
… and N14
Harald: the impetus that led to this proposal for N04: when network conditions change, you need to be able to move to a different network without a full ICE restart
… which needs to keep quality metrics
Bernard: right, that's part of the mobile use case
Peter: I would agree that ICE forking is lower priority, although I wouldn't want to paint us in a corner where we can't support it
Jan-Ivar: in the spirit of incremental approaches, could these be built upon the existing RTCIceTransport object?
… I brought this up on the issue
Samir: the initial implementation was built on RTCIceTransport
… it makes it easier to deal with bundling; but there is a question of ensuring that candidates can be pruned
… this could be solved by specific configuration options
Jan-Ivar: same question for you Peter, assuming we forego ICE forking for now
Peter: yes, I think that should be possible; e.g. preventing to re-use IceGatherer
… not sure if the proposal works without an IceGatherer object at all
… it might be possible to make it so that if you don't construct an IceGatherer, one is done automatically for you
Henrik: wrt moving events and methods to the existing RTCIceTransport interface and the risks of a race…
Jan-Ivar: I commented on the issue on this - I don't think there can be a race; if you add your event handler directly after sLD, you won't have a race
<dom> s|comment on the issue|[commented on the issue](sam-vi/
Harald: we have prototype code for ICE Controller
… it's likely that ICE Controller could be implemented on top of WebICE though
… so maybe this needs more experimentation
Peter: I don't think we need to choose quite yet - I think coming up an incremental plan based on feedback we received today would be a good next step
Harald: I'm hearing consensus on the problems to be solved, and that these are reasonable approaches to experiment with