Web Real-Time Communications Working Group Teleconference

29 Oct 2012

See also: IRC log


andy_hutton, adambe, hta, stefanh, derf, burn, dan_druta, richt, anant, dom, juberti, jim, matthew, ekr, fluffy, Magnus, Vidhya
Harald, Stefan
adambe, markus, JimB, martin


<inserted> scribenick: adambe

hta: the best use of face-to-face time is to be specific
...: people pay more attention campared to mailing lists
... we've tried to make the agenda specific
... we might go with presenters proposal
... there also be discussions
... we should record off-topic topics and move them to the AOB session

stefanh: fist session is about API functionality we haven't addressed yet
... this might be the least concrete item on the agenda
... what de we need to do now and what can we postpone

API functionalities missing in PeerConnection API

Stefan's slides

stefanh: API topics
... width, hegiht sent over a PeerConnection
... priority
... is media flowing?
... how much bandwith is used
... how much can be used
... how to reject media streams
... echo control
... one thing missing on the slides
... should we support security descriptions

(a use case is presented)

scribe: one peer sends video with two different resolutions to two other peers
... are people ok with this?

fluffy: we should support this

burn: you're showing the entrence and the consumer side

stefanh: yes

hta: there are multiple ways to achive this reuslt
... first question is.. do we need to support this?
... the answer is yes
... then how should we solve it

stefanh: this is the most difficult slide in my deck
... application knows about the receiving side
... or the receiving side could tell the sender
... third possibilty is that the app isn't involved

matthew: even if the receiver is involved the sender needs to be involved

justin: there needs to be some higher limit

matthew: there might be a lot of contstraints

fluffy: this is one of the reasons we need constraints

stefanh: we shouldn't modify the sdp unless we have very good reasons

hta: we currently don't use a speakers queue

ekr: what do we expect to happen if the receiver asks for a different aspect ratio
... should the PeerConnection rescale
... ?

justin: the sender might need to chose the closest thing

hta: we want people to separate the case where you really require something compared to the case where you really can't live without something

<matthew> side comment: generally we either need to fully specify the SDP the is used to negotiate things like this OR we need contraints. unfortunately when there's contraints, we *also* need to fully specify the SDP that flows between them... unless we choose to have it *only* be an API issue ("direct manipulation" vs. "constraint setting")

stefan: is the sending app or sending PeerConnection informe

justin: information about the consumer dimensions could be sent as a session description

fluffy: the connection to the video tag is interesting
... would it be reasonable to adjust the send size to the consumer size?
... this was a driver for renegotiation callback

martin: you don't have the consumer in the initial negotiation

fluffy: it's not always true

derf: css also impacts this

fluffy: a video tag might be specified with a size or not

stefanh: there could be several video tags that wants to affect the sender side

hta: there's a ultimate fallback video size

burn: smart browsers might behave differently

matthew: smart browser decisions must be overidden if the developer wants to do something else

burn: if we have an API then the browser must take input from the API into account... otherwise the browser can do its magic

stefanh: what is the conclusion?
... the receiving side can influence, but we also need an API to let the developer tweak

ekr: do we want information from the receiver to bubble back all the way to the camera on the sender side?

burn: a browser can use any valid value in a contraint range

matthew: if several consumers (local, remote).. who wins?

ekr: some constrains are enforced when set and some constrains seems to be enforced continiously

burn: we might need a constraints that says "do not change video size"

stefanh: recap on proposed conclusion
... receiving side as driver with an API for the app to influence

matthew: there's two ways to implement this

fluffy: we don't know if we'll use RTCP or SDP to negotiate video size

matthew: even if we go with RTCP we need an API to influence how the browser should behave

stefanh: let's move on
... next slide
... if the receiver detaches a stream from the consumer.. what should happen,
... ?

ekr: what if sender offers something that the receiver doesn't want?

stefanh: we'll get to that in a later slide

fluffy: we have two operations.. mute (undetectable from the sender side) and to say "don't send me packets"

justin: how do you say that I only want a subset of the offered set of streams
... we don't have a good way to express that today

stefanh: next slide
... requeseting a certain bandwith
... there's an IETF draft talking about this

DanD: I propose a priority param to addStream()
... we can discuss how the priority information should be handeled but we should decide on a mechanism to specify the intent

stefanh: addStream() is only called once per stream, it's a problem

DanD: we're the right group to specify the priority intent

burn: intent means it might not happen

<martin> hmm...s/constraint/pref/g ?

<ekr> ?

<martin> that works too

<matthew> there's a difference between "preferences" and "absolute limits"

<matthew> i "prefer" 720P, but there's no way i ever want >2560 horizontal pixels

<burn> no, constraint != intent

DanD: at least the browser sholud be able to say that I marked you packets in a certain way

<burn> intent == hint, discussions about hints lead to a decision to support both mandatory and optional constraints

<matthew> if you just go with direct manipulation then the API only has "settings", and the difference between "constraint" and "preference" is JS code

<matthew> but the moment you let the two ends talk to each other via possibly-unmodified SDP blobs then you need places on the monster where it can be poked with a stick

Göran: we should describe this case in more detail

scribe: can the network trust the diff-serv code points set by the app

fluffy: in some environment you can't
... they might provide information
... but a separate request is needed to verify

Göran: we need more discussion around use cases

fluffy: I think this should be a topic for the AoB section

(people agree)

hta: new topic - Security for Qos flow labels

ekr: there might be two API surfaces needed: tie constraints to media flow, (missed the other one)

DanD: I wan't a request that should be handeled by the trusted environment

matthew: even if diff-serv cps isn't supported a system may enforce priority anyhow (we need to support this case as well)

hta: new topic for AoB - Control of the DSCP interface

stefanh: next slide
... other side of the same coin
... feedback on bw..

martin: I thin we need generic feedback on constraints set
... there might be existing mechanisms
... for others we might need to add something

matthew: e.g., if video has highter priority over slides, the app might want to know if the slides can't be sent at all

stefanh: next slide
... sender side pause/resume
... we currently have enable/disable on MediaStreamTrack
... every consumer is affected
... we could have enabel/disable on PeerConnection

burn: we don't need that since you could clone media streams

hta: we implemented enable and disable as affecting the associasion between a stream and track

stefanh: next topic, AGC

fluffy: the text on the slides looks good to me

<martin> conclusion here was that we don't need to prioritize the agc/noise settings, it's not a preference, it's a setting

<martin> it might also be necessary to change this on the fly

burn: what happens if you turn AGC on and it's not available

fluffy: it should be a on/off setting

derf: I hope there's a way to enumerate settings

<martin> we need a way to enumerate settings, for sure

fluffy: we should require that

<martin> agc/noise is a property of a track

hta: AGC should be on track level

stefanh: next topic
... rejecting streams

martin: we need to know if multiple streams are offered
... we need basic methods on the session description

derf: example: you offer me 10k streams.. how do I reject

DanD: it's not only if your device can do something.. there's also a question if your service can support this

martin: we need to know what you get before you can say what you want to accept

hta: if we have a mechanisms for the receiver to turn off a stream at any point.. would it be sufficient to turn it off rather than not accepting it?

matthew: it might be a resource question

ekr: I interpret it as: you set the remote description and you see what streams are offered

stefanh: is the sending app informed about a rejected streams
... ?

hta: if the use case for rejecting streams is to save resources then the sender needs to know

matthew: it's important how quickly can this be done

fluffy: can someone send a proposal to the list?

(I missed to scribe a lot of stuff while I was talking)

stefanh: next thing - AEC

fluffy: one param you need is what's going to the speakers

<martin> new issue that is related to this, and one that I will bring up later when we talk about this: SDP describes RTP sessions. The one m= section can have multiple tracks. How do we learn of these other than just waiting for arrival of packets on one of those?

<hta> adambe said: I sent a proposal to the list with inbound and outbound streams as different objects, which might make this discussion easier; missed having feedback on that.

<matthew> another issue that's related to the last slide and not yet decided (and different in different implementations) is what to offer/send before user consent on a sending device occurs. one approach is to say "recvonly" and change to "sendrecv" when the consent happens, the other is to say "sendrecv" and send muted audio + black video until consent is received. recvonly->sendrecv takes the extra round trip to enable.

(scribe is missing a lot of stuff here)

burn: why do we need API for this?

fluffy: you might want to turn it off

burn: could this be a browser control?

someone: music is a use case where you wan't to turn EC off

matthew: broadcast is also a use case where you want to turn AEC off

martin: if the browser and user wants differents things, the user setting should win

stefanh: final slide - SDES
... wait until RTCWeb has discussed this

ekr: yes

stefanh: that was easy
... summary slide

martin: you could get two different tracks - one with AEC and one without

hta: new AoB (MC TF) to topic - Multiple open of camera/mic?

martin: AGC, NR should be fairly simple

burn: rather than doing this right now we should sumarize what we have decided the last hour and update the summary

hta: there are things that needs to be discussed and some that just can be edited in

Acoustic Echo Cancellation

stefanh: we're 15 minutes before schedule

<markus> coffee break

<hta> anyone remote actually here?

SDP handling

Cullen's slides on SDP handling

<markus> getting started, markus is the scribe

<markus> fluffy showing slides

issues: what SDP do createOffer and createAnswer create and how they can be manipulated before passing them back to the browser

matthew:some changes are ok from SDP syntax perspective but not for the browser to act upon

goals: clear definition of SDP and error handling rules

how are new ICE candidates added to the set SDP

Matthew: when I ask which SDP is in use is it the one I have set or can it be something slightly different?

constraints: can be used to enable common use cases, but do not solve what can be changed

burn:constraints can affect what SDP gets generated in the first place so it does not need to be modified anymore

fluffy: the place to define SDP use is in the JSEP draft - latest draft has a start

<AndyHutton> remote audio not working

matthew:state transitions need to be taken into account, can't call everything multiple times

<matthew> specifically, setRemoteDescription("answer") is restricted to only-call-once even though we are saying that enforcement of these transitions is up to the JS

what can be changed between create* and setLocal/Remote

use cases: remove codec, change bw limit etc.

justin: enable/disable bundle is one case

adam:rejecting audio, getting video - is that with munging SDP?

fluffy:no, if we have explicit API then don't use SDP mungling

matthew:have a detailed set of cases

fluffy:grouping of cases:

fluffy:don't do RTCP mux via settings, not SDP

matthew:ptime for codecs

matthew:want to fix it

matthew:will take my cases to the list

[several people agree most things being discussed should be done via other mechanisms than changing SDP]

burn: nervous about limiting too much what in SDP can be modified

fluffy:changing SDP on the way between browsers can be done flexibly, changing between create* and set* in the same browser more limited

justin:don't understand why there is big difference between different manipulation "loops"

fluffy:positive list of what can be changed is needed, not the list of what can't be

provide a list of changeable things that is *guaranteed* to work

matthew: errors need to be specific on what is wrong

[argument over what the current spec (JSEP?) says...]

burn: many kinds of modifications will be needed before sending SDP out

burn:what you send out does not match with what you set with setLocal?

<ekr> who was speaking?

matthew:jsep-02 has fixed the ice password issue

fluffy:need to have the use cases that motivate changes

justin:needs coming up in the future anyway...

how much and what state does createOffer create? devices with HW codecs?

<matthew> jsep-02 fixed the "createOffer is optional" language, though whether or not createOffer creates state is true or not is questionable

<ekr> matthew: though, there's clearly an implication of that b/c of the validity window of the offer.

<ekr> i.e. during the callback.

<matthew> indeed

martin:set down the list of things that MUST be possible to change

burn:is everything else MUST reject?

people have different interpretations...

explicit list of things that MUST be changeable, explicit list of things that MUST not be changed and what falls in between the browser need to explicitly error report if it won't support it

<martin> there is a list of things that we MUST be able to change; there is a list of things that MUST NOT be able to change, which trigger an error; everything else the browser MUST either accept or reject with an error

<matthew> silent failure is incompatible with the assert on the 2nd slide. either the browser fires an error (invalidating the assert) or it is lying to you in order to make that assert.

[the above three comments (markus, martin, matthew) try to record the consensus in the meeting]

AndyHutton: Configuration/settings? how do they relate to the MUST, MUST NOT, ...

fluffy:start with the list of use cases to get the MUST-be-changeable list

matthew:listing MUST-items...

[cullen taking notes...]

adam: we will have APIs for some of these things anyway

fluffy:if we have an API to control what createO/A gives, is that not enough, do we still need to change SDP for those "features"

justin, matthew: there may be cases where SDP change is still needed

fluffy: use cases are still not clear

burn:do not worry about MUST-list anymore because anything on the MUST NOT list can still work...

still sensing consensus on having three lists: 1. MUST be changeable, 2. MUST NOT be changeable, 3. (default) Browser MUST give an error if it does not support it

fluffy:next issue

when can two different video flows use the same m-line

Proposal: all codec parameters are the same, "content-label" is the same, are in same MediaStream

(hta, fluffy, martin debate the details)

next: How does createOffer know to offer receiveOnly flow?

want to receive video but don't have video camera

justin has a proposal, mind to write it down here?

matthew: can you put "send" in SDP before getting user consent?

or do you first have to use receiveOnly and add sendReceive on a separate O/A

ekr: how do we correlate multiple offered video m-lines to the multiple video streams the answerer has

next topic: how does createOffer decide to offer a data channel?

should OfferToExchangeData constraint be added?

matthew: data is a great idea, but SCTP is horrible.

fluffy:take this to the IETF

tim:SCTP was decided in Feb

<ekr> matthew: how do you really feel about SCTP?

DanD:data is easy within a single app, but trapezoid between two apps is more difficult

martin:issues could come also if the other device running the "same" app has constraints

consensus: don't add this constraint

next: DTMF

will be discussed tomorrow with a proposal

next: How long is SDP from createOffer/Answer valid?

matthew:90 seconds would be an ultimate timeout

use case: the SDP is sent to the server for modification

should it be valid beyond the duration of the callback function

ice candidates etc. time out in matter of 10s of seconds

hta: time-to-live for the session description?

matthew:can createOffer be called again after getting the modified SDP back from the server?

matthew:proposes that createAnswer is valid only for the duration of callback and no longer

consensus: it must be valid at least for the duration of the callback function

ACTION: ekr to take follow-up to the list

slides about rollback and error handling left for now

(now lunch break until 1:30)

Implementation status

<fluffy> Cullen is scribing

HTA is presenting what chrome is doing

TURN and opus are scheduled to be in M24 all going well

DTMF is waiting on this group

Anant has a demo of firefox

THe demo allows login, gives a list of users, then you can call one of the other users

Have getusermedia, have peerConnection,

expectation to not have behind a flag in firefox 19

currently in firefox 18 in nightly builds behind a flag

have a fairly complete version of the DataChannel

showed cool file sharing with drag and drop using DataChannel


ICE but no TURN

prototype of Identity working

Doing desktop first, then working on mobile

VP8, opus, and G.711 as codecs

HTA: There was a test web even 2 days ago

dom: The idea of these is to get lots people to develop and contribute test cases

… presentation of theory of testing

… Event led to 404 test cases

General error handling principles

ScribeNick: JimBarnett

Anant's slides on error handling

Anant: exceptions vs error callbacks. exceptions when you can detect error without blocking the main thread. Covers only very simple errors. In all other cases, error callback. So as policy, favor error callbacks because they don't block the main thread.

Cullen: so lots of functions will do both?

Anant: yes

EKR: there's no way to get around exceptions occasionally.

Tim: what do you do if you pass in something that is not a function as the error callback?

Cullen: there are things that you should never see in production code, and those that have to be caught at runtime. The former are programming errors. We're interested in input errors.

Anant: what goes in spec is a list of things that must be exceptions and what should be callbacks. This will be the result of consensus of browser implementations.

Justin: so programming errors will be exceptions, bad SDP from the other side will be an error callback.

Anant: exceptions will be only for development. They shouldn't occur in deployments at runtime. In exceptions should include name and message, which all platforms support We should standardize the names of exceptions, message can be platform-specific, thought should be human readable. stack and linenumber are useful where available. We will inherit from Error object to make SDPErrorObject.

ekr: we should create new attribute for SDP error, rather than overloading linenumber.

Anant: yes, having different names helps. We should standardize on the ones that need to

be machine readable. Human-readable ones can be platform-specific.

Dom: W3C policy is to re-use existing names as much as possible.

Anant: we will use same object for error callback and exception. So name and message must be present in object passed to error callback. As with exceptions, human-readable properties can be platform-specific. For SDP errors may want to create a new property. Error callbacks should be mandatory. In current spec, they're optional. That makes it easy to make sloppy errors. If they're mandatory, at least you get an exception if you forget to define the error callback. It should never be the case that there's an error and nothing happens.

... In CreateOffer, exceptions: Invalid_callback, invalid_constraints, invalid_state.

Dom: these can be webIDL type mismatch errors. We don't need to specify them separately.

Anant: we can move invalid state to a callback (for cases where app violates the state machine that we define.) So we won't need to define any exceptions for (most?) functions. setLocal and setRemote can have invalid_sdp as error in callback. Assume that the success callback in set remote isn't called until the description has been fully applied.

Cullen: if have setRemote with provisional SDP and then later will apply final SDP. Consider the case where are parsing SDP and acting on it as you go along. Do you have to roll back? It may be hard to do that.

ekr: setRemote shouldn't generate callback until it's complete and in correct state.

Justin: we need a separate error to signal case that media system is hopelessly busted.

Cullen: we need two kinds of errors. Implementation will know whether the situation is

hopeless or not.

Harald: Invald SDP indicates that you have been able to rollback, so it's not fatal.

Cullen: when SDP fails a syntax check and no state has been changed, as opposed to case where what you thought was a camera turns out to be a mouse.

ekr: there are cases where the error is reversible, but you're still screwed.

Harald: let's put Anant's proposal into the spec, and work out some of these details later.

Anant: we can leave the decision on which errors are fatal to the UA. Further discussions of specific error cases on the list.

Dan: are error callbacks mandatory?

Dom: sloppy programmers can always put in a no-op error callback. On the other hand, if we make them mandatory, we will have to include them in our examples.


So decided that invalid_state will be a callback no exception

<dom> Error Types defined in DOM 4

Decided that for SDP, will add a sdpLineNumber

decided all errors callbacks will not be optional

Call flows Walk-through

Justin: when you call setLocal that's when UA starts gathering ICE candidates. Get callbacks as each candidate is gathered, another when all are gathered. If call createOffer before any candidates are gathered, it will have just your local address.

Justin: if you want to do trickle, have to get offer first and then gather candidates. So must get getOffer callback before all candidates are gathered.

Cullen: what things can cause gathering to begin? What indicates how many candidates to gather? How do you know when you're done.

Matthew: it has to be a settable parameter whether createOffer callback can fire before you have enough candidates to produce valid SDP. I don't like to get a success callback when the offer is not valid yet.

Cullen: one model is you can decide to wait until SDP is complete. Or we can say that you get callback immediately, then get another when you have all the candidates.

Harald: must have ICE candidates to be valid.

Cullen: but local address suffices for validity.

Justin: in Jingle can send offer without any candidates and it's fine.

ekr: it won't work with SIP.

Adam: we should have matching examples and callflows, including trickle.

Stefan: the Chrome implementation seems to work, with and without trickle. So it should be our reference.

Justin: we discussed calling createOffer and setLocal early to start allocation, and then add streams later

Cullen: we don't have mechanism for declaring dummy streams. What if had new method: gather n candidates.

Justin: that's like setLocal with empty m=lines.

Cullen: calling app has pretty good idea of how many candidates it will need. At least an upper bound.

ekr: two proposals for pre-allocation: - a direct instruction to peerconnection, or some sort of dummy SDP via createOffer and setLocal.

Cullen: we have 6 issues to discuss, all at least as big as this.

Harald: I'd like to get an overview of all the questions. for this issue, we have two questions: how to know when SDP is sendable, and how to do pre-allocation.

Matthew: what happens with resource reservation? What happens when you do all this gathering and find out that the camera isn't available any more?

Cullen: next issue. How does receiving side find out what's in the offer, so that it can show the user: Alice is calling and she wants audio and video.

Matthew: when you call setRemote, you have no idea how many times onaddstream will be called.

Cullen: general model is a callback for each structure, and a final callback saying that you won't get any more. questions: when do you know that you've gotten all mediastreams? When do you know you have all the tracks? When do the callbacks fire with respect to the setRemote callback?

ekr: at what point do side effects take effect w.r.t the callback? I think it must be before.

Justin: when does peerconnection.remotestreams get populated? When you get a stream, the tracks should be filled in already.

Matthew: parse them into the array, and then call processingcomplete.

Justin: how about a callback saying the remotestreams value has changed? (It would be an array value, so this callback would be called only once.)

Adam: it's more convenient to get the new stream in the callback, rather than having to parse the array to see what's different.

Justin: but it's important to know when the changes are complete, rather than getting changes one at a time.

State machines

Justin's slides

<inserted> ScribeNick: martin

juberti: [presenting on states]

martin: does back to gathering happen for moving from "relay" to "all"?

juberti: you probably already have all the necessary candidates

matthew: is there any way to remove candidates?

juberti: ice restart

ekr: can we have multiple components with one completed and the other one with no candidates

juberti: we are in checking until all components have resolved

fluffy: checking encompasses frozen

matthew: restart doesn't affect active flows

juberti: yes

matthew: why not just describe each of the states as they relate to the underlying states

fluffy: this is part of what I prepared for the last call on this topic and we rejected that

hta: there is no harm in playing audio while video is failing and restarting
... is there any difference betwen connected and completed?

juberti: middleboxes might require the updated offer that would be triggered from the completed transition

martin: is this application driven or not, can the browser add new candidates and contine?

juberti: restart is required by RFC 5245

fluffy: the application is going to need to be involved

juberti: if you get a new NIC (e.g. WiFi) you might just trickle that

hta: transitions to starting are tied to user actions

martin: how does the new WiFi candidate fit into this?

juberti: that would trigger a transition to connected

fluffy: there is an implication that disconnected might transition to failed, in the case where you were connected and you disconnect then something failed

juberti: proposes changing name of "starting" to "noo"

<hta> IceConnectState -> IceConnectionState

juberti: remove onicegatheringchange, and provide just onicecandidate to fire when the gathering state is changed

derf: propose to rename iceConnectState to iceConnectionState or something

* missed the name conclusion

ekr: is this same as the other state machine, just with the states merged?

juberti: yes

<hta> martin, name conclusion was to use IceConnectedState rather than IceConnectState (I think - my ears are going)

matthew: set...(answer) can't be done twice?
... why isn't createOffer and createAnswer shown on the diagram?

juberti: once you set...(answer), you may have removed some critical state for the offer, which invalidates some of the answers

[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.137 (CVS log)
$Date: 2012/11/15 12:10:30 $