WebRTC TPAC 2015 F2F Day 2

Friday 30 Oct 2015


See also: IRC log


Dominique Hazaël-Massieux, Shijun Sun, Peter Thatcher, Varun Singh, Eric Rescorla, Adam Roach, Martin Thomson, Cullen Jennings, Mathieu Hofman, Alexandre Gouaillard, Andrew Hutton, Vivien Lacourba, Daniel Burnett, Stefan Håkansson, Harald Alvestrand, Taylor Brandstetter, Anssi Kostiainen, Ningxin Hu, Chia-Hung Tai, Tzu-Hao Kuo
Attending remotely
Maire Reavy, Randel Jesup, Justin Uberti, Jan-Ivar Bruaroey, Erik Lagerway, Bernard Aboba, Robin Raymond, Andreas Pehrson
Harald Alvestrand, Stefan Håkansson
Dominique Hazaël-Massieux, Alexandre Gouaillard, Shijun Sun, Anssi Kostiainen, Daniel Burnett


[agenda bash]

Placement: Resolution of issues from Thursday's Transceiver preso

Transceiver Part 2 slides from Peter

Peter: coming back on a couple of new options for rollback
... Option I is a refinement of the F from yesterday

Peter: [description on slide 2]

Martin: there is 2 things that create a transceiver: adding a track and receiving a remote description
... when you do a rollback, you rollback only those created by the remote description

Martin: if you have one that was created by the combination of these 2 things, you only reset the one created by the remote side of things

ekr: the ordinary rollback behavior is to return the state you were before
... here, it's only half way backtracking

martin: we discussed this yesterday and had an option to rollback everything since the last stable state
... but that includes things that were done even before reaching stable state

cullen: when we said rolling back to 0 time, there are different states that need to be rolled back
... some states are linked to the offer/answer, others aren't
... rollback should be only about the former
... things that haven't been applied to the offer/answer state machine shouldn't be impacted

martin: the problem of doing otherwise is that a rollback rewrites history even further back to stable state

ekr: with such an approach, do we actually need rollback?

martin: we need it, and we implement it that way

cullen: [missed comment about pranswer]

martin: we're unlikely to implement pranswer

peter: we have it now, but we haven't implemented transceiver yet

cullen: I think we're at the point where we need to discuss whether to remove it

peter: I'd love to remove it

cullen: if we're going to reopen these discussions, there are other decisions that would need to be revisited

peter: that probably needs to be discussed separately in any case
... presenting the 2nd option, option J
... which is forbidding rollback after setting description and calling addTrack after

ekr: the main reason of rollback is that sometimes you need to apply a description before you can decide whether it's good for you

hta: we discussed other cases yesterday beyond that
... if we don't support rollback before or after addtrack (a "no return" point), it means that some error cases will have to be handled by tearing down the connection

ekr: which error cases?

hta: setRD is async, so things can happen between the call and the promise resolution that makes you think "I shouldn't have done that"

ekr: the Transceiver is created at the moment of the promise resolution of setRD

hta: the question is whether you can get in trouble between addTrack and setLD

ekr: I'd like to see a concrete set of events

juberti: if there is no compelling use case, I think option J is nice and simple
... we could always come back to this if there are use cases that require a more complex approach

ekr: do setRD, make sanity checks, do addTrack

martin: but sanity checks can be long and thus addTrack might need to be queued

cullen: the simplest scenario is: I send an offer, I add a track, the answerer doesn't understand my offer for any reason, and we need to rollback

ekr: but that's for setLD, not RD

juberti: I think for RD, you'll want to do your sanity checks before addTrack

hta: option J is looking good because it leaves us room for adding more stuff later if needed

cullen: this is deprecating rollback in a bunch of use cases

adamR: but we haven't identified any such use case

ekr: in the trivial case where I have an outsdanding offer with voice and I add video, and the audio call gets accepted, I'm now in onnegotiationneeded because it doesn't match the negotiation state
... the question is what should I do if the offer is rejected?
... how would you rollback?
... [describing an example on the whiteboard]
... A creates a datachannel, we exchange an offer/answer and get back to stable state
... then I add track audio, send an offer
... and then I add an video track
... if instead of an answer, I'm getting a rollback, it would be have to get to negotiationneeded state
... and this is addressed by option J
... but the other case of stuff happening after setRD and addTrack doesn't seem very plausible
... but even if the plausible case I described, it's not even clear how you can react
... stepping back a little, I'm not even sure what rollback is useful for

cullen: adding/removing rejected streams is easy, modifying them is much harder

ekr: but recovering from a stream change will take a lot of logic

martin: the remote rollback case is harder to make

ekr: so we have a complex system to signal situations that are complex to recover from
... not sure how useful that is

juberti: rollback is a way to reject things that we immediately realize we don't want
... it's not clear that there is a value to use it for things lower down in the chain

ekr: one way of dealing with avoiding loops of negotiations is to freeze the latest stable session in a super-stable / immutable state

peter: we need to distinguish remote and local rollback

cullen: I think we all agree we need local rollback

martin: local is (mostly) useful for apply and reject
... option J will be enough if we deal with activateSender and look at other state changes (at least removeTrack)

cullen: the only thing that rollback affects is what is part of the negotiation

ekr: jesus what a mess

adamR: either the app is in charge of keeping track of state changes, or the browser gives affordance to it

martin: option I was for the latter

juberti: but option J gives us more freedom to evolve on this

martin: it's a cop-out
... option I was rolling back just the receivers

hta: my suggestion is let's adopt J, and someone wanting to propose something else will need to provide a use case

martin: I don't think only handling addTrack is enough

peter: any other modification (e.g. activate or remove) is not surprising

cullen: another way to put it is: if you modify the transceiver created by the remote description and do a rollback, you'll get surprises

peter: the only thing that can change a transceiver is addTrack

martin: not if I addTrack before setRD, and then do a removeTrack

peter: this would not connect the two transceivers

martin: I think JSEP say they would
... when you create an answer, you go through the m-lines in the offer and find one that matches your track and reuse that transceiver

ekr: it would be very surprising otherwise

peter: not to me

juberti: it would not be that surprising
... it's not clear that the browser can make useful matching up based on track types

martin: so you're backtracking on track matching in JSEP

juberti: we can move stuff from JSEP to the application layer
... JSEP has some very complicated logic that is not necessarily full enumerated

cullen: this matters a lot of legacy interop

adamR: this is revisiting multiple years of history and discussion

<mt> http://rtcweb-wg.github.io/jsep/#rfc.section.5.3.1

juberti: I can tell you from experience that the matching up algorithm is extremely complex and the browser is not always in a good position to know what was meant

<mt> paragraph 5, http://rtcweb-wg.github.io/jsep/#rfc.section.5.3.1

juberti: we can maybe simply consider peter's case for the matching

martin: I suggest we simply stick to JSEP's rules
... but that argues that option I is a better match to the JSEP model

hta: so are we converging on getting a pull request for Option I
... Peter will provide a pull request, and martin will review/contribute

<vivien> Slide 4 Can we change MIDs on an m-line

Peter: looking at the other issue we discussed: changing MIDs on an m-line
... Justin said making mids immutable would make JSEP much simpler

cullen: this sounds good, but we have to deal with the mid exposed in candidates
... we should have a note to that effect

<juberti> Candidates should get GCed when m= section is cleared

peter: this has only effect on JSEP

juberti: I think JSEP probably needs to understand the concept of transceiver to do the matching to SDP

Changing dictionaries to interfaces

RTCRtpSender support objects -> interfaces #350

martin: we have a bunch of dictionaries returned by getParameters, and setParameters accept these dictionaries
... when looking at setCodecPreferences, having to deal with dictionaries being wrong vs requiring an interface, the latter is much better
... since interfaces guarantees that the object is built by the browser, you get much better control
... in general, domain objects are recommended to be built from interfaces rather than dictionaries to avoid this type of issume

(now looking at file modifications)

martin: in my pull request 350, I distinguished between dictionaries that are under control of the application, and interfaces that the browser need to control to restrict input from the app
... going through the various existing usage in getParameters / setParameters, most were cases where the browser should not leave access to the developer
... going through this, it wasn't clear whether making ssrc changeable is a good idea
... it was important beforehand, not sure it is useful

peter: I meant to be mark it as readonly
... there are use cases where it would be useful, but there was no consensus to do so

martin: I only focused on setParameters, but this probably affects other parts of the spec, especially stats

jan-ivar: we should not make too much generalization either; the case for setParameters is clear because you're getting objects from the browser before resending it to it
... it's not clear that the same applies to stats

dom: I think the case is quite clear for get/setParameters
... we would have to look at a proposal to evaluate stats

hta: speaking as stats editor, it's important to have serialization — but we can get that for interfaces too
... I wonder how that would play with ids

<Bernard> The question is whether interfaces will paint us into a corner - where the entire API will be unextendable.

<Bernard> These changes are being made based on the assumption that attributes that are readonly in exisrting objects will remain that way... or that this will be consistent between objects. That is not true in general.

<Bernard> Capability objects and Parameter objects are the similar or in some cases identical... dictionaries can serve both purposes but interfaces with restrictions cannot.

<Bernard> So effectively imterfaces are poisoning the object space in a way that will require massive changes later on... for no useful purpose.

shijun: dictionaries can vary, where interfaces can't

martin: if there are capabilities that browsers can or can't do, we should change the interfaces accordingly

varun: but for stats, there is data the browser may not be able to provide at some point

<Bernard> But the interface can't do multipile things at once....

<Bernard> The same attribute cannot be readonly and readwrite, for example....

shijun: I think we need to think more about this; at least the comparison with stats gives me pause

<Bernard> Yet that is exactly what is required in many cases if the same dictonary is used differently in different methods.

martin: I agree that the timing aspect of stats makes a good argument as to why to use dictionaries instead of interfaces

peter: if later one we decide to make ssrc writable?

martin: ... we add it to the dictionary

peter: so it would be fairly simple?

<Bernard> You'd have to change the attribute from readonly to readwrite - whereas with a dictionary you would not have to make that kind of change.

martin: yes; there are more complicated examples, e.g. when we want to change a sequence of interfaces to something else

jan-ivar: you could deal with that by making the attribute writable instead of readable (and still need to use setParameters)

<Bernard> There are cases where attributes are readonly within some methods and writeable in others. Interfaces cannot deal with this.

<Bernard> Yes.

dom: so this PR makes choices about things we would need to evolve later — are we comfortable with those?

martin: from my reading, the only constraints it creates are on things that are firmly under the browser control

peter: looking at the example, does this change requires this "copy(params)" call?

martin: yes

[discussion as to why the getParameters returns an immutable object]

martin: my assumption had been that setParameters only needed partial application

ekr: I'm still not sure what's the point of having the readonly fields
... I'm confused

hta: I think it's clear that is no consensus on #350

martin: I think we're close to making progress

hta: I don't understand the claim that getting an object from an interface with non read-only parameters, you could not change them

martin: the issue is that there are expectations that tweaking values from a platform objects should have side effects
... in my model getParameters returns the browser view of its state
... and then you have to provide your own view

ekr: but how does this help?

martin: I'm open to adopting the model of a modifiable platform object for setParameters

[consensus this would be better]

martin: I'll revise the pull request accordingly

Bug Bash for WebRTC-1.0

Bug Bash for WebRTC-1.0 slides

harald: bug status
... there are still 52 bugs, (152 were closed)
... and the oldest bug is around 1 year old
... if you look at the categories of bugs
... you will find many (see slide) but we want to fix all of them, except the functionality wishes
... of course, we might not deal with all of them today .....
... as we did for media capture and stream, we first remove all the editorial stuff
... first bug: #150 and #151: closing a peerconnection
... there is a question regarding what is the best practice for promises? return error / failing, or not?

justin: I thought we had already decided that we would fail them.

jesup: there were a couple a question regarding what happen after you close the peer connection
... we don t really care anymore after a close. because anything produced would not be usable.

fluffy: i do not say it s right, but right now we are collecting errors, and making statistics on them, so I am in favor of failing it.

martin: but you re calling close first, so you re biasing your stats

fluffy: right, i m just describing what we do right now.

mathieu: if we have an error, the application then get the option to deal with it or not.

ekr: can we make the promises cancelable?

martin: there are only two states right now, not a third cancelable state.

harald: our options are either fail or not fail. no cancel option

ekr: ok

martin: that s my rea

jan-ivar: the general consensus is that in cases where you shutdown or refresh, you don t want JS activity

justin: I think it s a different use case. I think failing would be nicer, if you want to do things like UI cleaning.

jib: <more comments on the close case>

ekr: if we don t fail, there is no option at all for the app. It seems to be a pretty strong argument in favor of resolving the promise. Of course, app can screw up, but hey, c est la vie. Plus it s already in the specs.

adam: bad user code is bad user code, we can t control it.

harald: plus PC will be closed at that time, so they can t shoot themselves in the foot anymore, really.

<juberti> +1 to leave it

jib: it looks like there is no case where close would be called by the browser. so I m ok with resolving the promise.

harald: show of hand?

all in favor or abstening

harald: ok , next problem
... #337 and #338: webrtc vs JSEP
... the suggested guideline are: if it s about what the SDP looks like, defer to JSEP, if it s about the JS API, write it in webrtc

harald: specific issues: how to reference JSEP in webrtc specs, how to reach agreement between teams, and what are the permitted changed to SDP in JS (between CreateOffer and SetLocalDescription)

peter: there is a sync issue, and sometimes we need to reference things that are not there yet.

fluffy: yes, like any sync problem, we'll deal with it. Eventually we will converge, it's ok. I'm happy to do whatever is needed here [as chair of IETF rtcweb group]

ekr: is there any reason why we can cross reference symbols between webrtc and JSEP?
... I'm happy to help

dom: i'm happy to help too

fluffy: i don think we need to work too much on JSEP => WEBRTC, those are rarely needed

dom: one of the difficulty when i started to review the spec, there is the question of maintenaining referencing, but there are also implicit references
... like handling ICE candidates

fluffy & dom: a scrum is in order

harald: dom and ekr are in charge of dealing with the leakage

harald: last bug
#334 consider using the stream API
back when we made the original decision, DC were modeled after web sockets,
however, streams API was not ready.

denis: revisiting it now, it seems that streams API is still quite not ready
... proposing to leave it for later (close it now, mark it "LATER")
... everybody OK with that?

all: OK

harald: next

harald: #257 ICE candidate should have accessors

harald: solved by <class name>. Closed
... #263 Overriding default ICE servers

<different discussions about it being about turn, return, ICE servers ......>

ekr: let's separate the use cases. Here we are speaking about ICE servers, which are provided by the app. return, which behaves more like a proxy, that the application do not have access to today, is different.

mathieu: I'm not ok with browser overriding ICE servers (STUN, turn) provided by the app.

fluffy: I think this bug was about to decide wether JS can decide to let the browser get control over ICE servers.
... wether it is overriding one list (app) by another (browser), or merge both lists.
... but this bug is only about overriding.

dom: what downside is there to use user provided server?

justin: if you use non-app-provided ICE server, you might bust th app.

adam: we can propose default servers, but make it explicit to the app with a GiveMeTheDefaultServersYouKnowAboutPlease() api available.

harald: here is a matrix of all the possible cases.
... <drawing on the board>

justin: so where are we with this?

fluffy: summarizing: return and proxy are not available to the app
... however in this bug, we propose an API that allows the app to query the browser for known ICE servers, and do whatever they want with it.

ekr: in terms of security, this is the same surface.

harald: so ... we do not override?

mathieu: yes, do not override.

dan: do not AUTOMATICALLY, silently, override.

harald: so we all agree on this?

martin: I have a pull request :-D

Harald: I feel ekr and martin are in favor ... ?

ekr: i m in favor of everything that close this conversation faster.

fluffy: it s something we wanted to do for a long time, we just never came around to decide HOW to do it

justin: I disagree.

<dom> [to clarify for my later self, the reason having this as a list of STUN servers rather than a a-priori-less-fingerprintable boolean "use configured stun" is that these STUN servers can be trivially exposed via stats after ICE gathering, without any user involvement]

harald: so we rejected the idea of letting the browser to inject ICE servers just because they feel like it.

adam: the other use case is return, and we do not need a JS API for that.

fluffy: agreed.

justin: in the case where you have multiple level of NAT, it s not clear people can handle it with ICE servers.

fluffy: someone report yesterday that the amount of <...> is pretty low

justin: i said that, right

ekr: you are misunderstanding return. It does not involve a normal TURN server, and in spirit it should be seen as a proxy.

fluffy: I mean this could work without return.

martin: the pull request is done, we could merge this right now.

harald: which interface shall we put it on

martin: PeerConnection (and it s static, which will also been fun to implement, but hey, my happy life)

harald: everybody satisfied with this?

fluffy: yes

justin: yes

ekr: fine

haralrd: ok, merged
... #265 add privacy/security review questions
... need volunteer for review and text proposals

ekr: is the rule obvious from this document, or will need further input from IETF docs?

harald: obvious.

ekr: hum, oh, uh ..... I guess I can do it .......

fluffy: why? it's an insane amount of work!

ekr: well, really? well, .. i'm not going to do it then :D

fluffy: the AI for you should be: read the document and decide if you want to do it.

ekr: agreed.

fluffy: we should close that bug anyway

alex: ekr reads it, decides, and if he decides NOT to do it, we reopen it.

ekr, fluffy, harald: OK

<adjourned, 1 hour for lunch>

<resuming the bug list>

harald: #305 (7 more bugs to go)
... Describes what happen when media changes
... this is a remote variant of the "variable bitrate camera" discussion.
... clarification: is the problem a sender side problem (MSTrack violating constraints in the PC) or a receiver side problem (incoming data violating constraints set on the remote track) or both ?

mathieu: what is a constraint on the remote track ?

harald: nothing in the spec prevents you from applying constraint on the remote track.

mathieu: if you apply constraints on a track, local or remote, the browser might do some processing to make it fit the contsraints
... and I believe FF implemented ApplyConstraint (jib)

jib: currently we only support ApplyConstraint for local tracks

stefan: Spec says that you cannot act on constraints applied to remote tracks. Or that browser should ignore it....

mathieu: app can signal it to the remote side.

shijun: that s a protocol level thing

adam: at the application level, you do what you want, the specs only deals with local constraints.

harald: martin, can you elaborate on the bug

martin: sure, constraints at the session level, might be not overlapping with constraints on the track.

fluffy, ekr: it is already defined in JSEP.

matrin: right, I m just asking someone to explicit this case [in the webrtc spec]

ving: can you repeat the original question

matrin: you have (in the sdp) constraints both at t he session level and at the track level.

martin: the intersection might or might not be empty.

<ekr> https://tools.ietf.org/html/draft-ietf-rtcweb-jsep-12#section-3.5.2

martin: and each call to apply constraint can create a problem

ekr: i just pasted the corresponding text from JSEP. I think it should be in our spec as well.

harald: so the scaling policy is documented in JSEP.

ekr: that s correct, but we need to make a PR, because there are blank spots in the JSEP spec still.

<discussions about scaling up policies, and how to deal with errors>

harald: it seems that the text in JSEP is the right thing to import. Do you have a volunteer.

fluffy: I'll do it

harald: just import it from JSEP.

fluffy: wait, no, i don;t know where to put it!

<juberti> ...back...

<welcome back>

<discussion about wether we should handling scaling up on the Sender>

ekr: peter is saying, in an ORTC world, ....

peter: no, the RTPSender just sends whatever the size of the track is.

ekr: but for simulcast, you are forcing size in. ok, it s scaling down and not upscaling, but you still do not respect what the size of the track is.

fluffy: so where would you want to put that (peter)
... ?

<juberti> seems like a sender property

martin: what is fundamentally wrong with putting in on the sender?

peter: i prefer it on the transceiver. But i m not going to fight to death on it. Would your solution lead to renegotiation?

fluffy: no.

harald: sense of the room, and alternatives ?

<juberti> sender

harald: senders has it.

stefan: who has the AI to make the pull?

fluffy: I'll take it.

harald: #308 RTCRTPSender.get/setParameters is underspecified

<vivien> RTCRtpSender.get/setParameters is underspecified #308

peter: I can't think of anything that would go outside the negotiated params.

adam, fluffy: we have several things that constraint the bitrate.

fluffy: if we can detect an error, please detect it.

martin: this is only one of the issue in this bug.
... on the setting of parameters, shall we use getParameters? Is it done as a promise resolve?
... I think that the other adam (B) has something for that, but .... I'm not sure we can close this one today.

<juberti> sorry!

harald: so setting the parameters is immediate, but the action triggered by the setting can take some time

dom: so if you set and get, you get back the values you've just set?

peter: if we really wan tot know if things were done, we need to return a promise, but ....

martin: so what is the conclusion? The parameters are set right away, but taken into account only at a future (undefined) time?

xxxx: yes

peter: the question remains if we want a promise to be resolved at that undefined time.

martin: codec setting example

dan: so what happen if you make a set, and then a get, before the set finished?

adam: you get whatever is there at the get time, even if there is a pending operation.

martin, ekr: then you lost the ability to get the old parameters.

peter: that seems for a lot of complexity, while we have only one use case.

adam: the other problem is that it is an expending set.

peter: we are close to 1.0 ...
... the set can't expand forever.

ekr: do we reject at set time? Why don t we make it async, and that s it.

harald: ok, set return a promise.

martin: i m not in favor of having another queue. I don t want get to return a promise.

harald: who is taking responsibility for this?

jib: note: you can t throw and return a promise

martin: i think we should ask Byron to identify what we missed.

peter: and also that we really do not have anything going out.

martin: I did answer that.

adam: I'll take care of the PR

harald: ok, next slide.
... #340 'RTCPriorityType not aligned with names in rtcweb-transport

fluffy: so we re going with the tsvwg naming, right?

harald: yes.
... next #341, [FIPS-180-3] has been replaced by [FIPS-180-4]
... next #343

RTCDegradationPrefence vs RTCQualityPreference

peter: thought we decided on Degradation

harald: ok, peter, you ll take it
... next #345 #346 - create DataChannel param Error

shall we return n InvalidValue?

scribe: ok, next
... #358 Remove mid from sender receiver

peter: we talked about it yesterday, and then i realized it s easy, and there is no reason not to

harald: ok, done. next. last slide.
... and that s the only thing i m actually sure about, this is really the last slide.

<back to the agenda>

ekr: agenda proposal: focus on 1.0, keep NV for later if needed.

fluffy: proposing to put simulcast on the list since we made progress on it yesterday.

martin: yeah, most of the things we solved in the last hour we would have never solved on the mailing list.

<jesup> It I heard someone reference me, couldn't hear what was said. "lot of people happy..."

<vivien> Adding an accessor for the browser-configured ICE servers #365

<vivien> ^^ the issue that was just discussed

<dong_Cm> shalll we still have the section 'Introduction to ORTC', today?

<dom> dong_Cm, it's not clear yet — it depends on how much progress on the 3 additional topics we manage to make

<vivien> m-line match up doesn't work with RtpSender.mid or RtpTransceiver.mid #363

Peter: on issue #363

Fluffy: should set the mid after offer/answer

Peter: if transceiver has no mid, how to match with the m-line?
... addTrack can lead to new transceiver or choose the existing transceiver if mid match

MT: can check the transceivers and choose one without using mid

Peter: offer created by the transceiver reflects what is created already, so the mid should be set already when ceateOffer

MT: think #3 is okay

Peter: no need to create the sender when addTrack, the transceiver can then be replaced
... #4, remove mid and use a new transCeiver with the same sender
... mid uniquely identified the transCeiver

Fluffy: why #2 won't work?

Peter: not deterministic, so will need more rules

Adam: based on the order the tranCeivers are created and the type of the transceiver (audio or video)

Fluffy: follow the JSEP definition, based on the order of the transceiver creation

Peter: rollback, mid get reset?
... need time to think through the jsep matching rule, jsep has no transceiver concept yet
... mid proposed at create offer, and set/final at setlocal/remote.

MT: need to think through all corner cases

Peter: take the PR

Stefan and Fluffy added notes to the issue

Harald: PR #317 - data channel in workers

<vivien> Make RTCDataChannels Transferable #317

<jesup> :-)

Harald: marked as "later" already

Adam: easy to implement, easy to be compliant

Dom: plan to implement in FF?

Adam: yes even if not in the spec

Dom: that's what I wanted to know

Dom: worry about security aspect

Ekr: not think through yet

Stefan: assign to Ekr for security review

Dom: would it be in 1.0 if there is no security issue?

<jesup> This is a significant ask by DataCHannel users (see list posting by Faaborg(sp), and games users)

looks like the depedency on the security review

<jesup> "significant benefit to "

<jesup> ok

<jesup> thanks

<vivien> [Moving back to Simulcast discussion]

Fluffy: would like to discuss simulcast issues, and decide what to include in 1.0, e.g. framerate, maxbitrate, framesize, ...
... should start with framerate and bitrate

<vivien> RTP Payload Format Constraints draft-pthatcher-mmusic-rid-02

Justin: introduce frameratescale

Adam: or use an upper bound

Peter: framerate scale is better, a lot of cameras changes framerates

Justin: how this interact with the degradationperference?

<dom> [we're looking at the smallest set of properties for WebRTC 1.0, not dealing with all possible corner cases]

Bernard: for max framerate for each track, you can use separate clones

Dom: we are looking at the minimm set of encoding params for simulcast in 1.0. We don't need to look at all params at the point

Varun: smulcast should have streams with different properties, sender should choose which ones to send

Adam: adding a control to framerate is a good thing? then we can talk about what to put there

<jesup> framerate control makes sense

<vivien> +1 to jesup

<vr000m> +1 as well

Harald: if not converge, we should leave it out

<jesup> I generally happy with simple solutions, and if there are less-than-great corner cases, that's ok. In some cases, the app can push us out of them by noticing we're there from stats. In other cases we may have live with a corner case. I'm'm ok with that.

<vivien> from what I heard I think max-framerate is enough

Peter: should address the question from Justin on the degradation

<jesup> I'm pretty much ok with maxbitrate, and fill smaller layers first before higher layers. Send higher layer when there are "enough" bits to make sense to send it; up to the browser

Harald: Justin to help clarify the relationship with the degradationpreference, and will have discussion next week at IETF, Ekr and Justin will drive the decision

(coffee break, we resume at 16:00)

Media capture in Worker

Media capture in Worker slides presented by Tzu-Hao

Tzu-Hao: introducing mediacapture-worker
... the spec in in a GH repo: https://w3c.github.io/mediacapture-worker/
... the purpose of the proposal is to solve the following problem:
... no efficient way to do video processing on the Web Platform currently

[ showing a diagram of the current building blocks ]

Tzu-Hao: must use canvas element to modify, which is inefficient

Tzu-Hao: the problems: redundant canvas element, processing in the main thread, only pull-based mechanism

Martin: in FF, we implemented capture in canvas
... canvas.captureString API

<jesup> http://mozilla.github.io/webrtc-landing/canvas_demo.html and http://mozilla.github.io/webrtc-landing/canvas_filter_demo.html

Martin: the current situation is not pretty, I agree with the presenter
... we want a mechanism where we can quarantee frame processing
... so that a frame is processed only once
... the third problem is that all the processing happens in the main thread
... better have a mechanism to do video processing on the worker thread
... core concepts:
... MediaStream with Worker
... for each frame a browser can dispatch an event at the worker
... with design the problem is how a developer can access the video frame data from the event dispatched at the worker
... the proposal is that in the event there are two properties: inputBitmap and outputBitmap
... web developers can access the frame data through inputBitmap
... the spec extends the ImageBitmap interface
... we want to get rid of the hidden canvas and ImageData in the current processing pipeline

<jesup> can someone update us on what slide he's on as he goes? Thanks

Martin: only one copy of the real video source exists

ImageBitmapsFactories interface is extended

shijun: how can you quarantee you get every frame only once?
... how can you ensure you can process the data in realtime?

peterT: if the processing takes a really long time, e.g. face tracking, what happens?

Tzu-Hao: worker has a queue and if the queue is almost full the process will stop dispatch to the worker

Martin: the basic model is, you take video frames from the underlying track, and you have an object you mirror into worker, in which you can manipulate the frame data
... advantages of canvas is that you have access to all canvas APIs, so this means those optimization would not be available
... you could do things like take the image rendered on canvas and apply shaders there

mathieucitrix: you can draw to an offscreen canvas, and it stays in GPU

Tzu-Hao: use case: video editing
... also camera effects, stream manipulation, computer vision, depth
... demo

Cullen: how is this tied to the depth images

Related: Worklets, aka Processors aka Isolated Workers


mathieucitrix: this is very important to me, I want to help improve the spec

[ asking if there is someone who does *not* care about the use cases presented ]

[ sees no hands raised ]

cullen: using processors is compelling, but having an issue for having invisible workers

Tzu-Hao: discussing open issues
... that's a wrap

Media capture from DOM elements

Martin: capture from element, status:
... implemented in FF, close to complete

<jesup> http://mozilla.github.io/webrtc-landing/canvas_demo.html and http://mozilla.github.io/webrtc-landing/canvas_filter_demo.html are the demos

Martin: have one open issue, thanks to Paul that's resolved soon
... next steps: consider progressing to CR

[ showing slides ]

Martin: unless someone else implements, do not want to advance to CR
... not on a Chrome roadmap AFAIK
... Microsoft seems to have a plan to implement, not scheduled yet

Harald: any test cases?

Martin: no w-p-t tests
... a compelling use case: put together an MCU in Firefox

<vivien> Media Capture from DOM Element Editor's draft

Martin: Firefox Nightly has this, and FF 43-ish

<vivien> Google said in https://code.google.com/p/chromium/issues/detail?id=524218#c3 "We started discussing possible implementations here: https://goo.gl/EtsTBG"

Martin: merge one PR to get a spec ready to publish an updated WD

Harald: how do capture from canvas vs capture from <video> tag compare
... canvas is hugely useful for a variety of use cases

<vivien> Landed officially in Firefox 43 https://developer.mozilla.org/en-US/docs/Web/API/HTMLCanvasElement/captureStream#Browser_compatibility

Screen capture / screen sharing

[ presenting slides, screen sharing on slide 3]

<vivien> Media Capture Screen Sharing repository

Martin: last time we discussed the distinction between window and application
... implemented this in the spec and figured our the distinction is not much different from your window

Martin: REMOVE ALL constraints

<vivien> [ shijun give a thumb up ]

<vivien> [ discussion of the use case of enterprise products having an option that forbids sharing the whole screen, and a possible unsafe mode that would also include sharing the browser ]

cullen: requirement is if javascript can somehow indicate never to share unsafely

mt: others may not have the same policy as cisco
... open an issue on this

cullen: when i ask for permission what do i get?

mt: browser is responsible for presenting ux, and contains complete list of browser, monitor, etc.

adam: if we went finer-grained about what we don't want to share we couldn't knock out all apps that way

cullen: you could get most for most users
... so you want to ensure that user gets to control?

mt: virtually all the issues here are related to selection constraints by apps
... we plan to include browers in the list but don't have all the machinery yet

ekr: don't think filtering this list will help at all because the app will suggest the user allow everything
... is it okay to all of you that user has to do this selection?

peter: yes, but not sure about distinctien between apps and tabs

shijun: want to be able to filter out certain resoultion sizes or crazy framerates. want to keep resolution from surface (no up or down scaling)

mt: you don't want to do resoluttion constraint, just keep what is provided. if someone asks for a restriction, then the track itself will be downscaled.

cullen: on the list of windows you might not want to display small windows

mt: that's the browser's responsibility

peter: you don't want app to even downscale at all?

cullen: good use case for this is in classrooms where teacher is getting many videos from all the students to make sure they stay on task

shijun: any dsp processing would damage the output, just give us the raw original stream

mt: but that would change the way mst works

hta: nice thing about keeping to the mst deifiniton is that then any downstream process expecting an mst can use it

shijun: might not implement rigth away
... what about framerate

mt: won't offer a rate that browser can't handle

cullen: can you do less than 1 fps?

mt: yes, very important for some use cases

hta: i worry about taking away selection constraints because it makes the selection list too long unles the browser displays this intelligently

cullen: show me your UI. that will determine whether this is okay or not

mt: think we can do this well. have some mockups that are pretty good.

hta: other issues?

mt: permissions API adds a "screenshare" label with no extra parameters. Do not display allow state, only deny

shijun: what about notification?

mt: on screen indicators, easy revocation, if you need more raise an issue

cullen: if you capture app and it goes full screen, then what

mt: PPT case? Yes. Challenge was if you selected a window and then go full screen it generates a new window for that, so you lose it. You're sharing the old one, which might be gray. issue #29 is "powerpoint is special". we think the browser needs to just deal with this.

shijun: if you open PPT in full window it could even go to a different screen (projector) and that's what you want to get

mt: this may be simple to describe. monitor for new windows. if new window for app you're sharing is full screen that's it.

cullen: would also be nice in full screen sharing mode, sharing whole monitor, to be able to share the other monitor

mt: we have indicator for what is being shared. thinking about having that show a menu where you can select the source.

cullen: if you watch screen sharing failure, multi screen is the most interesting case

mt: may need some text on this in the doc

hta: what is implemenatiotn status?

mt: both ff and chrome have some sort of capability that doesn't match this API at all. No concrete plans to implement this. UX folks in particular thought this was complex, so still under discussion.

cullen: what about MSFT?

shijun: not this year. Next year I don't know (laughs). This is uunder my domain.

peter: I'd have to go ask at Google. New owner for screencasting and will have to ask.
... will ask what is the plan for a non-extension and how does the picker include all possibiliites and still remain usable

mt: may want to do frequency sorting for list of apps so the most likely shows up at the top of the list

adam: yes, we have to deal with many many options

hta: main outstanding item seems to be UX design

[various jokes about listing PPT first ]

hta: time to adjourn?

cullen: one question: what to comumnicate to rtcweb chairs for next week's meeting?

hta: finish jsep!!!

cullen: authors, authors, authors

[and we are done!]

Summary of Action Items

[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.140 (CVS log)
$Date: 2015/11/08 18:31:37 $