<hta2> I don't see anyone on the testing webex - are you in the meeting webex?
<fluffy> stefan, are you on IM here ?
<dom> hta2, can you say something?
<fluffy> IM us back as we are not hearing you
<hta2> Sorry, not looking at all the screens all the time while waiting - did you get the sound check for me OK?
<inserted> ScribeNick: dom
<hta2> Sound from the meeting room is very low.
<lgrahl> Ouch!
<hta2> Sounds like you have too many devices with the loudspeaker on in the room.
<hta2> if someone can turn the camera to point at the speaker, that might be nice.
Huib: I work for Google; used to
work for Opera and have been working with Web technologies for
a long time
... have been working on finalizing the WebRTC technology and
we have been working with testing to that end
... making the Web work reliably is a key aspect of providing
APIs ot the Web
... WebRTC 1.0 has been a long road, with many specs both APIs
and protocols at W3C and IETF
... lots of efforts to get there
... Web Platform Tests is a great tool for testing Web
technologies, but it is not sufficient for WebRTC given the
complexity of WebRTC, and the number of underlying low level
protocols
... So Google took up the idea of Alex to work on a more
comprehensive system - KITE, an open source project we formally
announced last week
... Alex will tell us more about it
... Beyond spec and testing, the third part if compliance - all
browsers have their own challenges in that space
... We've made progress in Chrome in the past year - compliance
on getStats, constraints, RTCRtpReceiver
... The major upcoming milestone is unified plan, planned for
Q1 2018
... getting to 1.0 is critical to avoid getting too many
proprietary or native solutions deployed for WebRTC
... The major engineering effort in WebRTC in Chrome is around
reliability - which is much harder than spec compliance
... other browsers are probably in the same situation
... Soares will report on his work on WPT, sponsored by
Google
Soares: quick update on the
status of compliance testing in WPT
... We have ~1300 tests - added 1000 in the past few
months
... We have added a tool to calculate coverage for WebRTC
<hta2> Repeat: If it is possible to turn the camera towards the speaker, that would be good.
Soares: We extracted all the normative statements
<hta2> Thanks!
Soares: the WebRTC 1.0 spec is about 70% tested at this stage!
<lgrahl> Can you talk louder? Sound is cut off a lot.
Soares: Some of the challenges to
reach full coverage
... tests are currently written based on the June editors draft
- quite a few updates since then, so the tests are a bit out of
sync
... some race conditions make test difficult
... WebRTC describe steps that happen in parallel - sometimes
hard to trigger specific concurrent operations
DanBurnett: you noted 10% untestable assertions - what's your thinking on that?
Soares: the untestable ones are bound to the constraint of what is exposed in plain JavaScript
DanBurnett: if these assertions come from the spec, the spec may need to be changed to avoid those
Alex: Some errors can't be dealt
with at the JavaScript level - e.g. errors linked to hardware
error which can't be triggered
... re race conditions, dealing with timing and promise-based
algorithms is hard
Jan-Ivar: we may need to fix the spec if the timing is not deterministic
Alex: re untestable, we count
them as "tested" in the 70%
... regarding the new 97 steps added since June, many of them
come from error reported out of the testing effort
... in the remaining path toward full coverage, 90% of it is
easy; we expect to reach 100% in Q1 2018
... once we reach 100% for webrtc-pc
Soares: note that coverage
doesn't mean they are all correct - the tests need to be
verified by implementors and editors
... one possible topic is how to keep track of coverage over
time
... we're looking at automation with web driver &
permissions API
... there are other specs that need testing efforts: we are on
track on testing mediacapture-main next - other specs are
further down in the pipeline
[Slides: Real-Time Communication Testing Evolution with WebRTC]
Alex: beyond testing the
JavaScript compliance, there is a lot more that is needed to
properly test an RTC system
... JS API compliance is part of the standardization process -
but the tests are standalone tests, testing a single browser at
a time
... Until WebRTC, this was probably sufficient
... but for WebRTC, you need tests that test interop across
browsers
... you also need to test intersection with protocols and the
networking layer (e.g. ICE, NAT traversal)
... So beyond compliance testing (with automation, stand alone
tests, one browser), we need to move to interop testing (still
with automation, but on two synchronized instances of the test,
with 2 or more browsers)
<lgrahl> Sorry for nagging but sound is cut off a lot again. :/
Alex: For automating
interactions, Web Driver helps quite a bit
... But we need improvements there - e.g. extensions to
integrate permissions management in Web Driver
... Once you've obtained permissions, you need ways to emulate
captured media
... browsers have already ad-hoc ways to do that, but we want a
standardized Web driver approach to it
... So Web Drivers help quite a bit, but it doesn't help when
it comes to run synchronized tests across browsers
... lots of tools hvae been developed to address part of the
problem - which confirms the problem exists and is
important
... how bad is it when you try testing interop across browsers
/ OS - leading to ~500 situations to test
... only very few of these situations had been automated so
far
... existing solutions have limited reach in terms of browser
versions and OS
... eg in the enterprise market, Windows 7 is still a key OS to
test for
... different browsers have different level of support for Web
Drivers, and even more so when taking into account the
permissions extension
... to avoid regression tests or ensure newer versions of
browsers are as compliant / interoperable as possible, bugs
need to be caught early in the release process (i.e. in
~nightly builds)
... depending on resources, your needs on
freedom/convenience/price vary quite a bit
... Taking all these dimensions into account, what you can
achieve with existing solutions is always limited
... Testing WebRTC feels like death by overwork, hence we
created this interop testing engine named "Karoshi Interop Test
Engine", aka KITE
... it is designed to be as generic as possible
... meant to work with any client, any platform, any
back-end
... we architected this by designing 4 independent components:
a selenium grid, a test engine, tests themselves, and a
reporting dashboard
... tests and dashboard aren't completely independent since the
dashboard needs to know what to expect from the tests
... Tests have to encapsulated into a Java class
... configuration files are used to describe which tests are
run in what browsers, on what selenium nodes
... we also record timing information to detect performance
issues
<lgrahl> Sound (sorry) :)
[demo of KITE dashboards and how they allow to look into details of runs]
scribe: This level of testing has
already allowed us to identify and report bugs
... we have plans to extend this to mobile browsers (will come
to open source project), to electron and other native platforms
(in the cosmo proprietary version)
... we will bring additions for SFU testing over time
... we are very open to all suggestions
... we want to make this tool as useful as possible - please
file issues on our github repo
Cullen: first, thanks, this is amazing
<hta2> the sound has become worse and worse over the period of the presentation, with mike cutting out randomly. Now Cullen's voice is much better than Dr. Alex' was.
Cullen: what are the next steps to expand this
Alex: Web Driver handling is key;
support in Edge is important to reach 100%
... Tests for IETF protocols are a big piece of the picture
<hta2> the new mike sounds better, at least at first.
Alex: a key topic for people is call quality
<hta2> but now it's doing the same thing again.
Alex: media quality is another
topic - but hard to measure automatically
... What I want is have a tool that moves beyond compliance to
quality, incl network and media resilience
... As far as standardization is involved, the most difficult
part will be around the number of protols
Cullen: the key is identifying key use cases - it's impossible to test all the situations covered by the RFCs
Huib: from Google side, we've
sponsored the development of KITE to enable the proper testing
of interop testing
... getting the important use cases exposed is key
... we want the baseline scenarios to be right
dom: how do you write an actual test case?
<lgrahl> The mic seems to work decent if the person talks loud.
Alex: we need to improve the documentation on this; this is upcoming in the next few weeks
dom: how hard/easy is it to run WPT tests in KITE?
Alex: we're looking into making it easier
<hta2> now the sound is completely gone.
Huib: but this not the main use case for KITE - the expectation is that this will run in the WPT dashboard
<lgrahl> Thanks dom for summarising - sadly I couldn't hear much. :)
[coffee break until 10:45]
[reviewing slides]
Adopt test as you commit policy
Philip: I work from Stockholm
Google office on testing
... WPT is big
... WebRTC test suite is in pretty big shape
... To keep the test suite in good shape, I have a proposal to
keep them in sync as the spec evolves
... We have an annotated version of the spec based on test
coverage rwaldron.github.io/webrtc-pc/
... the coverage of that spec is one of the better known of the
platform
... The WebRTC test suite is run daily on the WPT dashboard
wpt.fyi
... currently has lots of red, but most of it is linked to
automation of getUserMedia (or lack thereof)
... both for permission prompt and fake media generation
... so this should get much greener soon for Chrome and Firefox
soon
... we have a new piece of infrastructure to have tests to
drive themselves via Web Driver
... we plan to extend the Web Driver API to manage permission
prompt and to bring that to the WebRTC test case
... the WebRTC test suite is being imported into browser test
suites (at least Chromium, Gecko, Webkit)
... also test case updates get automatically upstreamed
... My proposal today relates to an approach groups have
started to adopt
... where a normative change to the spec comes with an
associated pull request on the test suite
<pthatcher> Sure...except my slides are next.
Philip: we hope to increase the production of tests based on that policy - this is already having a clear impact
<pthatcher> Also.. .when I do scribe, do I have to be as detailed as you? I don't know if I'm capable of typing that quickly.
Philip: we have passed 50% of specs who have adopted the policy
danburnett: there is a discussion
in the issue - e.g. what to do if you don't know how to write
test case
... also, we should understand if/when that policy applies -
for CR, it makes plenty of sense; for new work, there may be
too much change early on
Philip: the proposal here is to
do this for WebRTC 1.0 spec
... some groups have decided CR is a good time to enforce
this
... but clearly it can't wait until CR
Cullen: at a conceptual level,
sounds great
... the practicality of it is a bit worrisome - will it slow us
down in closing bugs? it will improve quality for sure
... but maybe before adopting it, we should play with it to see
how it works out
Philip: note that the policy is
not about adding a hard requirement so much as encouraging it
as a practice
... this may slow down the spec work, but would increase the
rate of interop
Youenn: fixing the platform is
more important than fixing the spec
... this simplifies implementation in browser vendors
<inserted> ScribeNick: pthatcher
Dom: how many people have experience writing tests cases, among people who make changes to the text?
Cullen: I don't
(a few hands saying they do?)
foolip: the person who wants it
merged is going to want to see the test added
... but the test can be written by someone else
dan: Idea for policy: for a PR to be merged, there must be a test. But the editors can make a judgement call to be able to merge without a test.
foolip: This is like tests in a code review. It's a cultural change.
Harald: feels safe to try
this;
... if we don't try it, we won't get there
jan-ivar: What's the granularity?
Harald: I'm just talking for volume testing purposes
foolip: ... Can't be adding huge
new things; so shouldn't need huge new tests for small
changes
... just small tests for small changes
dan: maybe we should wait until we have a certain test coverage threshold
Bernard: a call for consenus on issue 1617 to provide a test for changes unless you convince the editors it doesn't apply to you
lots of thumbs up
no thumbs down
RESOLUTION: We adopt a test as you commit policy as describe in issue 1617 - we may revisit if it proves too complex in practice
Noted: a consensus to move ahead, with Cullen's coveat that we can revisit this if we find it doesn't work well
foolip: I'll go write that PR
I can't scribe any more :)
<dom> ScribeNick: dom
RTCPriorityType combines relative bitrate with QoS priority, which applications may not want
PeterT: we discussed as an
interim before
... right now, you can't control separately priority and
QoS
Adding relativeBitrate parameter to RTCRtpEncodingParameters
Peter: the PR provides a more granular control on RtpEncodingParameters, splitting bitrate and QoS priorities, with a more flexible control on bitrate
Varun: relativeBitrate is that related to what?
PeterT: to other streams
Varun: so the top bitrate is one?
Cullen: there was confusion about rate being used and max
PeterT: same as currently, but with more control on the relative values
Harald: I worry about this
... what you're saying now is that if the congestion point is
at the point of having DSCP glitches, you won't change
anything
... e.g. if you want to prefer audio over video and get into
congestion, the relative bitrate will not get you what you want
(e.g. throwing video)
petert: if we focus first on splitting qos vs bitrate - what's your opinion?
hta: from a process perspective,
the IETF specs have already passed last call, so that's a
concern
... from a practical perspective, the risk is that we open up
many cases that make no sense for users that wouldn't do the
right thing
PeterT: there are 2 type of
users: those that don't care and those that do
... having one knob helps the former, having two helps the
latter
Harald: the relative bitrate control seems useless to me; but changing the spec of control we already agreed upon worries me
Bernard: exploring one of the cases that match this question: audio conf with low priority file transfer; but there is no reason to keep its bitrate low
harald: the current spec says that when there is congestion, the low priority @@@
stefan: we should have much more
data about actual usage nowadays
... I would like to see proof of need of this
... to get data on how this actually behaves
PeterT: we have an application
that wants to use this field
... but when trying to implement it, the fact that the two are
combined was blocking us
Varun: we came across this as well
PeterT: we want to turn on DSCP markings to experiment with them, but we can't do that if this is coupled with bitrate
Cullen: assuming we would agree to do this (i.e. it's not useless), what would we need to change in specs?
<foolip> who wants to merge https://github.com/w3c/webrtc-pc/pull/1657?
Peter: so imagine we have one 4 level knob for QoS, and a 4 level knob for bitrate
Cullen: I don't think such a change would have to impact the RTCWeb-transports draft
HTA: RTCWeb transports is where 1/2/4/8 is defined
PeterT: I'm happy with only doing the split of knobs - the relative bitrate thing is just a nice to have
<hta2> https://tools.ietf.org/html/draft-ietf-rtcweb-transports-17#section-4
Cullen: I'm not against it; I'm not sure I'm convinced yet
PeterT: so the model would be to keep the current knob, and add two "experts" one for separate control that would override the other
Cullen: splitting the knobs would impact the rtcweb-transports draft
PeterT: Let's look into this offline and come back to this tomorrow
Need for Initial Bitrate by the Application/RtpSender? #1635
Varun: the IETF congestion control wanted lots of info to help with initialization
Cullen: I think the IETF would have concerns about this being misused by attackers in JavaScript
Bernard: RMCAT could send a liaison statement if they want to push for the W3C WG to adopt this
PeterT: my recommendation was pushing this for post 1.0
Varun: that's fine by me - but I
want to emphasize that people are complaining about the lack of
current optimal starting situation
... I'll reach out to the RMCAT WG to let them know they should
a formal request
PeterT: but even if and when they do, it's not clear where we would put it
Bernard: we'll put that issue on the icebox for now
<fippo> fluffy: won't those security concerns apply to what is currently possible in chrome with sdp munging?
audioLevel of tracks, both send and receive? #1194
PeterT: this can be done with Web Audio
Cullen: but not when the stream is isolated
PeterT: recommendation is not adding this
Bernard: no objection
Adding more values to RTCIceTransportPolicy Enum #1644
PeterT: 2 different issue - one about network permission separate from getUserMedia, the other about giving more control for the app to constraint their routing
Varun: use cases including
routing data through or not VPN
... another thing is to enable a relay-first approach to help
with establishing a call quickly
Dom: so if we don't do this, what should we recommend to the OP?
PeterT: so for the first bit, we
need to look into the permissions thing
... for the second, give more justification / use case
Cullen: maybe a way to address the second thing is a way to expose from which route the origin was loaded
PeterT: it may make sense to close this issue and open one on data-channel only permission, and then maybe one discussion exposing more routing info for selecting ice candidates
behaviour of offerToReceive* set to false when there is a local track #1586
<fippo> i probably owe a PR here. Its a mess. (dom: note that so its on record :-))
JIB: this case would only happen in the case where one of the party is not a browser
RESOLUTION: we don't change how offerToReceive*: false behaves in the spec
Soares: discovered thing while
testing isolated media
... right now, the way the permission grant doesn't take into
account the fact that permission is granted not to an origin,
but to a peer
DanBurnett: this needs to be described in webrtc not getusermedia
Dom: but getusermedia probably
needs a hook in its permission check
... so I guess we do need to modify the specs
Soares: I can take a stab at it
Alex: who will review?
Dom: DanB+JIB for getUsermedia, MartinT & EKR for webrtc, …
Bernard: this is input to our
discussion tomorrow on WebRTC NV
... we're doing it today due to time constraints of the
presenter
Sergio: this is meant as a conversation trigger for WebRTC NV
[slide SVC Use cases]
[slide Content Adaptation (II)]
Cullen: decoding complexity seems
a bit different from the rest
... e.g. based on codec support
... e.g. it might be that stereo videos would use different
layers for the 2 eyes
Sergio: the most clear need is turning on/off temporal vs spatial scalability
PeterT: I think we have everything in RtpEncodingParameters, except for the spatial/temporal switches
Varun: there is also the question
of priority - which layer should switch to which priority
... not entirely sure if that level of control is needed
Cullen: I agree with these
requirements
... when it comes to image size, we care about the
thresholds
... we code to specific sizes for specific phones
... right now we work around this via stats
... but it would be more convenient to have direct control over
this
DanB: This all looks great; but
how doable is it? how standardized are the controls across SVC
codecs?
... every codec seems to be doing it a bit differently
... can we define a predictable outcome?
PeterT: I believe if we just add the high level switches for scalability (temporal and spatial), yes
Sergio: agreed; it would be hard to do low-level control because of this variability, but high level control seems very feasible
Bernard: there is an implicit
assumption in what PeterT said: that a decoder can always
decode what an encoder sends
... that's true for VP9, but it may not be true for all
current, let alone future codecs
PeterT: if you set the scalability bit, if the codec doesn't support it or if the recipient can't support, then you just use simulcast
DanB: that's what's at the heart of my comment - can this actually be implemented?
cullen: there aren't so many SVC codecs, and they've all been designed similarly - so yes, this should be doable
Bernard: AV1 has a similar
syntactic structure simliar to H264
... it may break the assumptions of VP8/VP9 of encoder //
decoder
<burn> scribenick: burn
[Varun presents slides]
<dom> WebRTC Stats Issues (30 atm)
Issue 99/PR 251: Security and privacy considerations
s;Issue 99/PR 251: Security and privacy considerations;Topic: Issue 99/PR 251. Security and privacy considerations;
Dom: for the security-impacting features in Stats that are beyond what webrtc-pc itself introduces, we will send the APIs to the TAG with a proposal
Bernard: "network layer data" is vague. I suggest describing "statistics related to packets lost/received, etc."
Varun: counters might be better
<dom> Cullen: some of the stats exposed here can fingerprint a specific network based on its characteristics, which can help determine two peers are on the same local network
Varun: regarding location access, we need feedback
Cullen: what can you do with this that you can't do with HTTPS?
Dom: by exposing rtt between peers, given the location of one you may be able to infer location of the other
HTA: can tell distance between clients, not just clients and servers
Dom: and these stats can be used without the user knowing it. Data channels alone with this could be used to triangulate.
Varun: and this is useful for CDNs.
Bernard: candidate pairs also give you RTTs.
Dom: we will expose the problems that exist and discuss possible mitigations
Next step would be to merge this PR (after Varun updates).
<dom> Discuss caching and consistency of getStats() return #177
Varun: PR 243 is related
JIB: PR says when you cannot cache
Dom: since this says For Example, the MUST should be lower case
<dom> Consistent marker for "non-active" object? #131
<dom> Added 'objectDeleted' attribute #262
JIB: if we keep around stale data, under what conditions would they go away?
<dom> https://github.com/w3c/webrtc-stats/issues/235
Varun: this came up before, but we haven't seen any problems with it. Issue 235
<dom> Is keeping stats around a memory problem? #235
Varun: we haven't seen it, but others might.
JIB: Why can't we just grab stats just before closing an object?
HTA: sometimes an object is destroyed by remote control, e.g. setRD that closes some of the connections.
Varun: also a third party might be monitoring and not be privy to the actions taken by the application.
JIB: what about a time limit?
HTA: for a long time now the
limit has been the life of the PC
... we could add an API for stats clearing.
Adam: what about an API to register a capture for anything you would care about?
Varun: if you are a third-party monitor this is too hard to make work
JIB: this is a slippery slope. Maybe we need something else in reality, like a session ended event. Let's solve the right problem.
HTA: we have event transitions in theory. Maybe in NV we can consider other options I will send by email.
JIB: FF doesn't work like this, so we want to make sure this is necessary before we do it.
Dom: does that mean issue 235 should be a CR blocker?
JIB: we will review and get back
to you.
... this is related to sender vs. receiver stats
Varun: back to 262, should we do it?
JIB: that's where we had concerns
Varun: if 262 depends on 235, which is in icebox, we have a problem
Dom: if either blocks CR we need to discuss.
JIB: yes, we need to review and propose an alternate
Dom: are we marking 235 as CR blocking and linking it to this issue?
(agreement)
and Jan-Ivar owes a response to it.
JIB: using replaceTrack will
replace the counters on the sender side but not on the receiver
side
... builds on next topic of Issue 230/PR 272.
... idea is to have counters on senders and receivers.
varun: not having track info there is a problem with replaceTrack because track stats go away in this approach. We need the stats to stay around.
JIB: this sounds CallStats specific. In general we have assumed that with a PC we don't get a done event for every action.
Varun: intercepting WebRTC APIs is problematic.
JIB: this is not an issue for
original JS app code, since their code causes the actions and
thus knows.
... instrumentation is an edge case, albeit an important
one.
... how often do we need to know the counts before and after a
replaceTrack?
Varun: switching front to back cameras, or screen capture, are very good examples.
JIB: getting tough to parse stats now, since you have to know which objects have been deleted.
Varun: we should separate the two
issues / the two PRs. Getting an event would be nice, but we
hear users asking to be able to call getStats at the end, after
objects have been ended.
... this has not been an issue so far because we haven't had
lots of tracks lying around. Hard to know how this will change
going forward.
JIB: the issue seems to be removing track stats, right? Is there a problem with adding the sender and receiver stats?
varun: not at all. It's the removal of the track stats that would be an issue.
JIB: okay, let's focus on the addition first.
varun: +1
JIB: I will update the PR
JIB: this is editorial only since the API is unaffected.
Varun: no objections. What would be required now if we get depth stats?
JIB: same as what we have. Might be a bit more prose.
Cullen: how does extensibility work?
Varun: easier with an enum.
JIB: depth is simpler because it inherits off the existing dictionaries
HTA: another extensibility piece to track.
JIB: any objections?
Varun: let's make sure this is JS observable.
(no objections)
Cullen: (asks about how it would look)
HTA: we have two alternatives because it's not really clear why someone would prefer one over the other
Cullen: maplike is easier to use, but a string could be parsed as well.
HTA: (missed comment on strings). With maplike you would have to iterate.
Dom: any privacy or security
issues?
... say, bleaching of JS?
Varun and Cullen: yes, but there's not much you know.
JIB: record is closer to a JS object so might be passed by reference, but don't know for sure.
(agreement this is useful)
(but not on which to choose)
JIB will review and comment
Bernard: why during audible portions?
Varun: if concealment is noise or content is important.
Cullen: there is an RFC that defines concealed.
Bernard: in XR block
Varun: but it doesn't say whether
it's in an audible section or not.
... packet loss, audible sample, or not are the three
factors
Bernard: RFC 7294
Varun: there are many issues like this that are not controversial but are lacking expert review.
Cullen: we were only supposed to
reference existing stats and not create new ones, so maybe
these need to be removed if they are unclear.
... we were only supposed to add stats that are clear to the
spec editor.
Dom: and this is not a CR blocker? why not?
Varun: virtually everything in
getStats is optional anyway. We may just provide a date cutoff
for something going in.
... as soon as all the CR blocking items are in we will move on
even without these iffy items going in
Dom: should warn people of what might go in/not go in.
Varun: conclusion is we need a citeable reference that all parties agree on.
Dom: Cullen is right that the experts are not likely in our group.
Cullen: this specific metric has
problems.
... the concealment algorithm introduced audio, but that's
about all. Get IETF to define the details if you really want
it. Concealed seconds is useful, but the high/low energy
distinction is of questionable use.
Varun: this is in a grey area where I don't see the value but that doesn't mean it isn't useful to someone else.
hta: we have lost samples that
occur in the middle of speech, so you might want to replay
rather than filling in.
... typical suggestion by someone but we can't decide whether
to refuse or find a better definition.
burn: the burden needs to be on the suggester.
Dom: let's ask for a more specific justification and reference.
<dom> Stats and Isolated Streams
HTA: we know this can be an issue. So either we try to mitigate, or we just remove the stat. The latter is my suggestion.
Cullen: security arch doc actually says to do that.
HTA: I will prepare a PR for not reporting the stat in this case.
Bernard: an issue for csrcs as
well
... in webrtc-pc
Dom: will the PR address CSRCs as well, or just volume in isolated streams?
HTA: i will do the stats part.
Dom: I will file a new issue for CSRCs
HTA: I will do the PR then.
<dom> ScribeNick: DrAlex_
<DrAlex> jib speaking
<DrAlex> starting with jib tickets in the absence of peter
sometimes the values that were preset are different from the current values
example, capture rate might fluctuate up to a preset of 60 fps
live volume vs actual volume settings
is another example.
camera motor pan has a target, and a current value as well.
cullen: let s take th pan example
we need both values
jib: yes but one is not a constrainable
cullen: agreed
dan: we already discussed this
this should be what it is SET to
hence the name.
we can still discuss wether we also need a current value or something, but the result of getSettings() should be what i was SET too
bernard: the downside is that if we had a current value, people would start pooling it.
jib: the spec has special frame rate wording.
dan: there may be a precision discussion we may have across
all measurements.
that is also true for framerate
which can fluctuate around a target velue
but your setting is what it is, and doe snto change.
cullen: it s ok to report error if you make a mistake.
or if there is a problem.
you should not set constraint, but when you do, and use exact, you are expecting errors when things is not exact.
camera rates do not fluctuate. However browsers get frames at different rates.
jib: cameras, e.g. on mac, have modes. if you ask for an arbitrary rate you night not get it.
it is my opinion JS should count the frame rate themselves.
cullen: i do not think there is a way for app to compute FPS, really.
jib: ok, so for motor pan .....
do we speak about target, settings, values
?
cullen: actually: what do we have again ?
<inserted> ScribeNick: DrAlex
issue 466
the answer is "it shoul dbe whatever the last setting was"
for previous issue.
<stefanh> was there any conclusion on 470?
<stefanh> audio is really bad
conclusion of 470 the answer is "it shoul dbe whatever the last setting was"
<dom> (i.e. the target setting, not the live value)
settings shoul now belong to the track
<stefanh> +1 to hta's comment
of course source have settings, but source can have multiple settings, from the user point of view (and also for security reason) the settings should be on the track
jib
it would be very easy to see the track as a rescuer object.
should the track only reflect the source settings
or shoul dit handle all the scaler.
dan: when we spoke about it initially, we had decided that the sink would be the one holding the processing (scaling here)
jib: we have now additional sink: canvas capture, remote track .. we have never came up to define constraint on those. e.g. echo cancellation is audio only .....
with that, I think that if we can word this in such a way that tracks can have different settings we are still all in agreement
cullen: isn t it like that already ?
consensus on leaving it as is
<DrAlex_> harald: we decided last time we had two ways to do it.
<DrAlex_> one sends nothing
<DrAlex_> one sends black frame.
<DrAlex_> cullen: while I think we should specify it so people know what to expect, but it s an implementation detail.
<DrAlex_> sergio: in the case of simulcast
<DrAlex_> we also have that kind of problem, when we kind of "mute" the higher resolution in case of congestion. In the IETF there is a solution.
<DrAlex_> bernard: i just looked it up. there is nothing about enabled=false
<DrAlex_> ericsson: it does not matter if it s muted or enabled=false, it should be the same behavior.
<DrAlex_> <everybody agrees>
<DrAlex_> jib: so ... no action item ?
<DrAlex_> cullen: two different settings. one is resulting in silence / black to be send the second one stops rtp packets.
<DrAlex_> bernard: track=null results in no rtp sent.
the track could be enabled, while the source be muted.
(jib)
adam: the specs says balck frames and silence shoul dbe send. but it does not specify wether you send only one such black frame, one per second, or full stream of black frames at 30fps.
youenn: safari sends one frame per second, just to make sure something is coming.
so does firefox.
<audio setting in the room>
jib: do have a motion toward consensus on this.
consensus
495 / 472 Video dimension constraint
<stefanh> What was decided on 441?
441: one frame per second.
scenario: sender has 16:9 receiver wants 1:1
<stefanh> BTW, the "one frame per second" was an advice, right?
shall we add pixels (letterbox) or removing piixels (crop)
proposal: new constraint resizeMode
what a bout default
cullen: ok, what abou tnone?
peter: not allowed to do any resizing
cullen: ok, love it.
peter: there is still the question of default
cullen "trash my video"
jib: my one concern is that we are moving to a "rescaler API" we argued shoul dbe on the sink. if we re speaking about access to the source ......
i like crop'n scale
cullen: i originally though like you. but justin convinced me that for screen sharing cannot let any pixel to be thrown.
jib: different spec
adam: surveillance video also need to keep all pixels.
jib: ok
peter: to you point that it s become a size and rescale API, yes, and that s what we aim at.
Dan: i do not think it conflicts with anything else.
dom: one confusing thing, there are many different constraints.
<stefanh> FWIW I like Peter's proposal 'resizeMode'
dom: we should start making the distinction between the constraints used through applyConstraint, and those used in GUM, ....
<dom> objectFit
<dom> +1
Harald: we should reuse the names from CSS
peter: agreed. i just could not remember them when i wrote those slides.
harald: i ll dig them up
jib: so ....
the implementation concern is not w3c"s priority, but .... for browser vendors, there are risks.
peter: this should ONLY be used for aspect ratio.
if the aspect ratio does nto change, it is not intended to be used.
<stefanh> my understanding was that resizeMode would be used also when width/height was constrained
jib: we might end up rescaling twice then
peter: yeah i guess
jib: do we want a 4th value "preserve aspect" as per cullen suggestion?
peter: i ve never heard anybody asking for that.
jib: any other comment ?
consensus in the room
dan: the conversation was going fine, so i was DL the audio system info
jib: is there a pr
<dom> consensus to add a resizeMode constraint
peter: no, not yet
<dom> JIB will produce a PR
<stefanh> consensus on default mode?
jib: ok, i ll write the PR.
<stefanh> crop-and-scale?
jib: with the new constraint we have a way forward, but we still need to define a default.
cullen: constriant has a mechanism: minimize a istance function
youenn: no scaling at all, so we would prefer minimize processing.
jib vs cullen round 5
on constrint.
constraint
proposal: unless people specifically requires resizeMode, the default is to let the browser choose
consensus
<dom> Chair: Bernard_Aboba, Stefan_Hakasson_(remote)
<dom> Agenda: https://www.w3.org/2011/04/webrtc/wiki/November_6_-_7_2017
<dom> Slides
<dom> ScribeNick: fluffy
On slide for "Issue 478"
Questions abotu what to do. Not many questions asnwered since last time
This would be a new field on track or something like that
Seems where we are is people previously thought this was generally a good idea but wanted to see specifics before they agreed to accept or not
Argument for putting it in a track was that it needed to go to multiple things, like recorder, so that made it better in track and sender
<dom> cullen: another question is whether we have enough and the right categories
<dom> ... e.g. having 30 (sport, ...) vs 2 modes, this is very different conversation
Main driving use case was indicate that video should be treated like screen cast
Question does this go on tracks or sinks
Concerns raised about auto setting as this implies it is a hint not a setting
PeterB: If you make an app that
does things like echo cancelation and voice enhancements, they
mess up things like music
... browser has no sane way to set sane defaults for music vs
speach
... as more things get added to the browser, the music app will
need to know to turn it off if we don't have something like
this
Question about is auto value of enum or antoher flag.
Dan: We have an "auto" setting for constrainable properties. This would allow the browser to use the seperate hint about media type (music/speach) to do this
Adam: propose that is what we have today and this is what no constraint means
Dan: OK, perhaps don't need auto
Agrement in room we do not need to add Auto
Question about if put it souce / sink
JanIvar: lean towards source as it is descriptive of the track
Adam: we don't have controll of all the sinks so better not to it there. The fact that it might look like a camera, but it is a source. So want to put on a Track.
Proposal put it on track. (and be great if we could hear this from Randal if he is OK with this)
Questioon what happens if you have a track and you change the value of the hint.
Dan: as long as browser is operatinng with constraints, it can do what it wants and use info inculding this hint to decide how to process it
JanIva: Set echo cancel false. Then set hint to voice. This will not chang echo cancel as it was explicitly set. But if it had not been explicitly set, then this hint might chang the echo cance
Question about impact on GUM and extentiosn that don't know about it
Dan: Do we need to say anything about what extention specs must say? and other thing how does this change implemetation of existing specs.
Adam: we will have some hints, and if you exptend the base spec, you start with an empty bucket of items and so there is no impact of adding this
Dom: we should have words on if
you are a sink you need to consdier the impact of hints
... do we try and bring that into GUM 1.0 or if this is a next
version thing
... answer depends on who plans to implement
Bernard: one place this made a big differnces is when we do AV1 which has screen content encoding - prob bit longer than 6 months
Question how would conflict setting be addressed
Dan: There may be different places where you set the track. If you set it once and set it again, it replaces the previos. How much info carries to toher side. Questions on where you can set it.
JanIvar: any idea that this would cary across RTP over a clone.
Bernard: the decode knows what tools were used to encode but does not need to set the hint. Or the app might know from JS to turn it on. Would not assume it is carried over.
Dan: what about clone
DOm: wes should copy hint
JanIvar: this is a weak hint and
you send it to web audio, hint gets dropped
... if there was no processing, the hint would be lost
<burn> ouch
We need to avoid slipper slope of speach/child
Dom: how to we deal with timeline of changes to attribute and when it happens to media
PeterT: no gaurentee of when it applies. It's a hint
What are audio category
speach and music seems to be the winners
Dom: so does speach and music
corespond to what we need to adjust.
... is there a ref set of categories we could use from
elsewhere to guide this
fluffy: existing codec have speach and music
Agrement in room: speach and music
Moving on to question to video categories
Bernard: even thought video came from screen, if it was a game, it might be motion not slides
<dom> https://wicg.github.io/mst-content-hint/#video-content-hints uses "motion" and "detail"
Adam: are we dupliating existing constraints ?
Fluffy: propose motion, animation, detail
<burn> (sorry about the echo - trying to find it)
Bernard: might effect the pallet used for encoding
Room is good with having at least motion detail
Question do we add third category of animation
Varun: will this confuse
people
... will people know what detail is
Dom: we will have to provide good definitions
<stefanh> webrtc-pc talks about framerate/resolution https://w3c.github.io/webrtc-pc/#dom-rtcdegradationpreference
Dan: if input is a movie, don't
change white balance or de nosie
... but if it is a camera would be good to do
Varun: what if person says want motion and detail
Dan: think abotu this as use case "this is what is going wrong", and thus "this is the hint you need"
<burn> varun asked what about motion and no-motion
<burn> then Peter said no-motion doesn't need a codec
Fluffy: think abotu example with comptuer vision and no codec
Adam: why do need the hint
Dom: hint is only usefull if there is something delegated to the browser
Adam: can we do this already
PeterT: for many things yes, we have direct control, but for some new things we don't have, having a hint helps
Varun: hints should not be on get user media
<dom> Accelerated Shape Detection in Images I mentioned earlier
JanIvar: the gum makes a track,
and then things would allow setting it
... setting low framerate is not good way to say prefer high
res
Varun: in screen capture of games, want high fps and good detail
JanIvar: track sits between soruce and sink. It is a way to configure source but it is also may have to become an API for sync too - for example downscale
Back to quesiton on do we add thrid animation category ?
Dan: What is the use case
Fluffy: arch fly through with synthetic artificial images
Varun: because it is articfical mostion, we want to see all of it in games and such
PeterT: had asked what you do
because of this
... hint will not cover all use cases
Fluffy: would turn off pre processing of white balance, de-noise, etc. Would pass differnt paramters to AOM1 codec.
soares: want a way to turn off much of the pre and post processing
varun: want fine things or don't mess with motion
Adam: we are duplicating what might have in constraint
Fluffy: this is a "do what I mean" not "do what I said" interface
Dom: hard to know we need for do what I mean. Less convinced this model is good
<stefanh> What would the default be between motion and detail? "balanced"?
Dan: sugest we need more work on why this is needed
PeterT: two things going forward. 1) recovince ourselves we are OK with this model 2) do we want to add a third
JanIvar: like to see some examples of things we use this for that we don't want to do with direct controlls
Varun: people do not want to figure out the the constraints and instead just leave it empty and have the hint do the right thing
<dom> [we need a boolean "doTheRightThing" constraint]
PeterT: uefull to have more specific of how an browser might implement this and how apps might use it
<burn> [a Do What I Mean API is perfect, of course. If we had a Jarvis we could ask him to create that API]
Moving on to Issue 1533
<dom> Clarify whether RTCRtpContributingSource members are live. #1533
JanIvar presenting: We could get live objects where you get an object and get live values of it.
This does not work becuse you don't get new particpats. And get old values for person that left.
Conclusion: live objects are not usefull
fluffy: why did we make it an interface
Adam: so do we have to poll
JanIvar: yes
... used to be one field, but now two. Why did we split
Dom: to make it cleaner ???
Agreed no live objects
Moving on to splitting priority up to two controlls
PeterT: made PR to deal with this
<dom> RTCPriorityType combines relative bitrate with QoS priority, which applications may not want. #1625
<dom> Let javascript set different priorities for bitrate and DSCP markings. #1659
dom: prefer clean break
JanIvar: is there an impact on data channel
Bernard: think it is OK
Fluffy: on behalf of HTA, do we have people that set it seperate
PeterT: Yes, we willhave people that set marking but not bitrate
<burn> Cullen: and we will need to set bitrate.
Varun: prefer ratio
bitrates
... would use bitrate prioriyt with the 4 values
Dom: was trying to get to if the bitrate priories in 4 bin was worth doing now and would used. Is there enough value that we do this
Vaarun: fine with 8 4 2 1 locking
Summary: If the IETF is ok with peters changes, then WebRTC is in favour of making this changes
<adambe> scribenick: adambe
bernard: 15 issues relates to peer identity
fluffy: a lot of issues on peer
identity are bogus
... no one is working on it
burn: Who's working on implementing (peer identity?)
bernard: We should look at interop requirements
fluffy: Our current text about
two implementation is wrong
... it's more correct to say that there will be more than one
browser
dom: Is there a difference?
bernard: It's also a question about how code is integrated in a browser
dom: The JS API is a small part
of this
... What we want to verify is that it actually works to send
media between browsers
burn: It's also important for independent people to be able to to implement the spec
bernard; Our edge-stack is fundamentally different
fluffy: It's hard to define independent implementations is our case; and it doesn't buy us much
dom: We could probably get away with only testing the JS API, but I think we have a moral duty to do more (i.e. interop on media level)
bernard: Do we need to change our text in the spec about "independent implementations"?
burn: We need to file an issue to update this text
AP on burn to file an issue
fluffy: It's in our charter that we should demonstrate media interop
bernard: Moving to other specs in
the group
... We should have wider review of screen capture spec
... We need to check with the "depth" editors on the
status
... The major action items are around screen capture
jib: Several browsers have some kind of implementation of screen captuer
<dom> Screen Capture editors draft
burn: The API should be standardized
fluffy: That's the easy part; the hard part is the privacy aspects
bernard: We should include "screen capture" spec in the interim meetings
AP on chairs: Bring screen capture in to the interm agendas
Topic is: WebRTC stats status
varun: summarizes the current issues
soares: It's quite straight
forward to test the stats stuff
... We just need to put in the work
dom: Have you considered testing
the life cycle of stats objects?
... We don't need to check the exact value of metrics. Just
that they are not zero
varun: It's not easy to do
... That seems like browser testing to me
fluffy: It would be really nice to at some point test a bunch of stats between a set of browsers and verify that the result makes sense
varon: That kind of activity could be part of the "ultimate goal"
jib: Make sure we distinguish
between API tests and "browser functionality tests"
... What about non-MTI stats?
varun: We're focusing on the
MTI-ones at the moment
... Do we want allow experiments with stats?
fluffy: we can't disallow
it
... Don't use prefixes on names
varun: Google has an old and a new API. Experiments are currently done in the old API
Discussion about experiments with origin trials
scribe: and different approaches on doing experiments with stats
<dom> ScribeNick: vr000m
fine grained control for media, SVC, simulcast support.
Cullen: talking about Simulcast
support.
...: we have level of simulcast support today, however, we do
not have everything.
Peter Thatcher: for the server to negotiate simulcast, for the endpoint to receive simulcast (i.e., run the transcoding bridge on the browser)
Cullen: Fine grain media stack
control, we could take a high level api is the pc spec, today.
The ORTC stuff is moving away from SDP, but it still follows
the O/A.
...: so we should expose all the minimal features to the
javascript.
Pether Thatcher: ideas on WebRTC next steps
Cullen: can do codecs via webassembly should explore that
Alex: Explore the idea of using EME for hardware crypto
PeterT: QUIC recap
...: QUIC features for datachannels
... QuicTransport
... QuickStream discussion about ordered and unordered
Adam: there should be a way to get an event when data is acked on the send side
jib: did you use "async for" from whatwg streams
peter thatcher: there are some roadblocks.
<jib> "async for" == for-await-of https://github.com/tc39/proposal-async-iteration
DECISION: Add a constructor to the RTCIceTransport to the WebRTC CR
Peter can write other docs for Quic and other changes
<soareschen> Cullen: How to make ICE easier?
<dom> ScribeNick: soareschen
Jan on issue 434
<dom> Disable user media by default in cross-origin iframes #434
<dom> Feature Policy
Chrome has different syntax for iframes, i.e. allow="microsoft"
<dom> allow attribute
Jan: We should change the language in the spec for feature policy
Related issue: 440
Adam: If Chrome is implemented that way and mozilla is considering, we should revisit that
Dom: agreement
... Do mozilla expected to do both?
Jan: just the attribute
Dom: we can still move to PR if
implementers already align with the changes
... reopen issue 440
Jan: does absence of allow='microphone' means disallowed?
Dom: we could just reuse the sandbox
Peter: more scary proposal
<dom> Dom: I'll make sure to follow up internally to see if feature policy can move to a more formal status
way to get audio/video over any transport
Peter: low level way: stick in a
track and get out a video frame
... for decoder, provide a frame, and at future time will go
into a sink
Cullen: the frame need to include details such as timestamp
Does this mean the stream need to be ordered?
Peter: only needed for key frames
Dom: there are requests getting this feature into service workers
Vr00m & Peter: flexibility of inspecting metadata of encoded frames
Vr000m: Supporting SVC over QUIC?
Cullen: JS is fast enough to do SVC in JS
Vr000m: more metadata such as
bitrate and target
... we can get a lot more control over this
... doing simulcast?
Peter: We could have 3
decoders
... We can get extra metadata such as layer for simulcast
Vr000m: Is it only one frame one frame?
Peter: You can have your own framing scheme for multiple frames in one stream
Cullen: When decoding one frame may be decoded into many frames
Peter: Haven't thought through for multiple frames
Dom: dictionary format for codec
metadata
... are only codecs exposed are codecs that can be specified as
dictionary?
Peter: there are two ways, not sure which one is better
Vr000m: on jitter buffer
Peter: you can have control with your own jitter buffer implementation
in JS
Vr000m: want to control jitter buffer underneath to send only one frame
Cullen: we can inspect timestamp metadata
Peter: we can specify frame dependency
Vr000m: if there is a jitter buffer underneath I want more control over it
Peter: example for audio
encoder/decoder
... audio jitter buffer is different. Sink is pulling at
regular rate regardless of whether it is filled
... if there is nothing you'll get silence
Vr000m: For video, we want to pass partial frames to video decoder
Peter: for media over QUIC streams, we can experiment more easily
without having to standardize first
Dan: did you talk about isolation?
Cullen: we can't have isolated media streams for this
Peter: one way is that the encoder can encrypt
Cullen: Or we can have isolated JS that does decoding
Jan: Audio worklet can run
limited JS in audio thread
... Decoding video on main thread isn't desirable
Peter: technically we can implement encoder/decoder and jitter buffer in JS
only difference is you can't get access to native or hardware implementations
Cullen: You can build your own jitter buffer in JS, but most would use a default native implementation
Peter: option B: high level
send/receive example
... require mapping down to QUIC stream
Dom: Youenne have some security concerns
Cullen: if isolation is a
concern, can tell the encoder to pass only encrypted SRTP
packet
... For decoder, pass SRTP packet to it which connect directly
to sink
... People who use this API probably care less about
isolation
<dom> Media Source Extensions
??: People might say we don't need another JS interface for audio/video pipeline
Peter: does WebAudio has encoders?
No
Cullen: Scary ICE proposal
... API to send STUN requests
... All the stuff stay the same. On the other hands we have API
to nofify STUN requests
Things complicated for ICE is that we have two peers that test connection independently
Cullen: Move complexity of ICE to
JS. no need to do it billaterally
... no need check list and gatherer state
... JS shouldn't able to modify STUN packet content
... browser would keep track of best one found
... but JS can decided which one to use
... we can find paths that ICE can't currently find
vr000m: Is it because current ICE have different priority?
Cullen: we don't want to allow JS to construct custom STUN packet to exploit the server
Peter: as long as there is
limitation on how to send STUN packets it should be fine
... if we call send too fast, browser just return error
Cullen: browser always send STUN response to STUN packets
Peter: we also want notification responses
Cullen: problem of ICE is it only
discover 2-way paths, not one way
... enterprises often have firewalls and want 1-way path
Peter: there are strict rules on how much/fast you can send in ICE, and we would still make those apply
Cullen: we are very careful to keep the security properties of ICE
Peter: we follow the ICE rules for punch etc
Cullen: I am even willing and able to write a test for this
Bernard: wrap up
Cullen: for encoder/decover we prefer option A over B
Peter: we want to write an ICE transport doc
Adam and Dan: the testing PR #1657 has been merged, and there is now a testing.md with help
Bernard: do we have conclusion on SVC?
Cullen: we agree to not add new functionality for current spec
Dom: we should consider one
document for each new feature
... we need new documents for QUIC stream and ICE transport
extension
... encoder/decoder might have much more impact beyond
WebRTC
Cullen: the proposal would be have a subset for the ICE protocol in IETF, then a W3c document for API access
Project snowflake for new ICE protocol
that's all for TPAC!