Meeting minutes
SongXu: our meeting is about Network Emulation Tool in Browser Developer Tools
… our agenda: some background of this work and goals
… overview of existing tools that are relevant to similar functionalities
… followed by discussions on the models & requirements
Background & Goal
Song: During TPAC, there was interest showed around network condition emulation
… we want to review concrete proposal to improve network emulation tools for web developers
… and identify standardization opportunities for trace format
… our initial roadmap on this was to standardization efforts around trace format in Q2 2021
… the landscape analysis too
… have use cases defined in Q3 2021, along with emulation methodology
Quick Overview about existing tools
Song: WebPageTest, LPP Network Trace Tools and Throttle
WPT
LarryZhao: WebPageTest (aka WPT) - a web performance tool providing diagnostic under a variety of conditions
… it helps find performance issues, based on test run from a different set of locations around the world
… You can also pick a type of connection - e.g. LTE
… the documentation shows that this is done using NetEm and dummynet
… netem is an option enabled in the linux kernel, e.g. with packet delay, loss, duplication, reordering
… When WPT simulates network, it works at the network level directly
LPP Network Trace Tools
LarryZhao: we heard from Jonas on this before
… it features 3 important parts: trace capture, emulator and a file format
… it is presented as a chrome extension
… [shows demo of LPP]
… it utilizes the Chrome Dev Tool Protocol
… so it operates at the level of application network
… it only has effects on the Web site you visit via Google Chrome
… that's a major difference between WPT and LPP - network level vs app level
Throttle
LarryZhao: Similar to WPT, from sitespeed.io to simulate slow network conditions
… it tool works at the network level
… very similar to WPT
Sudeep: is the Throttle tool similar to traffic control (tc) tools?
LarryZhao: it is based on TC
Arpit: how about the Mac version?
Song: they use the same core library across Mac/Linux
Piers: it uses dummynet underneath afaik
… throttle uses both pfctl & dummynet
… to provide something similar to tc & netem
Gap Analysis
Song: some of the tools can only emulate limited network conditions
… but it's useful to be able to test under network switch (e.g. from WIFI to LTE)
… with WebRTC & cloud gaming scenarios, you need access to more conditions (e.g. package dropped, different loss rates)
… there are features common across tools: up & down bandwidth, latency / rtt
… how much interop is needed for these features? a standardized trace format would help in that case
Use cases
Song: I've collected use cases that would make use of this
… one is test automation - a developer could test their sites under various real life network conditions previously captured
… WPT has a collection of such traces
… Another use case around test & development
Jonas: this is one use case we've considered
… Our main intention was to use it for standards Web application testing to get a feeling how the connection is in different parts of the world
… we're studying extensions for MPEG dash, cloud gaming, real-time media
Song: another use case is for PWA testing
Sudeep: the input on this is also available on the github repo
Sudeep: testing scenarios where some requests randomly timeout has proved difficult in the context of an app experience
… and testing robutness depending on the frequency of these timeouts
… Another thing they mentioned is testing how service workers react to these varying network conditions, e.g. when using background sync APIs
… having a way to automate this testing in local environments would go a long way
… the last item is emulating online/unreliable/offline states
… there can be scenarios where a given request will go fine, but then another will fail due to a e.g. a proxy redirect
… this creates delays in how the web app will load & render
… and this can't be dealt with today at the level of browser developer tools
Models or requirements
Song: Opening the floor to discuss what features of emulation, what parameters to monitor, what methods to use to emulate
Ali: I've added two slides to discuss other use cases
testing http streaming clients
Ali: we use network emulation to test streaming scenarios
… HTTP streaming is very popular, consumes a lot of bandwidth and triggers a lot of research
… we depend quite heavily on network emulation tools
… at a high level, the model is that a streaming client fetches media segments from a server
… based on information known in advance on what to request when to request
… the stream is available in different resolutions & bitrates, allowing to adapt the stream (hence the name adaptive streaming)
… the most important parameter to determine what bitrate is bandwidth evaluation
… there can be others (e.g. CPU & memory)
… but the general case is the network is the main bottleneck
… the client needs to measure its active bandwidth and predict what the right segment will be
… the measure is done based on the data being fetched
… but this can be pretty noisy given the different sizes of segment, the impact of round trips, etc
… what most clients do is that they smoothed out the value (e.g. average, percentiles)
… some clients make prediction out of this
… this gets done over & over during the streaming session
… my understanding is that LPP can be used to help with this
… in terms of measuring and predicting the bandwidth, although not about how the client should behave
… This leads to a number of requirements:
… network emulation is a big part of our testing in our daily life
… we usually set different upload & download conditions
… with different network congestion
… our main focus is on bandwidth throttling, we're mostly ignoring e.g. packet loss or re-ordering
… when dealing with live streaming, we need throttling on short time scales too
… Reproducibility is a very important requirement for us
… we need to be able to test these conditions across different devices & content
… this means it should be easy to design programmatic traces, including modifications to existing traces (e.G. scale up the bandwidth)
… e.g. just having to adjust a scale factor
… likewise being to scale the timescale on which it runs
… the files should be small, easy to edit and share
Song: thanks for the excellent input
Ali: we heavily use the built-in throttling tool in Chrome - it has proved easy to script, but it doesn't have all the features we want
… we would be more than happy to switch to a better tool
Jonas: the scaling aspect is a good one to add
Ali: yes, both in terms of magnitude & time
… sometimes, we design a trace that we just want to replicate under different scales
Jonas: that indeed helps with sensitivity tests
… With regard to streaming, I agree that LPP shouldn't be involved in how the estimations and predictions are used
… we're exploring different type of behaviors of what we can do with these traces - of course not forcing them on any one
… we're planning to publish a set of examples that demonstrate different type of optimizations (e.g maximizing quality vs minimizing buffering)
Sudeep: how do know which use cases would benefit the most when we start emulating network conditions?
… are there use cases that have no solution to manage the quality experience e.g. under congestion control?
… how do cloud gaming deal with these?
Piers: WebRTC has P2P traffic
… not sure how Chrome throttling applies to that
Sudeep: I tried it but it didn't seem to work
Piers: another question is network emulation in the context of newer congestion algorithms like BBR which are delay based
Ali: I mentioned we didn't care much about packet loss & reordering - that's true for HTTP 1.1 & HTTP 2 since they're based on TCP
… but as we start testing HTTP3 at some point in the future, these things might become more relevant - hard to say how much at this point
… but as QUIC replaces TCP, we will be interested to access these aspects too
… that aspect will also be more relevant for cloud gaming & webrtc
Next steps
Song: I'll bring these topics we've identified to our github repo
Jonas: in terms of trace format standardization, it was scheduled for Q2 which is about finished
Song: yes, this will have to be revisited
… probably more of a Q3 now
Jonas: I guess the schedule depends on the interest around standardizing such a format
Dom: which producers / consumers of such a format would need to be involved in these discussions?
Ed: is there a timeframe for getting feedback on trace format standardization?
Song: I would like to say by the end of July
Piers: has there been an assessment of other commercial & open source approaches in this space?
Song: we did part of this stuff in the past, but we didn't look at commercial implementations yet
Piers: there are quite a few network emulators out there, I'm not sure about their format
… hotnet, NS (?)
… they're often programmatic rather than declarative
… in streaming papers, they tend to use traces from the FCC and other sources
… but they tend to be just CSV files
Sudeep: in a last meeting, two tools were mentioned: crowdad and network management labs
piers: indeed - a new one is emerging from Standford, Puffer (?) that provides anonymized logs of clients
… there are a number of repositories traces outthere
Jonas: Crawdad seems like a mix of formats
Sudeep: it makes sense to look indeed at existing formats & repositories
Song: thank you all for joining this meeting
… let's continue discussions in github