WHATWG Stream?
| Plug |
Construct |
| None |
VideoFrame
(in WebCodecs) |
 |
Stream of VideoFrame
(used by VTG, MSTP) |
 |
Stream of encoded chunks
(VideoEncoder + TS) |
 |
MediaStreamTrack
(in WebRTC) |
Why the difference?
- Requirements:
Streams propagate backpressure signals through the chain.
Some media processing scenarios require additional control signals (e.g. configure, flush, reset).
- WHATWG Streams seen as too complicated.
- History: WHATWG Streams did not exist when WebRTC started.
Creating a stream of video frames
From scratch:
VideoFrame stream
From camera:
getUserMedia()
MSTP
From a received stream:
WTReceiveStream
VideoDecoder TS
Notes:
- RTCDataChannel could be used as well.
- Also possible from containerized media but no API to demux the media to create a stream of encoded chunks, so up to the application.
Processing video frames
Idea is to use a TransformStream that takes a stream of video frames as input and produces another stream of video frames as output.
TransformStream
Processing can be chained if needed:
TransformStream
TransformStream
TransformStream
Actual transformation logic can use pure JavaScript, WebGPU, WebNN, WebAssembly, etc.
Sending/Rendering final video
Render to display:
VTG
<video>
Or:
JS + <canvas>
... but implementing a full video player in JavaScript is no easy task! (e.g. sync, accessibility, controls)
Send somewhere:
VideoEncoder TS
WTSendStream
Note: RTCDataChannel could be used as well
Measuring jitter
- When using a stream of
VideoFrame, timestamp may be used to track the frame through the processing pipeline.
- But no way to know when a frame is rendered onto the display.
requestVideoFrameCallback does not report the frame's timestamp.
- Best is to use
requestVideoFrameCallback to detect frame changes, then copy rendered video frames onto a canvas, and analyze pixels to identify the frame. The color overlay encodes the timestamp.
➜ Works more or less and introduces additional processing time...
Time stats
Typical run with overlay and H.264 encoding/decoding
| Timer |
Frames |
Min. |
Max. |
Avg. |
Median |
| overlay |
104 |
4 |
45 |
10 |
7 |
| encoding |
104 |
15 |
245 |
20 |
17 |
| decoding |
104 |
1 |
23 |
1 |
1 |
| queued |
104 |
0 |
288 |
26 |
13 |
| end2end |
104 |
24 |
338 |
58 |
151 |
| displayed during |
101 |
20 |
59 |
38 |
39 |
Times in milliseconds
Key takeaway
Interconnections between APIs is not straightforward:
- Different architectures followed
e.g. use of streams or internal queues
- Hard to understand when copies are made and time is lost
e.g. read back from WebGPU to create VideoFrame
- No clear owner, not a top priority for groups
- Not really part of test suites
- Each world requires its own set of skills
e.g. pipeline layout and memory alignment in WebGPU
Other takeaways
- Streams and backpressure signals are confusing
... until they no longer are!
- Support for new APIs is flaky
... e.g. VideoFrame closure across workers
- WebGPU is as complex as it seems
A report from Dom and François on their explorations on video processing during Geek Week 2022.