TPAC breakout session on efficient audio/video processing.

Efficient audio/video processing

François Daoust – @tidoust
Dominique Hazaël-Massieux – @dontcallmedom

Context

Recurring discussions around ways to process audio/video streams efficiently
Spans a number of groups including:
- APA WG
- Audio WG
- GPU for the Web CG
- Immersive Web WG
- Machine Learning CG
- Media & Entertainment IG
- Media WG
- Timed Text WG
- WebRTC WG
- WICG

Use cases

Media stream analysis

Barcode reading
Face recognition
Gesture/Presence tracking
Emotion analysis
Speech recognition
Depth data streams processing

Media stream manipulation

Funny hats
Voice effects
Background removal or blurring
In-browser composition
Augmented reality
Non-linear video edition

Media stream production/consumption

Custom codecs

Generic need

Some way to hook into the media pipeline and be able to process frames efficiently:

on the CPU in workers
on the GPU
using WebAssembly
using machine learning processors
using dedicated APIs that hook into the hardware when possible

Efficiently?

Some of the use cases can already be implemented, but typically restricted to processing on the CPU through a Canvas. Goal is to:

Avoid duplication of bytes in memory as much as practical
Make it possible to run multiple operations on a stream

Some questions

How to map between MediaStreamTrack and Streams?
Best way to allow WebAssembly to access encoded and/or raw media?
Best way to inject an overlay on a frame by frame basis?
How to inject custom codec support in the media pipeline?
What mechanisms should be standardized?

Goals

Refine scope: add new use cases? drop use cases?
Clarify need to address the problems: is it worth it?
Identify existing/possible proposals
Assess willingness to progress these proposals
Identify group(s) who could work on them

Let's look at...

Possible performance gains
Practical experience gained with work on AudioWorklet and on Mozilla's FoxEye
WebRTC API for insertable Streams of Media

References

Thank you!

Questions?