W3C/SMPTE Joint Workshop on Professional Media Production on the Web

Hello everyone. I'm Qiang Fu from Bilibili group. My topic is video transcoding in web browsers.

I will introduce several feasible ways to achieve the goal and provide a suggestion for Web API to make video transcoding easier for web developers.

Bilibili is based in China and our application provides video content for users. You may think of it as YouTube in China.

Okay. Now let's begin.

Why we need to focus on the topic? To guide a new way using browsers, we need it.

The first thing is about video preview. Some video formats like AVI or FLV cannot be played directly by HTML. Yes. We can implement a player by WebAssembly or JavaScript for these videos but what if we transcode them to MP4 or WebM format, then we can use a video tag to play in HTML. It is simple.

The other thing related to the details about media production. For some videos, only small adjustments are required. We could offer help to change to change the resolution and frame rate or some other parameters of a video to meet the (?) for uploading. Our goal is to get a new way to, via transcoding.

The basic steps for transcoding can be concluded as the following: passing the input file to a demuxer to get to the streams and encoded video chunks. Then passing the video chunks to a decoder to get the video frame. Then maybe we need to perform some operations on the frame. After passing it the opposite way, finally we get the output file.

I summarized the process for ffmpeg, but how can we use it in browser?

There is a popular way for web developers to experience video transcoding. WebAssembly can help. After compiling the source code of ffmpeg to WebAssembly, run it in a browser and give it a command, we get the video we need. that's really simple at first sight.

But WebAssembly looks like a black box and developers don't care about the inner details. Let's see how it works.

Although WebAssembly is a black box, we still need some preparations. First of all, familiarity with ffmpeg. Then we need to study how EMScripten works and the best basic design concepts. The last step is to run WebAssembly in browsers.

After you finish the steps and get to demo a few common questions will be there. Why it is so slow, why the same command can give such huge difference on performance between browsers and OS. How could I make it faster?

The answer is tough. WebAssembly has done improvements, just like supporting SIMD and multi-threads. To enjoy these new features, we usually check the configure of EMScripten and ffmpeg. And luckily the source coder of ffmpeg should be modified. It is really professional and tough for web developers.

Let's talk about this other way where we can divide the process of transcoding into different parts. Each part may have a unique solution.

For example, the demuxer and muxer can be realized by JavaScript, encoder and decoder can use WebCodecs API simply. The video frame can be drawn on canvas or WebGL.

Tested in an integrated way, everything looks fine. We splice each part like Lego blocks.

I searched the GitHub for some repositories. Finally, I could transcode the video in the integrated way. The demuxer and decoders are integrated as a video player. To reduce the size of WebAssembly. I imported some libraries for ffmpeg and derived some files. The output of the player is video frames in RGB or YUV format. I pass it to WebCodecs API to get the encoded video chunk. Then the chunks flow through a muxer for making video in WebM for watching. Finally I got the local video I need. Sounds perfect.

So integrated way seems flexible enough. I could even change the frame in canvas by JavaScript, just like what filters do in ffmpeg. WebCodecs is cool. And it has option for hardware acceleration. The developers don't need to focus on the details.

The only concern is the muxer. Every muxer is different and I have to collect solutions for supporting MP4, WebM and other formats independently. It takes a lot of time.

What if an official muxer API is provided, that's my suggestion. Its design can follow the WebCodecs.

The Web will get the following benefits. First, it will complete the roadmap of media processing in browsers. And if we want to popularize some video formats, like WebM, we should reduce the difficulty for producing them.

The most valuable thing for me is to provide a better coding experience. The source code of ffmpeg is really tough and you need a lot of time to become a real expert. That's all my contents. Thank you for watching.