W3C/SMPTE Joint Workshop on Professional Media Production on the Web

Hello everyone. My name is Kevin Streeter, and I'm here to talk about some of the challenges that exist bringing desktop class creative applications to the web. In this talk, we'll begin with a broad summary of the many technical areas we need to work through to create powerful creative tools. Then we will dive into a few selected topics to get a better feel for the kind of capabilities we need from the web to make it all possible. Let's get started.

Creators are accustomed to having the full power of desktop class hardware at their fingertips. How can we build applications that have the same level of performance and expressivity, but also having all the benefits of a web based experience?

Here are some of the ways that applications take advantage of traditional desktop platforms.

These applications have full access to 64-bit CPUs and performance or instruction sets such as AVX.

They have access to large amounts of memory.

They have full access to disk hardware and the local file system.

They use low level APIs for performance and efficiency.

They have end to end support for color management and color spaces that allow for wide color gamuts and high dynamic range.

Desktop software has access to input hardware, such as styluses, pens, and other devices.

They have access to modern media decoding and encoding, particularly video encoding and decoding.

They have access to modern GPU pipelines that support both graphics and compute.

And finally, they have access to the kind of quality of experience capabilities such as clipboards, accessibilities and other things that really make a tool work for the creative professional.

The first area that we're going to dive deeper into is the use of WebAssembly.

Creative media tools tend to use a mix of new and existing media processing code. The existing code is typically written in C++. This code tends to be a mix of compute intensive and memory intensive operations. Importantly, not all of this code can simply be ported to a GPU.

Sometimes that's because we haven't really found a good GPU-based solution for the operation, but frequently it's also because we have working code and we'd really like to continue to use it.

These code bases have been heavily optimized over time, and they've been optimized in a couple of different ways. They've been optimized to take advantage of 64-bit processing, and they've also been optimized to take advantage of SIMD instruction sets that are present on most desktop class hardware.

If we want to do these types of things on the web, we've got a few challenges today. One is that in-browser support for WebAssembly today is all 32-bit. That not only limits the processing capabilities, but it also limits the total amount of memory that we have access to.

SIMD is not currently supported in most browsers. There are some proposals around SIMD operations, and there are support for that in some browsers, but it's still very early in terms of availability of that capability.

One important aspect of this is that the SIMD proposals that are currently being investigated really do emphasize portability over performance. And what this means is that we no longer really have access to the lowest level of instruction sets, and that does have an impact on overall performance.

The next important area I'd like to discuss is color.

Color management, wide color gamut, and high dynamic range capabilities are critical for pro media production. This is true regardless of the type of media, whether it's video photography or print.

Consumer devices such as phones and tablets increasingly support wide color gamut and high dynamic range, making the ability to deliver correct color to these devices even more important.

It's critical to keep in mind that color is an end-to-end problem and that we have to solve for that across all aspects of the system. That includes formats, pixel processing, and rendering.

Now, today, standards such as CSS 4 are helping make it possible to support extended color spaces, but this isn't supported broadly in all browsers today.

Also these color management capabilities are not supported by all web modules. For example, today there isn't really a way to ensure that WebGL is going to display wide color gamut content correctly.

Last, it's really important to note that different industries have their own standards around color. What that means is that the web platform needs to be inclusive if we want to support the creation of tools that target these industries.

Next let's talk about disks and file systems.

Desktop creative tools use the local disk heavily and rely on access to the local file system. These tools use disks for variety of reasons. They use them to store documents and projects. They use them for importing and exporting content in and out of the application, and they use them for scratch space and for caching.

These tools have been optimized for disk I/O very carefully, and they try to minimize both the total number of I/O operations and the total amount of data transferred.

Today's browser-based file access doesn't really give write access to the local file system. And the APIs are fairly high level, meaning it's hard to ensure that the I/O is optimal.

Current proposals such as Origin Private File System are promising, but need to allow for efficient import and export with a host file system.

In the end, many low level capabilities don't really have an analog in the web platform. For example, there's no support for sparse files or zero copy file I/O.

The last topic I'd like to dive deeper on is hardware encoding and decoding of media.

Pro media production tools often use more specialized media formats than in a typical consumer application. For example, a photography tool will use formats like TIFF, JPEG 2000, and a variety of different raw image formats. Similarly, video tools will use things like REDCODE, ARRIRAW, or ProRes.

Now for imaging applications, we can get away with using WebAssembly to do encoding and decoding, especially if we've tackled some of the performance issues that we spoke about earlier in the talk. But video is much harder since maintaining good frame rates during playback, scrubbing, and other aspects of authoring are really, really important.

Now today, the ability to get hardware accelerated video is present in the web through the video element, and soon we'll have access through an API such as WebCodecs. But a problem with these APIs is that they only work with mainstream consumer-oriented formats like AVC and VP8 and VP9.

Now, unfortunately there is an unavoidable challenge with media codecs, is that many of them are encumbered by intellectual property issues, making it much harder to ensure they are available on all devices all the time.

So what we may need here is some way to access pluggable or modular hardware acceleration capabilities, so when this capability is present on a device, the application can get access to it.

Last slide. I hope you enjoyed this talk. I'd like to give a quick thank you to all the folks at Adobe that helped me prepare this material. And in particular, I'd like to thank Sean Voisen for his insights.

See you at the workshop.