W3C/SMPTE Joint Workshop on Professional Media Production on the Web

Hello, I'm Peter and I'm experimenting with creating music in the web browser.

And what I want to show you here is a web application for creating music in the web browser. This video here is also being recorded in that same application. It's a camera view here on the bottom right. And on the left side, here it is code for generating the music.

This is actually a live coding studio, where you have the notes to be played here in comments, and you have the durations and the velocities. And on the right side here is a language called AssemblyScript, which compiles to WebAssembly.

And what I have here is the instruments. There are actually no recorded instruments here. It's just generated in real time and you have all the math and all the code here for generating the sound.

When I actually (playing note on keyboard) play here on my MIDI keyboard. (playing keyboard) The code here on the right side is running and calculating all the samples for the instruments in real time. (playing the keyboard)

I have lots of instruments here. I have a string (string instrument notes playing) which you can actually see the code for down here. I'm not going to go into the code right now. That has to be another session. (playing the keyboard)

You can create lots of interesting instruments here. (flute instrument notes playing) That's the flute and also I have a drum here. (drum beat playing)

And actually I have this little beat that is recorded here so I can start playing it. (musical beat playing)

Why not record something while I'm at it? (playing bass notes over musical beat)

And what you just saw there was that after I recorded, it actually inserted the code here on the left side here. You got all the notes I played and they did the durations and the velocities.

This is a full environment for creating music. You have recording and creating the instruments, you got all you need here actually for creating a complete piece of music.

But the very important thing for making this a pleasant experience to work with for a musician and a keyboard player (playing the keyboard) is that you actually get instant response from the MIDI keyboard when you play. And it's just a matter of milliseconds. If it takes like 10 milliseconds, then it's too much.

On this Mac that I have here, that's no problem. It just works as it should. It's just a low end MacBook Air. It's not problem at all, but it's a challenge across systems, this latency.

For example, if you go to a Windows computer, you generally get good performance, but you get this lag of 10 milliseconds, which is really too much to have a good playing experience. It's okay for listening, but maybe also for gameplay. But actually when you are using this as a musician you need that instant response.

And also you see that, for example, a laptop running on battery. Also, this is with Windows. It's not problem with the Mac. Then you actually hear that the sound is degraded. There are clicks, there are... sometimes it just stops and it's not the experience it should be. These are things that really have to be improved in order to make this, because the possibilities are very exciting here. Just, this is a very simple application, but you could create full replacements for what we have in desktop apps today for creating music.

I really think there are some interesting possibilities there when you expand that area onto the web.

And also across browsers, for example, in Firefox you don't have the MIDI support. It's a very good AudioWorklet implementation. AudioWorklet is the technology that can bind this WebAssembly code to the Web Audio API. And makes it possible to have this low latency, this instant response when you play. This works very good on Firefox, but you don't have any MIDI connectivity. You have to find other solutions for connecting a MIDI keyboard to Firefox.

And this is also important in order to get this fully usable experience is that you have this working across browsers.

But other than that, I think this is very interesting technology. I've created quite a bit of music with it so far. I use it for all my music now, and I think it's great and I really hope that we will see this stable and performant and you get this low latency across all browsers or platforms.

That's the point I really want to make today. And I hope it can get some attention in the future. Thank you for watching.