Work in progress. See http://www.w3.org/2011/audio/track/actions/26

We haven't been clear enough on this. What we want is for JavaScript processing to happen *only* in workers. Doing anything on the same context as mouse and keyboard events are processed and where scripts can easily be blocked for 100s of milliseconds by layout reflows is simply a no-go.

I completely agree that workers are a better approach for lower-latency, smaller buffer sizes.  But there is a cost to the developer to being required to use a web worker because the JavaScript state is completely isolated from the main JS thread.  Thus it will require more complex code, and some applications might not even be practical.

Some developers have expressed concerns that JavaScriptAudioNode *only* happens in workers.

While I agree that workers are slightly more cumbersome to work with than regular callbacks, I think that there are some risks with supporting both methods:

1) It's generally confusing to have two options for doing the same thing. It's quite likely that developers will pick the "wrong" solution just because they copied from an example or tutorial that used the alternative that wasn't optimal for the actual use case at hand. 

2) I suspect that the callback based approach will be more sensitive to browser/system variations (e.g. different painting/event/animation architectures), which means that it's more likely that someone will design an app on his/hers system+browser combination of preference, and then it will suffer from latency problems on another system+browser combination. This is less likely to be the case if people are always forced to use workers, where the problem should be less pronounced.

3) Since the non-worker option is generally simpler to grasp, it's more likely to be used in more apps than it should.

While I agree that it's not a good idea to do time-critical heavy lifting like audio on the main thread, sometimes there isn't much choice: emulators and other virtual machines, ports of existing code (more possible but could be very difficult as the original code may have strong shared state with the audio code), and such.

These kind of programs are quite challenging as it is, I don't think a few bad eggs should make the lives of those developers worse than it already is by having to do expensive tricks like sending the audio data to the worker with postMessage and maintaining the callback system themselves.

I always find these discussions about creating bad practices a bit frustrating, people will make bad choices, no matter how well we design things. For me, it doesn't mean that we shouldn't try to avoid making things so that developers want to do bad things, but actively making actual use cases harder to keep some people from making stupid decisions is counter-productive, IMHO.

You give them the rope, some will hang themselves, some will build a bridge.

(In reply to comment #5)

> expensive tricks like sending the audio data to the worker with
> postMessage and maintaining the callback system themselves

That's not how a worker-based AudioNode would work, it would be a callback in the worker that can read directly from the input and write directly to the output.

There are things on the main thread that are not interruptible (layout and event handlers being the most obvious) so it's only luck if one is able to run the callback often enough. I can't speak for any other implementors, but I'm fairly certain it would fail horribly in Opera, as other pages running in the same process can't be expected to write code to avoid long-running scripts or expensive re-layouts.

> That's not how a worker-based AudioNode would work, it would be a callback in
> the worker that can read directly from the input and write directly to the
> output.

Exactly, and if the audio processing takes place in the main thread, you have no way of knowing when the callbacks in the worker occur. Hence you have to devise your own callback system to sync with the one going on in the worker, and send data over to the worker using postMessage, being a very ineffective solution for a case that's already very vulnerable. Not to mention that it's difficult to implement without ending up with weird edge-case race conditions.

> I'm fairly certain it would fail horribly in Opera, as other pages running in
> the same process can't be expected to write code to avoid long-running scripts
> or expensive re-layouts.

Of course, it's impossible to predict what's going on other pages, but that applies to drawing and other things as well, to achieve the best results, users have to close other tabs unless the browser has some multi-threading going on with different tabs.

But I beg to differ that Opera would fail horribly. In my sink.js [1] (a library to allow raw access to audio cross-browser), I have a fallback using the audio tag, you can take a look at an example of how it runs on Opera here [2] (a demo I made for last Christmas). The result is bearable, even though wav conversion and data URI conversion sucks the CPU dry. There are glitches every 0.5s or so, due to switching the audio tag but that's only because the onended event triggering the next clip fires a significant time after the audio has actually finished playing.

[1] https://github.com/jussi-kalliokoski/sink.js
[2] http://audiolibjs.org/examples/bells.html

(In reply to comment #7)
> > That's not how a worker-based AudioNode would work, it would be a callback in
> > the worker that can read directly from the input and write directly to the
> > output.
> 
> Exactly, and if the audio processing takes place in the main thread, you have
> no way of knowing when the callbacks in the worker occur. Hence you have to
> devise your own callback system to sync with the one going on in the worker,
> and send data over to the worker using postMessage, being a very ineffective
> solution for a case that's already very vulnerable. Not to mention that it's
> difficult to implement without ending up with weird edge-case race conditions.

The solution is to not do audio processing in the main thread and to post the state needed to do it in the worker instead. This seems trivial to me, do you have a real-world example where it is not?

> > I'm fairly certain it would fail horribly in Opera, as other pages running in
> > the same process can't be expected to write code to avoid long-running scripts
> > or expensive re-layouts.
> 
> Of course, it's impossible to predict what's going on other pages, but that
> applies to drawing and other things as well, to achieve the best results, users
> have to close other tabs unless the browser has some multi-threading going on
> with different tabs.
> 
> But I beg to differ that Opera would fail horribly. In my sink.js [1] (a
> library to allow raw access to audio cross-browser), I have a fallback using
> the audio tag, you can take a look at an example of how it runs on Opera here
> [2] (a demo I made for last Christmas). The result is bearable, even though wav
> conversion and data URI conversion sucks the CPU dry. There are glitches every
> 0.5s or so, due to switching the audio tag but that's only because the onended
> event triggering the next clip fires a significant time after the audio has
> actually finished playing.
> 
> [1] https://github.com/jussi-kalliokoski/sink.js
> [2] http://audiolibjs.org/examples/bells.html

That's really cool, but not at all the same. If you generate 500 ms second chunks of audio, blocking the main thread with layout for 100 ms is not a problem. With the current JavaScriptAudioNode the block size can go as small as 256, which is only 5 ms at 48Khz. To never block the main thread for more than 5 ms is not a guarantee we can make.

To get an idea of how long a layout reflow can take, visit http://www.whatwg.org/specs/web-apps/current-work/ wait for it to load and then run javascript:alert(opera.reflowCount + ' reflows in ' + opera.reflowTime + ' ms')

On my very fast developer machine, the results are:

22 reflows in 19165.749624967575 ms

That means that in the best case (all reflows took the same amount of time) the longest reflow was 871 ms.

With https://dvcs.w3.org/hg/audio/raw-file/tip/webaudio/specification.html I got these results:

7 reflows in 79.2522840499878 ms

Unfortunately the longest reflow isn't exposed, but even if it's "only" 12 ms that means that it's not hard to go way beyond that on a slower machine, like a smartphone.

(In reply to comment #8)
> (In reply to comment #7)
> > > That's not how a worker-based AudioNode would work, it would be a callback in
> > > the worker that can read directly from the input and write directly to the
> > > output.
> > 
> > Exactly, and if the audio processing takes place in the main thread, you have
> > no way of knowing when the callbacks in the worker occur. Hence you have to
> > devise your own callback system to sync with the one going on in the worker,
> > and send data over to the worker using postMessage, being a very ineffective
> > solution for a case that's already very vulnerable. Not to mention that it's
> > difficult to implement without ending up with weird edge-case race conditions.
> 
> The solution is to not do audio processing in the main thread and to post the
> state needed to do it in the worker instead. This seems trivial to me, do you
> have a real-world example where it is not?

No, I'm afraid I don't, and there probably aren't too many around (anymore, developers know better these days). But the emulator case still stands, there are already emulator environments written in JS. [1] [2] [3] [4]

> That's really cool, but not at all the same. If you generate 500 ms second
> chunks of audio, blocking the main thread with layout for 100 ms is not a
> problem. With the current JavaScriptAudioNode the block size can go as small as
> 256, which is only 5 ms at 48Khz. To never block the main thread for more than
> 5 ms is not a guarantee we can make.

Obviously the developers need to adjust their buffer sizes to work in the main thread, buffer sizes of 256 samples in the main thread with JS are (for now) a bit unrealistic — given the complexity and threading that comes with this API — and fails currently in Chrome as well (although I'm not sure why, the CPU usage is only 2% or so, but I suppose this is a thread communication latency issue), in fact any buffer size under 2048 makes the JSNode glitch horribly. If the developer expects her application to work in a mobile phone as well, she'll have to adjust that buffer size further. Indeed, I once proposed that the buffer size argument of the JSNode would be optional so that the browser could make a best approximation of what kind of a buffer size could be handled on a given setup. [5]

It's not like we can prevent people from doing their audio processing in the main thread. What we can do, however, is give them proper tools to do that in a minimally disruptive way for the user experience.

[1] http://fir.sh/projects/jsnes/
[2] http://gamecenter.grantgalitz.org/
[3] http://www.kingsquare.nl/jsc64
[4] http://bellard.org/jslinux/ (has no use case for audio - yet)
[5] http://lists.w3.org/Archives/Public/public-audio/2012AprJun/0106.html

Not quite the same thing as audio processing, but we're trying to limit
all the new XHR features in main thread to async only. Sync is occasionally easier
to use, but it will just fail (cause bad user experience) in the main thread.
(And yes, the change to limit certain sync behavior to workers only broke
various libraries.)

Similarly, JS audio processing is guaranteed to fail on the main thread
in reasonable common cases. So, IMHO, all the JS audio processing
should happen in background threads.

(In reply to comment #11)
> Not quite the same thing as audio processing, but we're trying to limit
> all the new XHR features in main thread to async only. Sync is occasionally
> easier
> to use, but it will just fail (cause bad user experience) in the main thread.
> (And yes, the change to limit certain sync behavior to workers only broke
> various libraries.)
> 
> Similarly, JS audio processing is guaranteed to fail on the main thread
> in reasonable common cases. So, IMHO, all the JS audio processing
> should happen in background threads.

I agree it should, but I don't think it will. What should an emulator/VM developer do? Render off main-thread as well? MSP API would have been a perfect fit for that use case, given it's ability to process video as well... Analyzing the byte code of those existing games and other programs and isolating the audio code to another thread doesn't sound very feasible.

(In reply to comment #12)
> I agree it should, but I don't think it will. What should an emulator/VM
> developer do? Render off main-thread as well? 
Probably

> MSP API would have been a perfect
> fit for that use case, given it's ability to process video as well... Analyzing
> the byte code of those existing games and other programs and isolating the
> audio code to another thread doesn't sound very feasible.
That is a limitation in the WebAudio API which should be fixed.

There has been a lot of discussion about a Worker-accessible canvas API and I expect one to be created fairly soon. Then we'll have a solution for VMs and emulators that really works. I agree that providing main-thread JS audio processing is just setting authors up to fail.

(In reply to comment #12)
> I agree it should, but I don't think it will. What should an emulator/VM
> developer do? Render off main-thread as well?

I guess it depends a lot on what kind of system you want to emulate, and to what extent you need CPU cycle exact state coherence (e.g. do you want to emulate a few popular games, or do you want to implement a fully functional virtualization of a machine?).

For instance for implementing a SID chip for a C=64 emulator, I'd imagine that you can simply post time-stamped messages from the main thread to an audio worker (possibly batched as a time-stamped command buffer per frame or whatever), where the worker implements all SID logic. A similar solution should work for the NES sound HW too, I guess.

For emulating the Paula chip on the Amiga, you'd have more problems since it uses DMA for accessing CPU-shared memory. On the other hand, I think you should be able to come quite far by setting up a node graph that effectively emulates the Paula chip using automation for timing etc, eliminating a lot of the problems that you would otherwise have with a 100% JS-based mixer.

In any event, this does strike me as a show-stopper for having audio processing in workers. Especially given that machine emulators are usually quite odd, both in terms of architecture and actual use cases.

(In reply to comment #15)
> In any event, this does strike me as a show-stopper for having audio processing
> in workers.

As Philip pointed out: should read "this does NOT strike me as a show-stopper" ;)

(In reply to comment #16)
> (In reply to comment #15)
> > In any event, this does strike me as a show-stopper for having audio processing
> > in workers.
> 
> As Philip pointed out: should read "this does NOT strike me as a show-stopper"
> ;)

All right, I'm running out of points to defend my case, especially since I don't have a horse of my own in the race. :) And if it's one or the other, I prefer the audio processing is done only in workers instead of the main thread (obviously), but I still think it'd be wise to have both.

(In reply to comment #17)
> (In reply to comment #16)
> > (In reply to comment #15)
> > > In any event, this does strike me as a show-stopper for having audio processing
> > > in workers.
> > 
> > As Philip pointed out: should read "this does NOT strike me as a show-stopper"
> > ;)
> 
> All right, I'm running out of points to defend my case, especially since I
> don't have a horse of my own in the race. :) And if it's one or the other, I
> prefer the audio processing is done only in workers instead of the main thread
> (obviously), but I still think it'd be wise to have both.

I agree with Jussi.  Quite honestly it's a lot simpler for developers to have access to the complete JS state while doing the processing.  If people are willing to work with larger buffer sizes, then quite reasonable things can be done in the main thread.

If so, then "larger buffer sizes" should be a hard requirement. On my fairly powerful desktop computer a layout could block for at least 871 ms. The closest power-of-two at 48Khz is 65536, i.e. over a second. With that amount of latency it doesn't seems very useful.

(In reply to comment #19)
> If so, then "larger buffer sizes" should be a hard requirement. On my fairly
> powerful desktop computer a layout could block for at least 871 ms. The closest
> power-of-two at 48Khz is 65536, i.e. over a second. With that amount of latency
> it doesn't seems very useful.

What? Why would it be a hard limit? Hard limits aren't very future-friendly. Should setTimeout have a minimum timeout limit of 871ms as well? Or requestAnimationFrame?

Developers have to be conscious about performance and avoiding layout reflows anyway, why should this API be any different?

(In reply to comment #20)
> Developers have to be conscious about performance and avoiding layout reflows
> anyway, why should this API be any different?

One problem is that developers don't control all the pages that might possibly be sharing the same thread. No browser puts every page on its own thread. So even if you write your page perfectly, you're still vulnerable to latency caused by poorly-written pages sharing your main thread.

(In reply to comment #20)
> Developers have to be conscious about performance and avoiding layout reflows
> anyway, why should this API be any different?

I'd also like to add to this discussion that you can't really compare glitches in graphics/animation to glitches in audio. In general we (humans) are much more sensitive to glitches in audio than to frame drops in animation. You can usually get away with a 100 ms loss in an animation every now and then, but you can't as easily get away with a 1 ms glitch in your audio.

Most systems (DVD, DVB etc) prioritize audio over video. This can be seen when switching channels on some TV boxes for instance, where video stutters into sync with the continuous audio - it's hardly noticeable, but it would be horrible if it was the other way around (stuttering audio).

In other words, an audio API should provide continuous operation even under conditions when a graphics API fail to do so.

(In reply to comment #21)
> (In reply to comment #20)
> > Developers have to be conscious about performance and avoiding layout reflows
> > anyway, why should this API be any different?
> 
> One problem is that developers don't control all the pages that might possibly
> be sharing the same thread. No browser puts every page on its own thread. So
> even if you write your page perfectly, you're still vulnerable to latency
> caused by poorly-written pages sharing your main thread.

Of course, but this argument is just as valid against having an audio API at all, after all the developer can't anticipate what else is running on the user's computer aside from the browser. For all the developer knows, the API might be running on a mobile browser with all cores (or maybe just one) busy. Throwing more threads at it doesn't necessarily solve the problem of not being able to anticipate all situations.

(In reply to comment #22)
> (In reply to comment #20)
> > Developers have to be conscious about performance and avoiding layout reflows
> > anyway, why should this API be any different?
> 
> I'd also like to add to this discussion that you can't really compare glitches
> in graphics/animation to glitches in audio. In general we (humans) are much
> more sensitive to glitches in audio than to frame drops in animation. You can
> usually get away with a 100 ms loss in an animation every now and then, but you
> can't as easily get away with a 1 ms glitch in your audio.
> 
> Most systems (DVD, DVB etc) prioritize audio over video. This can be seen when
> switching channels on some TV boxes for instance, where video stutters into
> sync with the continuous audio - it's hardly noticeable, but it would be
> horrible if it was the other way around (stuttering audio).
> 
> In other words, an audio API should provide continuous operation even under
> conditions when a graphics API fail to do so.

Yes, this is why it is preferable to run audio in a real time / priority thread where possible, but it's not always possible, maybe due to the system or the nature of the application.

(In reply to comment #23)
> (In reply to comment #21)
> Of course, but this argument is just as valid against having an audio API at
> all, after all the developer can't anticipate what else is running on the
> user's computer aside from the browser. For all the developer knows, the API
> might be running on a mobile browser with all cores (or maybe just one) busy.
> Throwing more threads at it doesn't necessarily solve the problem of not being
> able to anticipate all situations.

True, but there is a significant difference between running several threads on a single core (preemptive scheduling should give any thread CPU quite often), and running several pages in a single thread (a callback may have to wait for seconds).

(In reply to comment #25)
> (In reply to comment #23)
> > (In reply to comment #21)
> > Of course, but this argument is just as valid against having an audio API at
> > all, after all the developer can't anticipate what else is running on the
> > user's computer aside from the browser. For all the developer knows, the API
> > might be running on a mobile browser with all cores (or maybe just one) busy.
> > Throwing more threads at it doesn't necessarily solve the problem of not being
> > able to anticipate all situations.
> 
> True, but there is a significant difference between running several threads on
> a single core (preemptive scheduling should give any thread CPU quite often),
> and running several pages in a single thread (a callback may have to wait for
> seconds).

Still, no matter what we do, in some cases audio will not work as expected, will miss refills, and there's nothing we can do about it. What we can do, however, is give the proper tools to handle these situations, including main thread audio processing that doesn't have to resort to manually transferring audio to a worker (or graphics to the main thread either), because that's even more expensive and likely to fail.

(In reply to comment #26)
> 
> Still, no matter what we do, in some cases audio will not work as expected,
> will miss refills, and there's nothing we can do about it. What we can do,
> however, is give the proper tools to handle these situations,
Yes


> including main
> thread audio processing 
Why. This is what we should probably explicitly prevent to increase the
likelihood for well-designed apps.

I definitely favor helping developers "do the right thing", so I also prefer focusing exclusively on JavaScriptAudioNode processing in workers (instead of the main thread). Regardless, we all seem to agree that support for doing processing in workers needs added. 

If we do decide to support both, I suggest requiring some sort of explicit opt-in for running on the main thread. This way developers would at least be less likely to gravitate to it by default.

(In reply to comment #26)
> Still, no matter what we do, in some cases audio will not work as expected,
> will miss refills, and there's nothing we can do about it.

With JS audio produced in Workers, the browser should be able to make audio work reliably in any situation short of complete overload of the device.

With JS audio on the main thread, audio will start failing as soon as a badly-behaving page happens to share the same main thread as the audio page.

That is a huge difference.

> What we can do,
> however, is give the proper tools to handle these situations, including main
> thread audio processing that doesn't have to resort to manually transferring
> audio to a worker (or graphics to the main thread either), because that's even
> more expensive and likely to fail.

For use-cases such as "run an emulator producing sound and graphics", the best solution is to provide Worker access to canvas as well as audio. Then you can have reliable audio and a reliable frame rate as well.

Are there any other use-cases that are problematic for audio production in Workers?

(In reply to comment #29)
> (In reply to comment #26)
> > Still, no matter what we do, in some cases audio will not work as expected,
> > will miss refills, and there's nothing we can do about it.
> 
> With JS audio produced in Workers, the browser should be able to make audio
> work reliably in any situation short of complete overload of the device.
> 
> With JS audio on the main thread, audio will start failing as soon as a
> badly-behaving page happens to share the same main thread as the audio page.
> 
> That is a huge difference.
> 
> > What we can do,
> > however, is give the proper tools to handle these situations, including main
> > thread audio processing that doesn't have to resort to manually transferring
> > audio to a worker (or graphics to the main thread either), because that's even
> > more expensive and likely to fail.
> 
> For use-cases such as "run an emulator producing sound and graphics", the best
> solution is to provide Worker access to canvas as well as audio. Then you can
> have reliable audio and a reliable frame rate as well.
> 
> Are there any other use-cases that are problematic for audio production in
> Workers?

If Workers get access to canvas, at least I can't think of a (valid) reason why anyone would want to process audio in the main thread. :)

(In reply to comment #28)
> I definitely favor helping developers "do the right thing", so I also prefer
> focusing exclusively on JavaScriptAudioNode processing in workers (instead of
> the main thread). Regardless, we all seem to agree that support for doing
> processing in workers needs added. 
> If we do decide to support both, I suggest requiring some sort of explicit
> opt-in for running on the main thread. This way developers would at least be
> less likely to gravitate to it by default.

+1

Yeah, I can tell this talk about an emulator producing audio on the main thread and sending off the audio data to a worker is related to JS GBC using XAudioJS with the MediaStream Processing API in use. :P

I personally have to use the main thread for compatibility reasons with legacy browsers lacking web worker support. Essentially creating a second version of the emulator for webworker capable browsers seems like a big hack. Sending audio off from the main thread to the worker is very easy to do, but the i/o lag is off the charts (almost a half a second in some cases), as seen with experimentation with the mediastream api.

To see what it would look like to sync audio from the main thread to the worker:
main thread: https://github.com/grantgalitz/XAudioJS/blob/master/XAudioServer.js#L142
and
https://github.com/grantgalitz/XAudioJS/blob/master/XAudioServer.js#L406

worker: https://github.com/grantgalitz/XAudioJS/blob/master/XAudioServerMediaStreamWorker.js

The question to be asked: Why not allow the js developer to select either a web worker or main thread usage for outputting audio? Locking audio to one or the other seems to only limit options here.

The whole discussion about the main thread being very poor for audio streaming seems to be somewhat overplayed. Proper web applications that use the main thread should ration their event queue properly. We already have better audio stream continuity than some native apps: http://www.youtube.com/watch?v=H7vt5svSJiE

> For use-cases such as "run an emulator producing sound and graphics", the best
> solution is to provide Worker access to canvas as well as audio. Then you can
> have reliable audio and a reliable frame rate as well.


What about lack of hardware acceleration? That'll block audio as long as gfx is on the same thread as audio, which in my case is a must for low lag sync.

(In reply to comment #33)
> The whole discussion about the main thread being very poor for audio streaming
> seems to be somewhat overplayed. Proper web applications that use the main
> thread should ration their event queue properly.

See comment #21.

(In reply to comment #34)
> > For use-cases such as "run an emulator producing sound and graphics", the best
> > solution is to provide Worker access to canvas as well as audio. Then you can
> > have reliable audio and a reliable frame rate as well.
> 
> What about lack of hardware acceleration? That'll block audio as long as gfx is
> on the same thread as audio, which in my case is a must for low lag sync.

If you must have audio on the same thread as graphics, and that causes glitches because the device is simply overloaded, then so be it; that is unavoidable. See comment #29.

An even bigger blocker for me actually is the keyboard events. I cooked up a webworker based version of JS GBC long ago and noticed at the time that I had to pass keyboard event notifications to the webworker, causing one to experience lag with input as well. It's the being forced to pipe so much from the UI to the worker that makes webworker usage for my case entirely pointless at the moment. A webworker is going to need access to almost as many APIs as the main thread if we're going to perform real time processing with low latency that involves various forms of dynamic input. Adding thousands of lines of code to make a single-threaded application perform worse in a web worker versus the UI thread seems silly to me.

Also the UI thread being blocked from other tabs is experienced on a per-browser basis for me at least, with Chrome in the clear actually. I do experience things like js timers dropping from 4 ms to 500 ms intervals when I do things like try to hover over the mac os x dock or change the volume, so forcing multi-tasking within the js environment via workers seems like the "fix" for firefox at least.

(In reply to comment #36)
> Also the UI thread being blocked from other tabs is experienced on a
> per-browser basis for me at least, with Chrome in the clear actually.

It may depend on how many tabs you have open and what they contain, but sooner or later Chrome will put multiple tabs onto the same main thread.

(In reply to comment #37)
> (In reply to comment #36)
> > Also the UI thread being blocked from other tabs is experienced on a
> > per-browser basis for me at least, with Chrome in the clear actually.
> 
> It may depend on how many tabs you have open and what they contain, but sooner
> or later Chrome will put multiple tabs onto the same main thread.

True

To sum up my argument: We are currently not provided all the resources inside a webworker to attain full independence from the main thread yet, and dependence on the main thread kills us with i/o lag.

(In reply to comment #38)
> To sum up my argument: We are currently not provided all the resources inside a
> webworker to attain full independence from the main thread yet, and dependence
> on the main thread kills us with i/o lag.

A possible solution for emulators like the JS GBC emulator (which I really like!): move the audio emulation part to a Web worker (making it independent from the main thread), and post time-stamped audio HW register writes from the main thread to the audio Web worker (should be quite compact data?). That way you would be glitch free even in cases of i/o lag. This assumes that you can do the audio HW emulation independently from the rest of the machine, but for simple "8-bit" sound HW, I think it can easily be done.

(In reply to comment #39)
> (In reply to comment #38)
> > To sum up my argument: We are currently not provided all the resources inside a
> > webworker to attain full independence from the main thread yet, and dependence
> > on the main thread kills us with i/o lag.
> 
> A possible solution for emulators like the JS GBC emulator (which I really
> like!): move the audio emulation part to a Web worker (making it independent
> from the main thread), and post time-stamped audio HW register writes from the
> main thread to the audio Web worker (should be quite compact data?). That way
> you would be glitch free even in cases of i/o lag. This assumes that you can do
> the audio HW emulation independently from the rest of the machine, but for
> simple "8-bit" sound HW, I think it can easily be done.

One possible problem with that is we would need to synchronize up to once every two clocks at a sample rate of 4194304 hertz (One sample per clock cycle for LLE emulation). We also would need to synchronize back part of the audio state, as we expose some of the state back in the emulated hardware registers.

(In reply to comment #40)
> One possible problem with that is we would need to synchronize up to once every
> two clocks at a sample rate of 4194304 hertz (One sample per clock cycle for
> LLE emulation).

My idea here would be to use time-stamped commands (such as CYCLE:REGISTER=VALUE), and batch up the commands in a typed array buffer that is flushed (sent to the worker) once per frame for example. The worker could run at its own clock with a small latency w.r.t the main thread (I doubt that it would be noticeable).

> We also would need to synchronize back part of the audio state,
> as we expose some of the state back in the emulated hardware registers.

Read-back is trickier of course. I know very little about the GB/GBC hardware. My experience is mostly from the SID chip (from the C=64), which allows you to read back values from one of the oscillators. Here, I think I would just emulate a very small sub-set of the SID chip in the main thread (those readable registers were almost never used, and if they were used, it would typically be for random number generation).

Anyway, these were just some ideas. Perhaps worth trying, perhaps not...

(In reply to comment #41)
> (In reply to comment #40)
> > One possible problem with that is we would need to synchronize up to once every
> > two clocks at a sample rate of 4194304 hertz (One sample per clock cycle for
> > LLE emulation).
> 
> My idea here would be to use time-stamped commands (such as
> CYCLE:REGISTER=VALUE), and batch up the commands in a typed array buffer that
> is flushed (sent to the worker) once per frame for example. The worker could
> run at its own clock with a small latency w.r.t the main thread (I doubt that
> it would be noticeable).
> 
> > We also would need to synchronize back part of the audio state,
> > as we expose some of the state back in the emulated hardware registers.
> 
> Read-back is trickier of course. I know very little about the GB/GBC hardware.
> My experience is mostly from the SID chip (from the C=64), which allows you to
> read back values from one of the oscillators. Here, I think I would just
> emulate a very small sub-set of the SID chip in the main thread (those readable
> registers were almost never used, and if they were used, it would typically be
> for random number generation).
> 
> Anyway, these were just some ideas. Perhaps worth trying, perhaps not...

Except latency for reading back would quite literally kill the emulator. Also the emulator checks how many samples it's under running by in real time do it can run more clocks over the base clocking amount to make sure we actually run 100% speed without this speed adjustment via listening to the sample count, browsers would be underrunning the audio like crazy, as all browsers cheat setInterval timing it seems (aka never call it enough even with extra CPU time free). This was a major reason for the single threaded design, as async buffering from the main thread causes us lag in the speed determination portion, which is unacceptable. It's this measurement that also allows us to prioritize CPU over graphics when hitting the 100% CPU usage wall. We do frame skip by holding off blitting until the end of iteration.

(In reply to comment #42)
> ...

Fair enough.

A bit off topic, but could we add nodes that generate specific waveforms? Like a sine/triangle/square/sawtooth/LSFR White Noise generator node?

Typo: meant to say LFSR instead ( http://en.wikipedia.org/wiki/Linear_feedback_shift_register )

Reducing the bitwidth of the LSFR results in interesting effects some sound devs might like. As a bonus, programmable audio has a *very* download low bandwidth cost, so it's optimal for instantly loading apps.

And obviously an arbitrary length waveform buffer bank node would be awesome as well (So the dev can create custom waveforms themselves to repeat over and over).

I know we have normal nodes for audio buffers, but can we do direct access on them?

Hi Grant,

(In reply to comment #44)
> A bit off topic, but could we add nodes that generate specific waveforms? Like
> a sine/triangle/square/sawtooth/LSFR White Noise generator node?

I think you are looking for https://dvcs.w3.org/hg/audio/raw-file/tip/webaudio/specification.html#Oscillator

Ideally, if you have any comment not related to a specific issue, discussion on public-audio@w3.org is a better idea than hijacking the comment thread ;)

(In reply to comment #48)
> Hi Grant,
> 
> (In reply to comment #44)
> > A bit off topic, but could we add nodes that generate specific waveforms? Like
> > a sine/triangle/square/sawtooth/LSFR White Noise generator node?
> 
> I think you are looking for
> https://dvcs.w3.org/hg/audio/raw-file/tip/webaudio/specification.html#Oscillator
> 
> Ideally, if you have any comment not related to a specific issue, discussion on
> public-audio@w3.org is a better idea than hijacking the comment thread ;)

Oh wow, I never noticed you had that implemented already. I'm wondering where the white noise osc is?

Grant, it seems to me that there are at least two options for main-thread audio generation even if there's no JavaScriptAudioNode.

1. Generate your audio into AudioBuffers and schedule these to play back-to-back with AudioBufferSoruceNodes. (I haven't tried if the WebKit implementation handles this gapless, but I don't see why we shouldn't support this in the spec.)

2. Generate your audio into AudioBuffers and postMessage these to a WorkerAudioNode. If ownership of the buffer is transferred it should be cheap and there's no reason why this should incur a large delay, particularly not half a second like you've seen. That sounds like a browser bug to be fixed.

In both cases one will have one new object per buffer to GC, in the first case it's a AudioBufferSourceNode and in the second case it's the event object on the worker side.

(In reply to comment #50)
> Grant, it seems to me that there are at least two options for main-thread audio
> generation even if there's no JavaScriptAudioNode.
> 
> 1. Generate your audio into AudioBuffers and schedule these to play
> back-to-back with AudioBufferSoruceNodes. (I haven't tried if the WebKit
> implementation handles this gapless, but I don't see why we shouldn't support
> this in the spec.)
> 
> 2. Generate your audio into AudioBuffers and postMessage these to a
> WorkerAudioNode. If ownership of the buffer is transferred it should be cheap
> and there's no reason why this should incur a large delay, particularly not
> half a second like you've seen. That sounds like a browser bug to be fixed.
> 
> In both cases one will have one new object per buffer to GC, in the first case
> it's a AudioBufferSourceNode and in the second case it's the event object on
> the worker side.

Option 2 is not viable, due to I/O lag between webworkers and the main thread. I tried webworker audio gen with the MediaStreamProcessing API (Experimental API by roc, he even had builds for it) and sent buffers from main->worker and latency was around 1/3 a second or more.

Option 1 does not make the situation for gapless audio any better here. We're just making it harder to push out audio. The browser knows best when to fire audio refills. Forcing the JS code to schedule audio will make audio buffering and drop outs worse.

There's always some lag with comm to a webworker, and losing the direct ability to be called for refill on the same thread for audio makes the entire web app vulnerable to lag spikes in webworker communication. I saw this with the MediaStream API, where sync backs would be bunched up too much causing too much inconsistency and thus gaps.

(In reply to comment #51)
> Option 1 does not make the situation for gapless audio any better here. We're
> just making it harder to push out audio. The browser knows best when to fire
> audio refills. Forcing the JS code to schedule audio will make audio buffering
> and drop outs worse.

I don't follow this. You're already using some JS scheduling to drive the progress of the emulator (requestAnimationFrame? setInterval?). Each step of the emulator generates some audio samples that you want to play back as soon as possible. So stuffing those samples into an output pipe somehow should be good for you.

Or are you actually trying to drive the progress of the emulator off the audio clock somehow?

(In reply to comment #53)
> (In reply to comment #51)
> > Option 1 does not make the situation for gapless audio any better here. We're
> > just making it harder to push out audio. The browser knows best when to fire
> > audio refills. Forcing the JS code to schedule audio will make audio buffering
> > and drop outs worse.
> 
> I don't follow this. You're already using some JS scheduling to drive the
> progress of the emulator (requestAnimationFrame? setInterval?). Each step of
> the emulator generates some audio samples that you want to play back as soon as
> possible. So stuffing those samples into an output pipe somehow should be good
> for you.
> 
> Or are you actually trying to drive the progress of the emulator off the audio
> clock somehow?

I drive off both. I use the audio clock and setInterval. If I used setInterval, almost every browser wouldn't run the code enough, so the right thing to do is to drive with the audio clock in addition to the setInterval.

What frequency do you need to run the code at? With setInterval you should be able to get 4ms.

(In reply to comment #55)
> What frequency do you need to run the code at? With setInterval you should be
> able to get 4ms.

I use a ms setInterval as the base driver. The rate of the setInterval is not the problem. The problem lies with the fact that setinterval was designed, rightfully so, to skip a call if there was a buildup of delays. Audio gen by this method should never be conducted without audio clok heurisitics as a result of this. Proper audio gen should always check to see the respective buffering levels, as a system can skip a setInterval callback due to unexpected latency. Also some browsers, like google chrome, have A LOT of jitter in their setInterval and skip an unnecessarily large amout of setInterval calls. If I disabled audio clock input, we would be hearing a lot of crackles and pops across the board in every browser if we simply just timed by setInterval.

*I use a 4 ms timer already, in addition to the less often occurring web audio callbacks for 2048 samples frames (this is also the min frames per callback adobe flash uses).

Is the writing part of the Audio Data API --- mozSetup, mozWriteAudio, mozCurrentSampleOffset --- what you really want?

(In reply to comment #51)
> Option 1 does not make the situation for gapless audio any better here. We're
> just making it harder to push out audio. The browser knows best when to fire
> audio refills. Forcing the JS code to schedule audio will make audio buffering
> and drop outs worse.

It seems to me that you're not really interested in doing audio *processing* in the audio callback (which is what it was designed for). Am I right in assuming that you're looking for some kind of combination of an audio data push mechanism and a reliable event mechanism for guaranteeing that you push often enough?

AFAICT, the noteOn & AudioParam interfaces were designed for making it possible to schedule sample accurate audio actions ahead of time. I think that it *should* be possible to use it for providing gap-less audio playback (typically using a few AudioBuffers in a multi-buffering manner and scheduling them with AudioBufferSourceNodes). The problem, as it seems, is that you need to accommodate for possible jittering and event drops, possibly by introducing a latency (e.g, would it work if you forced a latency of 0.5s?).

Would the following be a correct conclusion?:

- Audio processing in JavaScript should be done in workers.
- We need a reliable main-context event system for scheduling audio actions (setInterval is not up to it, it seems).

> - We need a reliable main-context event system for scheduling audio actions
> (setInterval is not up to it, it seems).

There can't be any truly reliable way to schedule audio actions on the main thread.

I doubt you can do better than setInterval plus some way to measure audio progress, such as mozCurrentSampleOffset.

(In reply to comment #52)
> There's always some lag with comm to a webworker, and losing the direct ability
> to be called for refill on the same thread for audio makes the entire web app
> vulnerable to lag spikes in webworker communication. I saw this with the
> MediaStream API, where sync backs would be bunched up too much causing too much
> inconsistency and thus gaps.

I haven't looked into the worker communication lag part, but I don't really see a fundamental reason why it has to be a problem.

On the other hand, doing audio callbacks on the main thread will always be problematic, and I don't see how it could ever be solved. I envision a design where the audio mixer is running in a separate thread (or possibly multiple threads for graphs with independent processing chains). Whenever a node is encountered that needs data from a main thread callback, the mixing thread will have to halt and wait until the main thread callback has fired and finished, which is very sensitive to other things that may be going on in the main thread (e.g. a setInterval/requestAnimationFrame event).

I'd much more prefer a solution that does not require the audio mixing thread to be halted just to fire an event on the main thread.

(In reply to comment #59)
> Would the following be a correct conclusion?:
> 
> - Audio processing in JavaScript should be done in workers.
[…]

I think that is a reasonable conclusion, yes - although there may be objections by e.g Philip who seemed to be quite adamant about it (See Comment #2). 

Any more suggestions on how this could be made obvious to developers (as you mentioned in Comment #4)? Would it suffice to have the default interface be constructed in the way suggested by Chris in http://lists.w3.org/Archives/Public/public-audio/2012JanMar/0225.html - and requiring a conscious choice *not* to use a worker?

(In reply to comment #62)
 - and
> requiring a conscious choice *not* to use a worker?

That tends to not work. If some API is available, whether it is bad or not, 
it will be used. (sync XHR in the main thread is a good example)

(In reply to comment #62)
> (In reply to comment #59)
> > Would the following be a correct conclusion?:
> > 
> > - Audio processing in JavaScript should be done in workers.
> […]
> 
> I think that is a reasonable conclusion, yes - although there may be objections
> by e.g Philip who seemed to be quite adamant about it (See Comment #2). 

Nope, I think that JavaScript processing should be done in workers, and only workers.

(In reply to comment #64)
> > (In reply to comment #59)
> > > Would the following be a correct conclusion?:
> > > 
> > > - Audio processing in JavaScript should be done in workers.
> > […]
> > 
> > I think that is a reasonable conclusion, yes - although there may be objections
> > by e.g Philip who seemed to be quite adamant about it (See Comment #2). 
> 
> Nope, I think that JavaScript processing should be done in workers, and only
> workers.

I might have misunderstood Marcus' point then. 

Marcus, was your conclusion that it should be done in workers (and only there, hence a MUST) or that it should be done in workers (and could be done in mail thread at your own risks)?

(In reply to comment #65)
> (In reply to comment #64)
> > > (In reply to comment #59)
> > > > Would the following be a correct conclusion?:
> > > > 
> > > > - Audio processing in JavaScript should be done in workers.
> > > […]
> > > 
> > > I think that is a reasonable conclusion, yes - although there may be objections
> > > by e.g Philip who seemed to be quite adamant about it (See Comment #2). 
> > 
> > Nope, I think that JavaScript processing should be done in workers, and only
> > workers.
> 
> I might have misunderstood Marcus' point then. 
> 
> Marcus, was your conclusion that it should be done in workers (and only there,
> hence a MUST) or that it should be done in workers (and could be done in mail
> thread at your own risks)?

Sorry for the vague language. What I meant was that JavaScript processing MUST be done in workers, and only in workers, never in the main thread.

Marcus wrote (In reply to comment #66):
> Sorry for the vague language. What I meant was that JavaScript processing MUST
> be done in workers, and only in workers, never in the main thread.

Ah, OK, thanks for the clarification. I guess we do have something approaching consensus here. Jussi seemed unsure at first but given Comment #30, the doubts seem to have been dispelled. Ditto for Grant. 

I will send a Call for Consensus on this to the list early next week - in particular I'd like to give ChrisR a chance to develop on what he said in Comment #3: “Some developers have expressed concerns that JavaScriptAudioNode *only* happens in workers.”

(In reply to comment #58)
> Is the writing part of the Audio Data API --- mozSetup, mozWriteAudio,
> mozCurrentSampleOffset --- what you really want?

Yes

(In reply to comment #59)
> (In reply to comment #51)
> > Option 1 does not make the situation for gapless audio any better here. We're
> > just making it harder to push out audio. The browser knows best when to fire
> > audio refills. Forcing the JS code to schedule audio will make audio buffering
> > and drop outs worse.
> 
> It seems to me that you're not really interested in doing audio *processing* in
> the audio callback (which is what it was designed for). Am I right in assuming
> that you're looking for some kind of combination of an audio data push
> mechanism and a reliable event mechanism for guaranteeing that you push often
> enough?
> 
> AFAICT, the noteOn & AudioParam interfaces were designed for making it possible
> to schedule sample accurate audio actions ahead of time. I think that it
> *should* be possible to use it for providing gap-less audio playback (typically
> using a few AudioBuffers in a multi-buffering manner and scheduling them with
> AudioBufferSourceNodes). The problem, as it seems, is that you need to
> accommodate for possible jittering and event drops, possibly by introducing a
> latency (e.g, would it work if you forced a latency of 0.5s?).
No, 0.5 seconds is trash. Needs to be no worse than 100 ms or it sounds like poop to the user.
> 
> Would the following be a correct conclusion?:
> 
> - Audio processing in JavaScript should be done in workers.
> - We need a reliable main-context event system for scheduling audio actions
> (setInterval is not up to it, it seems).
Main thread needs access to audio clock to drive audio correctly. Doing audio generation by Date object is bad design. Need actual input from the browser that we played x number of samples just now. Making me generate multiple buffers to schedule with no respect to the machine's actual buffering seems bound to fail.

(In reply to comment #60)
> > - We need a reliable main-context event system for scheduling audio actions
> > (setInterval is not up to it, it seems).
> 
> There can't be any truly reliable way to schedule audio actions on the main
> thread.
> 
> I doubt you can do better than setInterval plus some way to measure audio
> progress, such as mozCurrentSampleOffset.

Funny story, mozCurrentSampleOffset is essentially non-functional on Linux Firefox. Timing audio off of it has been broken since it was introduced.

The callback for more samples method is best because it tells me exactly how many samples it needs. With mozCurrentSampleOffset and mozWriteAudio we were actually exposed to each OS's buffering edge cases, and still are! I have the user agent sniff for windows Firefox for instance to insert a shadow 100 ms of extra buffering otherwise windows Firefox stops the audio clock. This is in contrast to Mac Firefox being perfect at all buffer levels.

Why even a faulty mozCurrentSampleOffset is better than no buffering determination: Being forced to use time stamps to generate audio from the lack of audio clock access though would be much much worse, as things like sleep events and time zone changes will kill the timed logic.

(In reply to comment #61)
> (In reply to comment #52)
> > There's always some lag with comm to a webworker, and losing the direct ability
> > to be called for refill on the same thread for audio makes the entire web app
> > vulnerable to lag spikes in webworker communication. I saw this with the
> > MediaStream API, where sync backs would be bunched up too much causing too much
> > inconsistency and thus gaps.
> 
> I haven't looked into the worker communication lag part, but I don't really see
> a fundamental reason why it has to be a problem.
> 
> On the other hand, doing audio callbacks on the main thread will always be
> problematic, and I don't see how it could ever be solved. I envision a design
> where the audio mixer is running in a separate thread (or possibly multiple
> threads for graphs with independent processing chains). Whenever a node is
> encountered that needs data from a main thread callback, the mixing thread will
> have to halt and wait until the main thread callback has fired and finished,
> which is very sensitive to other things that may be going on in the main thread
> (e.g. a setInterval/requestAnimationFrame event).
> 
> I'd much more prefer a solution that does not require the audio mixing thread
> to be halted just to fire an event on the main thread.

Make a mozAudio like access to the API for the main thread. Only sync up when the main thread pushes new audio or requests for the buffering amount. Firefox is known to play samples even through GC pauses with mozAudio. It allows the js dev to pass samples to the browser AND let the browser handle the buffering for it as well without issuing callbacks.

Also why using the Date object is bad: the sample rate might not be exactly what is reported, so long running synth might be affected by the error margin. Only audio clock sniffing would account for this right (mozCurrentSampleOffet and the firing of each jsaudionode event).

Being that there's clock skew for the computer's own clock and that master clocks don't always divide evenly into 44100 hertz. Long running synths without actual audio clock metrics will be affected by date and time readjustment when the computer fixes its skewed clock by contacting an atomic clock date server.

(In reply to comment #64)
> (In reply to comment #62)
> > (In reply to comment #59)
> > > Would the following be a correct conclusion?:
> > > 
> > > - Audio processing in JavaScript should be done in workers.
> > […]
> > 
> > I think that is a reasonable conclusion, yes - although there may be objections
> > by e.g Philip who seemed to be quite adamant about it (See Comment #2). 
> 
> Nope, I think that JavaScript processing should be done in workers, and only
> workers.

Problem lies with the web worker being restricted from accessing all the APIs the main thread has access too. If I house everything in the worker, doing so will mess up some other components not audio related.

(In reply to comment #69)
> No, 0.5 seconds is trash. Needs to be no worse than 100 ms or it sounds like
> poop to the user.

Obviously. I meant it more as an exercise: How low can you practically go? (out of curiosity)

(In reply to comment #75)
> (In reply to comment #69)
> > No, 0.5 seconds is trash. Needs to be no worse than 100 ms or it sounds like
> > poop to the user.
> 
> Obviously. I meant it more as an exercise: How low can you practically go? (out
> of curiosity)

Doesn't matter, because we would not be continuing from the previous stream, and the lack of audio clock. Firing sample buffers based on time rather than audio clock is something I would never do. I'd rather pipe in audio to the web worker and force the users of browsers that have to do that experience high copy overhead for the piping with the audio lag. I'd always rather deal with crazy audio lag than arbitrarily schedule new audio that's not driven from previous stream endings.

Let's not forget that placing the js code in a worker does not solve the blocking issue. JS code can still be very inefficient in a worker and block the worker for a long period of time. GC pauses apply too. Inefficient JS can cause blocking for longer periods than what the UI would normally block for. Putting canvas support in a worker can also block out audio as well, as non-accelerated users are very common with most browsers today.

I'd recommend mozAudio style, with no callbacks, but rather a mozCurrentSampleOffet function to determine the audio clock position. This would allow the API to not have to wait on callbacks for a fill. Blocking would only occur at the write function and offset read back function.

The mozAudio style being for a special js node that only acts as a source rather than a processing node.

If you implement a mozAudio style API as a source only node, then I'd be fine with the js audio node being locked to a worker, as long as the mozAudio style API node is accessible from the main thread too.

(In reply to comment #59)
> AFAICT, the noteOn & AudioParam interfaces were designed for making it possible
> to schedule sample accurate audio actions ahead of time. I think that it
> *should* be possible to use it for providing gap-less audio playback (typically
> using a few AudioBuffers in a multi-buffering manner and scheduling them with
> AudioBufferSourceNodes). The problem, as it seems, is that you need to
> accommodate for possible jittering and event drops, possibly by introducing a
> latency (e.g, would it work if you forced a latency of 0.5s?).

I think a solution that uses setInterval() to schedule frequent JS callbacks, checks AudioContext.currentTime to measure the progress of the audio clock, and uses AudioBufferSourceNodes to queue playback of generated audio buffers, should work as well as any other, providing the Web Audio implementation is adequate. Incremental GC might also be required.

(In reply to comment #81)
> (In reply to comment #59)
> > AFAICT, the noteOn & AudioParam interfaces were designed for making it possible
> > to schedule sample accurate audio actions ahead of time. I think that it
> > *should* be possible to use it for providing gap-less audio playback (typically
> > using a few AudioBuffers in a multi-buffering manner and scheduling them with
> > AudioBufferSourceNodes). The problem, as it seems, is that you need to
> > accommodate for possible jittering and event drops, possibly by introducing a
> > latency (e.g, would it work if you forced a latency of 0.5s?).
> 
> I think a solution that uses setInterval() to schedule frequent JS callbacks,
> checks AudioContext.currentTime to measure the progress of the audio clock, and
> uses AudioBufferSourceNodes to queue playback of generated audio buffers,
> should work as well as any other, providing the Web Audio implementation is
> adequate. Incremental GC might also be required.

I seriously don't like the idea of using setInterval callbacks to drive the callback procedure instead of an actual callback derived from the audio clock. I thought we were about reducing power consumption as well? Deriving callbacks from the audio clock means we don't waste checks with setInterval, as we'd need to run setInterval faster than the buffer scheduling rate versus at the rate for an audio clock derived callback. I still don't like the idea of driving with a float based timestamp, I feel chills from possible implementation errors, and one off errors.

Can we add callbacks for when a buffer is done playing? I could drive off that then. Also auto-attaching (scheduled attaching) another buffer to the end of a previously playing buffer would avoid checking currentTime in the first place.

(In reply to comment #83)
> Can we add callbacks for when a buffer is done playing?

No, because they will fire too late for you to queue the next buffer and get seamless playback.

(In reply to comment #82)
> I seriously don't like the idea of using setInterval callbacks to drive the
> callback procedure instead of an actual callback derived from the audio clock.

You're going to have to schedule timeouts early anyway to maximise the chance that your callback runs in time to schedule the next buffer seamlessly, so accuracy doesn't matter, so using the audio clock instead of the system clock doesn't matter since there will be negligible drift over the intervals we're talking about.

I think there's been a misunderstanding that somehow the JavaScript code rendering audio in a JavaScriptAudioNode callback will block the audio thread!  This is not the case.  An implementation should use buffering (producer/consumer model) where the JS thread produces and the audio thread consumes (with no blocking).  This is how it's implemented in WebKit.

Additionally, the JS callbacks should all be clocked/scheduled from the audio system (in the implementation), and not rely on setTimeout() or require client polling/querying of a timestamp from javascript (which is a much less ideal approach).

(In reply to comment #84)
> (In reply to comment #83)
> > Can we add callbacks for when a buffer is done playing?
> 
> No, because they will fire too late for you to queue the next buffer and get
> seamless playback.
> 
> (In reply to comment #82)
> > I seriously don't like the idea of using setInterval callbacks to drive the
> > callback procedure instead of an actual callback derived from the audio clock.
> 
> You're going to have to schedule timeouts early anyway to maximise the chance
> that your callback runs in time to schedule the next buffer seamlessly, so
> accuracy doesn't matter, so using the audio clock instead of the system clock
> doesn't matter since there will be negligible drift over the intervals we're
> talking about.

I would be listening on a supposed finish event of buffers ahead of the last one scheduled. One wouldn't wait until the system is dry to refill. I'm talking about firing events for multiple buffers queued, so that if we run past a low point in the queue, we can prep and load new buffers. Using setInterval timers is not fool proof here, as some browsers that may implement the web audio api spec might not allow high res js timers. Numerous browsers either clamp heavily or have too much timer jitter (Opera people, I'm looking at you with that timer jitter on os x...).

crogers: I see what you're getting at. I'm just trying to make sure we all understand the audio input from the js side can be blocked inside a worker or on the main thread either way. Slow JS is slow JS.

Also how would sleep events interact with the currentTime counting? Would it be paused or would it clock with the computer's time system? For the audio to be foolproof it requires pure time clocking from the audio card/chip (So sleep events would pause it), otherwise audio buffering state can be thrown off by time changes or sleep events.

(In reply to comment #85)
> I think there's been a misunderstanding that somehow the JavaScript code
> rendering audio in a JavaScriptAudioNode callback will block the audio thread! 
> This is not the case.  An implementation should use buffering
> (producer/consumer model) where the JS thread produces and the audio thread
> consumes (with no blocking).  This is how it's implemented in WebKit.

How does this work in a subgraph similar to this?:

+------------+      +---------------------+      +------------------+
| SourceNode |----->| JavaScriptAudioNode |----->| BiquadFilterNode |
+------------+      +---------------------+   +->|                  |
                                              |  +------------------+
+------------+      +---------------------+   |
| SourceNode |----->|    AudioGainNode    |---+
+------------+      +---------------------+

(hope this ASCII art works)

I assume that without the input from the SourceNode, the JavaScriptAudioNode will not be able to produce anything (hence its callback will not be fired until enough data is available), and likewise the BiquadFilterNode can not produce any sound until data is available from both the JavaScriptAudioNode and the AudioGainNode.

In other words, if the JavaScriptAudioNode callback in the main thread is delayed by a setInterval event, for instance, i guess that at least the BiquadFilterNode (and all nodes following it?) will need to halt until the JS callback gets fired and finished so that it has produced the necessary data for the graph to continue?

I guess that the lower part of the graph (source + gain) could produce data ahead of time while the JS node is blocking, but I assume that it could be problematic in some cases (e.g. if there are loops or other intra-node dependencies, such as a panner node somewhere controlling the pitch of the source node), so perhaps it's simpler to just let it be blocked too?

(In reply to comment #87)
> (In reply to comment #85)
> > I think there's been a misunderstanding that somehow the JavaScript code
> > rendering audio in a JavaScriptAudioNode callback will block the audio thread! 
> > This is not the case.  An implementation should use buffering
> > (producer/consumer model) where the JS thread produces and the audio thread
> > consumes (with no blocking).  This is how it's implemented in WebKit.
> 
> How does this work in a subgraph similar to this?:
> 
> +------------+      +---------------------+      +------------------+
> | SourceNode |----->| JavaScriptAudioNode |----->| BiquadFilterNode |
> +------------+      +---------------------+   +->|                  |
>                                               |  +------------------+
> +------------+      +---------------------+   |
> | SourceNode |----->|    AudioGainNode    |---+
> +------------+      +---------------------+
> 
> (hope this ASCII art works)
> 
> I assume that without the input from the SourceNode, the JavaScriptAudioNode
> will not be able to produce anything (hence its callback will not be fired
> until enough data is available), and likewise the BiquadFilterNode can not
> produce any sound until data is available from both the JavaScriptAudioNode and
> the AudioGainNode.
> 
> In other words, if the JavaScriptAudioNode callback in the main thread is
> delayed by a setInterval event, for instance, i guess that at least the
> BiquadFilterNode (and all nodes following it?) will need to halt until the JS
> callback gets fired and finished so that it has produced the necessary data for
> the graph to continue?

No, this is not the case.  We're talking about a real-time system with an audio thread having realtime priority with time-constraints.  In real-time systems it's very bad to block in a realtime audio thread.  In fact no blocking calls are allowed in our WebKit implementation, including the taking of any locks.  This is how pro-audio systems work.  In your scenario, if the main thread is delayed as you describe then there will simply be a glitch due to buffer underrun in the JavaScriptAudioNode, but the other graph processing nodes which are native will continue processing smoothly.  Obviously the glitch from the JavaScriptAudioNode is bad, but we already know that this can be possible due to things such as setInterval(), GC, etc.  In fact, it's one of the first things I described in some detail in my spec document over two years ago.  Choosing larger buffer sizes for the JavaScriptAudioNode can help alleviate this problem.

(In reply to comment #88)
> (In reply to comment #87)
> > (In reply to comment #85)
> > > I think there's been a misunderstanding that somehow the JavaScript code
> > > rendering audio in a JavaScriptAudioNode callback will block the audio thread! 
> > > This is not the case.  An implementation should use buffering
> > > (producer/consumer model) where the JS thread produces and the audio thread
> > > consumes (with no blocking).  This is how it's implemented in WebKit.
> > 
> > How does this work in a subgraph similar to this?:
> > 
> > +------------+      +---------------------+      +------------------+
> > | SourceNode |----->| JavaScriptAudioNode |----->| BiquadFilterNode |
> > +------------+      +---------------------+   +->|                  |
> >                                               |  +------------------+
> > +------------+      +---------------------+   |
> > | SourceNode |----->|    AudioGainNode    |---+
> > +------------+      +---------------------+
> > 
> > (hope this ASCII art works)
> > 
> > I assume that without the input from the SourceNode, the JavaScriptAudioNode
> > will not be able to produce anything (hence its callback will not be fired
> > until enough data is available), and likewise the BiquadFilterNode can not
> > produce any sound until data is available from both the JavaScriptAudioNode and
> > the AudioGainNode.
> > 
> > In other words, if the JavaScriptAudioNode callback in the main thread is
> > delayed by a setInterval event, for instance, i guess that at least the
> > BiquadFilterNode (and all nodes following it?) will need to halt until the JS
> > callback gets fired and finished so that it has produced the necessary data for
> > the graph to continue?
> 
> No, this is not the case.  We're talking about a real-time system with an audio
> thread having realtime priority with time-constraints.  In real-time systems
> it's very bad to block in a realtime audio thread.  In fact no blocking calls
> are allowed in our WebKit implementation, including the taking of any locks. 
> This is how pro-audio systems work.  In your scenario, if the main thread is
> delayed as you describe then there will simply be a glitch due to buffer
> underrun in the JavaScriptAudioNode, but the other graph processing nodes which
> are native will continue processing smoothly.  Obviously the glitch from the
> JavaScriptAudioNode is bad, but we already know that this can be possible due
> to things such as setInterval(), GC, etc.  In fact, it's one of the first
> things I described in some detail in my spec document over two years ago. 
> Choosing larger buffer sizes for the JavaScriptAudioNode can help alleviate
> this problem.

Hmm? Convolution with big kernels is just as, if not more, suspectible to glitch as a JS node, so do you mean that if any of the nodes fails to deliver, the others still keep going?

It seems to me that the current behavior in the WebKit implementation is that if the buffer fill stops happening in time, it will start looping the previous buffer, whereas when a convolution node fails to deliver, it's just glitch and jump all over the place. Is this correct?

Seems a bit weird to treat parts of the graph differently, but I think I might have misunderstood something.

Having the audio thread never make blocking calls is good, but it leaves open the spec/API question of how a node should behave when one of its inputs can't produce data in a timely manner. It is possible to have the node pause while the input catches up, and it's possible to fill the input with silence as necessary. I think there are valid use-cases for both, especially when you throw media element and MediaStreams into the mix.

Either way, it needs to be spelled out in the spec.

I am still rather uncomfortable with this issue, not only because we have not yet reached  consensus, but because I haven't really seen the right points made on either side of the debate.

On the "must happen in Workers" side, I mostly see (in e.g Marcus' Comment #3) the idea that we should make sure not to let developers do anything bad. I am not fully confortable with that - thought our job was to enable and empower, and if need be to document best practices. There *were* a few mentions that limiting custom processing to workers would solve other issues (see recent thread on the delay caused by the custom nodes) - which are more interesting IMHO and should probably be developed.

On the other side of the debate, Chris mentioned a few times that developers were not comfortable with using workers - something that should be taken seriously if we are to follow our pyramid of constituencies - developers over implementors over spec designers; but IIRC we haven't really validated with a significant sample of developers that it would indeed be an issue. I know the W3C doesn't really do user testing, but that may be a good way to prove (or disprove) Chris' point.

(In reply to comment #91)
> On the "must happen in Workers" side, I mostly see (in e.g Marcus' Comment #3)
> the idea that we should make sure not to let developers do anything bad. I am
> not fully confortable with that - thought our job was to enable and empower,
> and if need be to document best practices.

I don't understand what those best practices would be, since Web developers aren't in a position to avoid the problems with a main thread callback API. They cannot control what goes on in other tabs, and as I mentioned in comment 9 and comment 19, those tabs can block the main thread for longer than even the maximum buffer size allowed by JavaScriptAudioNode.

That being said, perhaps we should begin to focus on what a WorkerAudioNode would look like, since that's what we want to actually implement.

Many points have been raised for and against running JS audio code in web workers. This is an important issue for developers (like me) and so I thought having a summary of the points raised thus far might be  useful [with some extra notes in square brackets]. Hope this helps reach consensus.

Please add/correct/clarify as you see fit.

== Arguments for JS audio in workers ==

1. Audio calculation will not be interrupted by the goings on on the main thread such as GC, graphics, layout reflow, etc.

2. Possibility for tighter integration with the audio thread. Perhaps we will be able to remove the one buffer delay currently in place? [Seems unlikely based on what ChrisR says - "cannot have blocking calls in system audio callbacks". Also cannot risk JS code in high priority system callbacks.]

3. Permits multiple audio contexts to operate independent of each other.  If all JS audio is on the main thread, then the JS nodes of all audio contexts are multiplexed on to the same thread. [Thus an offline context that uses a JS node will interfere with a realtime context if all JS audio code runs on the main
thread.]

4. With a "main thread only" design for JS audio nodes, applications will not be able to control the effects on audio of the browser multiplexing multiple tabs into a single thread - i.e. audio generation/processing can be affected by what happens in other browser tabs, with no scope for developer control.

== Arguments against JS audio in workers ==

1. API access to JS audio code will be severely limited. We may need to lobby for access to specific APIs (ex: WebCL).

2. More complicated to program. DOM and JS state access will require additional communication with the main thread.

3. Event communication latency between the main thread and workers is currently high (up to 0.5s). The communication overhead is also dependent on OS characteristics and consistent behaviour between OSes may be hard to implement. [Perhaps because workers are intended for "background" tasks and so minimizing the communication latency for events from the main thread to a worker isn't a priority for browser vendors?]

4. JS audio in workers cannot be synchronously executed with the high priority audio thread for security reasons. So the same delay in the current webkit implementation of JS audio nodes will likely apply.

5. Some applications such as device emulators become hard or impossible to program if all JS audio were to be forced to run in workers.

6. [If JS audio code is to be exclusively run in workers, then that demands that browser implementations cannot include support for the web audio api unless they also have support for web workers. Mobile browsers may choose not to support web workers but will likely want to support web audio.]

== Options ==

1. Permit JS audio nodes that run in the main thread *and* have a separate node type to run code in workers. A suggested con is that developers will tend to take the easier route which in this case (i.e. JS on main thread) is less reliable.  OTOH, "give them some rope and some will hang themselves and others
will build bridges" (Jussi).

2. It may be possible and acceptable to use AudioBufferSourceNodes to implement alternative schemes for "pumping" audio to the output, albeit at higher latencies.

3. [Live with JS on main thread for v1 of the spec and postpone worker nodes to later versions after prototyping. JS nodes on main thread are quite usable with long enough buffers.]

Resolution at the 2013-03-27 f2f meeting:

1. We will add a way to use ScriptProcessorNode with workers

2. The creation method will be similar to what was proposed in MediaStreamProcessing

See: http://www.w3.org/TR/streamproc/#stream-mixing-and-processing
and example 9 

  document.getElementById("out").src = new ProcessedMediaStream(new Worker("synthesizer.js"));

http://people.mozilla.org/~roc/stream-demos/worker-generation.html

3. We will not specify a buffer size, the UA will choose appropriately.

I remember writing code for my js audio lib to support the mediastream api and being required to output audio in a different thread: https://github.com/grantgalitz/XAudioJS/blob/master/XAudioServerMediaStreamWorker.js

It worked with an experimental firefox build that was published. Problem is, that there's up to a quarter second latency that kills. The end user of a browser will notice a giant time delay between an in-game event and the audio for it.

I have no problems with having a spec that has off thread audio, just don't cripple stuff by removing the on-thread audio. It's been mentioned many times that there is work on canvas support in-workers, so that it can be done alongside the audio out of the UI thread. The problem I have with that is it complicates keeping the audio libraries separate from the code that uses it. I want to support legacy audio APIs that use the main UI thread, and this will complicate that greatly.

And when I say "worked with," I do mean it was able to play gapless audio being streamed in from the UI thread to the worker. The only issue was latency for passing the stream around.

Changed title of the issue to reflect renaming of JavaScriptAudioNode to ScriptProcessorNode.

Web Audio API issues have been migrated to Github. 
See https://github.com/WebAudio/web-audio-api/issues

Closing. See https://github.com/WebAudio/web-audio-api/issues for up to date list of issues for the Web Audio API.