ISSUE-74: (Bugzilla 17361) RealtimeAnalyserNode design

RealtimeAnalyserNode

(Bugzilla 17361) RealtimeAnalyserNode design

State:
CLOSED
Product:
Web Audio API
Raised by:
Philip J├Ągenstedt
Opened on:
2012-05-16
Description:
The RealtimeAnalyserNode has so many problems that we will put them into a single issue.

The use case appears to be probing/polling the signal for visualization, where it does not matter if all of the signal is available. It also doesn't modify the output, so it need not delay the processing at all, like JavaScriptAudioNode would.

The problems identified with the current spec are:

1. It is undefined how multi-channel input maps to time/frequency data, which are both single arrays.

2. The layout/order of the frequency bins is undefined. Are the negative frequencies included?

3. It is undefined what happens if the array to the getters has more elements than frequencyBinCount.

4. smoothingTimeConstant is defined only as "A value from 0 -> 1 where 0 represents no time averaging with the last analysis frame." How does it affect time/frequency domain data? What is an analysis frame and which is the last?

5. If frequencyBinCount == fftSize / 2, why is it exposed at all?

6. minDecibels/maxDecibels are undefined. Do minDecibels/maxDecibels control the output of getByteFrequencyData, or do they describe it? Are the parameters only used for getByteFrequencyData()? If so, why are they not arguments to that method? How does the Uint8 range (0-255) map to the decibel range (minDecibels-maxDecibels)?

7. What happens if fftSize is set to something that is not a power of two? Are there any limits? Are 1 and 2^32 both valid values?

8. If the fftSize is initially set to 1, then changed to 2^32, what should the getters do? To not restrict this requires unbounded buffering to handle an arbitrarily large fftSize.

9. It's not at all clear why there are Uint8Array getters, instead of simply a frequency domain and time domain getter, both as Float32Array.

For the use case we're aware of, this can be simplified greatly. We'd prefer an interface that just allows probing the most recent time domain data as an AudioBuffer and leave it up to the Web developer to perform the FFT by other means. A fast, generic FFT function can be very useful not only for visualization, but also for synthesis, filters etc. In the absence of a native FFT implementation (which could be part of another specification - perhaps add it to the Math object), a custom JavaScript FFT implementation will most likely suffice for most applications.

For example:

interface AudioProbe : AudioNode {
// get the most recent data available.
AudioBuffer getData();
}

// in AudioContext, the size must be given up-front and cannot change
AudioProbe createAudioProbe(in unsigned long bufferSize);

Depending on how https://www.w3.org/2011/audio/track/issues/28 is resolved, we could simply have an attribute "AudioBuffer data" that is guaranteed to be stable while the script is executing, to avoid the use of a getter function altogether.
Related Actions Items:
No related actions
Related emails:
  1. [web-audio-api] (RealtimeAnalyserNode): RealtimeAnalyserNode design (#86) (from notifications@github.com on 2013-09-11)
  2. Re: Next teleconference : 6th June (from Olivier.Thereaux@bbc.co.uk on 2012-06-04)
  3. Re: Next teleconference : 6th June (from mage@opera.com on 2012-06-04)
  4. Re: [Agenda] W3C Audio WG Teleconference, 30th May 2012 (from olivier.thereaux@bbc.co.uk on 2012-05-30)
  5. Re: Aiding early implementations of the web audio API (from cwilso@google.com on 2012-05-23)
  6. Re: Aiding early implementations of the web audio API (from mage@opera.com on 2012-05-23)
  7. Re: Summary of issues (from crogers@google.com on 2012-05-22)
  8. Summary of issues (from mage@opera.com on 2012-05-22)
  9. Re: Aiding early implementations of the web audio API (from mage@opera.com on 2012-05-21)
  10. Audio-ISSUE-74 (RealtimeAnalyserNode): RealtimeAnalyserNode design [Web Audio API] (from sysbot+tracker@w3.org on 2012-05-16)

Related notes:

[MikeSmith]: https://www.w3.org/Bugs/Public/show_bug.cgi?id=17361

5 Jun 2012, 13:44:50

Display change log ATOM feed


Matthew Paradis <matthew.paradis@bbc.co.uk>, Raymond Toy <rtoy@google.com>, Chairs, Chris Lilley <chris@w3.org>, Staff Contact
Tracker: documentation, (configuration for this group), originally developed by Dean Jackson, is developed and maintained by the Systems Team <w3t-sys@w3.org>.
$Id: 74.html,v 1.1 2019/11/12 13:32:42 carcone Exp $