Bug 17390 - Loop start/stop points
Loop start/stop points
Status: CLOSED FIXED
Product: AudioWG - OBSOLETE - Moved to Github
Classification: Unclassified
Component: Web Audio API - OBSOLETE - See Github
unspecified
PC All
: P2 normal
: TBD
Assigned To: Chris Rogers
public-audio
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2012-06-05 12:10 UTC by Michael[tm] Smith
Modified: 2012-12-18 13:07 UTC (History)
4 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Michael[tm] Smith 2012-06-05 12:10:53 UTC
Audio-ISSUE-12 (Joe Berkovitz): Loop start/stop points [Web Audio API]

http://www.w3.org/2011/audio/track/issues/12

Raised by: Joe Berkovitz
On product: Web Audio API

Hi folks,

At the F2F in January we discussed the need for a more powerful looping construct in AudioBufferSourceNode than simply "loop the whole sound", in order to support UC6 (wavetable synthesis) and other user cases.  In particular, basic wavetable synths require the ability to start an audio buffer playing and then have it enter a loop in some arbitrary subrange of that sample [m..n].  Trying to achieve this via explicit scheduling of various subranges of a sample by calling noteGrainOn() feels unworkable since sample-accurate transitions from one subrange to another could not be achieved in this way -- and at any rate, it would be very unwieldy. The idea is that noteOn() should perform seamless repetition of the looped subrange of a sample as an integral part of its behavior.

Somehow this idea has been lost, probably due to the coarse-grain definition of looping in the UC document as simply "Playing / Looping sources of audio". However it's an essential requirement for wavetable synthesis and my concern is that games and synths will be seriously compromised without it.

I am proposing that there be separate loopStart and loopEnd properties of AudioBufferSourceNode that are sample-accurate.  If respectively set to 0 and the length of the sample, these will cause the entire sample to be looped.  If set to offsets M and N, these will cause the sample to be played by noteOn() and noteGrainOn() as if the sample consisted of contiguous subranges [0..M-1], [M..N], [M..N], ... 

Thanks!

...Joe
Comment 1 Olivier Thereaux 2012-06-07 16:04:29 UTC
Discussion thread, 30Apr - 8 May:
http://lists.w3.org/Archives/Public/public-audio/2012AprJun/thread.html#msg80

Discussion thread, 23-24 May:
http://lists.w3.org/Archives/Public/public-audio/2012AprJun/thread.html#msg397

Comment from Joe Berkovitz, 24 May 2012:
«
In my original proposal I had suggested that loopStart/loopEnd be set to 0 and the sample length in order to loop the entire sample (iff the "loop" attribute is set on the AudioBufferSourceNode).

After some thought I think that while this behavior obviously makes sense when loopEnd is the sample length, it would be also advantageous to define that whole-sample-looping occurs when loopStart == 0 and loopEnd == 0, and that these in fact be those attributes' default values. This provides a guaranteed way to achieve the whole-sample-looping behavior regardless of what underlying AudioBuffer is set into the AudioBufferSourceNode.
»
Comment 2 Joe Berkovitz / NF 2012-07-11 18:00:56 UTC
We're looking to make some progress on porting our synth architecture to the audio API and the lack of loop start/stop points is a gating problem for us.

My understanding from past discussion has always been that this issue was in scope for v1 since it is a basic feature of wavetable synthesis and is documented in our WG use cases.
Comment 3 Chris Rogers 2012-07-30 19:32:01 UTC
(In reply to comment #2)
> We're looking to make some progress on porting our synth architecture to the
> audio API and the lack of loop start/stop points is a gating problem for us.
> 
> My understanding from past discussion has always been that this issue was in
> scope for v1 since it is a basic feature of wavetable synthesis and is
> documented in our WG use cases.

Joe, I agree this is important for implementing sample-playback synths (SoundFonts, etc.).  The one point I'd like to make sure we figure out is how to deal with the sample data *after* the loopEnd attribute.  For example, in SoundFonts the loopEnd is often (but not always) set to end of the total sample data.  But when it's not, upon receiving a note-off command, the loop is expected to "exit" and proceed with sample playback all the way through to the end of the sample data (finally playing the sample data after the loopEnd).
Comment 4 Joe Berkovitz / NF 2012-07-30 21:05:59 UTC
(In reply to comment #3)
> Joe, I agree this is important for implementing sample-playback synths
> (SoundFonts, etc.).  The one point I'd like to make sure we figure out is how
> to deal with the sample data *after* the loopEnd attribute.  For example, in
> SoundFonts the loopEnd is often (but not always) set to end of the total sample
> data.  But when it's not, upon receiving a note-off command, the loop is
> expected to "exit" and proceed with sample playback all the way through to the
> end of the sample data (finally playing the sample data after the loopEnd).

In my experience the scenario of "loop exits" is quite rare with actual instrument samples, because one usually has few viable choices for internal loop points that are glitch-free -- making it that much less likely that a loop endpoint will also feed into (or be grafted onto) a naturally recorded and well-timed note release. Instead, a release envelope is applied to the continuing loop, beginning at the time that the musical effect of a "note off" is desired (i.e. the start of the release). The actual noteOff() on the AudioBufferPlaybackNode can be scheduled for the time at which the release envelope reaches an inaudible level.

So I feel that support for these less-common loop modes is not needed in V1, given the use case of playback of looped instrument samples.

However, we should certainly consider how to deal with loop modes.  One suggestion is that we adopt a [future?] loopMode attribute that is enumerable between these various options:
1. disable looping
2. once loop is entered, play continuously until noteOff()
3. once loop is entered, exit loop end after noteOff() and play rest of sample

#3 raises a sticky point. As I implied above, samplers tend to equate the concept of "note off" with the start of a release phase, not with the cessation of sound.  So it might also be worth considering whether the Web Audio API's noteOff() is the right way of conveying this concept of starting the release, particularly in the loop-exit case where exiting the loop occurs before sound ceases. Given the above definitions of loopMode, one would have to call noteOff() at very different times in the case of #2 and #3, because in #3 noteOff() doesn't actually stop the sound: it merely begins the process of stopping the sound, and in a rather difficult-to-predict way (since one doesn't easily know where in the loop one is going to be at some specific time, given the vagaries of playbackRate, etc).  It might be better to add a new, schedulable AudioParam whose purpose is to indicate that any internal loop is to be exited.

Make that another argument to attack this in V2.

Finally, some samplers also support the ability to optionally crossfade the end of the loop into the start of the loop with a specified time window, and also to "bounce" the loop by playing it successively backwards and forwards.  To me these are V2 features although the crossfade would be very welcome.
Comment 5 Joe Berkovitz / NF 2012-07-31 00:22:43 UTC
Note that the noteOff terminology issue raised above is tracked in Bug 18442.
Comment 6 Joe Berkovitz / NF 2012-08-29 21:18:02 UTC
I've been corresponding with Chris Rogers, and we have agreed on an approach that defers some of his previous suggestions to a V2 API.  So here's a concrete spec that lines up with this thinking:

The following elements are added to the AudioBufferSourceNode interface:

    const unsigned short LOOP_ENTIRE_SAMPLE = 0;  // looping applies to entire sample
    const unsigned short LOOP_START_END = 1;        // looping repeats from loopStart to loopEnd until stop() is called
    // (In the future, may support other modes, e.g. exit loop and play to end after stop() is called
    // or crossfade at start/end repeat boundary)

    // looping mode to be employed if the existing loop attribute is true. Default value is LOOP_ENTIRE_SAMPLE.
    attribute unsigned short loopMode;

    // start and endpoints as zero-based frame offsets within AudioBuffer for an *inclusive* loop range.
    // Only used if loop == true and loopMode == LOOP_START_END
    attribute unsigned long loopStart;
    attribute unsigned long loopEnd;

The new loopMode attribute determines the behavior of playback when the loop flag on the node is true.  The default value of LOOP_ENTIRE_SAMPLE preserves the legacy behavior of looping the entire sample. Setting loopMode to LOOP_START_END, however, will cause loopStart and loopEnd to drive looping playback as follows:

- loopStart and loopEnd are clipped to the range [0..buffer.length-1]
- the sample frame at element [loopStart] of the underlying AudioBuffer is considered the successor of the element [loopEnd]. Thus the playback order of all frames in the buffer is taken as [0..loopEnd-1], [loopStart..loopEnd-1], [loopStart..loopEnd-1], ...
- if playback is attempted at an offset at loopEnd or greater by a call to startGrain(), no playback occurs
- if stop() is called, output from the node ceases at the requested time regardless of what sample frame offset is being rendered

Note that loopStart/loopEnd are integral frame offsets in the buffer, not time offsets in seconds. That's because loops are really properties of the wave data in the underlying buffer, which makes them independent of playbackRate and thus not defined in terms of temporal units. Changing the playbackRate of a sample with loop points will alter the pitch of the sample, but not the nature of its loop.
Comment 7 Joe Berkovitz / NF 2012-08-29 21:24:00 UTC
(In reply to comment #6)
BTW the use of an inclusive endpoint as opposed to exclusive is geared to common usage of loop point ranges in popular music synth applications. This is somewhat at odds with range endpoints in functions like String.substring(), so I wanted to at least expose the reasoning here.
Comment 8 Joe Berkovitz / NF 2012-09-12 16:57:28 UTC
Amending to reflect discussion:

no loop mode in V1
start and end == 0 implies whole-sample loop and these are default values
clamping of start/end reflctd in loop behavior not readable values
type of start/end attributes to be float to support future subsample loops
non-subsample impls to round start/end to integral values

> I've been corresponding with Chris Rogers, and we have agreed on an approach
> that defers some of his previous suggestions to a V2 API.  So here's a concrete
> spec that lines up with this thinking:
> 
> The following elements are added to the AudioBufferSourceNode interface:
> 
>     const unsigned short LOOP_ENTIRE_SAMPLE = 0;  // looping applies to entire
> sample
>     const unsigned short LOOP_START_END = 1;        // looping repeats from
> loopStart to loopEnd until stop() is called
>     // (In the future, may support other modes, e.g. exit loop and play to end
> after stop() is called
>     // or crossfade at start/end repeat boundary)
> 
>     // looping mode to be employed if the existing loop attribute is true.
> Default value is LOOP_ENTIRE_SAMPLE.
>     attribute unsigned short loopMode;
> 
>     // start and endpoints as zero-based frame offsets within AudioBuffer for
> an *inclusive* loop range.
>     // Only used if loop == true and loopMode == LOOP_START_END
>     attribute unsigned long loopStart;
>     attribute unsigned long loopEnd;
> 
> The new loopMode attribute determines the behavior of playback when the loop
> flag on the node is true.  The default value of LOOP_ENTIRE_SAMPLE preserves
> the legacy behavior of looping the entire sample. Setting loopMode to
> LOOP_START_END, however, will cause loopStart and loopEnd to drive looping
> playback as follows:
> 
> - loopStart and loopEnd are clipped to the range [0..buffer.length-1]
> - the sample frame at element [loopStart] of the underlying AudioBuffer is
> considered the successor of the element [loopEnd]. Thus the playback order of
> all frames in the buffer is taken as [0..loopEnd-1], [loopStart..loopEnd-1],
> [loopStart..loopEnd-1], ...
> - if playback is attempted at an offset at loopEnd or greater by a call to
> startGrain(), no playback occurs
> - if stop() is called, output from the node ceases at the requested time
> regardless of what sample frame offset is being rendered
> 
> Note that loopStart/loopEnd are integral frame offsets in the buffer, not time
> offsets in seconds. That's because loops are really properties of the wave data
> in the underlying buffer, which makes them independent of playbackRate and thus
> not defined in terms of temporal units. Changing the playbackRate of a sample
> with loop points will alter the pitch of the sample, but not the nature of its
> loop.
Comment 9 Olivier Thereaux 2012-09-13 09:56:03 UTC
(In reply to comment #8)
> Amending to reflect discussion:
> 
> no loop mode in V1
> start and end == 0 implies whole-sample loop and these are default values
> clamping of start/end reflctd in loop behavior not readable values
> type of start/end attributes to be float to support future subsample loops
> non-subsample impls to round start/end to integral values

Thanks for the summary Joe. Adding it to the list of things to integrate in the spec.
Comment 10 Joe Berkovitz / NF 2012-10-10 18:20:34 UTC
Amending my earlier inaccurate comment about loopEnd -- I believe this index should be exclusive of the loop endpoint, not inclusive.
Comment 11 Chris Rogers 2012-11-13 23:29:02 UTC
Fixed:
https://dvcs.w3.org/hg/audio/rev/9d6ce45332e6
Comment 12 Olivier Thereaux 2012-12-18 13:07:20 UTC
Changeset looks good. 

Having seen no objection in over a month since changeset, closing.

I am assuming from the prose that the presence of loopStart/loopEnd will simply be ignored if loop is set to 0 (or not set). Feel free to reopen if you object and/or think there should be a warning or exception thrown then.