Bug 15747 - Echo cancellation API
Summary: Echo cancellation API
Status: RESOLVED FIXED
Alias: None
Product: WebRTC Working Group
Classification: Unclassified
Component: WebRTC API (show other bugs)
Version: unspecified
Hardware: All All
: P2 enhancement
Target Milestone: ---
Assignee: Cullen Jennings
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-01-27 18:30 UTC by Randell Jesup
Modified: 2012-05-18 14:55 UTC (History)
5 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Randell Jesup 2012-01-27 18:30:52 UTC
Proposal for the API to control Echo Cancellation in WebRTC:

We need options to enable and disable echo cancellation in WebRTC.  By
default, echo cancellation should be on for audio channel selections
that imply the likely possibility of echo, and off for ones where echo
is unlikely (audio input and output both going to a headset).  The
system would be allowed to change the EC state in response to input
routing or mute changes, or even to changes in the captured signal;
when that happens the onmediaoptionschange notification would occur.

The app could set the echoCancellation state.  Open Question: should
this block future automatic changes?  If so, should there be a way to
restore them, perhaps by using a tri-state setting (on, off, auto)?

Since we should future-proof this API against needing more options in
thie future, we will put the API for controlling echo cancellation in
a media options object.

Add:

interface MediaOptions {
   attribute boolean echoCancellation;
}

Open issue: should we expose the amount of echo cancellation or
expected echo levels?  I know the GoCast people thought this might be
useful.

If we add it, it should be relatively generic and not firmly tied to
the underlying implementation we currently have (and there are two in
the webrtc code drop):

  attribute float expectedERL;

Basically, the aggressiveness of the echo cancellation should be
dependent on the expected ERL.  This would map to the 'routing'
parameter in the webrtc.org 'mobile' EC block, and maybe to the
'SuppressionLevel' in the regular EC block.  (Or do we need to
separate the concepts?)  Alternatively, we could expose something
closer to the enumerated API in webrtc.org code.


Add to PeerConnection:
{
...
    attribute MediaOptions mediaOptions;
    attribute Function? onmediaoptionschange; 
}
Comment 1 Harald Alvestrand 2012-01-29 20:33:03 UTC
(reply-all does not update bugzilla; reply to bugzilla is impossible.)

Questions not answered yet:
- to which audio streams does echo cancellation apply, if there are several? Should we ignore it and say "the implementation will do the best it can across all streams"?
- should the echo options be part of some other option object; alternatively should the mediaOptions be renamed to something more generic?
- is it correct to place this option on the PeerConnection?
Comment 2 Timothy B. Terriberry 2012-01-29 21:40:23 UTC
(In reply to comment #1)
> - is it correct to place this option on the PeerConnection?

I don't think it is. Echo cancellation should occur using output audio as close to the speakers as possible to cancel echo in input streams as close to the microphone as possible. It would seem to me the appropriate place would be either a LocalMediaStream or the corresponding audio track.

Even if (especially if!) you just planned to play back the stream locally, you'd want echo cancellation enabled. Alternatively, if didn't want to play or transmit the stream, but, e.g., just wanted to record it, or use it for visualization, or something else, you'd want to be able to shut echo cancellation off. No PeerConnection is involved in any of these cases.

> - to which audio streams does echo cancellation apply, if there are several?
> Should we ignore it and say "the implementation will do the best it can across
> all streams"?

If you put the option at the LocalMediaStream level, I'd say "all of them" (assuming we even allow more than one). If you put it at the track level, then this question goes away.
Comment 3 Randell Jesup 2012-01-30 07:19:40 UTC
(In reply to comment #2)
> (In reply to comment #1)
> > - is it correct to place this option on the PeerConnection?
> 
> I don't think it is. Echo cancellation should occur using output audio as close
> to the speakers as possible to cancel echo in input streams as close to the
> microphone as possible. It would seem to me the appropriate place would be
> either a LocalMediaStream or the corresponding audio track.
> 
> Even if (especially if!) you just planned to play back the stream locally,
> you'd want echo cancellation enabled. Alternatively, if didn't want to play or
> transmit the stream, but, e.g., just wanted to record it, or use it for
> visualization, or something else, you'd want to be able to shut echo
> cancellation off. No PeerConnection is involved in any of these cases.

That's correct; so we can move this to the LocalMediaStream.  We can assume that the EC for input track (or tracks) knows what's being played out on the hardware.  Avoiding the app telling the system what might be causing echo is important since we'd like to have the EC cancel echo generated by *any* source, not just outputs from within one JS app.  In systems where we can, we'll even use a system EC since it has access to other processes writing to the audio drivers.

One could move it to the track(s) instead.  I'm not sure the added granularity is worth the added complexity; especially as this is the "override the default handling" knob.  The usecase for this would be multiple audio inputs in a single stream, where one is a headset and the other is the computer speakers and mic. However, in almost all cases those two users would be close together, and so the mic of the headset user would pick up echoes from the speakers on the computer.

So my vote is "on the LocalMediaStream".

> 
> > - to which audio streams does echo cancellation apply, if there are several?
> > Should we ignore it and say "the implementation will do the best it can across
> > all streams"?
> 
> If you put the option at the LocalMediaStream level, I'd say "all of them"
> (assuming we even allow more than one). If you put it at the track level, then
> this question goes away.

Agree.
Comment 4 Timothy B. Terriberry 2012-01-30 07:32:43 UTC
(In reply to comment #3)
> One could move it to the track(s) instead.  I'm not sure the added granularity
> is worth the added complexity; especially as this is the "override the default
> handling" knob.  The usecase for this would be multiple audio inputs in a

The other argument for putting it on a LocalMediaStream is that this makes it specific to something that gets handed back by getUserMedia(), which is I think the only place we want to expose this knob. You could make a LocalAudioMediaStreamTrack that extends AudioMediaStreamTrack as well, but it's not as clear that that's what you'd find in the audioTracks list on a LocalMediaStream (i.e., I don't think you can express that in the IDL).
Comment 5 Cullen Jennings 2012-03-19 16:21:40 UTC
I think this can be added once we get the constraints API sorted out.
Comment 6 Cullen Jennings 2012-05-18 14:55:24 UTC
This has been added in git to getusermedai spec