User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36

Steps to reproduce:

Join a webrtc meeting on for example https://letsmeet.no, http://meet.jit.si or https://demo.mediasoup.org.

Actual results:

After a short while, could be 10 seconds, could be 2 minutes. The Firefox user will start to echo back the sound from the other users in the meeting.

As a Firefox user, I have tested this with a Sennheiser headset with a mechanical mute, and the other users get their own audio back from me even tho my mic is muted. This is reproducable on both Linux and Windows.

Expected results:

The audio from other peers should not be sent back out.

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Hi Paul, could you please help triage this one?

This means that the reverse stream somehow goes back into the PeerConnection. Not something I've seen however.

What would be useful is a mode that would dump the audio from all inbound PeerConnection, outbound PeerConnection, mic input, speaker output, and AEC reverse stream. We don't have this however, so we'll have to do it the old school way.

Reporter, if you're willing to go the extra mile to help fixing this, you can try the following:

• Go to about:config, search for security.sandbox.content.level, and set it to 0
• Restart Firefox
• Start a call like you usually do
• Open about:webrtc, and click "Start AEC Logging"
• Talk for a bit, etc., like it's a normal call
• After the problem has reproduced, wait for a bit
• Go to about:webrtc again, and click the button to stop the AEC logging. This indicates a path on your computer where you can find a bunch of .wav and .dat (sometimes sorting the folder by date helps finding those)
• Zip those files and attach them here
• Go to about:config again, restore the value for security.sandbox.content.level (there is a button to restore it to its default value)
• Restart Firefox to apply this sandbox level change

So I gathered 2 different examples of this happening, but the files are 100-200 mb, so get the files here:
https://majd.eu/firefox/

I also took a screen capture of the test where you can see the effect in play:
https://www.youtube.com/watch?v=MaHMLjUdqTQ

In the video, I am the left-most user, and you can see me mechanically muting my mic. The white border around my video means that I have become active speaker according to the server, and you can also see my audio indicator moving while I am muted.

This seems to vary based on headsets, we have an example where the change of headset fixes the issue. We have examples from different headsets on different computers where the echo is present. We have observed the issue even in Firefox ESR on Debian (68).

Please tell me if there is any more you need from me, this is a very important issue for us to get fixed as this currently breaks an entire meeting if a buggy Firefox user joins.

(In reply to Håvar Aambø Fosstveit from comment #4)

So I gathered 2 different examples of this happening, but the files are 100-200 mb, so get the files here:
https://majd.eu/firefox/

I also took a screen capture of the test where you can see the effect in play:
https://www.youtube.com/watch?v=MaHMLjUdqTQ

In the video, I am the left-most user, and you can see me mechanically muting my mic. The white border around my video means that I have become active speaker according to the server, and you can also see my audio indicator moving while I am muted.

Thanks for all the files, this helps a lot. I can hear/see the issue very clearly indeed, opening aec_out_linear_0-3.wav shows you talking and asking to repro to the other people in the call. Then you touch your mute button, this produces a crazy discontinuity, then silence.

What I find weird is that there is positive DC offset of 0.05 in the signal of your microphone (i.e. the audio wave form is not centered on 0, I'm attaching an image with annotations). I've reproduced your setup here, with the hardware I have, and it's perfectly centered on 0 (as it should be), so it's not an artifact of our recording mechanism. I'll get you a build shortly with a bit of processing to remove this, as it might well help, if you're willing to try.

This seems to vary based on headsets, we have an example where the change of headset fixes the issue. We have examples from different headsets on different computers where the echo is present. We have observed the issue even in Firefox ESR on Debian (68).

Please tell me if there is any more you need from me, this is a very important issue for us to get fixed as this currently breaks an entire meeting if a buggy Firefox user joins.

Can you list headsets that work and headsets that don't work, maybe alongside computers you've tried? It's quite frequent for us to find buggy hardware (sound card, headphones, earbuds, etc.), we usually buy them and debug, it's usually faster, granted we can reproduce.

Now a couple questions to help me diagnose, since I see your problem, but can't repro myself yet:

What is the behavior if your don't use your headset mute switch, but instead use the mute switch in the browser, can you reproduce then? I see the app you're using in the youtube video has a mute switch.

This is different: in the first case (what you demonstrate in the video), then we don't know that the microphone is muted. We just happen to receive silence on the audio input. As far as we know, it could just be that you're very quiet in a very quiet room. We can't really detect this, so we still send audio to other people (however it compresses extremely well since it's silent, so it's not crazy problematic, nothing compared to video). After that the echo canceler is very confused about the DC offset it seems, and after processing, a non trivial amount of reverse stream (= the other people in the call, mixed together) appears in the mic stream.

However if you mute in software, then we know that there is no input. In Firefox, all the audio buffers are marked as being "silent" and various optimizations happen. For example, the input processing doesn't happen as much, since we know there is no input being recorded. I don't know if we still send audio to the other people.

Both ways should work of course, I'll now work on a speculative fix. Thanks again for taking the time to provide all the info here!

A side effect is that this high-passes the signal somewhat, but we're dealing
with voice here, it doesn't matter.

Builds will be at https://treeherder.mozilla.org/#/jobs?repo=try&revision=2bb16bc2954911dd13f60c2492817800ef9c5052, you can click green letter B on the right hand side, find the zip, tar.gz, dmg, etc. for your preferred platform, and then unzip/untar/etc. and run firefox.exe or firefox like usual. This build is similar to a Nightly build, with the patch attached here on top.

I have tested again with different scenarios.

Tested audio hardware with echo:
Sennheiser PC360D (with USB connection) (Debian testing)
Sennheiser PC363D (with USB connection) (Windows 10/Debian testing usbid: 1395:1000, 5.4.0-4-amd64 Debian 5.4.19-1)
Unconfirmed Logitech cam mic (no model number)

Tested audio hardware without echo:
Sennheiser PC360D (3,5 mm) (Debian testing)
Sennheiser PC363D (3,5 mm) (Debian unstable)
Logitech Brio cam mic
Lifecam Studio cam mic
Lifecam Cinema cam mic
Sades A6
Logitech C615 cam mic

There seems to be something about the USB dongle with Sennheiser. We have reports from more users about echo issues, but have no models for those at the moment, sorry. I even tested the Sennheiser PC363D headset with USB on Firefox on Android, and the echo was there. (As a side note, there is no echo in Chrome)

I tested it also by setting audio output to a completly different set of speakers, while keeping the Sennheiser mic as input, and the echo was still there.

If I pause the audio track the echo disappears, but when I resume the track the echo is back in full force, and does not seem to "reset" back to start conditions where it is lower. If I tear down the audio track, and reestablish it, the echo seems to be "reset" to start conditions.

Will test the new build tomorrow.

Builds (all 64bits, I can get 32bit builds if needed):

• Linux https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/f4x9R08sRf-ZkPDr2kIY8A/runs/0/artifacts/public/build/target.tar.bz2
• macOS (not notarized, will not run on catalina, let me know if this is a problem) https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/a1KDIEwzQNezJ2iZUh2lWQ/runs/0/artifacts/public/build/target.dmg
• Windows https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BUgL2vPlR4S17z6TFi1I4Q/runs/0/artifacts/public/build/target.zip

It's the same builds as linked above, but our CI system can be a bit complex to navigate, so here are the direct links.

Thanks for the breakdown by device. It seems clear to me that this comes from particular devices, let me know if the build above fixes it.

Asking you to mute via the app was to characterize the issue better, of course we need this to work regardless of what users do!

Preliminary testing shows this bug as fixed. If you want, I can join a live session for showing before/after.

I have verified by going between versions, and the custom built one has no echo, while the vanilla one has a clear echo almost immediately. Have tested with 3 participants so far, 2 of which had headsets with echo, but now has no echo.

Have now tested with 4 participants for over an hour in both Linux and Windows, where 3 of us tested both vanilla and nightly. No echo at all from any of us with nightly, great audio the entire time.

Very good to hear, thanks for taking the time to report and follow up on this. I'll now clean up my patch and get it landed.

What I want to understand is why this is a problem in Firefox and not in Chromium-based browser. Supposedly we're using the same components for this (from the webrtc.org codebase). But I think we might want to merge this patch right now since it does solve the issue.

I think that webrtc.org might have a dc blocker as well, but maybe we're not activating it or something.

At first glance the echocancellation is initialized here: https://github.com/chromium/chromium/blob/e7582c3ed8d3339f3ee190c75879facc4cc99694/media/webrtc/audio_processor.cc#L185

It's probably the high-pass then, I'll activate it.

The high-pass filter helps fixing the DC bias on input, which is critical for
the AEC to work well.

It's not controllable by authors, enabled if any of AEC, AGC or Noise
suppression is activated, disabled otherwise, and therefore not taken into
account when activating the pass-through mode.

media.getusermedia.hpf_enabled allows controlling it's activation. This
defaults to true.

https://hg.mozilla.org/mozilla-central/rev/f116a9aef090

Would it be possible to verify this is now working well in Firefox Nightly, that can be installed alongside a regular Firefox?

I have downloaded latest nightly in Debian and Windows, and can verify that it works very well, no echo.

Thank you very much for all you've done to help us fix this. I'll now nominate this bug to be in the next release.

Comment on attachment 9139500 [details]
Bug 1627313 - Activate an HPF on getUserMedia microphone input, when processing is enabled. r?achronop

Beta/Release Uplift Approval Request

• User impact if declined: Potential persistent echo regardless of the OS and computer, depending solely on the audio hardware being used, with a reasonably common brand of headset.
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce: I have received help from the reporter to confirm this is fixed. This requires specific hardware to reproduce and have tested myself with a hacked up Firefox build that contains code (that I'm not willing to ship) to simulate a similar problem at the hardware level.
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This seem like a lot of code, but is just boilerplate (copy/paste from the other similar surrounding functions). This is just enabling another processing pass in the WebRTC audio input processing component. The difference is that this particular processing pass is not controllable from javascript, which makes it simpler.

If this is still too scary for uplift, the other patch can be used as well, but this patch is going to be used in the long run.

• String changes made/needed: none

This sounds good, looking forward to it being in stable!

Comment on attachment 9139500 [details]
Bug 1627313 - Activate an HPF on getUserMedia microphone input, when processing is enabled. r?achronop

Thanks for the context about the size of the patch. FWIW, it would have been nice to better understand the trade-offs of taking this patch vs. the other one attached to the bug. It's a bit tough to reason about which is better to take in an uplift at this point in the cycle based on what's currently in the bug. That said, we've still got a bit of time left this cycle for any issues to shake out and the scope of the patch seems fairly contained, so I'll approve it for 76.0b5.

https://hg.mozilla.org/releases/mozilla-beta/rev/0a072dacccf5

(In reply to Ryan VanderMeulen [:RyanVM] from comment #24)

Comment on attachment 9139500 [details]
Bug 1627313 - Activate an HPF on getUserMedia microphone input, when processing is enabled. r?achronop

Thanks for the context about the size of the patch. FWIW, it would have been nice to better understand the trade-offs of taking this patch vs. the other one attached to the bug. It's a bit tough to reason about which is better to take in an uplift at this point in the cycle based on what's currently in the bug. That said, we've still got a bit of time left this cycle for any issues to shake out and the scope of the patch seems fairly contained, so I'll approve it for 76.0b5.

Both are functionally equivalent for the purpose of fixing this bug (and both have been tested to fix the bug by the reporter), but what has been uplifted better integrates with our existing audio input processing, that comes from the webrtc.org code base, that other browser use. The patch that has been uplifted puts us more in line with what other implementations do. In terms of risk, it means we go closer to an audio input processing solution for voice that is known to work well for this case, instead of adding a custom solution on top of the existing processing.

Thanks for taking this, in any case!

Hi,

I wasn't able to reproduce the issue but according to Comment 20 the fix has been verified by the reporter on Firefox 77.0a1.
Havar, could you please also verify the fix on the latest Firefox Beta? You can download it from here:
https://www.mozilla.org/en-US/firefox/channel/desktop/

Thank you!

I can verify that it works in the latest Firefox Beta. I tested first with stable, and then with beta. Stable had echo, beta did not.

Thank you for verifying!