Steps to reproduce: Do a webrtc audio+video call between desktop and android. The sound from the desktop's microphone is played on the android device approximately 1s after the video. The sound in the other way (from the android device's microphone to the desktop machine) doesn't seem significantly delayed (it's in sync or almost in sync with the video). This was tested with Nexus 4 android phones, and reproduced by at least 2 people.

Note: we're using builds based on Alder, which include the a/v sync bug fix. The desktop machines are all Macs. We've also reproduced using S3, and also a galaxy note.

Reproduced with Linux Desktop (Ubuntu 12.10) and LG Nexus 4.

Jesup asked me during the webrtc apps call if this is only with the mac builtin microphone or also with the headset's microphone. I've just tested, it also happens when using the headset's microphone. Someone (Maire?) also suggested turning off echo cancellation. It didn't help either.

I just tested using the gum test page as well on Android - this actually has nothing to do with the full stack. It's a gUM bug in the Android space.

Added roc and kinetik, since we're down to a more restricted case of <android audio drivers> <OpenSL ES> webrtc.org audio capture MediaEngineWebRTCAudio::Process() MediaStreamGraph() Audio element cubeb <android audio drivers>

We're not using cubeb for audio output on Android yet (we need bug 698328, bug 837564, at least). The sydneyaudio backend that we are currently using uses a 1s buffer on Android (and much less (~55ms) on Gonk).

That's very suspicious... 1 second is the apparent delay we're seeing. Padenot or kinetik, can you figure out a way to find out if the output buffer is running a 1s delay when we hear this problem? (Perhaps padenot's delay-monitoring stuff)

A quick test would be to try setting media.use_cubeb to true in about:config and seeing if it helps, that should be using a ~100ms buffer (+/- whatever Android ends up forcing you to use).

I'm going to investigate today.

(In reply to Matthew Gregan [:kinetik] from comment #9) > A quick test would be to try setting media.use_cubeb to true in about:config > and seeing if it helps I tested this. The audio and video streams are still out of sync. I was under the impression that flipping that preference slightly improves the situation (with the delay reduced from one second to half a second), but as I have no way to actually time the delay, it's possible this improvement is just coming from my imagination or hopes.

(In reply to Florian Quèze [:florian] [:flo] from comment #11) > (In reply to Matthew Gregan [:kinetik] from comment #9) > > A quick test would be to try setting media.use_cubeb to true in about:config > > and seeing if it helps > > I tested this. The audio and video streams are still out of sync. > I was under the impression that flipping that preference slightly improves > the situation (with the delay reduced from one second to half a second), but > as I have no way to actually time the delay, it's possible this improvement > is just coming from my imagination or hopes. Florian - How does this compare to when you test this on the gum test page with Audio and Audio+Video - http://mozilla.github.com/webrtc-landing/gum_test.html?

So, this brings the latency down by using the OpenSL cubeb backend, an requesting a latency of 20ms. Also, it reduces the cubeb buffer size.

(In reply to Jason Smith [:jsmith] from comment #12) > Florian - How does this compare to when you test this on the gum test page > with Audio and Audio+Video - > http://mozilla.github.com/webrtc-landing/gum_test.html? It seems similar. Audio and video are out of sync by almost 1s.

Isn't playback of WebRTC stuff done through the OpenSL backend in the WebRTC code? Or is that only used on the capture side?

Capture side only. Output is through assigning a stream to a media element.

Unfortunately, when I tried this patch on my optimized/non-DEBUG build, it didn't seem to effect the in-call latency much. There was still about a 1-second delay. :-( Try builds with the patch are available from the B links in <https://tbpl.mozilla.org/?tree=Try&rev=679457b301c3>.

I only tested gUM, and I saw a lot of improvement. I'll be testing with a real call today.

(In reply to Dan Mosedale (:dmose) from comment #17) > Unfortunately, when I tried this patch on my optimized/non-DEBUG build, it > didn't seem to effect the in-call latency much. There was still about a > 1-second delay. :-( Try builds with the patch are available from the B > links in <https://tbpl.mozilla.org/?tree=Try&rev=679457b301c3>. I've just tested these try builds. I see (or rather hear) a dramatic improvement, both on the gum page and during calls. The audio latency is short enough that I likely wouldn't notice it if I didn't have the phone and the laptop on the same desk.

I wonder if the difference is that I didn't tweak anything echo canceller-related, either in the builds, the prefs, or on phone. Florian, is it possible that those tweaks that you had been playing with before will still in effect?

s/will still/were still/

(In reply to Dan Mosedale (:dmose) from comment #20) > I wonder if the difference is that I didn't tweak anything echo > canceller-related, either in the builds, the prefs, or on phone. Florian, > is it possible that those tweaks that you had been playing with before will > still in effect? I actually had media.peerconnection.aec_enabled set to false on my phone. I've just tested this again with the pref back to its default value, and I still get the same results as in comment 19.

Another thing to mention here: there seems to be some audio drift. When the call starts, audio and video are in sync or almost in sync. After a few minutes the audio and video from the phone arrive out of sync on the desktop, and the delay video and audio keeps increasing over time. The sound and video from desktop are still in sync even after a while.

Having a Mac build without this patch call an Android build with it seems to usually / mostly be lower latency and sounds great. However, if the Mac build also has this patch, the audio received on the Android phone from the Mac has regular irritating glitches. I suspect most people who've tried so far weren't testing with both of their builds patched. Tweaked patch forthcoming.

This ifdefs the bufferLimit change to only happen on Android, and leaves it alone on other platforms.

Try server build in progress: https://tbpl.mozilla.org/?tree=Try&rev=8b3e029ff9fe

I'm not totally convinced that these changes are quite as good as before this patch on the sound quality front, though I'm not at all sure how that could be. It may well indeed just be coincidental network quality issues. However, the latency is different better with the try build on both ends. On balance, this feels better. Since if gather data in the next day or so that suggests this was the choice, we can back it out, or simply fall back to the previous build, I'm landing on alder. (The reasoning mentioned here is what we agreed to go forward with earlier today).

Pushed to alder: https://hg.mozilla.org/projects/alder/rev/7f758499c51e

Something like this is likely needed for uplift, so I've added [android-trunk-needed]. How closely it relates to this patch is likely to depend on the high-level audio strategy for Android going forward...

Goodness, let's try comment 27 again: Since if gathering data in the next day or so that suggests this was the wrong choice, we can back it out, or simply fall back to the previous build, I'm landing on alder. Typing is hard, I'm going shopping.

gcp - Was this fixed when we landed on central?

Actually, nevermind. This is definitely still reproducible, in fact it's a few seconds in delay on apprtc.

Putting into blocking? to run some tests to see how bad this is to figure out if we need to block or not

QA Wanted for comment 33

Using Nightly (06/02) on Mac OS X (10.8.3) and Nightly (06/02) on Android (4.1) on an HTC One and Galaxy S4 (Android 4.2), and a Galaxy Note II (Android 4.2.2) and Chromium (29.0.1527.0 (203622)), I am unable to discern any troubling ~1s± delay in audio and video between clients using http://apprtc.appspot.com as a test application for connecting clients. The latency difference I saw between audio and video sync between desktop and mobile device was rather minuscule very much less than a <1s on desktop and Android.

Sounds good. Closing as wfm then.