Platform: Debian Jessie Repro: - Start FF Nightly on Linux (though crash happens all the way back thru at least 25) - Open Web Console - run the following line: window.addEventListener("gamepadconnected", function () {}, false); Expected: undefined return Actual: Crash Important stack frames (taken from my local debug build which has a lot of changes but this still looks vaguely valid): #0 0x00007fb3594fc49d in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fb3594fc341 in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:137 #2 0x00007fb353e6c53a in ah_crap_handler (signum=11) at /share/code/mozbuild/mozilla-central/toolkit/xre/nsSigHandlers.cpp:88 #3 0x00007fb353e74dd5 in nsProfileLock::FatalSignalHandler (signo=11, info=0x7fff3f996570, context=0x7fff3f996440) at /share/code/mozbuild/mozilla-central/profile/dirserviceprovider/src/nsProfileLock.cpp:190 #4 <signal handler called> #5 0x00007fb34b2f6ef5 in udev_list_entry_get_next () from /lib/x86_64-linux-gnu/libudev.so.1 #6 0x00007fb34b2fba84 in udev_monitor_filter_update () from /lib/x86_64-linux-gnu/libudev.so.1 #7 0x00007fb31f1f98a1 in udev_monitor_enable_receiving () from /lib/x86_64-linux-gnu/libudev.so.0 #8 0x00007fb354f9a2a7 in AddMonitor (this=0x7fb323532000) at /share/code/mozbuild/mozilla-central/hal/linux/LinuxGamepad.cpp:228 #9 Startup (this=0x7fb323532000) at /share/code/mozbuild/mozilla-central/hal/linux/LinuxGamepad.cpp:251 #10 mozilla::hal_impl::StartMonitoringGamepadStatus () at /share/code/mozbuild/mozilla-central/hal/linux/LinuxGamepad.cpp:365 #11 0x00007fb354b267a3 in mozilla::dom::GamepadService::AddListener (this=0x7fb3228cd650, aWindow=aWindow@entry=0x7fb322c34800) at /share/code/mozbuild/mozilla-central/dom/gamepad/GamepadService.cpp:110 #12 0x00007fb354711ecc in nsGlobalWindow::EnableGamepadUpdates (this=0x7fb322c34800) at /share/code/mozbuild/mozilla-central/dom/base/nsGlobalWindow.cpp:9306 #13 0x00007fb35452ae1a in nsEventListenerManager::AddEventListenerInternal (this=this@entry=0x7fb32243e3e0, aListener=..., aType=<optimized out>, aTypeAtom= 0x7fb3425031c0, aTypeString=..., aFlags=..., aHandler=false, aAllEvents=false) at /share/code/mozbuild/mozilla-central/content/events/src/nsEventListenerManager.cpp:360 #14 0x00007fb35452af52 in nsEventListenerManager::AddEventListenerByType (this=this@entry=0x7fb32243e3e0, aListener=..., aType=..., aFlags=...) at /share/code/mozbuild/mozilla-central/content/events/src/nsEventListenerManager.cpp:542 #15 0x00007fb35452afa1 in nsEventListenerManager::AddEventListener (this=this@entry=0x7fb32243e3e0, aType=..., aListener=..., aUseCapture=aUseCapture@entry=false, aWantsUntrusted=aWantsUntrusted@entry=true) at /share/code/mozbuild/mozilla-central/content/events/src/nsEventListenerManager.cpp:1077 #16 0x00007fb354712556 in AddEventListener (aWantsUntrusted=true, aUseCapture=false, aListener=0x7fff3f997d38, aType=..., this=0x7fb32243e3e0) at /share/code/mozbuild/mozilla-central/content/events/src/nsEventListenerManager.h:232 #17 nsGlobalWindow::AddEventListener (this=this@entry=0x7fb322c34800, aType=..., aListener=aListener@entry=0x7fb3228508e0, aUseCapture=aUseCapture@entry=false, aWantsUntrusted=..., aRv=...) at /share/code/mozbuild/mozilla-central/dom/base/nsGlobalWindow.cpp:9024 #18 0x00007fb355068c44 in mozilla::dom::EventTargetBinding::addEventListener (cx=0x7fb348218c40, obj=..., self=0x7fb322c34800, args=...) at /share/code/mozbuild/mozilla-central/obj-gamepad/dom/bindings/EventTargetBinding.cpp:63 #19 0x00007fb355056e89 in mozilla::dom::EventTargetBinding::genericMethod (cx=0x7fb348218c40, argc=<optimized out>, vp=0x7fb33fb1f520) at /share/code/mozbuild/mozilla-central/obj-gamepad/dom/bindings/EventTargetBinding.cpp:325 #20 0x00007fb355c499e0 in js::CallJSNative (cx=0x7fb348218c40, native=0x7fb355056c38 <mozilla::dom::EventTargetBinding::genericMethod(JSContext*, unsigned int, JS::Value*)>, args=...) at /share/code/mozbuild/mozilla-central/js/src/jscntxtinlines.h:220 #21 0x00007fb355c469de in js::Invoke (cx=cx@entry=0x7fb348218c40, args=..., construct=construct@entry=js::NO_CONSTRUCT) at /share/code/mozbuild/mozilla-central/js/src/vm/Interpreter.cpp:463 #22 0x00007fb355c380f2 in Interpret (cx=cx@entry=0x7fb348218c40, state=...) at /share/code/mozbuild/mozilla-central/js/src/vm/Interpreter.cpp:2502 #23 0x00007fb355c467bc in js::RunScript (cx=cx@entry=0x7fb348218c40, state=...) at /share/code/mozbuild/mozilla-central/js/src/vm/Interpreter.cpp:420 #24 0x00007fb355c47a50 in js::ExecuteKernel (cx=cx@entry=0x7fb348218c40, script=script@entry=..., scopeChainArg=..., thisv=..., type=<optimized out>, evalInFrame=..., result=result@entry=0x7fff3f998c60) at /share/code/mozbuild/mozilla-central/js/src/vm/Interpreter.cpp:611 #25 0x00007fb355bf8dbd in js::EvaluateInEnv (cx=cx@entry=0x7fb348218c40, env=env@entry=..., thisv=..., thisv@entry=..., frame=..., chars=..., length=<optimized out>, filename=filename@entry=0x7fb3564b2e9c "debugger eval code", lineno=lineno@entry=1, rval=rval@entry=...) at /share/code/mozbuild/mozilla-central/js/src/vm/Debugger.cpp:4279 #26 0x00007fb355c0f8ba in DebuggerGenericEval (cx=cx@entry=0x7fb348218c40, fullMethodName=fullMethodName@entry=0x7fb3567a97f0 "Debugger.Object.prototype.evalInGlobalWithBindings", code=..., evalWithBindings=evalWithBindings@entry=EvalHasExtraBindings, bindings=..., options=..., vp=..., dbg=dbg@entry=0x7fb32339a800, scope=scope@entry=..., iter=iter@entry=0x0) at /share/code/mozbuild/mozilla-central/js/src/vm/Debugger.cpp:4408 #27 0x00007fb355c101fe in DebuggerObject_evalInGlobalWithBindings (cx=0x7fb348218c40, argc=<optimized out>, vp=<optimized out>) at /share/code/mozbuild/mozilla-central/js/src/vm/Debugger.cpp:5251

Did more research. This works fine on Ubuntu 12.04 using udev 175. Crashes on Debian jessie/sid with udev 204.

Tested on Ubuntu 13.10 VM with udev 204. Works fine. So much for the udev versioning theory.

This is a test executable I built out of udev.h and the relevant parts of LinuxGamepad.cpp to see if I could repro the crash outside of firefox. It seems to run fine, printing out GAMEPAD twice when I have two gamepads hooked up.

Solved! It seems that the assumption of ABI compatibility between libudev0 and libudev1 is no longer valid. So, if a system is running udev 204 (libudev.so.1) but tries to access it from udev 175 (libudev.so.0) ABI, things explode. libudev0 was removed from ubuntu as of ubuntu 13.04, which is why we're not seeing it there. However, it's still around on debian, and since the array contains libudev.so.0 first, that gets loaded, ABI's don't match, boom. I removed the libudev0 package from my system, things came up fine. My best guess to fix this is try loading .1 before .0? I'm not sure how many distros that will affect though. It'd be nice if we could have some sort of ABI checking on this. I also have no idea why this worked in my PoC executable since that used the same dylib loading order, so it seems like it should've seen the same crash? No idea.

:jld brought up a good point, we could use RTLD_NOLOAD tests to see whether the library already came in elsewhere, so that we aren't always loading possibly conflicting symbols.

Comment on attachment 831633 [details] [diff] [review] Patch 1 (v1) - Use RTLD_NOLOAD to test whether libraries are already loaded Review of attachment 831633 [details] [diff] [review]: ----------------------------------------------------------------- So the comments about ABI compat were localized to our (gamepad) usage of it. The issue here is that we wound up loading udev.0 when we already had udev.1 loaded, so we overwrote some symbols and then confused udev? This seems sane.

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #7) > So the comments about ABI compat were localized to our (gamepad) usage of > it. The issue here is that we wound up loading udev.0 when we already had > udev.1 loaded, so we overwrote some symbols and then confused udev? Yeah, the ABI compat is still fine, which is the reason my proof of concept executable actually worked, since it was ONLY loading libudev.so.0 since there were no other dependencies. It's the symbol mismatches across libraries due to something else bringing in libudev.so.1 first that caused the issue. Will land and file for aurora uplift, since gamepad defaults on as of 27.

Given this is not a recent regression no need to track for firefox 27 in particular. Although we will consider for uplift depending on where we are in the cycle and risk/reward analysis. Please let me know if I am missing on any information on why this would be a blocker for Fx27 and feel free to renom if needed.

(In reply to bhavana bajaj [:bajaj] from comment #11) > Given this is not a recent regression no need to track for firefox 27 in > particular. Although we will consider for uplift depending on where we are > in the cycle and risk/reward analysis. > > Please let me know if I am missing on any information on why this would be a > blocker for Fx27 and feel free to renom if needed. I actually managed to crash this all the way back through FF25. That said, it requires turning the pref on in 25/26 (we keep the pref on in aurora/nightly) I guess it doesn't need to uplift until we figure out exactly when we're permanently turning this on.

Unfortunately, we don't have access to Debian Jessie, since we usually test on Ubuntu. I tried a few times, but the bug doesn't reproduce on Ubuntu (as specified in above comments too). Anthony, does anyone on your side have access to Debian Jessie to verify this bug?

(In reply to Ioana Budnar, QA [:ioana] from comment #13) > Anthony, does anyone on your side have access to Debian Jessie to verify > this bug? Not immediately, you can just download the ISO and set up a new VM though from debian.org.

Since nobody had access to Debian Jessie, I got that VM with the latest x64 version of it. Unfortunately I still can't reproduce this issue. I tried with Firefox 27 and the 11/11 Nightly (Firefox 28.0a1, both regular and debug) and I always get "undefined" instead of a crash. Could anyone who reproduced this issue before verify the fix?

Untracking for QA testing given comment 15. Thanks for trying, Ioana.

Yeah, this may be really, really hard to QA without having to also try and set up the same weird packaging situation which I doubt will happen in most "normal" distros. This was mostly a function of me running on weird bleeding edge stuff, and it looks like it's even fixed on the bleeding edge now.