Bug 1505536 added Android x86-64 debug builds, including artifact builds, but the artifact builds crash on startup. Apparently libmozglue.so (and others?) are 32 bit. https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=215704969&repo=try&lineNumber=1584 [task 2018-12-06T06:31:39.094Z] 06:31:39 INFO - 12-06 06:31:25.610 E/GeckoLoader( 2420): Couldn't load /data/app/org.mozilla.geckoview.test-1/lib/x86_64/libmozglue.so: java.lang.UnsatisfiedLinkError: dlopen failed: "/data/app/org.mozilla.geckoview.test-1/lib/x86_64/libmozglue.so" is 32-bit instead of 64-bit [task 2018-12-06T06:31:39.094Z] 06:31:39 INFO - 12-06 06:31:25.610 E/GeckoLoader( 2420): Library exists but couldn't load! [task 2018-12-06T06:31:39.094Z] 06:31:39 INFO - 12-06 06:31:25.610 E/GeckoLoader( 2420): Couldn't load /data/user/0/org.mozilla.geckoview.test/lib/libmozglue.so: java.lang.UnsatisfiedLinkError: dlopen failed: library "/data/user/0/org.mozilla.geckoview.test/lib/libmozglue.so" not found [task 2018-12-06T06:31:39.094Z] 06:31:39 INFO - 12-06 06:31:25.610 E/GeckoLoader( 2420): Couldn't load /data/app-lib/org.mozilla.geckoview.test/libmozglue.so: java.lang.UnsatisfiedLinkError: dlopen failed: library "/data/app-lib/org.mozilla.geckoview.test/libmozglue.so" not found [task 2018-12-06T06:31:39.095Z] 06:31:39 INFO - 12-06 06:31:25.610 E/GeckoLoader( 2420): Couldn't load /data/data/org.mozilla.geckoview.test/lib/libmozglue.so: java.lang.UnsatisfiedLinkError: dlopen failed: library "/data/data/org.mozilla.geckoview.test/lib/libmozglue.so" not found [task 2018-12-06T06:31:39.095Z] 06:31:39 INFO - 12-06 06:31:25.610 D/GeckoLoader( 2420): Copying lib/x86_64/libmozglue.so to /data/user/0/org.mozilla.geckoview.test/files/lib/libmozglue.so [task 2018-12-06T06:31:39.095Z] 06:31:39 INFO - 12-06 06:31:25.630 D/EGL_emulation( 1798): eglMakeCurrent: 0x73767b080660: ver 2 0 (tinfo 0x73767b1b53e0) [task 2018-12-06T06:31:39.095Z] 06:31:39 INFO - 12-06 06:31:25.640 D/skia ( 1394): --- SkAndroidCodec::NewFromStream returned null [task 2018-12-06T06:31:39.095Z] 06:31:39 INFO - 12-06 06:31:25.650 E/SurfaceFlinger( 1039): ro.sf.lcd_density must be defined as a build property [task 2018-12-06T06:31:39.096Z] 06:31:39 INFO - 12-06 06:31:25.650 D/ ( 2420): HostConnection::get() New Host Connection established 0x73767aa687e0, tid 2420 [task 2018-12-06T06:31:39.096Z] 06:31:39 INFO - 12-06 06:31:25.660 D/GeckoScreenOrientation( 2420): updating to new orientation PORTRAIT_PRIMARY [task 2018-12-06T06:31:39.096Z] 06:31:39 INFO - 12-06 06:31:25.680 D/ ( 2420): HostConnection::get() New Host Connection established 0x73767aa686c0, tid 2441 [task 2018-12-06T06:31:39.096Z] 06:31:39 INFO - 12-06 06:31:25.680 I/OpenGLRenderer( 2420): Initialized EGL, version 1.4 [task 2018-12-06T06:31:39.096Z] 06:31:39 INFO - 12-06 06:31:25.680 D/OpenGLRenderer( 2420): Swap behavior 1 [task 2018-12-06T06:31:39.096Z] 06:31:39 INFO - 12-06 06:31:25.680 D/EGL_emulation( 2420): eglCreateContext: 0x737688ad1980: maj 2 min 0 rcv 2 [task 2018-12-06T06:31:39.097Z] 06:31:39 INFO - 12-06 06:31:25.680 D/GeckoLoader( 2420): Marking /data/user/0/org.mozilla.geckoview.test/files/lib/libmozglue.so as executable. [task 2018-12-06T06:31:39.097Z] 06:31:39 INFO - 12-06 06:31:25.680 E/GeckoLoader( 2420): Couldn't load /data/user/0/org.mozilla.geckoview.test/files/lib/libmozglue.so: java.lang.UnsatisfiedLinkError: dlopen failed: "/data/data/org.mozilla.geckoview.test/files/lib/libmozglue.so" is 32-bit instead of 64-bit [task 2018-12-06T06:31:39.097Z] 06:31:39 INFO - 12-06 06:31:25.680 E/GeckoLoader( 2420): Load diagnostics: LOAD mozglue: ABI: arm, x86_64: Data: /data/user/0/org.mozilla.geckoview.test, ax=false, ddx=false, -1x=false, -2x=false, nativeLib: /data/app/org.mozilla.geckoview.test-1/lib/x86_64, dirx=true, libx=true [task 2018-12-06T06:31:39.097Z] 06:31:39 INFO - 12-06 06:31:25.680 D/EGL_emulation( 2420): eglMakeCurrent: 0x737688ad1980: ver 2 0 (tinfo 0x7376921b58a0) [task 2018-12-06T06:31:39.097Z] 06:31:39 INFO - --------- beginning of crash [task 2018-12-06T06:31:39.097Z] 06:31:39 INFO - 12-06 06:31:25.680 E/AndroidRuntime( 2420): FATAL EXCEPTION: Gecko [task 2018-12-06T06:31:39.098Z] 06:31:39 INFO - 12-06 06:31:25.680 E/AndroidRuntime( 2420): Process: org.mozilla.geckoview.test, PID: 2420 [task 2018-12-06T06:31:39.098Z] 06:31:39 INFO - 12-06 06:31:25.680 E/AndroidRuntime( 2420): java.lang.RuntimeException: LOAD mozglue: ABI: arm, x86_64: Data: /data/user/0/org.mozilla.geckoview.test, ax=false, ddx=false, -1x=false, -2x=false, nativeLib: /data/app/org.mozilla.geckoview.test-1/lib/x86_64, dirx=true, libx=true [task 2018-12-06T06:31:39.098Z] 06:31:39 INFO - 12-06 06:31:25.680 E/AndroidRuntime( 2420): at org.mozilla.gecko.mozglue.GeckoLoader.doLoadLibrary(GeckoLoader.java:461) [task 2018-12-06T06:31:39.098Z] 06:31:39 INFO - 12-06 06:31:25.680 E/AndroidRuntime( 2420): at org.mozilla.gecko.mozglue.GeckoLoader.loadMozGlue(GeckoLoader.java:469) [task 2018-12-06T06:31:39.098Z] 06:31:39 INFO - 12-06 06:31:25.680 E/AndroidRuntime( 2420): at org.mozilla.gecko.GeckoThread.initGeckoEnvironment(GeckoThread.java:272) [task 2018-12-06T06:31:39.098Z] 06:31:39 INFO - 12-06 06:31:25.680 E/AndroidRuntime( 2420): at org.mozilla.gecko.GeckoThread.run(GeckoThread.java:415) [task 2018-12-06T06:31:39.099Z] 06:31:39 INFO - 12-06 06:31:25.680 E/AndroidRuntime( 2420): Caused by: java.lang.UnsatisfiedLinkError: dlopen failed: "/data/app/org.mozilla.geckoview.test-1/lib/x86_64/libmozglue.so" is 32-bit instead of 64-bit [task 2018-12-06T06:31:39.099Z] 06:31:39 INFO - 12-06 06:31:25.680 E/AndroidRuntime( 2420): at java.lang.Runtime.loadLibrary0(Runtime.java:977) [task 2018-12-06T06:31:39.099Z] 06:31:39 INFO - 12-06 06:31:25.680 E/AndroidRuntime( 2420): at java.lang.System.loadLibrary(System.java:1530) [task 2018-12-06T06:31:39.099Z] 06:31:39 INFO - 12-06 06:31:25.680 E/AndroidRuntime( 2420): at org.mozilla.gecko.mozglue.GeckoLoader.doLoadLibraryExpected(GeckoLoader.java:382) [task 2018-12-06T06:31:39.099Z] 06:31:39 INFO - 12-06 06:31:25.680 E/AndroidRuntime( 2420): at org.mozilla.gecko.mozglue.GeckoLoader.doLoadLibrary(GeckoLoader.java:408)

It looks like all of the libs are 32-bit arm: $ file *.so libfreebl3.so: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=f9ffb743d951408febae3243885a2988308e1563, stripped liblgpllibs.so: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=29aa3b63fc5e4e86623e88a650596c1dc8719b93, stripped libmozavcodec.so: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=96abb620146bfab11e7e54104156463e2ece059f, stripped libmozavutil.so: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=b776e457fa39aa50ac0270f6cf29bba3eb15a8d6, stripped libmozglue.so: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=ba1d33df794fe3c60a5fc89cf6c9ee38384a69cc, stripped libnss3.so: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=2f011acc53af1aae4c3b2917157445cc413e57f9, stripped libnssckbi.so: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=8335fc539f157ee0693c7a965cf71246ac00e609, stripped libplugin-container.so: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /system/bin/linker, BuildID[sha1]=0ea1e1ac5aab2a0fbee1fec557ffb3a0b4206b45, stripped libsoftokn3.so: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=9fe324ab9b2463b5b8c31d17214e08a4d234f71b, stripped libxul.so: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=80fad92f04b4c79de4fa6160901993f9e414af4e, stripped

Nick, do you know what is going wrong here? How can I narrow this down?

(In reply to Geoff Brown [:gbrown] from comment #2) > Nick, do you know what is going wrong here? How can I narrow this down? This is fairly clear: the packaged APK isn't grabbing the libraries from the correct TC job. We need to grow the logic at https://searchfox.org/mozilla-central/source/python/mozbuild/mozbuild/artifacts.py#996-1003 to take into account x86_64. Sorry that I didn't catch this first time around.

Thanks much Nick - that seems to do the job! I manually verified the libs in the target.apk's from: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=45a4a24433256522bf32235912068b92fbdffa87

Comment on attachment 9030568 [details] [diff] [review] expand _guess_artifact_job for android 64 bit Review of attachment 9030568 [details] [diff] [review]: ----------------------------------------------------------------- Makes sense to me.

Pushed by gbrown@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/71e418fed02b Use correct artifact jobs for android 64-bit artifact builds; r=nalexander