I am currently unable to build mozilla-central for Android on my macOS 10.12 machine.

The failure happens when compiling the lmdb-rkv-sys 0.8.2 crate:
cargo is running the build script for lmdb-rkv-sys, which then uses the cc crate to compile the LMDB C library.

The interesting part of the log seems to be this piece:

5:45.10 cargo:warning=dyld: Symbol not found: __ZN4llvm2cl23ParseCommandLineOptionsEiPKPKcNS_9StringRefEPNS_11raw_ostreamE
5:45.10 cargo:warning= Referenced from: /Users/mstange/.mozbuild/clang/bin/clang-7
5:45.10 cargo:warning= Expected in: /Users/mstange/.rustup/toolchains/stable-x86_64-apple-darwin/lib/libLLVM.dylib

My .mozconfig contains these lines (among others):

CC="/Users/mstange/.mozbuild/clang/bin/clang"
CXX="/Users/mstange/.mozbuild/clang/bin/clang++"

It looks like something is picking up the wrong libLLVM.dylib: It's trying to link .mozbuild/clang/bin/clang-7 against .rustup/toolchains/stable-x86_64-apple-darwin/lib/libLLVM.dylib instead of against .mozbuild/clang/lib/libLLVM.dylib.

I can reproduce on both macOS 10.13 and 10.14. I too have CC/CXX set to the clang recommended and installed by mach bootstrap. Using the system clang works around this problem, which is presumably why it doesn't happen in CI.

Running just this command passes for me: "/Users/mstange/.cargo/bin/sccache" "/Users/mstange/.mozbuild/clang/bin/clang" "-std=gnu99" "--target=arm-linux-androideabi" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "-I/Users/mstange/code/obj-m-fxr-android-opt/dist/system_wrappers" "-include" "/Users/mstange/code/fxr-gecko/config/gcc_hidden.h" "-DNDEBUG=1" "-DTRIMMED=1" "-I/Users/mstange/code/fxr-gecko/toolkit/library/rust" "-I/Users/mstange/code/obj-m-fxr-android-opt/toolkit/library/rust" "-I/Users/mstange/code/obj-m-fxr-android-opt/dist/include" "-I/Users/mstange/code/obj-m-fxr-android-opt/dist/include/nspr" "-I/Users/mstange/code/obj-m-fxr-android-opt/dist/include/nss" "-fPIC" "-include" "/Users/mstange/code/obj-m-fxr-android-opt/mozilla-config.h" "-DMOZILLA_CLIENT" "-Qunused-arguments" "-isystem" "/Users/mstange/.mozbuild/android-ndk-r17b/sysroot/usr/include/arm-linux-androideabi" "-isystem" "/Users/mstange/.mozbuild/android-ndk-r17b/sysroot/usr/include" "-gcc-toolchain" "/Users/mstange/.mozbuild/android-ndk-r17b/toolchains/arm-linux-androideabi-4.9/prebuilt/darwin-x86_64" "-D__ANDROID_API__=16" "-fstack-protector-strong" "-fno-short-enums" "-fno-exceptions" "-march=armv7-a" "-mthumb" "-mfpu=neon" "-mfloat-abi=softfp" "-mno-unaligned-access" "-fstack-protector-strong" "-fno-strict-aliasing" "-ffunction-sections" "-fdata-sections" "-fno-math-errno" "-fPIC" "-pipe" "-g" "-Oz" "-mno-outline" "-fno-omit-frame-pointer" "-funwind-tables" "-DMOZILLA_CONFIG_H" "--target=armv7-linux-androideabi" "-o" "/Users/mstange/code/obj-m-fxr-android-opt/armv7-linux-androideabi/release/build/lmdb-rkv-sys-e921df4ec972af65/out/mdb.o" "-c" "/Users/mstange/code/fxr-gecko/third_party/rust/lmdb-rkv-sys/lmdb/libraries/liblmdb/mdb.c"

I'm assuming that's because something (cargo?) sets DYLD_LIBRARY_PATH to something I'm not seeing?

Prepending DYLD_LIBRARY_PATH="$HOME/.rustup/toolchains/stable-x86_64-apple-darwin/lib/" to the above command reproduces the failure.

Rust 1.32.0 doesn't have this problem, but Rust 1.33.0 does. And when I build lmdb-rkv-sys in isolation with very verbose output, I see that 1.33.0 prepends DYLD_FALLBACK_LIBRARY_PATH to the rustc invocation that builds the lmdb-rkv-sys build script:

> export CC="/Users/myk/.mozbuild/clang/bin/clang"
> cd third_party/rust/lmdb-rkv-sys/
> cargo clean && cargo +1.33.0 -vv build
…
Compiling lmdb-rkv-sys v0.8.2 (/Users/myk/Projects/gecko/third_party/rust/lmdb-rkv-sys)
Running CARGO_PKG_NAME=lmdb-rkv-sys CARGO_PKG_REPOSITORY='https://github.com/mozilla/lmdb-rs.git' CARGO_PKG_VERSION_MINOR=8 CARGO_PKG_VERSION_MAJOR=0 CARGO=/Users/myk/.rustup/toolchains/1.33.0-x86_64-apple-darwin/bin/cargo CARGO_MANIFEST_DIR=/Users/myk/Projects/gecko/third_party/rust/lmdb-rkv-sys CARGO_PKG_HOMEPAGE= CARGO_PRIMARY_PACKAGE=1 CARGO_PKG_AUTHORS='Dan Burkert <dan@danburkert.com>' CARGO_PKG_VERSION_PRE= CARGO_PKG_VERSION_PATCH=2 CARGO_PKG_DESCRIPTION='Rust bindings for liblmdb.' CARGO_PKG_VERSION=0.8.2 DYLD_FALLBACK_LIBRARY_PATH='/Users/myk/Projects/gecko/third_party/rust/lmdb-rkv-sys/target/debug/deps:/Users/myk/.rustup/toolchains/1.33.0-x86_64-apple-darwin/lib:/Users/myk/lib:/usr/local/lib:/lib:/usr/lib' rustc --crate-name build_script_build build.rs --color always --crate-type bin --emit=dep-info,link -C debuginfo=2 -C metadata=83d2eb17ef3d795f -C extra-filename=-83d2eb17ef3d795f --out-dir /Users/myk/Projects/gecko/third_party/rust/lmdb-rkv-sys/target/debug/build/lmdb-rkv-sys-83d2eb17ef3d795f -C incremental=/Users/myk/Projects/gecko/third_party/rust/lmdb-rkv-sys/target/debug/incremental -L dependency=/Users/myk/Projects/gecko/third_party/rust/lmdb-rkv-sys/target/debug/deps --extern cc=/Users/myk/Projects/gecko/third_party/rust/lmdb-rkv-sys/target/debug/deps/libcc-c5f969f3e72193cf.rlib --extern pkg_config=/Users/myk/Projects/gecko/third_party/rust/lmdb-rkv-sys/target/debug/deps/libpkg_config-e6da1e3007c93d93.rlib

Running that build script then fails with a different symbol not found:

Running /Users/myk/Projects/gecko/third_party/rust/lmdb-rkv-sys/target/debug/build/lmdb-rkv-sys-83d2eb17ef3d795f/build-script-build
[lmdb-rkv-sys 0.8.2] TARGET = Some("x86_64-apple-darwin")
[lmdb-rkv-sys 0.8.2] HOST = Some("x86_64-apple-darwin")
[lmdb-rkv-sys 0.8.2] CC_x86_64-apple-darwin = None
[lmdb-rkv-sys 0.8.2] CC_x86_64_apple_darwin = None
[lmdb-rkv-sys 0.8.2] HOST_CC = None
[lmdb-rkv-sys 0.8.2] CC = Some("/Users/myk/.mozbuild/clang/bin/clang")
[lmdb-rkv-sys 0.8.2] CFLAGS_x86_64-apple-darwin = None
[lmdb-rkv-sys 0.8.2] CFLAGS_x86_64_apple_darwin = None
[lmdb-rkv-sys 0.8.2] HOST_CFLAGS = None
[lmdb-rkv-sys 0.8.2] CFLAGS = None
[lmdb-rkv-sys 0.8.2] CRATE_CC_NO_DEFAULTS = None
[lmdb-rkv-sys 0.8.2] DEBUG = Some("true")
[lmdb-rkv-sys 0.8.2] running: "/Users/myk/.mozbuild/clang/bin/clang" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "--target=x86_64-apple-darwin" "-Wall" "-Wextra" "-o" "/Users/myk/Projects/gecko/third_party/rust/lmdb-rkv-sys/target/debug/build/lmdb-rkv-sys-a1d6ff0710803295/out/mdb.o" "-c" "/Users/myk/Projects/gecko/third_party/rust/lmdb-rkv-sys/lmdb/libraries/liblmdb/mdb.c"
[lmdb-rkv-sys 0.8.2] cargo:warning=dyld: lazy symbol binding failed: Symbol not found: __ZN4llvm9DIBuilder17createCompileUnitEjPNS_6DIFileENS_9StringRefEbS3_jS3_NS_13DICompileUnit17DebugEmissionKindEybbb

Building with Rust 1.32.0, on the other hand, does not prepend DYLD_FALLBACK_LIBRARY_PATH to the rustc invocation:

> cargo clean && cargo +1.32.0 -vv build
…
Compiling lmdb-rkv-sys v0.8.2 (/Users/myk/Projects/gecko/third_party/rust/lmdb-rkv-sys)
Running rustc --crate-name build_script_build build.rs --color always --crate-type bin --emit=dep-info,link -C debuginfo=2 -C metadata=804700b471b5eb90 -C extra-filename=-804700b471b5eb90 --out-dir /Users/myk/Projects/gecko/third_party/rust/lmdb-rkv-sys/target/debug/build/lmdb-rkv-sys-804700b471b5eb90 -C incremental=/Users/myk/Projects/gecko/third_party/rust/lmdb-rkv-sys/target/debug/incremental -L dependency=/Users/myk/Projects/gecko/third_party/rust/lmdb-rkv-sys/target/debug/deps --extern cc=/Users/myk/Projects/gecko/third_party/rust/lmdb-rkv-sys/target/debug/deps/libcc-1bfc39f05156f088.rlib --extern pkg_config=/Users/myk/Projects/gecko/third_party/rust/lmdb-rkv-sys/target/debug/deps/libpkg_config-5888e3576dc8f315.rlib

And the build script succeeds:

Running /Users/myk/Projects/gecko/third_party/rust/lmdb-rkv-sys/target/debug/build/lmdb-rkv-sys-804700b471b5eb90/build-script-build
[lmdb-rkv-sys 0.8.2] TARGET = Some("x86_64-apple-darwin")
[lmdb-rkv-sys 0.8.2] HOST = Some("x86_64-apple-darwin")
[lmdb-rkv-sys 0.8.2] CC_x86_64-apple-darwin = None
[lmdb-rkv-sys 0.8.2] CC_x86_64_apple_darwin = None
[lmdb-rkv-sys 0.8.2] HOST_CC = None
[lmdb-rkv-sys 0.8.2] CC = Some("/Users/myk/.mozbuild/clang/bin/clang")
[lmdb-rkv-sys 0.8.2] CFLAGS_x86_64-apple-darwin = None
[lmdb-rkv-sys 0.8.2] CFLAGS_x86_64_apple_darwin = None
[lmdb-rkv-sys 0.8.2] HOST_CFLAGS = None
[lmdb-rkv-sys 0.8.2] CFLAGS = None
[lmdb-rkv-sys 0.8.2] CRATE_CC_NO_DEFAULTS = None
[lmdb-rkv-sys 0.8.2] DEBUG = Some("true")
[lmdb-rkv-sys 0.8.2] running: "/Users/myk/.mozbuild/clang/bin/clang" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "--target=x86_64-apple-darwin" "-Wall" "-Wextra" "-o" "/Users/myk/Projects/gecko/third_party/rust/lmdb-rkv-sys/target/debug/build/lmdb-rkv-sys-fb4a04cbaa8c4e1d/out/mdb.o" "-c" "/Users/myk/Projects/gecko/third_party/rust/lmdb-rkv-sys/lmdb/libraries/liblmdb/mdb.c"
[lmdb-rkv-sys 0.8.2] cargo:warning=/Users/myk/Projects/gecko/third_party/rust/lmdb-rkv-sys/lmdb/libraries/liblmdb/mdb.c:10073:33: warning: unused parameter 'env' [-Wunused-parameter]
[lmdb-rkv-sys 0.8.2] cargo:warning=mdb_env_get_maxkeysize(MDB_env *env)
[lmdb-rkv-sys 0.8.2] cargo:warning= ^
[lmdb-rkv-sys 0.8.2] cargo:warning=1 warning generated.
[lmdb-rkv-sys 0.8.2] exit code: 0

So this seems to be a regression caused by a change in Rust 1.33.0.

Thanks. I did rustup install 1.32.0 && rustup default 1.32.0 && rustup target add armv7-linux-androideabi and am building now. I think this unblocks me.

Yes, building with 1.32.0 worked.

This minimal testcase of a Rust crate that compiles a C library using the cc crate demonstrates the problem:

https://github.com/mykmelez/rust-c-library-testcase

And the problem occurs regardless of the build target. So it isn't Android-specific.

However, I can't reproduce it with clang 6.0 or 7.0 downloaded directly from the LLVM project (http://releases.llvm.org/download.html). Nor can I reproduce it with clang 6.0.1 or 7.0.1 installed by Homebrew. So it seems specific to the clang installation provided by mach bootstrap.

(In reply to Myk Melez [:myk] [@mykmelez] from comment #6)

This minimal testcase of a Rust crate that compiles a C library using the cc crate demonstrates the problem:

https://github.com/mykmelez/rust-c-library-testcase

And the problem occurs regardless of the build target. So it isn't Android-specific.

It also isn't specific to the lmdb-rkv-sys crate in mozilla-central. It also occurs with libloading, and it'll presumably occur with any crate that compiles a C/C++ library.

Nevertheless, Android is the target platform for which mach bootstrap recommends setting CC/CXX to mach bootstrap-provided clang (https://searchfox.org/mozilla-central/rev/358f816f63da072145c593e9e2ac36b7250ecd25/python/mozboot/mozboot/android.py#66-68), so that's where developers are most likely to encounter it.

However, I can't reproduce it with clang 6.0 or 7.0 downloaded directly from the LLVM project (http://releases.llvm.org/download.html). Nor can I reproduce it with clang 6.0.1 or 7.0.1 installed by Homebrew. So it seems specific to the clang installation provided by mach bootstrap.

This is presumably because clang from LLVM and Homebrew is statically linked.

Nick had the same problem yesterday, and what I told him was that a bug should be opened against cbindgen, because since it's responsible for the loading of libclang, it should ensure that the right libLLVM is going to be picked. That might mean adding the directory libclang is in to DYLD_LIBRARY_PATH on mac, and doing equivalent things on other platforms.

That being said, it would also be worth fiding out why rust is setting DYLD_FALLBACK_LIBRARY_PATH on its own, because that doesn't seem right either.

Whoops, I filed this upstream on rust-lang/cargo before I saw the last comment:

https://github.com/rust-lang/cargo/issues/6764

It isn't clear that DYLD_FALLBACK_LIBRARY_PATH is the culprit, however. Comparing the 2019-01-06 and 2019-01-07 Rust nightlies, which is when this seems to have regressed, the value of DYLD_FALLBACK_LIBRARY_PATH doesn't change. Following my steps to reproduce in that issue, the only difference in its value is the date in the nightly toolchain directory name:

2019-01-06: DYLD_FALLBACK_LIBRARY_PATH='/Users/myk/rust-c-library-testcase/target/debug/deps:/Users/myk/.rustup/toolchains/nightly-2019-01-06-x86_64-apple-darwin/lib'
2019-01-07: DYLD_FALLBACK_LIBRARY_PATH='/Users/myk/rust-c-library-testcase/target/debug/deps:/Users/myk/.rustup/toolchains/nightly-2019-01-07-x86_64-apple-darwin/lib'

Nick had the same problem yesterday

Not sure it was exactly the same, my error said "unable to find libclang" and then something about dlopen() failing while running the build script for the style crate.

In the end I fixed it by switching my package manager from MacPorts to Homebrew.

(In reply to Myk Melez [:myk] [@mykmelez] from comment #9)

Whoops, I filed this upstream on rust-lang/cargo before I saw the last comment:

https://github.com/rust-lang/cargo/issues/6764

It isn't clear that DYLD_FALLBACK_LIBRARY_PATH is the culprit, however. Comparing the 2019-01-06 and 2019-01-07 Rust nightlies, which is when this seems to have regressed, the value of DYLD_FALLBACK_LIBRARY_PATH doesn't change. Following my steps to reproduce in that issue, the only difference in its value is the date in the nightly toolchain directory name:

2019-01-06: DYLD_FALLBACK_LIBRARY_PATH='/Users/myk/rust-c-library-testcase/target/debug/deps:/Users/myk/.rustup/toolchains/nightly-2019-01-06-x86_64-apple-darwin/lib'
2019-01-07: DYLD_FALLBACK_LIBRARY_PATH='/Users/myk/rust-c-library-testcase/target/debug/deps:/Users/myk/.rustup/toolchains/nightly-2019-01-07-x86_64-apple-darwin/lib'

Did libLLVM.dylib have a different name before?

(In reply to Mike Hommey [:glandium] from comment #11)

Did libLLVM.dylib have a different name before?

Hmm, 2019-01-06 doesn't have that file in lib/, while 2019-01-07 does:

> cd ~/.rustup/toolchains && ls -l nightly-2019-01-06-x86_64-apple-darwin/lib/libLLVM.dylib nightly-2019-01-07-x86_64-apple-darwin/lib/libLLVM.dylib
ls: nightly-2019-01-06-x86_64-apple-darwin/lib/libLLVM.dylib: No such file or directory
-rw-r--r-- 1 myk staff 58531716 Mar 19 14:04 nightly-2019-01-07-x86_64-apple-darwin/lib/libLLVM.dylib

They both have it in lib/rustlib/x86_64-apple-darwin/lib/:

> cd ~/.rustup/toolchains && ls -l nightly-2019-01-06-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libLLVM.dylib nightly-2019-01-07-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libLLVM.dylib
-rw-r--r-- 1 myk staff 58531716 Mar 19 14:03 nightly-2019-01-06-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libLLVM.dylib
-rw-r--r-- 1 myk staff 58531716 Mar 19 14:04 nightly-2019-01-07-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libLLVM.dylib

There you go... there might also be a bug in rust or rustup there, one is not a symlink or hardlink of the other, so that's taking double the disk space. I'd still argue rust should avoid fiddling with *_LIBRARY_PATH, and that cbindgen should try harder to ensure the right libLLVM is used.

Alex, can you take a look at the discussion in this bug?

Sure yeah! It looks like this may be similar to https://github.com/rust-lang/rust/issues/59034 which is in theory fixed by https://github.com/rust-lang/rust/pull/59173 and should be in nightlies now

Yeah, that unfortunately only fixes it on Linux, because for some reason LLVM doesn't add the name for OSX. The code doing that is in an if (NOT APPLE) block:

https://github.com/llvm/llvm-project/blob/master/llvm/cmake/modules/AddLLVM.cmake#L535

http://reviews.llvm.org/D13841 has the initial implementation of this which already excluded Darwin targets. Not sure if there's a reason for that.

I think that the NOT APPLE condition should be removed from LLVM. llvm-config already makes the wrong assumption that the name is actually changed:

https://github.com/llvm/llvm-project/blob/master/llvm/tools/llvm-config/llvm-config.cpp#L375

I'd be happy to submit that patch, but I'm not on OSX, so somebody would need to test it and such.

Filed https://github.com/KyleMayes/clang-sys/issues/89

In the end, this turns out that recent change in location of libLLVM.dylib in rust unveiled an ages old problem that was fixed in cargo in https://github.com/rust-lang/cargo/pull/6355, but still affects rustup.

Opened https://github.com/rust-lang/rustup.rs/pull/1752.

It should be possible to build with rust 1.33 if you use:

CARGO=`rustup which cargo`

Should this patch remove the early-fail I added in bug 1542862?

I'll get bug 1542862 backed out.

https://hg.mozilla.org/mozilla-central/rev/c1f7767c6896