Building Firefox 60.2.1 or 60.2.2 with "--enable-geckodriver" using Clang or GCC is failing with: > 42:37.22 error: failed to run custom build command for `bzip2-sys v0.1.6` > 42:37.22 process didn't exit successfully: `/var/tmp/portage/www-client/firefox-60.2.2/work/firefox-60.2.2/ff/testing/geckodriver/./release/build/bzip2-sys-6f8d2d214145b4a9/build-script-build` (exit code: 101) > 42:37.22 --- stdout > 42:37.22 TARGET = Some("x86_64-unknown-linux-gnu") > 42:37.22 OPT_LEVEL = Some("2") > 42:37.22 TARGET = Some("x86_64-unknown-linux-gnu") > 42:37.22 HOST = Some("x86_64-unknown-linux-gnu") > 42:37.22 TARGET = Some("x86_64-unknown-linux-gnu") > 42:37.22 TARGET = Some("x86_64-unknown-linux-gnu") > 42:37.22 HOST = Some("x86_64-unknown-linux-gnu") > 42:37.22 CC_x86_64-unknown-linux-gnu = None > 42:37.22 CC_x86_64_unknown_linux_gnu = None > 42:37.22 HOST_CC = Some("/usr/bin/x86_64-pc-linux-gnu-gcc -std=gnu99") > 42:37.23 TARGET = Some("x86_64-unknown-linux-gnu") > 42:37.23 HOST = Some("x86_64-unknown-linux-gnu") > 42:37.23 CFLAGS_x86_64-unknown-linux-gnu = None > 42:37.23 CFLAGS_x86_64_unknown_linux_gnu = None > 42:37.23 HOST_CFLAGS = None > 42:37.23 CFLAGS = Some("-pipe -march=ivybridge -mtune=ivybridge -mno-xsaveopt -Wno-error=missing-prototypes -Wno-error=enum-compare -Wno-error=unused-function -Wno-error=deprecated-declarations -frecord-gcc-switches") > 42:37.23 DEBUG = Some("false") > 42:37.23 running: "/usr/bin/x86_64-pc-linux-gnu-gcc -std=gnu99" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "-pipe" "-march=ivybridge" "-mtune=ivybridge" "-mno-xsaveopt" "-Wno-error=missing-prototypes" "-Wno-error=enum-compare" "-Wno-error=unused-function" "-Wno-error=deprecated-declarations" "-frecord-gcc-switches" "-m64" "-I" "bzip2-1.0.6" "-DBZ_NO_STDIO" "-o" "/var/tmp/portage/www-client/firefox-60.2.2/work/firefox-60.2.2/ff/testing/geckodriver/./x86_64-unknown-linux-gnu/release/build/bzip2-sys-17a1b1ab7e70b8a0/out/bzip2-1.0.6/blocksort.o" "-c" "bzip2-1.0.6/blocksort.c" > 42:37.23 > 42:37.23 --- stderr > 42:37.23 thread 'main' panicked at ' > 42:37.23 > 42:37.23 Internal error occurred: Failed to find tool. Is `x86_64-pc-linux-gnu-gcc -std=gnu99` installed? > 42:37.23 > 42:37.23 ', third_party/rust/cc/src/lib.rs:1984:5 > 42:37.23 stack backtrace: > 42:37.23 0: 0x55c4612301de - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::h9aa5ffc216cd8bab > 42:37.23 1: 0x55c461222106 - std::sys_common::backtrace::print::h20333f3cf15259f2 > 42:37.23 2: 0x55c4612200dd - std::panicking::default_hook::{{closure}}::hd1cdeeefd64cf4be > 42:37.24 3: 0x55c46121fe50 - std::panicking::default_hook::h386fdb6b44148c45 > 42:37.24 4: 0x55c46122072c - std::panicking::rust_panic_with_hook::h655ee35696268a2c > 42:37.24 5: 0x55c4612204d9 - std::panicking::continue_panic_fmt::h9e6d68e1d0b2da5c > 42:37.24 6: 0x55c46122043d - std::panicking::begin_panic_fmt::h8726b3c9c9f319ff > 42:37.24 7: 0x55c461219ca1 - cc::fail::h9d6d9599e20fc10f > 42:37.24 8: 0x55c46120f07e - cc::Build::compile::h8192787e03a59719 > 42:37.24 9: 0x55c4612047e9 - build_script_build::main::h998515e8f8daa082 > 42:37.24 10: 0x55c461203c12 - std::rt::lang_start::{{closure}}::h44498c5e97f8754d > 42:37.24 11: 0x55c4612201e2 - std::panicking::try::do_call::hf328099dc8204c6c > 42:37.25 12: 0x55c46123cc69 - __rust_maybe_catch_panic > 42:37.25 13: 0x55c46121ad95 - std::rt::lang_start_internal::h3d854016d73be659 > 42:37.26 14: 0x55c461204963 - main > 42:37.26 15: 0x7fb6246c050a - __libc_start_main > 42:37.26 at ../csu/libc-start.c:308 > 42:37.26 16: 0x55c4612038f9 - _start > 42:37.26 17: 0x0 - <unknown> > 42:37.26 > 42:37.26 warning: build failed, waiting for other jobs to finish... We can build geckodriver in 62.x branch (i.e. 62.0.2 or 62.0.3). > # gcc -v > Using built-in specs. > COLLECT_GCC=gcc > COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-pc-linux-gnu/8.2.0/lto-wrapper > Target: x86_64-pc-linux-gnu > Configured with: /var/tmp/portage/sys-devel/gcc-8.2.0-r3/work/gcc-8.2.0/configure --host=x86_64-pc-linux-gnu --build=x86_64-pc-linux-gnu --prefix=/usr --bindir=/usr/x86_64-pc-linux-gnu/gcc-bin/8.2.0 --includedir=/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.0/include --datadir=/usr/share/gcc-data/x86_64-pc-linux-gnu/8.2.0 --mandir=/usr/share/gcc-data/x86_64-pc-linux-gnu/8.2.0/man --infodir=/usr/share/gcc-data/x86_64-pc-linux-gnu/8.2.0/info --with-gxx-include-dir=/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.0/include/g++-v8 --with-python-dir=/share/gcc-data/x86_64-pc-linux-gnu/8.2.0/python --enable-languages=c,c++,fortran --enable-obsolete --enable-secureplt --disable-werror --with-system-zlib --enable-nls --without-included-gettext --enable-checking=release --with-bugurl=https://bugs.gentoo.org/ --with-pkgversion='Gentoo 8.2.0-r3 p1.3' --disable-esp --enable-libstdcxx-time --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu --enable-multilib --with-multilib-list=m32,m64 --disable-altivec --disable-fixed-point --enable-targets=all --enable-libgomp --disable-libmudflap --disable-libssp --disable-libmpx --disable-systemtap --enable-vtable-verify --enable-libvtv --enable-lto --without-isl --enable-libsanitizer --enable-default-pie --enable-default-ssp > Thread model: posix > gcc version 8.2.0 (Gentoo 8.2.0-r3 p1.3) > # clang -v > clang version 7.0.0 (tags/RELEASE_700/final) > Target: x86_64-pc-linux-gnu > Thread model: posix > InstalledDir: /usr/lib/llvm/7/bin > Selected GCC installation: /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.0 > Candidate multilib: .;@m64 > Candidate multilib: 32;@m32 > Selected multilib: .;@m64

So the failure message is: > Internal error occurred: Failed to find tool. Is `x86_64-pc-linux-gnu-gcc -std=gnu99` installed? It looks like it searches for `x86_64-pc-linux-gnu-gcc` but `/usr/lib/gcc/x86_64-pc-linux-gnu` is installed? I don't know about the package ecosystem on Gentoo, but does a symlink here maybe help?

No, a missing symlink isn't the problem. See the failure when using Clang: > 49:11.30 error: failed to run custom build command for `bzip2-sys v0.1.6` > 49:11.30 process didn't exit successfully: `/var/tmp/portage/www-client/firefox-60.2.2/work/firefox-60.2.2/ff/testing/geckodriver/./release/build/bzip2-sys-545eaa606c062b5d/build-script-build` (exit code: 101) > 49:11.30 --- stdout > 49:11.30 TARGET = Some("x86_64-unknown-linux-gnu") > 49:11.30 OPT_LEVEL = Some("2") > 49:11.31 TARGET = Some("x86_64-unknown-linux-gnu") > 49:11.31 HOST = Some("x86_64-unknown-linux-gnu") > 49:11.31 TARGET = Some("x86_64-unknown-linux-gnu") > 49:11.31 TARGET = Some("x86_64-unknown-linux-gnu") > 49:11.31 HOST = Some("x86_64-unknown-linux-gnu") > 49:11.31 CC_x86_64-unknown-linux-gnu = None > 49:11.31 CC_x86_64_unknown_linux_gnu = None > 49:11.31 HOST_CC = Some("/usr/lib/llvm/7/bin/x86_64-pc-linux-gnu-clang -std=gnu99") > 49:11.31 TARGET = Some("x86_64-unknown-linux-gnu") > 49:11.32 HOST = Some("x86_64-unknown-linux-gnu") > 49:11.32 CFLAGS_x86_64-unknown-linux-gnu = None > 49:11.32 CFLAGS_x86_64_unknown_linux_gnu = None > 49:11.32 HOST_CFLAGS = None > 49:11.32 CFLAGS = Some("-pipe -march=ivybridge -mtune=ivybridge -mno-xsaveopt -Wno-error=missing-prototypes -Wno-error=enum-compare -Wno-error=unused-function -Wno-error=deprecated-declarations") > 49:11.32 DEBUG = Some("false") > 49:11.32 running: "/usr/lib/llvm/7/bin/x86_64-pc-linux-gnu-clang -std=gnu99" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "-pipe" "-march=ivybridge" "-mtune=ivybridge" "-mno-xsaveopt" "-Wno-error=missing-prototypes" "-Wno-error=enum-compare" "-Wno-error=unused-function" "-Wno-error=deprecated-declarations" "--target=x86_64-unknown-linux-gnu" "-I" "bzip2-1.0.6" "-DBZ_NO_STDIO" "-o" "/var/tmp/portage/www-client/firefox-60.2.2/work/firefox-60.2.2/ff/testing/geckodriver/./x86_64-unknown-linux-gnu/release/build/bzip2-sys-cdca6cdfca0915a9/out/bzip2-1.0.6/blocksort.o" "-c" "bzip2-1.0.6/blocksort.c" > 49:11.32 > 49:11.33 --- stderr > 49:11.33 thread 'main' panicked at ' > 49:11.33 > 49:11.33 Internal error occurred: Failed to find tool. Is `x86_64-pc-linux-gnu-clang -std=gnu99` installed? > 49:11.33 > 49:11.34 ', third_party/rust/cc/src/lib.rs:1984:5 > 49:11.34 stack backtrace: > 49:11.34 0: 0x558a4a6407ae - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::he72067c4756fc64f > 49:11.34 at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49 > 49:11.34 1: 0x558a4a631056 - std::sys_common::backtrace::print::h88c229e1b073f122 > 49:11.34 at libstd/sys_common/backtrace.rs:71 > 49:11.34 at libstd/sys_common/backtrace.rs:59 > 49:11.34 2: 0x558a4a62e4fd - std::panicking::default_hook::{{closure}}::hb998174544b4cbff > 49:11.35 at libstd/panicking.rs:211 > 49:11.36 3: 0x558a4a62e270 - std::panicking::default_hook::h2a69458a7b4a8d95 > 49:11.36 at libstd/panicking.rs:227 > 49:11.36 4: 0x558a4a62eb1c - std::panicking::rust_panic_with_hook::h1b0bf5031744c801 > 49:11.36 at libstd/panicking.rs:475 > 49:11.36 5: 0x558a4a62e719 - std::panicking::continue_panic_fmt::h46dd3d92ccb20144 > 49:11.36 at libstd/panicking.rs:390 > 49:11.36 6: 0x558a4a62e67d - std::panicking::begin_panic_fmt::hc0cc09bb1a210514 > 49:11.36 at libstd/panicking.rs:345 > 49:11.36 7: 0x558a4a62a5c1 - cc::fail::h4aba75b28ac954d2 > 49:11.37 8: 0x558a4a61ee5e - cc::Build::compile::h6369b354c0cac3a6 > 49:11.37 9: 0x558a4a613f63 - build_script_build::main::hc422d7e31efa21f1 > 49:11.37 10: 0x558a4a613332 - std::rt::lang_start::{{closure}}::h6819c502d40c00db > 49:11.37 11: 0x558a4a62e602 - std::panicking::try::do_call::hdbb9d33e960bebf1 > 49:11.37 at libstd/rt.rs:59 > 49:11.37 at libstd/panicking.rs:310 > 49:11.37 12: 0x558a4a64d209 - __rust_maybe_catch_panic > 49:11.37 at libpanic_unwind/lib.rs:105 > 49:11.38 13: 0x558a4a6301c5 - std::rt::lang_start_internal::h4fca12a3897276e1 > 49:11.38 at libstd/panicking.rs:289 > 49:11.38 at libstd/panic.rs:392 > 49:11.38 at libstd/rt.rs:58 > 49:11.38 14: 0x558a4a6140c3 - main > 49:11.38 15: 0x7ff096d5f50a - __libc_start_main > 49:11.38 at ../csu/libc-start.c:308 > 49:11.38 16: 0x558a4a613029 - _start > 49:11.38 17: 0x0 - <unknown> > 49:11.39 > 49:11.39 gmake[4]: *** [/var/tmp/portage/www-client/firefox-60.2.2/work/firefox-60.2.2/config/rules.mk:1017: force-cargo-program-build] Error 101 > 49:11.39 gmake[4]: Leaving directory '/var/tmp/portage/www-client/firefox-60.2.2/work/firefox-60.2.2/ff/testing/geckodriver' > 49:11.39 gmake[3]: *** [/var/tmp/portage/www-client/firefox-60.2.2/work/firefox-60.2.2/config/recurse.mk:73: testing/geckodriver/target] Error 2 > 49:11.39 gmake[3]: *** Waiting for unfinished jobs.... /usr/lib/llvm/7/bin/x86_64-pc-linux-gnu-clang exists (like the gcc counterpart). And if there would be a missing symlink or something like that, wouldn't this affect other 'rust packages'? But firefox is built successfully with geckodriver disabled. See https://bugzilla.mozilla.org/show_bug.cgi?id=1367995 (and https://github.com/alexcrichton/bzip2-rs/issues/30). Same issue but affecting "Windows".

Lets see what Ted thinks about this problem. Here has way more knowledge than me when it comes to build steps with Rust.

I fixed this in bug 1409276 which landed in the Firefox 63 timeframe. If you look at your failing log there you'll see: > 42:37.22 HOST_CC = Some("/usr/bin/x86_64-pc-linux-gnu-gcc -std=gnu99") You're presumably setting `HOST_CC` to that string, and the Rust cc crate is trying to find a binary with that name (including the space). This was fixed upstream in the cc crate, but the vendored version in ESR60 doesn't contain that fix: https://github.com/alexcrichton/cc-rs/commit/0e405f9e62ce1cb8132002febc55c1e31788c31d#diff-b4aea3e418ccdb71239b96952d9cddb6

Oh, shall we then try to vendor in the new version of the cc crate on the esr60 branch?

We could do that, it shouldn't be harmful. You'd need to run `cargo update -p cc && ./mach vendor rust`.

Thanks Ted, Upstream commit helped me to find https://bugzilla.mozilla.org/show_bug.cgi?id=1445528. For Gentoo I will just backport changes from "third_party/rust/cc/src/lib.rs" file and update ".cargo-checksum.json". First test run with Clang was successful. Henrik, I would appreciate if you could update so that we can drop our (Gentoo) patch with next ESR update and use a complete updated cc package.

Comment on attachment 9018182 [details] Bug 1496270 - Upgrade cc crate to 1.0.25. r?ted [ESR Uplift Approval Request] If this is not a sec:{high,crit} bug, please state case for ESR consideration: The cc crate had a bug where it would ignore compiler arguments in compilers specified in environment variables. This is basically a backport of https://hg.mozilla.org/integration/mozilla-inbound/rev/ee4bd7464329a1b284d484554e22fdc1ee6fb954. User impact if declined: None, only distributors and developers are affected Fix Landed on Version: 63 Risk to taking this patch: Low Why is the change risky/not risky? (and alternatives if risky): Failures only happen during compile time String or UUID changes made by this patch: None

I can approve and land this now, but note that it won't be included in the tagged 60.3.0 release (that's already built and ready to ship on Tuesday). It'll be available if you build directly from ESR60 tip or for the next tagged release (whatever that is).

Comment on attachment 9018182 [details] Bug 1496270 - Upgrade cc crate to 1.0.25. r?ted Fixes NPOTB code causing build bustage for downstream users. Approved for ESR60.