Kelsey Gilbert [:jgilbert] Reporter
	Description • 7 years ago

We're not quite done with GCC<6, so we can't rely on having GCC>=6 yet.

	Zsolt Fay [:zfay]
	Comment 1 • 6 years ago

Any updates on this matter? Jeff I'll assign this bug to you. 

Please let someone from the build duty team know when we should remove GCC<6. Does this sound good for you?

	Kelsey Gilbert [:jgilbert] Reporter
	Comment 2 • 6 years ago

My needs are met now, so I'm going to cede this cleanup to RelEng.

	Jordan Lund (:jlund)
	Comment 3 • 6 years ago

I see in Taskcluster scheduling, 4.9 still referenced in places: https://dxr.mozilla.org/mozilla-central/search?q=path%3Ataskcluster+gcc-4&redirect=false

That's all in-tree and self serve'able if you want to update.

If there are places it's used in Buildbot, we can ignore that and let it die with the rest of Buildbot infra.

	Jordan Lund (:jlund)
	Comment 4 • 6 years ago

(In reply to Jordan Lund (:jlund) from comment #3)
> I see in Taskcluster scheduling, 4.9 still referenced in places:
> https://dxr.mozilla.org/mozilla-central/search?q=path%3Ataskcluster+gcc-
> 4&redirect=false
> 
> That's all in-tree and self serve'able if you want to update.
> 
> If there are places it's used in Buildbot, we can ignore that and let it die
> with the rest of Buildbot infra.

jgilbert - I'm a bit out of touch with current state, did you need a hand updating gcc versions anywhere in-tree (e.g. particular tasks)? I see you have touched the above taskcluster/ci/* scheduling files before.

Or is this bug filed in case there are places outside of taskcluster/in-tree that you are not familiar with and you're just giving releng a heads up?

	Kelsey Gilbert [:jgilbert] Reporter
	Comment 5 • 6 years ago

This is just cleanup now that we require GCC-6.1+. Since we require 6.1+ now, anything that references <6.1 is out-of-date or orphaned, but I can't tell which. This was all outside my domain, so this is a note for others that there's still cleanup to be done, since I can't easily do it myself.

	Jordan Lund (:jlund)
	Comment 6 • 6 years ago

This is beyond buildduty scope (for now) to action. However, since they are familiar with CI and who owns what, filing here is good as they can escalate appropriately. Moving to Releng inbox so it's on the radar there.

	Bogdan Crisan [:bcrisan] (EEST - GMT + 3)
	Comment 7 • 6 years ago

Hi,

We have found this in triaging, and I was wondering if you are the guy or team that we should escalate this kind of problems, if not, can you point us to the right direction?

	John Ford [:jhford] CET/CEST Berlin Time
	Comment 8 • 6 years ago

(In reply to Bogdan Crisan [:bcrisan] from comment #7)
> Hi,
> 
> We have found this in triaging, and I was wondering if you are the guy or
> team that we should escalate this kind of problems, if not, can you point us
> to the right direction?

Hi Bogdan, I'm really not sure about this bug.

Greg, do you know if this is the correct component for this bug or which team would be responsible?

	Gregory Szorc [:gps]
	Comment 9 • 6 years ago

We have a dedicated bug component for this type of thing now.

	Nathan Froyd [:froydnj] Assignee
	Comment 10 • 6 years ago

We're going to see how many things break with the naive approach:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=efa5539a7d28f25d139bdfe95ee7ed48e5874037

	Nathan Froyd [:froydnj] Assignee
	Comment 11 • 6 years ago

OK, so the first problem this turns up is that the building gcc tries to use the system `as` for assembly, which fails horribly, because the system `as` is older than whatever gcc was compiled with, which is bad.  We can fix that with PATH adjustments or -B options; I went with the latter.

The second problem is that apparently the newly-built clang can't find an appropriate libstdc++, since it apparently wants 4.8, and the system libstdc++ is 4.7.2.  Presumably it was finding the 4.9 libstdc++ before, but I don't know why it can't find the 6.x libstdc++ now.  Different filesystem layout?

	Nathan Froyd [:froydnj] Assignee
	Comment 12 • 6 years ago

Sigh, this is some stupidity with GCC's target naming:

root@a3eb5e67219e:~/workspace/build/src/build/build-clang# /builds/worker/workspace/moz-toolchain/build/stage1/clang/bin/clang++ float-bug.c
pp -std=c++11 -v                          
clang version 5.0.1 (tags/RELEASE_501/final) (llvm/tags/RELEASE_501/final 320874)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /builds/worker/workspace/moz-toolchain/build/stage1/clang/bin
Found candidate GCC installation: /builds/worker/workspace/moz-toolchain/build/stage1/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.7
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.7.2
Selected GCC installation: /builds/worker/workspace/moz-toolchain/build/stage1/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
 "/builds/worker/workspace/moz-toolchain/build/stage1/clang/bin/clang-5.0" -cc1 -triple x86_64-unknown-linux-gnu -emit-obj -mrelax-all -disa
ble-free -disable-llvm-verifier -discard-value-names -main-file-name float-bug.cpp -mrelocation-model static -mthread-model posix -mdisable-
fp-elim -fmath-errno -masm-verbose -mconstructor-aliases -munwind-tables -fuse-init-array -target-cpu x86-64 -v -dwarf-column-info -debugger
-tuning=gdb -resource-dir /builds/worker/workspace/moz-toolchain/build/stage1/clang/lib/clang/5.0.1 -internal-isystem /builds/worker/workspa
ce/moz-toolchain/build/stage1/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0/../../../../include/c++/6.4.0 -internal-isystem /builds/wo
rker/workspace/moz-toolchain/build/stage1/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0/../../../../include/x86_64-linux-gnu/c++/6.4.0
 -internal-isystem /builds/worker/workspace/moz-toolchain/build/stage1/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0/../../../../inclu
de/x86_64-linux-gnu/c++/6.4.0 -internal-isystem /builds/worker/workspace/moz-toolchain/build/stage1/clang/bin/../lib/gcc/x86_64-unknown-linu
x-gnu/6.4.0/../../../../include/c++/6.4.0/backward -internal-isystem /usr/local/include -internal-isystem /builds/worker/workspace/moz-toolc
hain/build/stage1/clang/lib/clang/5.0.1/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -
internal-externc-isystem /usr/include -std=c++11 -fdeprecated-macro -fdebug-compilation-dir /builds/worker/workspace/build/src/build/build-c
lang -ferror-limit 19 -fmessage-length 140 -fobjc-runtime=gcc -fcxx-exceptions -fexceptions -fdiagnostics-show-option -o /tmp/float-bug-e890
bc.o -x c++ float-bug.cpp
clang -cc1 version 5.0.1 based upon LLVM 5.0.1 default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/builds/worker/workspace/moz-toolchain/build/stage1/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0/../.
./../../include/x86_64-linux-gnu/c++/6.4.0"
ignoring nonexistent directory "/builds/worker/workspace/moz-toolchain/build/stage1/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0/../.
./../../include/x86_64-linux-gnu/c++/6.4.0"
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
 /builds/worker/workspace/moz-toolchain/build/stage1/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0/../../../../include/c++/6.4.0
 /builds/worker/workspace/moz-toolchain/build/stage1/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0/../../../../include/c++/6.4.0/backw
ard
 /usr/local/include
 /builds/worker/workspace/moz-toolchain/build/stage1/clang/lib/clang/5.0.1/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
In file included from float-bug.cpp:1:
In file included from /builds/worker/workspace/moz-toolchain/build/stage1/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0/../../../../in
clude/c++/6.4.0/atomic:41:
/builds/worker/workspace/moz-toolchain/build/stage1/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0/../../../../include/c++/6.4.0/bits/a
tomic_base.h:35:10: fatal error: 
      'bits/c++config.h' file not found

Note the:

clang -cc1 version 5.0.1 based upon LLVM 5.0.1 default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/builds/worker/workspace/moz-toolchain/build/stage1/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0/../.
./../../include/x86_64-linux-gnu/c++/6.4.0"
ignoring nonexistent directory "/builds/worker/workspace/moz-toolchain/build/stage1/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0/../.
./../../include/x86_64-linux-gnu/c++/6.4.0"
ignoring nonexistent directory "/include"

which is dumb, because:

root@a3eb5e67219e:~/workspace/build/src/build/build-clang# find ~/workspace/moz-toolchain/build/stage1 -name c++config.h
/builds/worker/workspace/moz-toolchain/build/stage1/clang/include/c++/6.4.0/x86_64-pc-linux-gnu/32/bits/c++config.h
/builds/worker/workspace/moz-toolchain/build/stage1/clang/include/c++/6.4.0/x86_64-pc-linux-gnu/bits/c++config.h

So clang thinks the file should be under `x86_64-unknown-linux-gnu` (like everything else!), but no, we put it under `x86_64-pc-linux-gnu` for reasons inexplicable to me.  And no, saying --target x86_64-pc-linux-gnu doesn't work, because then we don't find bits that *are* under a `x86_64-unknown-linux-gnu` directory.

I can't tell if this is an artifact of where we built GCC or not.  I think it might be?

	Nathan Froyd [:froydnj] Assignee
	Comment 13 • 6 years ago

ni? to glandium, as he might want to see comment 10 and following.

	Nathan Froyd [:froydnj] Assignee
	Comment 14 • 6 years ago

Trying to be more explicit with our GCC target:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=da1d76370a8a3e33d102245465eac184ee08c7ab

	Nathan Froyd [:froydnj] Assignee
	Comment 15 • 6 years ago

Apparently we should be passing --build, too...

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c6559a3baa4f892f2c0cfa455584b46050d3c42f

	Nathan Froyd [:froydnj] Assignee
	Comment 16 • 6 years ago

GCC will pick up whatever `as` is first in PATH when trying to assemble
files.  It expects this `as` to have at least as many features as the
`as` detected at configure time when GCC was originally built.  We
should ensure that GCC is always picking up an appropriate `as` by
adding its base directory to the search path; otherwise, we get peculiar
assembler errors.

I don't know why GCC can't do this itself...

	Nathan Froyd [:froydnj] Assignee
	Comment 17 • 6 years ago

Otherwise we can wind up with terrible mismatches.

--build is necessary, otherwise we get some configure mismatches during the gcc
build, and we'll be trying to use a compiler that "can't generate object
files".

	Nathan Froyd [:froydnj] Assignee
	Comment 18 • 6 years ago

Firefox itself has moved on to GCC 6.x; we can move our toolchains along too.

	Nathan Froyd [:froydnj] Assignee
	Comment 19 • 6 years ago

Well, we at least get the toolchains to build correctly, but the fuzzing builds, at least, are unhappy. :(

	Mike Hommey [:glandium]
	Comment 20 • 6 years ago

Comment on attachment 8983188 [details] [diff] [review]
part 2 - be explicit about our GCC target and build machine

Review of attachment 8983188 [details] [diff] [review]:
-----------------------------------------------------------------

::: build/unix/build-gcc/build-gcc.sh
@@ +134,5 @@
>  }
>  
>  build_gcc() {
> +  # Be explicit about --build and --target so header and library install
> +  # directories are consistent. 

trailing whitespace.

	Mike Hommey [:glandium]
	Comment 21 • 6 years ago

Comment on attachment 8983189 [details] [diff] [review]
part 3 - move toolchains off GCC 4.9

Review of attachment 8983189 [details] [diff] [review]:
-----------------------------------------------------------------

I'm afraid, though, this may only hide the core issue in bug 1428158, because it will make the embedded libstdc++ have the same version as the system headers, making things work "by chance", and possibly break again when libstdc++ version changes again. So I'd pretty much like to see a try *without* this specific patch, and with everything else for android static analysis builds.

	Mike Hommey [:glandium]
	Comment 22 • 6 years ago

(because hopefully, part 2 of this series fixes the core issue)

	Nathan Froyd [:froydnj] Assignee
	Comment 23 • 6 years ago

(In reply to Mike Hommey [:glandium] from comment #21)
> I'm afraid, though, this may only hide the core issue in bug 1428158,
> because it will make the embedded libstdc++ have the same version as the
> system headers, making things work "by chance", and possibly break again
> when libstdc++ version changes again. So I'd pretty much like to see a try
> *without* this specific patch, and with everything else for android static
> analysis builds.

You're right, this doesn't quite fix everything; static analysis builds are still busted.  I think because LLVM symbols are somehow not getting resolved, but of course we can't actually get an error message about that:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=49fcfa3d369427dca14f74a3b3090252883dc0f0&selectedJob=181825018
https://treeherder.mozilla.org/logviewer.html#?job_id=181825018&repo=try&lineNumber=4331

I think this is *probably* just bug 1428518 comment 28, or something like that.

	Mike Hommey [:glandium]
	Comment 24 • 6 years ago

BTW, there's something weird about not getting the error messages. When I was reproducing on a loaner, it seemed like sccache is eating ld.so errors. Which gives a workaround to get more useful logs for this: disable sccache in your try push.

	Mike Hommey [:glandium]
	Comment 25 • 6 years ago

Actually, it's clang eating the error message :( which thankfully means we can fix that.

	Nathan Froyd [:froydnj] Assignee
	Comment 26 • 6 years ago

OK, rabbit hole time: on the above try push, I set up a loaner, and started to build.  The plugin tests naturally fell over.  From LD_DEBUG:

/builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/libclang-plugin.so: error: symbol lookup error: undefined symbol: _ZNSt3_V214error_categoryD2Ev

For whatever reason, we're using the system headers to build the plugin, rather than the headers distributed with clang.  But that means that we're going to want to link against symbols defined in the system libstdc++...except that we have a libstdc++.a distributed with clang, and we link the plugin with -static-libstdc++.  We're using -static-libstdc++ because that's what llvm-config says we should use, for better or worse.  This setup doesn't work, because we fail to resolve the above symbol in the libraries we're linking against, and we therefore fail to find it at runtime.

We can remove libstdc++.a, which causes the search to find the system's static libstdc++.a.  Rebuild, link, everything's fine.  During the tests, I get:

/builds/worker/workspace/build/src/clang/bin/clang-5.0: symbol lookup error: /builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/libclang-plugin.so: undefined symbol: _ZN5clang12ast_matchers8internal14HasNameMatcherC1ESt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS9_EE

OK, that's helpful information that we can do something with.  The clangASTMatchers library, which is presumably where we should have gotten that symbol from, has:

_ZN5clang12ast_matchers8internal14HasNameMatcherC2ESt6vectorISsSaISsEE

so we're running into the ABI breakage with GCC 5.  We compiled this clang on system with old GCC headers (though, curiously, we used GCC 6, so we should (?) have gotten the new headers...?).  Fine, no problem, I'll set -D_GLIBCXX_USE_CXX11_ABI=0.  That works, and we can use the plugin successfully.

But *all* of that came about because we're building with mismatched headers in clang vs. what we're using on the build system.  clang knows about the headers that came with it:

clang version 5.0.1 (tags/RELEASE_501/final) (llvm/tags/RELEASE_501/final 320874)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /builds/worker/workspace/build/src/clang/bin
Found candidate GCC installation: /builds/worker/workspace/build/src/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.4
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/6
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/6.3.0
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/6.3.0

It's not clear to me why clang is rejecting the 4.9.4 headers.

If we instead build with a clang built on GCC 6...presumably everything would be OK?

I'm not really sure how to solve this case as comment 21 wants it, so it's completely impervious to any changes in the toolchains or the systems on which the toolchains are built/run.  Maybe we have to figure out why clang isn't preferring its headers in this case and force it to choose otherwise?

	Nathan Froyd [:froydnj] Assignee
	Comment 27 • 6 years ago

(In reply to Nathan Froyd [:froydnj] from comment #26)
> I'm not really sure how to solve this case as comment 21 wants it, so it's
> completely impervious to any changes in the toolchains or the systems on
> which the toolchains are built/run.  Maybe we have to figure out why clang
> isn't preferring its headers in this case and force it to choose otherwise?

I think clang will choose the newest GCC version available.  Which makes a certain amount of sense, but I think in this case we actually want it to do something else?

	Mike Hommey [:glandium]
	Comment 28 • 6 years ago

It actually doesn't make sense, if it doesn't try to find the corresponding libstdc++.

	Nathan Froyd [:froydnj] Assignee
	Comment 29 • 6 years ago

We want our clang bootstrap to use the GCC headers we're building with,
not whatever sysroot it happens to find on the server we're building on.

The -gcc-toolchain argument we specify when building clang will also be
picked up by llvm-config, so we need to strip it out when building the
plugin.  Otherwise, we will get peculiar failures about not being able
to find C++ header files.

	Nathan Froyd [:froydnj] Assignee
	Comment 30 • 6 years ago

We do this for similar reasons as we did the previous GCC change.

The commits in this bug build and pass tests on their own, and they also enable
building Android static analysis builds with no weird C++ dependency things.
These commits imply that the clangs we build now require a libstdc++ with the
new libstdc++ ABI, but I think that's OK nowadays.

	Nathan Froyd [:froydnj] Assignee
	Comment 31 • 6 years ago

Firefox itself has moved on to GCC 6.x; we can move our toolchains along too.

	Mike Hommey [:glandium]
	Comment 32 • 6 years ago

Comment on attachment 8987199 [details] [diff] [review]
part 3 - force clang to always pick up its local GCC headers and libraries

Review of attachment 8987199 [details] [diff] [review]:
-----------------------------------------------------------------

::: build/autoconf/clang-plugin.m4
@@ +39,5 @@
> +    dnl
> +    dnl For a similar reason, we remove any -gcc-toolchain arguments, since the
> +    dnl directories specified by such arguments might not exist on the current
> +    dnl machine.
> +    LLVM_CXXFLAGS=`$LLVMCONFIG --cxxflags | sed -e 's/-isysroot [[^ ]]*//' -e 's/-gcc-toolchain [[^ ]]*//'`

It feels like we should change llvm-config to strip that in first place. Or set clang build flags in a way that doesn't alter llvm-config, if that exists. llvm-config apparently takes things from LLVM_CXXFLAGS, which is defined in cmake as "${CMAKE_CXX_FLAGS} ${CMAKE_CXX_FLAGS_${uppercase_CMAKE_BUILD_TYPE}} ${LLVM_DEFINITIONS}".
Unfortunately, CMAKE_CXX_FLAGS is how we pass the extra_cxxflags... sigh.

	Mike Hommey [:glandium]
	Comment 33 • 6 years ago

Comment on attachment 8987200 [details] [diff] [review]
part 4 - inform stage2/3 clang about gcc binutils

Review of attachment 8987200 [details] [diff] [review]:
-----------------------------------------------------------------

as mentioned in another bug, we should, in the longer term, make builds use the binutils package.

	Pulsebot
	Comment 34 • 6 years ago

Pushed by nfroyd@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/58e5f6f08c54
part 1 - ensure compatible binutils for GCC; r=glandium
https://hg.mozilla.org/integration/mozilla-inbound/rev/6f3d601d1ad8
part 2 - be explicit about our GCC target and build machine; r=glandium
https://hg.mozilla.org/integration/mozilla-inbound/rev/290c9bd3ead7
part 3 - force clang to always pick up its local GCC headers and libraries; r=glandium
https://hg.mozilla.org/integration/mozilla-inbound/rev/ac520651c6ee
part 4 - inform stage2/3 clang about gcc binutils; r=glandium
https://hg.mozilla.org/integration/mozilla-inbound/rev/3c0bda5f0564
part 5 - move toolchains off GCC 4.9; r=glandium

	Pulsebot
	Comment 35 • 6 years ago

Backout by nfroyd@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/b59bec39784a
Backout 3c0bda5f0564:58e5f6f08c54 for inadvertently stressing tooltool and busting this CLOSED TREE

	Mike Hommey [:glandium]
	Comment 36 • 6 years ago

Note that the gc and cctools failures are not about tooltool.

	Mike Hommey [:glandium]
	Comment 37 • 6 years ago

s/gc/gn/

	Nathan Froyd [:froydnj] Assignee
	Comment 38 • 6 years ago

We need to do this for similar reasons as the previous gcc patch.

	Nathan Froyd [:froydnj] Assignee
	Comment 39 • 6 years ago

I've renumbered this due to the gn change, but I've not bothered to attach all
the renumbered patches.

This patch is putting ld in a place where clang (and gcc) can find it, but this
version changes the location, so clang won't pick it up for builds where it's
not applicable.  Seems worth a re-review.  This change fixes the cctools bits
for the Mac build.

	Mike Hommey [:glandium]
	Comment 40 • 6 years ago

Comment on attachment 8992742 [details] [diff] [review]
part 2 - ensure gn builds with the right gcc flags

Review of attachment 8992742 [details] [diff] [review]:
-----------------------------------------------------------------

::: taskcluster/scripts/misc/build-gn-linux.sh
@@ +9,5 @@
> +GCC_DIR=$WORKSPACE/build/src/gcc/bin
> +export CC=$GCC_DIR/gcc
> +export CXX=$GCC_DIR/g++
> +export CFLAGS="-B $GCC_DIR"
> +export CXXFLAGS="-B $GCC_DIR"

IIRC we're using PATH to do that in other scripts, maybe do the same for consistence?

	Nathan Froyd [:froydnj] Assignee
	Comment 41 • 6 years ago

(In reply to Mike Hommey [:glandium] from comment #40)
> ::: taskcluster/scripts/misc/build-gn-linux.sh
> @@ +9,5 @@
> > +GCC_DIR=$WORKSPACE/build/src/gcc/bin
> > +export CC=$GCC_DIR/gcc
> > +export CXX=$GCC_DIR/g++
> > +export CFLAGS="-B $GCC_DIR"
> > +export CXXFLAGS="-B $GCC_DIR"
> 
> IIRC we're using PATH to do that in other scripts, maybe do the same for
> consistence?

Digression first: it turns out that a clang-7 toolchain job was added, and clang 7 apparently changed some bits so that CMAKE_C{,XX}_FLAGS, where we set -B $GCC_DIR, are not honored for bits of libfuzzer.  This breaks the libfuzzer build, specifically bits of compiling libcxx, since we can't find an assembler that understands some of the new syntax gcc uses.  I looked at some ways to fix this and basically failed.  I'm sure there's a good way to do it, I just couldn't find the magic cmake flags.

Making --build and --target explicit for binutils, like we do for gcc, fixes problems:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=35a34c66dab2949e6f1b9832ef0e0113afc622d7

Note that the "fix bits for libcxx in LLVM 7" is ineffectual; I can provide links if need be.

Making --target explicit, and matching the gcc --target, means that binutils installs bits in $GCC_DIR/x86_64-unknown-linux-gnu/bin also, whereas without --target, the binutils get dropped into $GCC_DIR/x86_64-pc-linux-gnu.  GCC, at least as we configure it, will look in x86_64-unknown-linux-gnu/bin, but not in x86_64-pc-linux-gnu/bin/.  Adding --build just seems nice for consistency.

In fact, making --build and --target explicit for binutils means that we don't even have to futz around with all the -B business.  The following try push drops the commits that specify -B in necessary places and everything is still green:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=fc0fb0663a2af950997d04887a7228d985d9d7e6

which is probably a good thing, as the continual necessity of passing -B would bite people who tried to depend on our gcc packages to compile other code.

I think that binutils change is reasonably obvious, so I'm going to drop part 1 and 2, renumber everything, carry over r+, and land the whole shebang.

	Pulsebot
	Comment 42 • 6 years ago

Pushed by nfroyd@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/b93c29157feb
part 1 - be explicit about our GCC/binutils target and build machine; r=glandium
https://hg.mozilla.org/integration/mozilla-inbound/rev/b4ebc9a500f4
part 2 - force clang to always pick up its local GCC headers and libraries; r=glandium
https://hg.mozilla.org/integration/mozilla-inbound/rev/47405e33747c
part 3 - inform stage2/3 clang about gcc binutils; r=glandium
https://hg.mozilla.org/integration/mozilla-inbound/rev/52d37e70b6f1
part 4 - move toolchains off GCC 4.9; r=glandium
https://hg.mozilla.org/integration/mozilla-inbound/rev/ae7ccb8c938d
part 5 - touch CLOBBER; r=me

	Pulsebot
	Comment 43 • 6 years ago

Pushed by nfroyd@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/8984f6868e57
follow-up - quiet clang-tidy content checking; r=me

	Pulsebot
	Comment 44 • 6 years ago

Backout by csabou@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/dcfd01a3010b
Backed out 6 changesets for causing build bustages on OS X Cross Compiled. CLOSED TREE

	Cosmin Sabou [:CosminS]
	Comment 45 • 6 years ago

Backed out for causing build bustages on OS X Cross Compiled.

Pushes that caused these problems with the clang and cctools-port: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&author=nfroyd@mozilla.com&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&tochange=8984f6868e572079cc991492124ba99747f0f681&fromchange=ae7ccb8c938d180f5777d4be8885cd425b711966

This has something to do with what Tom Prince explained here https://bugzilla.mozilla.org/show_bug.cgi?id=1471905#c1

But eventually looks like this was not the problem: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=dcfd01a3010b233490bafa693e51cd96f3ab5ce4&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-searchStr=cross&selectedJob=189282523

Failure log is the same as before: https://treeherder.mozilla.org/logviewer.html#?job_id=189282516&repo=mozilla-inbound&lineNumber=1030

	Mike Hommey [:glandium]
	Comment 46 • 6 years ago

After forcing a rebuild of cctools-port on the backout, everything should be back to normal.

	Cosmin Sabou [:CosminS]
	Comment 47 • 6 years ago

Yes, that eventually solved the bustages together with the backout. https://treeherder.mozilla.org/#/jobs?repo=autoland&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=success&filter-resultStatus=pending&filter-resultStatus=running&filter-classifiedState=unclassified&filter-searchStr=OS%20X%20Cross&tochange=3bbf774e9e48ddd2cd34692f3ae18016af32c640&fromchange=18f56efc409395f86ff7cbc5a59c645ec2360099&selectedJob=189289536

	Mike Hommey [:glandium]
	Comment 48 • 6 years ago

(In reply to Pulsebot from comment #42)
> https://hg.mozilla.org/integration/mozilla-inbound/rev/ae7ccb8c938d
> part 5 - touch CLOBBER; r=me

CLOBBER does nothing on automation (all jobs are clobbers), and local builds are not affected by these changes, so I don't know why you feel this needs a CLOBBER touch.

	Pulsebot
	Comment 66 • 6 years ago

Pushed by nfroyd@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/991777e081ff
part 1 - be explicit about our GCC/binutils target and build machine; r=glandium
https://hg.mozilla.org/integration/autoland/rev/c64bfaad8a2f
part 2 - force clang to always pick up its local GCC headers and libraries; r=glandium
https://hg.mozilla.org/integration/autoland/rev/675f73d41eb4
part 3 - inform stage2/3 clang about gcc binutils; r=glandium
https://hg.mozilla.org/integration/autoland/rev/5179c8066914
part 4 - sync up gcc-related toolchains and linux64-binutils binutils version; r=glandium
https://hg.mozilla.org/integration/autoland/rev/7c116f85ede6
part 5 - move toolchains off GCC 4.9; r=glandium
https://hg.mozilla.org/integration/autoland/rev/2f6199beeb71
part 6 - don't remove the libstdc++ files from the mingw build; r=glandium

	Pulsebot
	Comment 68 • 6 years ago

Pushed by nfroyd@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/468eed96e879
part 1 - be explicit about our GCC/binutils target and build machine; r=glandium
https://hg.mozilla.org/integration/autoland/rev/961ac7318786
part 2 - force clang to always pick up its local GCC headers and libraries; r=glandium
https://hg.mozilla.org/integration/autoland/rev/f3290e95b38d
part 3 - inform stage2/3 clang about gcc binutils; r=glandium
https://hg.mozilla.org/integration/autoland/rev/292ed1bb9143
part 4 - sync up gcc-related toolchains and linux64-binutils binutils version; r=glandium
https://hg.mozilla.org/integration/autoland/rev/858b68d306f2
part 5 - move toolchains off GCC 4.9; r=glandium
https://hg.mozilla.org/integration/autoland/rev/76a3b7b0c9d7
part 6 - don't remove the libstdc++ files from the mingw build; r=glandium