Closed Bug 1451104 Opened 7 years ago Closed 6 years ago

Remove GCC-4.9 from toolchains

Categories

(Firefox Build System :: Toolchains, enhancement, P3)

Product:
Firefox Build System
Component:
Toolchains
Type:
enhancement

Tracking

(firefox67 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla67
Tracking Flags:
Tracking Status
firefox67 --- fixed

People

(Reporter: jgilbert, Assigned: froydnj)

References

Details

Attachments

(6 files, 8 obsolete files)

Bug 1451104 - part 1 - be explicit about our GCC/binutils target and build machine; r=glandium
(deleted), text/x-phabricator-request
Details
Bug 1451104 - part 2 - force clang to always pick up its local GCC headers and libraries; r=glandium
(deleted), text/x-phabricator-request
Details
Bug 1451104 - part 3 - inform stage2/3 clang about gcc binutils; r=glandium
(deleted), text/x-phabricator-request
Details
Bug 1451104 - part 4 - sync up gcc-related toolchains and linux64-binutils binutils version; r=glandium
(deleted), text/x-phabricator-request
Details
Bug 1451104 - part 5 - move toolchains off GCC 4.9; r=glandium
(deleted), text/x-phabricator-request
Details
Bug 1451104 - part 6 - don't remove the libstdc++ files from the mingw build; r=glandium
(deleted), text/x-phabricator-request
Details
We're not quite done with GCC<6, so we can't rely on having GCC>=6 yet.
Blocks: 1444271
No longer blocks: 1444271
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
Any updates on this matter? Jeff I'll assign this bug to you. Please let someone from the build duty team know when we should remove GCC<6. Does this sound good for you?
Flags: needinfo?(jgilbert)
Assignee: nobody → jgilbert
My needs are met now, so I'm going to cede this cleanup to RelEng.
Assignee: jgilbert → nobody
Flags: needinfo?(jgilbert)
Priority: -- → P3
I see in Taskcluster scheduling, 4.9 still referenced in places: https://dxr.mozilla.org/mozilla-central/search?q=path%3Ataskcluster+gcc-4&redirect=false That's all in-tree and self serve'able if you want to update. If there are places it's used in Buildbot, we can ignore that and let it die with the rest of Buildbot infra.
(In reply to Jordan Lund (:jlund) from comment #3) > I see in Taskcluster scheduling, 4.9 still referenced in places: > https://dxr.mozilla.org/mozilla-central/search?q=path%3Ataskcluster+gcc- > 4&redirect=false > > That's all in-tree and self serve'able if you want to update. > > If there are places it's used in Buildbot, we can ignore that and let it die > with the rest of Buildbot infra. jgilbert - I'm a bit out of touch with current state, did you need a hand updating gcc versions anywhere in-tree (e.g. particular tasks)? I see you have touched the above taskcluster/ci/* scheduling files before. Or is this bug filed in case there are places outside of taskcluster/in-tree that you are not familiar with and you're just giving releng a heads up?
Flags: needinfo?(jgilbert)
This is just cleanup now that we require GCC-6.1+. Since we require 6.1+ now, anything that references <6.1 is out-of-date or orphaned, but I can't tell which. This was all outside my domain, so this is a note for others that there's still cleanup to be done, since I can't easily do it myself.
Flags: needinfo?(jgilbert)
This is beyond buildduty scope (for now) to action. However, since they are familiar with CI and who owns what, filing here is good as they can escalate appropriately. Moving to Releng inbox so it's on the radar there.
Component: Buildduty → General
Product: Infrastructure & Operations → Release Engineering
Hi, We have found this in triaging, and I was wondering if you are the guy or team that we should escalate this kind of problems, if not, can you point us to the right direction?
Flags: needinfo?(jhford)
(In reply to Bogdan Crisan [:bcrisan] from comment #7) > Hi, > > We have found this in triaging, and I was wondering if you are the guy or > team that we should escalate this kind of problems, if not, can you point us > to the right direction? Hi Bogdan, I'm really not sure about this bug. Greg, do you know if this is the correct component for this bug or which team would be responsible?
Flags: needinfo?(jhford) → needinfo?(gps)
We have a dedicated bug component for this type of thing now.
Component: General → Toolchains
Flags: needinfo?(gps)
Product: Release Engineering → Firefox Build System
QA Contact: catlee
We're going to see how many things break with the naive approach: https://treeherder.mozilla.org/#/jobs?repo=try&revision=efa5539a7d28f25d139bdfe95ee7ed48e5874037
OK, so the first problem this turns up is that the building gcc tries to use the system `as` for assembly, which fails horribly, because the system `as` is older than whatever gcc was compiled with, which is bad. We can fix that with PATH adjustments or -B options; I went with the latter. The second problem is that apparently the newly-built clang can't find an appropriate libstdc++, since it apparently wants 4.8, and the system libstdc++ is 4.7.2. Presumably it was finding the 4.9 libstdc++ before, but I don't know why it can't find the 6.x libstdc++ now. Different filesystem layout?
Sigh, this is some stupidity with GCC's target naming: root@a3eb5e67219e:~/workspace/build/src/build/build-clang# /builds/worker/workspace/moz-toolchain/build/stage1/clang/bin/clang++ float-bug.c pp -std=c++11 -v clang version 5.0.1 (tags/RELEASE_501/final) (llvm/tags/RELEASE_501/final 320874) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /builds/worker/workspace/moz-toolchain/build/stage1/clang/bin Found candidate GCC installation: /builds/worker/workspace/moz-toolchain/build/stage1/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0 Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.7 Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.7.2 Selected GCC installation: /builds/worker/workspace/moz-toolchain/build/stage1/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0 Candidate multilib: .;@m64 Candidate multilib: 32;@m32 Selected multilib: .;@m64 "/builds/worker/workspace/moz-toolchain/build/stage1/clang/bin/clang-5.0" -cc1 -triple x86_64-unknown-linux-gnu -emit-obj -mrelax-all -disa ble-free -disable-llvm-verifier -discard-value-names -main-file-name float-bug.cpp -mrelocation-model static -mthread-model posix -mdisable- fp-elim -fmath-errno -masm-verbose -mconstructor-aliases -munwind-tables -fuse-init-array -target-cpu x86-64 -v -dwarf-column-info -debugger -tuning=gdb -resource-dir /builds/worker/workspace/moz-toolchain/build/stage1/clang/lib/clang/5.0.1 -internal-isystem /builds/worker/workspa ce/moz-toolchain/build/stage1/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0/../../../../include/c++/6.4.0 -internal-isystem /builds/wo rker/workspace/moz-toolchain/build/stage1/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0/../../../../include/x86_64-linux-gnu/c++/6.4.0 -internal-isystem /builds/worker/workspace/moz-toolchain/build/stage1/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0/../../../../inclu de/x86_64-linux-gnu/c++/6.4.0 -internal-isystem /builds/worker/workspace/moz-toolchain/build/stage1/clang/bin/../lib/gcc/x86_64-unknown-linu x-gnu/6.4.0/../../../../include/c++/6.4.0/backward -internal-isystem /usr/local/include -internal-isystem /builds/worker/workspace/moz-toolc hain/build/stage1/clang/lib/clang/5.0.1/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include - internal-externc-isystem /usr/include -std=c++11 -fdeprecated-macro -fdebug-compilation-dir /builds/worker/workspace/build/src/build/build-c lang -ferror-limit 19 -fmessage-length 140 -fobjc-runtime=gcc -fcxx-exceptions -fexceptions -fdiagnostics-show-option -o /tmp/float-bug-e890 bc.o -x c++ float-bug.cpp clang -cc1 version 5.0.1 based upon LLVM 5.0.1 default target x86_64-unknown-linux-gnu ignoring nonexistent directory "/builds/worker/workspace/moz-toolchain/build/stage1/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0/../. ./../../include/x86_64-linux-gnu/c++/6.4.0" ignoring nonexistent directory "/builds/worker/workspace/moz-toolchain/build/stage1/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0/../. ./../../include/x86_64-linux-gnu/c++/6.4.0" ignoring nonexistent directory "/include" #include "..." search starts here: #include <...> search starts here: /builds/worker/workspace/moz-toolchain/build/stage1/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0/../../../../include/c++/6.4.0 /builds/worker/workspace/moz-toolchain/build/stage1/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0/../../../../include/c++/6.4.0/backw ard /usr/local/include /builds/worker/workspace/moz-toolchain/build/stage1/clang/lib/clang/5.0.1/include /usr/include/x86_64-linux-gnu /usr/include End of search list. In file included from float-bug.cpp:1: In file included from /builds/worker/workspace/moz-toolchain/build/stage1/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0/../../../../in clude/c++/6.4.0/atomic:41: /builds/worker/workspace/moz-toolchain/build/stage1/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0/../../../../include/c++/6.4.0/bits/a tomic_base.h:35:10: fatal error: 'bits/c++config.h' file not found Note the: clang -cc1 version 5.0.1 based upon LLVM 5.0.1 default target x86_64-unknown-linux-gnu ignoring nonexistent directory "/builds/worker/workspace/moz-toolchain/build/stage1/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0/../. ./../../include/x86_64-linux-gnu/c++/6.4.0" ignoring nonexistent directory "/builds/worker/workspace/moz-toolchain/build/stage1/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0/../. ./../../include/x86_64-linux-gnu/c++/6.4.0" ignoring nonexistent directory "/include" which is dumb, because: root@a3eb5e67219e:~/workspace/build/src/build/build-clang# find ~/workspace/moz-toolchain/build/stage1 -name c++config.h /builds/worker/workspace/moz-toolchain/build/stage1/clang/include/c++/6.4.0/x86_64-pc-linux-gnu/32/bits/c++config.h /builds/worker/workspace/moz-toolchain/build/stage1/clang/include/c++/6.4.0/x86_64-pc-linux-gnu/bits/c++config.h So clang thinks the file should be under `x86_64-unknown-linux-gnu` (like everything else!), but no, we put it under `x86_64-pc-linux-gnu` for reasons inexplicable to me. And no, saying --target x86_64-pc-linux-gnu doesn't work, because then we don't find bits that *are* under a `x86_64-unknown-linux-gnu` directory. I can't tell if this is an artifact of where we built GCC or not. I think it might be?
ni? to glandium, as he might want to see comment 10 and following.
Attached patch part 1 - ensure compatible binutils for GCC (obsolete) (deleted) — — Splinter Review
GCC will pick up whatever `as` is first in PATH when trying to assemble files. It expects this `as` to have at least as many features as the `as` detected at configure time when GCC was originally built. We should ensure that GCC is always picking up an appropriate `as` by adding its base directory to the search path; otherwise, we get peculiar assembler errors. I don't know why GCC can't do this itself...
Attachment #8983187 - Flags: review?(mh+mozilla)
Otherwise we can wind up with terrible mismatches. --build is necessary, otherwise we get some configure mismatches during the gcc build, and we'll be trying to use a compiler that "can't generate object files".
Attachment #8983188 - Flags: review?(mh+mozilla)
Attached patch part 3 - move toolchains off GCC 4.9 (obsolete) (deleted) — — Splinter Review
Firefox itself has moved on to GCC 6.x; we can move our toolchains along too.
Attachment #8983189 - Flags: review?(mh+mozilla)
Well, we at least get the toolchains to build correctly, but the fuzzing builds, at least, are unhappy. :(
Attachment #8983187 - Flags: review?(mh+mozilla) → review+
Comment on attachment 8983188 [details] [diff] [review] part 2 - be explicit about our GCC target and build machine Review of attachment 8983188 [details] [diff] [review]: ----------------------------------------------------------------- ::: build/unix/build-gcc/build-gcc.sh @@ +134,5 @@ > } > > build_gcc() { > + # Be explicit about --build and --target so header and library install > + # directories are consistent. trailing whitespace.
Attachment #8983188 - Flags: review?(mh+mozilla) → review+
Comment on attachment 8983189 [details] [diff] [review] part 3 - move toolchains off GCC 4.9 Review of attachment 8983189 [details] [diff] [review]: ----------------------------------------------------------------- I'm afraid, though, this may only hide the core issue in bug 1428158, because it will make the embedded libstdc++ have the same version as the system headers, making things work "by chance", and possibly break again when libstdc++ version changes again. So I'd pretty much like to see a try *without* this specific patch, and with everything else for android static analysis builds.
Attachment #8983189 - Flags: review?(mh+mozilla) → review+
(because hopefully, part 2 of this series fixes the core issue)
(In reply to Mike Hommey [:glandium] from comment #21) > I'm afraid, though, this may only hide the core issue in bug 1428158, > because it will make the embedded libstdc++ have the same version as the > system headers, making things work "by chance", and possibly break again > when libstdc++ version changes again. So I'd pretty much like to see a try > *without* this specific patch, and with everything else for android static > analysis builds. You're right, this doesn't quite fix everything; static analysis builds are still busted. I think because LLVM symbols are somehow not getting resolved, but of course we can't actually get an error message about that: https://treeherder.mozilla.org/#/jobs?repo=try&revision=49fcfa3d369427dca14f74a3b3090252883dc0f0&selectedJob=181825018 https://treeherder.mozilla.org/logviewer.html#?job_id=181825018&repo=try&lineNumber=4331 I think this is *probably* just bug 1428518 comment 28, or something like that.
BTW, there's something weird about not getting the error messages. When I was reproducing on a loaner, it seemed like sccache is eating ld.so errors. Which gives a workaround to get more useful logs for this: disable sccache in your try push.
Actually, it's clang eating the error message :( which thankfully means we can fix that.
OK, rabbit hole time: on the above try push, I set up a loaner, and started to build. The plugin tests naturally fell over. From LD_DEBUG: /builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/libclang-plugin.so: error: symbol lookup error: undefined symbol: _ZNSt3_V214error_categoryD2Ev For whatever reason, we're using the system headers to build the plugin, rather than the headers distributed with clang. But that means that we're going to want to link against symbols defined in the system libstdc++...except that we have a libstdc++.a distributed with clang, and we link the plugin with -static-libstdc++. We're using -static-libstdc++ because that's what llvm-config says we should use, for better or worse. This setup doesn't work, because we fail to resolve the above symbol in the libraries we're linking against, and we therefore fail to find it at runtime. We can remove libstdc++.a, which causes the search to find the system's static libstdc++.a. Rebuild, link, everything's fine. During the tests, I get: /builds/worker/workspace/build/src/clang/bin/clang-5.0: symbol lookup error: /builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/libclang-plugin.so: undefined symbol: _ZN5clang12ast_matchers8internal14HasNameMatcherC1ESt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS9_EE OK, that's helpful information that we can do something with. The clangASTMatchers library, which is presumably where we should have gotten that symbol from, has: _ZN5clang12ast_matchers8internal14HasNameMatcherC2ESt6vectorISsSaISsEE so we're running into the ABI breakage with GCC 5. We compiled this clang on system with old GCC headers (though, curiously, we used GCC 6, so we should (?) have gotten the new headers...?). Fine, no problem, I'll set -D_GLIBCXX_USE_CXX11_ABI=0. That works, and we can use the plugin successfully. But *all* of that came about because we're building with mismatched headers in clang vs. what we're using on the build system. clang knows about the headers that came with it: clang version 5.0.1 (tags/RELEASE_501/final) (llvm/tags/RELEASE_501/final 320874) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /builds/worker/workspace/build/src/clang/bin Found candidate GCC installation: /builds/worker/workspace/build/src/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.4 Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/6 Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/6.3.0 Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/6.3.0 It's not clear to me why clang is rejecting the 4.9.4 headers. If we instead build with a clang built on GCC 6...presumably everything would be OK? I'm not really sure how to solve this case as comment 21 wants it, so it's completely impervious to any changes in the toolchains or the systems on which the toolchains are built/run. Maybe we have to figure out why clang isn't preferring its headers in this case and force it to choose otherwise?
(In reply to Nathan Froyd [:froydnj] from comment #26) > I'm not really sure how to solve this case as comment 21 wants it, so it's > completely impervious to any changes in the toolchains or the systems on > which the toolchains are built/run. Maybe we have to figure out why clang > isn't preferring its headers in this case and force it to choose otherwise? I think clang will choose the newest GCC version available. Which makes a certain amount of sense, but I think in this case we actually want it to do something else?
It actually doesn't make sense, if it doesn't try to find the corresponding libstdc++.
Assignee: nobody → nfroyd
We want our clang bootstrap to use the GCC headers we're building with, not whatever sysroot it happens to find on the server we're building on. The -gcc-toolchain argument we specify when building clang will also be picked up by llvm-config, so we need to strip it out when building the plugin. Otherwise, we will get peculiar failures about not being able to find C++ header files.
Attachment #8987199 - Flags: review?(mh+mozilla)
We do this for similar reasons as we did the previous GCC change. The commits in this bug build and pass tests on their own, and they also enable building Android static analysis builds with no weird C++ dependency things. These commits imply that the clangs we build now require a libstdc++ with the new libstdc++ ABI, but I think that's OK nowadays.
Attachment #8987200 - Flags: review?(mh+mozilla)
Attached patch part 5 - move toolchains off GCC 4.9 (obsolete) (deleted) — — Splinter Review
Firefox itself has moved on to GCC 6.x; we can move our toolchains along too.
Attachment #8987201 - Flags: review+
Attachment #8983189 - Attachment is obsolete: true
Comment on attachment 8987199 [details] [diff] [review] part 3 - force clang to always pick up its local GCC headers and libraries Review of attachment 8987199 [details] [diff] [review]: ----------------------------------------------------------------- ::: build/autoconf/clang-plugin.m4 @@ +39,5 @@ > + dnl > + dnl For a similar reason, we remove any -gcc-toolchain arguments, since the > + dnl directories specified by such arguments might not exist on the current > + dnl machine. > + LLVM_CXXFLAGS=`$LLVMCONFIG --cxxflags | sed -e 's/-isysroot [[^ ]]*//' -e 's/-gcc-toolchain [[^ ]]*//'` It feels like we should change llvm-config to strip that in first place. Or set clang build flags in a way that doesn't alter llvm-config, if that exists. llvm-config apparently takes things from LLVM_CXXFLAGS, which is defined in cmake as "${CMAKE_CXX_FLAGS} ${CMAKE_CXX_FLAGS_${uppercase_CMAKE_BUILD_TYPE}} ${LLVM_DEFINITIONS}". Unfortunately, CMAKE_CXX_FLAGS is how we pass the extra_cxxflags... sigh.
Attachment #8987199 - Flags: review?(mh+mozilla) → review+
Comment on attachment 8987200 [details] [diff] [review] part 4 - inform stage2/3 clang about gcc binutils Review of attachment 8987200 [details] [diff] [review]: ----------------------------------------------------------------- as mentioned in another bug, we should, in the longer term, make builds use the binutils package.
Attachment #8987200 - Flags: review?(mh+mozilla) → review+
Blocks: 1428158
Pushed by nfroyd@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/58e5f6f08c54 part 1 - ensure compatible binutils for GCC; r=glandium https://hg.mozilla.org/integration/mozilla-inbound/rev/6f3d601d1ad8 part 2 - be explicit about our GCC target and build machine; r=glandium https://hg.mozilla.org/integration/mozilla-inbound/rev/290c9bd3ead7 part 3 - force clang to always pick up its local GCC headers and libraries; r=glandium https://hg.mozilla.org/integration/mozilla-inbound/rev/ac520651c6ee part 4 - inform stage2/3 clang about gcc binutils; r=glandium https://hg.mozilla.org/integration/mozilla-inbound/rev/3c0bda5f0564 part 5 - move toolchains off GCC 4.9; r=glandium
Backout by nfroyd@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/b59bec39784a Backout 3c0bda5f0564:58e5f6f08c54 for inadvertently stressing tooltool and busting this CLOSED TREE
Note that the gc and cctools failures are not about tooltool.
s/gc/gn/
We need to do this for similar reasons as the previous gcc patch.
Attachment #8992742 - Flags: review?(mh+mozilla)
I've renumbered this due to the gn change, but I've not bothered to attach all the renumbered patches. This patch is putting ld in a place where clang (and gcc) can find it, but this version changes the location, so clang won't pick it up for builds where it's not applicable. Seems worth a re-review. This change fixes the cctools bits for the Mac build.
Attachment #8992745 - Flags: review?(mh+mozilla)
Attachment #8987200 - Attachment is obsolete: true
Comment on attachment 8992742 [details] [diff] [review] part 2 - ensure gn builds with the right gcc flags Review of attachment 8992742 [details] [diff] [review]: ----------------------------------------------------------------- ::: taskcluster/scripts/misc/build-gn-linux.sh @@ +9,5 @@ > +GCC_DIR=$WORKSPACE/build/src/gcc/bin > +export CC=$GCC_DIR/gcc > +export CXX=$GCC_DIR/g++ > +export CFLAGS="-B $GCC_DIR" > +export CXXFLAGS="-B $GCC_DIR" IIRC we're using PATH to do that in other scripts, maybe do the same for consistence?
Attachment #8992742 - Flags: review?(mh+mozilla) → review+
Attachment #8992745 - Flags: review?(mh+mozilla) → review+
(In reply to Mike Hommey [:glandium] from comment #40) > ::: taskcluster/scripts/misc/build-gn-linux.sh > @@ +9,5 @@ > > +GCC_DIR=$WORKSPACE/build/src/gcc/bin > > +export CC=$GCC_DIR/gcc > > +export CXX=$GCC_DIR/g++ > > +export CFLAGS="-B $GCC_DIR" > > +export CXXFLAGS="-B $GCC_DIR" > > IIRC we're using PATH to do that in other scripts, maybe do the same for > consistence? Digression first: it turns out that a clang-7 toolchain job was added, and clang 7 apparently changed some bits so that CMAKE_C{,XX}_FLAGS, where we set -B $GCC_DIR, are not honored for bits of libfuzzer. This breaks the libfuzzer build, specifically bits of compiling libcxx, since we can't find an assembler that understands some of the new syntax gcc uses. I looked at some ways to fix this and basically failed. I'm sure there's a good way to do it, I just couldn't find the magic cmake flags. Making --build and --target explicit for binutils, like we do for gcc, fixes problems: https://treeherder.mozilla.org/#/jobs?repo=try&revision=35a34c66dab2949e6f1b9832ef0e0113afc622d7 Note that the "fix bits for libcxx in LLVM 7" is ineffectual; I can provide links if need be. Making --target explicit, and matching the gcc --target, means that binutils installs bits in $GCC_DIR/x86_64-unknown-linux-gnu/bin also, whereas without --target, the binutils get dropped into $GCC_DIR/x86_64-pc-linux-gnu. GCC, at least as we configure it, will look in x86_64-unknown-linux-gnu/bin, but not in x86_64-pc-linux-gnu/bin/. Adding --build just seems nice for consistency. In fact, making --build and --target explicit for binutils means that we don't even have to futz around with all the -B business. The following try push drops the commits that specify -B in necessary places and everything is still green: https://treeherder.mozilla.org/#/jobs?repo=try&revision=fc0fb0663a2af950997d04887a7228d985d9d7e6 which is probably a good thing, as the continual necessity of passing -B would bite people who tried to depend on our gcc packages to compile other code. I think that binutils change is reasonably obvious, so I'm going to drop part 1 and 2, renumber everything, carry over r+, and land the whole shebang.
Pushed by nfroyd@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/b93c29157feb part 1 - be explicit about our GCC/binutils target and build machine; r=glandium https://hg.mozilla.org/integration/mozilla-inbound/rev/b4ebc9a500f4 part 2 - force clang to always pick up its local GCC headers and libraries; r=glandium https://hg.mozilla.org/integration/mozilla-inbound/rev/47405e33747c part 3 - inform stage2/3 clang about gcc binutils; r=glandium https://hg.mozilla.org/integration/mozilla-inbound/rev/52d37e70b6f1 part 4 - move toolchains off GCC 4.9; r=glandium https://hg.mozilla.org/integration/mozilla-inbound/rev/ae7ccb8c938d part 5 - touch CLOBBER; r=me
Pushed by nfroyd@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/8984f6868e57 follow-up - quiet clang-tidy content checking; r=me
Blocks: 1477405
Backout by csabou@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/dcfd01a3010b Backed out 6 changesets for causing build bustages on OS X Cross Compiled. CLOSED TREE
After forcing a rebuild of cctools-port on the backout, everything should be back to normal.
(In reply to Pulsebot from comment #42) > https://hg.mozilla.org/integration/mozilla-inbound/rev/ae7ccb8c938d > part 5 - touch CLOBBER; r=me CLOBBER does nothing on automation (all jobs are clobbers), and local builds are not affected by these changes, so I don't know why you feel this needs a CLOBBER touch.

Sigh, this is now biting bobowen because the type_traits our clang is finding is the GCC 4.9 version, which isn't fully up-to-date wrt C++11. And the patches are not quite trivially relandable:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=d7091425c443b1ddf5f0289e9f97b2c6d3c5c376

because the valgrind image is too old (maybe we could hack around that) and the mingw jobs are failing because our bundled linker can't find libstdc++. (Maybe if those jobs installed gcc, everything would be OK?)

OK, the mingw builds appear to be because (host) clang is looking in...weird places for libstdc++:

[task 2019-03-05T21:13:34.694Z] 21:13:34     INFO -  Found candidate GCC installation: /builds/worker/workspace/build/src/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0
[task 2019-03-05T21:13:34.694Z] 21:13:34     INFO -  Found candidate GCC installation: /usr/lib/gcc/i686-linux-gnu/6
[task 2019-03-05T21:13:34.694Z] 21:13:34     INFO -  Found candidate GCC installation: /usr/lib/gcc/i686-linux-gnu/6.3.0
[task 2019-03-05T21:13:34.694Z] 21:13:34     INFO -  Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/6
[task 2019-03-05T21:13:34.694Z] 21:13:34     INFO -  Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/6.3.0
[task 2019-03-05T21:13:34.694Z] 21:13:34     INFO -  Selected GCC installation: /builds/worker/workspace/build/src/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0
...
[task 2019-03-05T21:13:34.733Z] 21:13:34     INFO -  attempt to open /builds/worker/workspace/build/src/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0/libstdc++.so failed
[task 2019-03-05T21:13:34.733Z] 21:13:34     INFO -  attempt to open /builds/worker/workspace/build/src/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0/libstdc++.a failed
[task 2019-03-05T21:13:34.733Z] 21:13:34     INFO -  attempt to open /lib/x86_64-linux-gnu/libstdc++.so failed
[task 2019-03-05T21:13:34.733Z] 21:13:34     INFO -  attempt to open /lib/x86_64-linux-gnu/libstdc++.a failed
[task 2019-03-05T21:13:34.733Z] 21:13:34     INFO -  attempt to open /lib/../lib64/libstdc++.so failed
[task 2019-03-05T21:13:34.733Z] 21:13:34     INFO -  attempt to open /lib/../lib64/libstdc++.a failed
[task 2019-03-05T21:13:34.733Z] 21:13:34     INFO -  attempt to open /usr/lib/x86_64-linux-gnu/libstdc++.so failed
[task 2019-03-05T21:13:34.733Z] 21:13:34     INFO -  attempt to open /usr/lib/x86_64-linux-gnu/libstdc++.a failed
[task 2019-03-05T21:13:34.733Z] 21:13:34     INFO -  attempt to open /usr/lib/../lib64/libstdc++.so failed
[task 2019-03-05T21:13:34.733Z] 21:13:34     INFO -  attempt to open /usr/lib/../lib64/libstdc++.a failed
[task 2019-03-05T21:13:34.734Z] 21:13:34     INFO -  attempt to open /builds/worker/workspace/build/src/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0/../../../libstdc++.so failed
[task 2019-03-05T21:13:34.734Z] 21:13:34     INFO -  attempt to open /builds/worker/workspace/build/src/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0/../../../libstdc++.a failed
[task 2019-03-05T21:13:34.734Z] 21:13:34     INFO -  attempt to open /builds/worker/workspace/build/src/clang/bin/../lib/libstdc++.so failed
[task 2019-03-05T21:13:34.734Z] 21:13:34     INFO -  attempt to open /builds/worker/workspace/build/src/clang/bin/../lib/libstdc++.a failed
...

[task 2019-03-05T21:13:34.734Z] 21:13:34     INFO -  attempt to open /builds/worker/workspace/src/gcc/lib64/libstdc++.so failed
[task 2019-03-05T21:13:34.734Z] 21:13:34     INFO -  attempt to open /builds/worker/workspace/src/gcc/lib64/libstdc++.a failed
...

So either we need to copy more files or possibly install gcc? It's not clear to me why this same problem wouldn't happen on, say, linux64 compiles.

Ah, the reason that comment 50 doesn't happen on linux64 compiles is that linux64 compiles pick up the clang/lib/libstdc++.so file, but that file is explicitly removed in the mingw case:

https://searchfox.org/mozilla-central/source/taskcluster/scripts/misc/build-clang-trunk-mingw.sh#324

Of course, I reviewed the patch that introduced that, so I ought to understand why we do that. I assume that it is to encourage the mingw target to pick the (target-specific?) libc++ in the same directory rather than the (host-specific) libstdc++? Or just to use libc++ instead of libstdc++? Jacek, do you know?

I think the Right Thing is to copy those files into a target-specific directory, rather than the general lib/ directory...I'm not sure if that will cause problem later down the line.

Flags: needinfo?(jacek)

I don't remember details, but I think it was because it wasn't compatible with host libstdc++.so and that caused problems on Taskcluster when using mingw clang as host compiler. Maybe we may get rid of this rm on top of your patches.

Flags: needinfo?(jacek)
Depends on: 1533145

PGO builds are consistently running into things like:

[task 2019-03-06T20:18:56.776Z] 20:18:56     INFO -  /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/mozalloc.h:131: undefined reference to `moz_xmalloc'
[task 2019-03-06T20:18:56.777Z] 20:18:56     INFO -  /tmp/lto-llvm-c957ca.o:/builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/mozalloc.h:131: more undefined references to `moz_xmalloc' follow
[task 2019-03-06T20:18:56.777Z] 20:18:56     INFO -  /builds/worker/workspace/build/src/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0/../../../../x86_64-unknown-linux-gnu/bin/ld: Dwarf Error: mangled line number section (bad file number).
[task 2019-03-06T20:18:56.777Z] 20:18:56     INFO -  /builds/worker/workspace/build/src/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0/../../../../x86_64-unknown-linux-gnu/bin/ld: Dwarf Error: mangled line number section (bad file number).
[task 2019-03-06T20:18:56.777Z] 20:18:56     INFO -  /builds/worker/workspace/build/src/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.4.0/../../../../x86_64-unknown-linux-gnu/bin/ld: Dwarf Error: mangled line number section (bad file number).
[task 2019-03-06T20:18:56.777Z] 20:18:56     INFO -  /tmp/lto-llvm-c957ca.o: In function `mozilla::dom::WorkerPrivate::WorkerPrivate(mozilla::dom::WorkerPrivate*, nsTSubstring<char16_t> const&, bool, mozilla::dom::WorkerType, nsTSubstring<char16_t> const&, nsTSubstring<char> const&, mozilla::dom::WorkerLoadInfo&)':
[task 2019-03-06T20:18:56.777Z] 20:18:56     INFO -  Unified_cpp_dom_workers1.cpp:(.text._ZN7mozilla3dom13WorkerPrivateC2EPS1_RK12nsTSubstringIDsEbNS0_10WorkerTypeES6_RKS3_IcERNS0_14WorkerLoadInfoE+0x673): undefined reference to `gMozillaPoisonValue'
[task 2019-03-06T20:18:56.777Z] 20:18:56     INFO -  /tmp/lto-llvm-c957ca.o: In function `operator new(unsigned int)':

Lots and lots of problems finding moz_xmalloc and gMozillaPoisonValue...but nothing else.

glandium, do you recall seeing anything like this when switching things over to clang? It puzzles me how a GCC switch here is affecting things, unless GCC 6 miscompiles clang, or something...?

Oh, oh, we're also maybe forcing the use of GNU ld by putting it first on the search path somehow. Is this us using an ld (from binutils) that clang doesn't expect, perhaps?

glandium, see comment 53, please.

Flags: needinfo?(mh+mozilla)

I've never seen this, but OTOH, that log says it's not using the ld from the binutils toolchain archive at all, like it should...

Flags: needinfo?(mh+mozilla)

(In reply to Mike Hommey [:glandium] from comment #55)

I've never seen this, but OTOH, that log says it's not using the ld from the binutils toolchain archive at all, like it should...

Ah, right, OK. So build jobs use some separate linux64-binutils thing, whereas the gcc packages build their own, separate binutils that is not version-aligned with linux64-binutils. But we need the clang bootstrap process, at least, to use a newer binutils than what the system provides.

There are at least two different ways around this:

  1. Make the GCC builds use the same version of binutils as the standalone binutils job.
  2. Make the GCC builds not build binutils themselves, but pull linux64-binutils and copy the binaries over.

The first one is expedient (and is almost trivially easy to write/verify). The second one is more robust, but also requires more extensive changes. In the interests of expediency (so people can start using all the C++11 things in, you know, C++11 headers), I would like to go with the first solution, with a commitment to follow up on the second solution. Sound good?

Flags: needinfo?(mh+mozilla)

Fair enough.

Flags: needinfo?(mh+mozilla)

Explicit is better than implicit, and helps ensure that GCC is always
using the binutils we built it with, rather than the system binutils.

We want our clang bootstrap to use the GCC headers we're building with,
not whatever sysroot it happens to find on the server we're building on.

The -gcc-toolchain argument we specify when building clang will also be
picked up by llvm-config, so we need to strip it out when building the
plugin. Otherwise, we will get peculiar failures about not being able to
find C++ header files.

Depends on D22879

Attachment #8983187 - Attachment is obsolete: true

We do this to encourage clang to find an new-enough linker instead of
the system one.

Depends on D22880

Attachment #8983188 - Attachment is obsolete: true
Attachment #8987199 - Attachment is obsolete: true

We're going to copy an x86_64-unknown-linux-gnu ld into the clang build,
which clang will then use in preference to things on PATH. We therefore
need to ensure that this ld is the same ld as would be used for other
builds, such as PGO. This change is the most expedient way to do that;
future work will make the gcc job(s) depend on linux64-binutils directly.

Depends on D22881

Attachment #8987201 - Attachment is obsolete: true

Firefox itself has moved on to GCC 6.x; we can move our toolchains along too.

Depends on D22882

History does not disclose why we needed this, but in the brave new GCC
6-compiled world, deleting these files means that host links can no
longer find libstdc++, which causes problems. Let's put the files back.

Depends on D22883

Attachment #8992742 - Attachment is obsolete: true
Attachment #8992745 - Attachment is obsolete: true

Uploaded new patches to phab. Parts 4 and 6 on phab are new, and parts 3 and 5 have been slightly tweaked from their corresponding patches here.

Looks like there's still some bits to iron out, but I've tried my chromium update with these patches and (aside from something I need to fix) the Linux builds are now working, thanks.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a1cc59ab891f46a46c3ad2c4b3eae0509c7787a6

Depends on: 1535142
Pushed by nfroyd@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/991777e081ff part 1 - be explicit about our GCC/binutils target and build machine; r=glandium https://hg.mozilla.org/integration/autoland/rev/c64bfaad8a2f part 2 - force clang to always pick up its local GCC headers and libraries; r=glandium https://hg.mozilla.org/integration/autoland/rev/675f73d41eb4 part 3 - inform stage2/3 clang about gcc binutils; r=glandium https://hg.mozilla.org/integration/autoland/rev/5179c8066914 part 4 - sync up gcc-related toolchains and linux64-binutils binutils version; r=glandium https://hg.mozilla.org/integration/autoland/rev/7c116f85ede6 part 5 - move toolchains off GCC 4.9; r=glandium https://hg.mozilla.org/integration/autoland/rev/2f6199beeb71 part 6 - don't remove the libstdc++ files from the mingw build; r=glandium

Backed out 6 changesets (bug 1451104) for toolchains bustage on a CLOSED TREE.

Backout link: https://hg.mozilla.org/integration/autoland/rev/779dcbea91ce69857c4aa3a9d1823905cb45ee03

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=success%2Cretry%2Cpending%2Crunning%2Ctestfailed%2Cbusted%2Cexception&selectedJob=233789124&searchStr=toolchains%2Copt%2Ctoolchain%2Cbuilds%2Cfor%2Clinux%2C64-bits%2Ctoolchain-linux64-clang-tidy%2Ctl%28clang-tidy%29&revision=2f6199beeb7158f0952fd029e3262a227e176220

Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=233789124&repo=autoland&lineNumber=7071

Log snippet:

[task 2019-03-14T02:34:24.851Z] -- Installing: /builds/worker/workspace/moz-toolchain/build/stage1/clang-tidy/lib/cmake/llvm/./FindSphinx.cmake
[task 2019-03-14T02:34:24.851Z] -- Installing: /builds/worker/workspace/moz-toolchain/build/stage1/clang-tidy/lib/cmake/llvm/./AddOCaml.cmake
[task 2019-03-14T02:34:24.851Z] -- Installing: /builds/worker/workspace/moz-toolchain/build/stage1/clang-tidy/lib/cmake/llvm/./LLVMProcessSources.cmake
[task 2019-03-14T02:34:24.852Z] -- Installing: /builds/worker/workspace/moz-toolchain/build/stage1/clang-tidy/lib/cmake/llvm/./AddLLVMDefinitions.cmake
[task 2019-03-14T02:34:24.852Z] -- Installing: /builds/worker/workspace/moz-toolchain/build/stage1/clang-tidy/lib/cmake/llvm/./AddLLVM.cmake
[task 2019-03-14T02:34:24.852Z] -- Installing: /builds/worker/workspace/moz-toolchain/build/stage1/clang-tidy/lib/cmake/llvm/./FindOCaml.cmake
[task 2019-03-14T02:34:24.852Z] -- Installing: /builds/worker/workspace/moz-toolchain/build/stage1/clang-tidy/lib/cmake/llvm/./HandleLLVMOptions.cmake
[task 2019-03-14T02:34:24.878Z] cd "/builds/worker/workspace/build/src/build/build-clang"
[task 2019-03-14T02:34:25.213Z] Traceback (most recent call last):
[task 2019-03-14T02:34:25.213Z] File "./build-clang.py", line 796, in <module>
[task 2019-03-14T02:34:25.213Z] osx_cross_compile)
[task 2019-03-14T02:34:25.213Z] File "./build-clang.py", line 426, in prune_final_dir_for_clang_tidy
[task 2019-03-14T02:34:25.213Z] raise Exception("Found unknown file %s in the final directory" % f)
[task 2019-03-14T02:34:25.213Z] Exception: Found unknown file /builds/worker/workspace/moz-toolchain/build/stage1/clang-tidy/x86_64-unknown-linux-gnu in the final directory
[fetches 2019-03-14T02:34:25.231Z] removing /builds/worker/workspace/build
[fetches 2019-03-14T02:34:25.657Z] finished
[taskcluster 2019-03-14 02:34:26.062Z] === Task Finished ===
[taskcluster 2019-03-14 02:34:27.796Z] Unsuccessful task run with exit code: 1 completed in 678.969 seconds

Flags: needinfo?(nfroyd)
Depends on: 1535350
Pushed by nfroyd@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/468eed96e879 part 1 - be explicit about our GCC/binutils target and build machine; r=glandium https://hg.mozilla.org/integration/autoland/rev/961ac7318786 part 2 - force clang to always pick up its local GCC headers and libraries; r=glandium https://hg.mozilla.org/integration/autoland/rev/f3290e95b38d part 3 - inform stage2/3 clang about gcc binutils; r=glandium https://hg.mozilla.org/integration/autoland/rev/292ed1bb9143 part 4 - sync up gcc-related toolchains and linux64-binutils binutils version; r=glandium https://hg.mozilla.org/integration/autoland/rev/858b68d306f2 part 5 - move toolchains off GCC 4.9; r=glandium https://hg.mozilla.org/integration/autoland/rev/76a3b7b0c9d7 part 6 - don't remove the libstdc++ files from the mingw build; r=glandium
Flags: needinfo?(nfroyd)
Blocks: 1571587
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: