This spun off of bug 1329593. :gcp, :ted suggested we add the buildchain used by Tor Browser as a tier 1 (?) compile target to Taskcluster. (Tor is compiling with mingw-cross to get reproducible builds, which seems to be an interesting topic for :arroway, CCing)

Adding :tjr as he is AFAICT currently concerned with setting up mingw-w64 build infrastructure to make it a tier 2 target.

We seem to break the mingw build fairly often, so if we're committing to support the Tor browser we should at a minimum ensure that their Windows builds work. I think getting these up and running in Taskcluster wouldn't be a huge effort, I stood up OS X cross-compiles a while ago (bug 935237) and most of the work was getting the toolchain and packaging sorted out, which I assume the Tor folks already have a handle on. The way we define Taskcluster tasks in the tree has changed a bit, but we have really good docs for how it works now: http://gecko.readthedocs.io/en/latest/taskcluster/taskcluster/how-tos.html#hacking-task-graphs The cross-mac build task definitions are here now: https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/build/macosx.yml Our Linux and Mac-cross builds build in a Docker container, defined here: https://dxr.mozilla.org/mozilla-central/source/taskcluster/docker/desktop-build/Dockerfile it's based on this base image: https://dxr.mozilla.org/mozilla-central/source/taskcluster/docker/centos6-build/Dockerfile + this one that just applies system updates (to keep the docker images smaller): https://dxr.mozilla.org/mozilla-central/source/taskcluster/docker/centos6-build-upd/Dockerfile For the mac-cross builds we didn't need too many additional dependencies, so I just added a handful of things to the desktop-build image and then put the rest in tooltool (cctools, the tools to build DMG files): https://dxr.mozilla.org/mozilla-central/source/browser/config/tooltool-manifests/macosx64/cross-releng.manifest If the tor mingw-cross builds need a lot of additional build tools we could create a new Docker image that derives from desktop-build and installs the additional build tools. If it's just a few things then we could use the desktop-build image and add things to it or put them in tooltool as appropriate. We tend to have toolchains in tooltool because we still have builds that don't run in Docker, but there's no real reason you'd have to use tooltool here, you could just bake the toolchain into the Docker image since these builds will only ever run in taskcluster. I'd be happy to offer advice and hand-holding here if someone wants to take this, it should all be self-service, and you can test new build types on try.

I'll take this, but I will need some help along the way. =) My intention, for Phase 1 at least, is to get a mingw build of FF going using a Docker container and into TaskCluster. It would be fantastic if we could promote this to Tier 1 (once we resolve the existing bugs that prevent compilation.) I'm following https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Cross_Compile_Mozilla_for_Mingw32 and encountering some errors, but have a few things to try yet. (Phase N would add a FF that is compiled & preffed to behave similar to TBB and run it through our test suite, Phase M would add proxy bypass tests; and so on.)

It is unclear if the scope of this bug includes building in Tor's reproducible build environment. Either way, we should still track it against our reproducible build efforts. FWIW, I'd like to move in the direction where the taskcluster taskgraph includes the build dependencies for Firefox. We could conceivably build all the build dependencies from source using a trusted base image then feed those into a trusted build image. Any time a dependency changes, it is recompiled from scratch and we get a new, deterministic build environment. This would essentially be recreating Tor's build environment but in a distributed execution environment. Bug 1313111 will likely be our first steps in this area.

T(In reply to Gregory Szorc [:gps] from comment #4) > It is unclear if the scope of this bug includes building in Tor's > reproducible build environment. Either way, we should still track it against > our reproducible build efforts. The main goal here was to make sure we don't unknowingly break building with mingw, thereby absolving the Tor people of needing to hunt and fix those regression. Making a reproducible build is a different, further step.

Yeah, I think the best bang for the buck here is just getting a mingw cross-compile stood up that is at least a close approximation of tor's build. We break the mingw build pretty frequently, so if they don't have to deal with that pain they'll be a lot better off. I haven't looked into the reproducible bits of their build, but those parts don't seem like things we're likely to break as often. If that turns out not to be true we could certainly stand up builds that use that machinery as a follow-up.

I think these patches have more to do with the build system, so I'm moving the bug. (I'm not a huge fan of the Task Configuration component because its scope is so wide.)

I wouldn't mind moving and/or renaming that component :)

Would it be worth investigating cross-compiling with clang-cl instead of mingw? It seems like that would be less likely to break than the mingw builds and would also be useful for us more generally in being able to switch to cross compiling our release builds.

Chrome's equivalent is happening here: https://bugs.chromium.org/p/chromium/issues/detail?id=495204

(In reply to Jeff Muizelaar [:jrmuizel] from comment #17) > Would it be worth investigating cross-compiling with clang-cl instead of > mingw? It seems like that would be less likely to break than the mingw > builds and would also be useful for us more generally in being able to > switch to cross compiling our release builds. It would, and we did. :) We talked with the Chrome engineer who is driving that effort. That project (compiling chrome on Linux for Windows) is a hobby project really, and there are a lot of things left to be done (especially in the linking area). So we're keeping an eye on it, and asked him to keep us in the loop, but until it matures a bit (and then we invest in making it reproducible) Tor won't be able to switch over.

(In reply to Tom Ritter [:tjr] from comment #19) > It would, and we did. :) We talked with the Chrome engineer who is driving > that effort. That project (compiling chrome on Linux for Windows) is a hobby > project really, and there are a lot of things left to be done (especially in > the linking area). Is there a list somewhere of what these things all are. I'd perhaps be interested in contributing to that effort.

(In reply to Jeff Muizelaar [:jrmuizel] from comment #20) > (In reply to Tom Ritter [:tjr] from comment #19) > > It would, and we did. :) We talked with the Chrome engineer who is driving > > that effort. That project (compiling chrome on Linux for Windows) is a hobby > > project really, and there are a lot of things left to be done (especially in > > the linking area). > > Is there a list somewhere of what these things all are. I'd perhaps be > interested in contributing to that effort. Yup! I don't feel comfortable pasting emails into bugzilla, but I'll add you to a gdoc that has as much info as I have, and can add anyone else who is likewise interested.

Comment on attachment 8881647 [details] Bug 1330608 Add the mingw32 build job (Work in Progress) https://reviewboard.mozilla.org/r/152810/#review159228 ::: taskcluster/scripts/misc/build-mingw-linux.sh:27 (Diff revision 8) > +mkdir mingw-w64-toolchain > +cd mingw-w64-toolchain > + > +wget http://ftp.gnu.org/gnu/binutils/binutils-2.27.tar.bz2 > +wget http://ftp.gnu.org/gnu/binutils/binutils-2.27.tar.bz2.sig > +gpg --keyserver hkp://pool.sks-keyservers.net --recv-key EAF1C276A747E9ED86210CBAC3126D3B4AE55E93 It turns out that sometimes you can't get the keys from the keyserver in the pool. Fun. My thinking is just pasting a giant public key blob at the top of this script (or in another file) and importing from there.

See build/unix/build-gcc/build-gcc.sh You should actually reuse the script from there, as well as the keys. If you need keys that are not in the tree already, please use https://github.com/glandium/pgpstrip to make them smaller. Also see the commit message for bug 1335667, which succintly describes the steps to verify the keys themselves.

Comment on attachment 8881647 [details] Bug 1330608 Add the mingw32 build job (Work in Progress) https://reviewboard.mozilla.org/r/152810/#review159252 ::: taskcluster/ci/toolchain/linux.yml:21 (Diff revision 10) > + run: > + using: toolchain-script > + script: build-mingw-linux.sh > + tooltool-downloads: public > + resources: > + - 'build/build-clang/**' this field lists the (extra) scripts used by the build script, and you're not using build-clang files. ::: taskcluster/docker/desktop-build/Dockerfile:71 (Diff revision 10) > # Install the screen package here to use with xvfb. > # Install bison to build binutils. > RUN yum install -y bison screen > > +# For the mingw toolchain > +RUN yum install -y gmp-devel mpfr-devel libmpc-devel scons flex See build/unix/build-gcc/build-gcc.sh, we shouldn't need gmp-devel, mpfr-devel or libmpc-devel. I'm not sure why you need to add flex, but I suspect it should work without. You don't need scons if you move building nsis to a separate job (which you should do ; please don't build multiple different tools in the same job) (Also, touching this file has the annoying side effect of breaking unrelated jobs, like valgrind, thanks to the non-reproduceability of docker) ::: taskcluster/scripts/misc/build-mingw-linux.sh:25 (Diff revision 10) > +mkdir -p $INSTALL_DIR > + > +mkdir mingw-w64-toolchain > +cd mingw-w64-toolchain > + > +wget http://ftp.gnu.org/gnu/binutils/binutils-2.27.tar.bz2 It seems like you're mostly repeating what build/unix/build-gcc/build-gcc.sh does ; it feels like there should be some sharing of some sort between the scripts... or maybe modify the build-gcc script to allow to build the mingw-cross one. ::: taskcluster/scripts/misc/build-mingw-linux.sh:100 (Diff revision 10) > +cd .. > +wget https://downloads.sourceforge.net/project/nsis/NSIS%203/3.01/nsis-3.01-src.tar.bz2 > +echo "559703cc25f78697be1784a38d1d9a19c97d27a200dc9257d1483c028c6be9242cbcd10391ba618f88561c2ba57fdbd8b3607bea47ed8c3ad7509a6ae4075138 nsis-3.01-src.tar.bz2" | sha512sum -c - > +bunzip2 nsis-3.01-src.tar.bz2 > +tar xf nsis-3.01-src.tar > +cd nsis-3.01-src > +# I don't know how to make the version work with the environment variables/config flags the way the author appears to > +sed -i "s/'VERSION', 'Version of NSIS', cvs_version/'VERSION', 'Version of NSIS', '3.01'/" SConstruct > +scons XGCC_W32_PREFIX=i686-w64-mingw32- ZLIB_W32=../zlib-1.2.11 SKIPUTILS="NSIS Menu" PREFIX=$INSTALL_DIR/ install > + > +cd .. > +wget http://dl.winehq.org/wine/source/2.0/wine-2.0.1.tar.xz > +wget http://dl.winehq.org/wine/source/2.0/wine-2.0.1.tar.xz.sign > +gpg --keyserver hkp://pool.sks-keyservers.net --recv-key DA23579A74D4AD9AF9D3F945CEFAC8EAAF17519D > +gpg --verify wine-2.0.1.tar.xz.sign > +unxz wine-2.0.1.tar.xz > +tar xf wine-2.0.1.tar > +cd wine-2.0.1 > +./configure --prefix=$INSTALL_DIR/ > +make > +make install > + > +cd .. > +git clone https://github.com/tomrittervg/fxc2.git > +cd fxc2 > +git checkout e00a32fbd3d5814b3f599c8528e37db9c1456803 # Asserts integrity of the clone (right?) > +make > +cp fxc2.exe $INSTALL_DIR/bin/ nsis, wine and fxc2 should go in separate jobs. ::: taskcluster/scripts/misc/build-mingw-linux.sh:128 (Diff revision 10) > +git clone https://github.com/tomrittervg/fxc2.git > +cd fxc2 > +git checkout e00a32fbd3d5814b3f599c8528e37db9c1456803 # Asserts integrity of the clone (right?) > +make > +cp fxc2.exe $INSTALL_DIR/bin/ > +cp d3dcompiler_47.dll $INSTALL_DIR/bin/ where does that file come from?

Comment on attachment 8873591 [details] Bug 1330608 Add the mingw32 build job (Work in Progress) https://reviewboard.mozilla.org/r/144966/#review159256 ::: browser/config/mozconfigs/win32/mingw32:29 (Diff revision 20) > +# RUSTC > +# CARGO > +. "$topsrcdir/browser/config/mozconfigs/common" > + > +# MinGW Stuff > +export CROSS_COMPILE=1 Shouldn't be required. ::: browser/config/mozconfigs/win32/mingw32:30 (Diff revision 20) > +# CARGO > +. "$topsrcdir/browser/config/mozconfigs/common" > + > +# MinGW Stuff > +export CROSS_COMPILE=1 > +ac_add_options --enable-application=browser That's the default. ::: browser/config/mozconfigs/win32/mingw32:33 (Diff revision 20) > +# MinGW Stuff > +export CROSS_COMPILE=1 > +ac_add_options --enable-application=browser > +ac_add_options --target=i686-w64-mingw32 > +ac_add_options --with-toolchain-prefix=i686-w64-mingw32- > +ac_add_options --enable-default-toolkit=cairo-windows That /shouldn't/ be required. ::: testing/mozharness/configs/builds/releng_base_windows_32_mingw_builds.py:1 (Diff revision 20) > +import os would be nicer if this was marked as a copy of an existing mozharness config. If you are using mercurial, use hg copy ; if you are using git, ping me.

Uh oh. I accidentally committed one of the patches but not the other and screwed mozreview up something bad. inter-diffs are going to be a mess, and the issues opened may not match the commit.... (In reply to Mike Hommey [:glandium] from comment #52) > this field lists the (extra) scripts used by the build script, and you're > not using build-clang files. Fixed. > See build/unix/build-gcc/build-gcc.sh, we shouldn't need gmp-devel, > mpfr-devel or libmpc-devel. > I'm not sure why you need to add flex, but I suspect it should work without. > You don't need scons if you move building nsis to a separate job (which you > should do ; please don't build multiple different tools in the same job) > > (Also, touching this file has the annoying side effect of breaking unrelated > jobs, like valgrind, thanks to the non-reproduceability of docker) scons is needed for nsis; and flex for wine. I can drop gmp-devel, mpfr-devel, and libmpc-devel - but should I leave scons and flex here? Or should I use some alternate method to supply them for those build tasks? > It seems like you're mostly repeating what build/unix/build-gcc/build-gcc.sh > does ; it feels like there should be some sharing of some sort between the > scripts... or maybe modify the build-gcc script to allow to build the > mingw-cross one. I have edited and rearchitected this script. I'm not sure you'll like the result - I'm open to doing it how you would prefer it, just let me know. > nsis, wine and fxc2 should go in separate jobs. Done > > +cp d3dcompiler_47.dll $INSTALL_DIR/bin/ > > where does that file come from? My local machine. But it's signed by Microsoft?

All of these issues seem to have disappears from mozreview. =( (In reply to Mike Hommey [:glandium] from comment #53) > > +export CROSS_COMPILE=1 > > Shouldn't be required. Removed. > > +ac_add_options --enable-application=browser > > That's the default. Removed. > > +ac_add_options --enable-default-toolkit=cairo-windows > > That /shouldn't/ be required. Removed. > ::: > testing/mozharness/configs/builds/releng_base_windows_32_mingw_builds.py:1 > (Diff revision 20) > > +import os > > would be nicer if this was marked as a copy of an existing mozharness > config. If you are using mercurial, use hg copy ; if you are using git, ping > me. I tried to do this... I'm not sure it took after I histedit+rolled though.

Comment on attachment 8884418 [details] Bug 1330608 Add the MinGW32 toolchain build to Taskcluster https://reviewboard.mozilla.org/r/155334/#review175532 ::: taskcluster/ci/toolchain/linux.yml:264 (Diff revision 15) > + using: toolchain-script > + script: build-mingw32-nsis.sh > + tooltool-downloads: public > + resources: > + - 'taskcluster/scripts/misc/tooltool-download.sh' > + - 'browser/config/tooltool-manifests/linux64/mingw32-components.manifest' See https://bugzilla.mozilla.org/show_bug.cgi?id=1391744

Okay, I think I have the toolchain half of this ready for review. There is some weirdness in the 'resources' section of the yml due to Bug 1391744, but other than that I think it's in good shape.

Comment on attachment 8884418 [details] Bug 1330608 Add the MinGW32 toolchain build to Taskcluster https://reviewboard.mozilla.org/r/155334/#review175970 ::: taskcluster/ci/toolchain/linux.yml:265 (Diff revision 15) > + script: build-mingw32-nsis.sh > + tooltool-downloads: public > + resources: > + - 'taskcluster/scripts/misc/tooltool-download.sh' > + - 'browser/config/tooltool-manifests/linux64/mingw32-components.manifest' > + toolchain-artifact: public/build/nsis.tar.xz Can I specify 'toolchains' here (like in taskcluster/build/ci/linux.yml) to indicate this toolchain relies on another toolchain?

Comment on attachment 8884418 [details] Bug 1330608 Add the MinGW32 toolchain build to Taskcluster https://reviewboard.mozilla.org/r/155334/#review177704 ::: browser/config/tooltool-manifests/linux64/mingw32-components.manifest:3 (Diff revision 17) > +[ > + { > + "version": "mingw32: gcc 5.1.0, binutils 2.27, mingw-w64 9c7c712240", why are you using a tooltool manifest for mingw gcc when you build one already? ::: build/unix/build-gcc/build-gcc.sh:100 (Diff revision 17) > } > > build_binutils() { > mkdir $root_dir/binutils-objdir > pushd $root_dir/binutils-objdir > - # gold is disabled because we don't use it on automation, and also we ran into > + ../binutils-$binutils_version/configure --prefix /tools/gcc/ $binutils_configure_flags I'm not a big fan of duplicating the binutils and gcc configure options in the various scripts that source this one. How about leaving the flags if none are passed in, and replace them otherwise? ::: build/unix/build-gcc/build-gcc.sh:109 (Diff revision 17) > } > > build_gcc() { > mkdir $root_dir/gcc-objdir > pushd $root_dir/gcc-objdir > - ../gcc-$gcc_version/configure --prefix=/tools/gcc --enable-languages=c,c++ --disable-nls --disable-gnu-unique-object --enable-__cxa_atexit --with-arch-32=pentiumpro > + ../gcc-$gcc_version/configure --prefix=/tools/gcc $gcc_configure_flags Note, you're not using build_gcc... ::: build/unix/build-gcc/build-gcc.sh:119 (Diff revision 17) > tar caf $root_dir/gcc.tar.xz gcc/ > popd > } > + > +build_gcc_and_mingw() { > + export install_dir=$root_dir/tools/gcc/ Having /home/worker/workspace/build in the install dir is suspicious. ::: build/unix/build-gcc/build-gcc.sh:125 (Diff revision 17) > + mkdir -p $install_dir > + > + export PATH=$PATH:$install_dir/bin/ > + > + cd $root_dir > + git clone git://git.code.sf.net/p/mingw-w64/mingw-w64 git clone -n ::: build/unix/build-gcc/build-gcc.sh:137 (Diff revision 17) > + ../gcc-$gcc_version/configure --prefix=$install_dir $gcc_configure_flags > + make $make_flags all-gcc > + make $make_flags install-gcc > + popd > + > + cd $root_dir you don't need to re-cd each time, popd brings you back there. ::: taskcluster/ci/toolchain/linux.yml:215 (Diff revision 17) > +linux64-mingw32-gcc: > + description: "GCC toolchain build for MinGW32 Cross Compile" > + treeherder: > + kind: build > + platform: toolchains/opt > + symbol: TL(mingw32-gcc) It feels like it would be better to have these jobs in a separate group than TL. I'm also tempted to tell you to move them in a separate yml file, but I'm not completely sold on this. ::: taskcluster/ci/toolchain/linux.yml:216 (Diff revision 17) > + description: "GCC toolchain build for MinGW32 Cross Compile" > + treeherder: > + kind: build > + platform: toolchains/opt > + symbol: TL(mingw32-gcc) > + tier: 1 This and the other jobs surely are not tier 1. ::: taskcluster/ci/toolchain/linux.yml:220 (Diff revision 17) > + symbol: TL(mingw32-gcc) > + tier: 1 > + worker-type: aws-provisioner-v1/gecko-{level}-b-linux > + worker: > + docker-image: {in-tree: desktop-build} > + max-run-time: 36000 it would probably be a good idea to find more reasonable max run times for these jobs, instead of cargo culting the too long values we're using on other jobs :) ::: taskcluster/ci/toolchain/linux.yml:225 (Diff revision 17) > + max-run-time: 36000 > + run: > + using: toolchain-script > + script: build-gcc-mingw32.sh > + resources: > + - 'build/unix/build-gcc/**' Please don't use wildcards. See 9a82fd89dff95e2a87f7d3818179b9218e1feb1b. ::: taskcluster/ci/toolchain/linux.yml:233 (Diff revision 17) > +linux64-mingw32-wine: > + description: "Wine build for MinGW32 Cross Compile" > + treeherder: > + kind: build > + platform: toolchains/opt > + symbol: TL(mingw32-wine) This is just wine, right? ::: taskcluster/scripts/misc/build-gcc-4.9-linux.sh:26 (Diff revision 17) > +$GPG --import $data_dir/33C235A34C46AA3FFB293709A328C3A2C3C45C06.key > +$GPG --import $data_dir/7F74F97C103468EE5D750B583AB00996FC26A641.key > +$GPG --import $data_dir/80F98B2E0DAB6C8281BDF541A7C8C3B2F71EDF1C.key > +$GPG --import $data_dir/90AA470469D3965A87A5DCB494D03953902C9419.key > +$GPG --import $data_dir/B215C1633BCA0477615F1B35A5B3A004745C015A.key > +$GPG --import $data_dir/B3C42148A44E6983B3E4CC0793FA9B1AB75C61B8.key Why are these keys suddenly necessary? ::: taskcluster/scripts/misc/build-gcc-mingw32.sh:24 (Diff revision 17) > +$GPG --import $data_dir/13975A70E63C361C73AE69EF6EEB81F8981C74C7.key > +$GPG --import $data_dir/33C235A34C46AA3FFB293709A328C3A2C3C45C06.key > +$GPG --import $data_dir/7F74F97C103468EE5D750B583AB00996FC26A641.key > +$GPG --import $data_dir/80F98B2E0DAB6C8281BDF541A7C8C3B2F71EDF1C.key > +$GPG --import $data_dir/90AA470469D3965A87A5DCB494D03953902C9419.key > +$GPG --import $data_dir/B215C1633BCA0477615F1B35A5B3A004745C015A.key > +$GPG --import $data_dir/B3C42148A44E6983B3E4CC0793FA9B1AB75C61B8.key I doubt you need more than one here. ::: taskcluster/scripts/misc/build-mingw32-fxc2.sh:20 (Diff revision 17) > + > +cd $WORKSPACE > + > +# -------------- > + > +git clone https://github.com/tomrittervg/fxc2.git git clone -n ::: taskcluster/scripts/misc/build-mingw32-fxc2.sh:30 (Diff revision 17) > +cp d3dcompiler_47.dll $INSTALL_DIR/bin/ > + > +# -------------- > + > +cd $WORKSPACE > +tar cf fxc2.tar gcc The implicit contract is that toolchain artifacts extract to a directory with the stem name of the archive. That is, unpacking foo.tar.xz should create a foo directory. You should also be able to use tar -acf foo.tar.xz instead of first create a tar and then xz it. ::: taskcluster/scripts/misc/build-mingw32-nsis.sh:37 (Diff revision 17) > +export PATH="$TOOLTOOL_DIR/gcc-mingw/bin:$TOOLTOOL_DIR/gcc-mingw/i686-w64-mingw32/bin:$PATH" > +mkdir -p $INSTALL_DIR > + > +cd $WORKSPACE > + > +echo "-----BEGIN PGP PUBLIC KEY BLOCK----- gpg --import <<EOF -----BEGIN PGP PUBLIC KEY BLOCK----- ... -----END PGP PUBLIC KEY BLOCK----- EOF ::: taskcluster/scripts/misc/build-mingw32-nsis.sh:78 (Diff revision 17) > +wget http://zlib.net/zlib-1.2.11.tar.gz > +wget http://zlib.net/zlib-1.2.11.tar.gz.asc > +gpg --verify zlib-1.2.11.tar.gz.asc why not use download_and_check from build-gcc.sh? (possibly moving it to a separate file) ::: taskcluster/scripts/misc/build-mingw32-nsis.sh:81 (Diff revision 17) > +gunzip zlib-1.2.11.tar.gz > +tar xf zlib-1.2.11.tar tar xaf ::: taskcluster/scripts/misc/build-mingw32-nsis.sh:89 (Diff revision 17) > +bunzip2 nsis-3.01-src.tar.bz2 > +tar xf nsis-3.01-src.tar tar xaf

Comment on attachment 8884418 [details] Bug 1330608 Add the MinGW32 toolchain build to Taskcluster https://reviewboard.mozilla.org/r/155334/#review177704 > why are you using a tooltool manifest for mingw gcc when you build one already? This manifest is used to build fxc2 and nsis with MinGW, so they need the mingw toolchain to run their job. If I can make one toolchain have a dependency on another toolchain I can eliminate this. > It feels like it would be better to have these jobs in a separate group than TL. > > I'm also tempted to tell you to move them in a separate yml file, but I'm not completely sold on this. Sure, I can put them wherever you like. What group should be in/symbol should I use? > This is just wine, right? Yup. I could yum install it into the image but I don't think we want wine in all our images.... > Why are these keys suddenly necessary? I added them because it seemed prudent to be futureproof. If a new gcc release is signed by one of the other keys (which gcc says they may be) our build will suddenly break. It may not be that difficult to figure it out and fix it, but... it seemed more correct to accept any of the keys gcc says the release may be signed by, instead of just the one that seems most commonly used.

(In reply to Tom Ritter [:tjr] from comment #90) > If I can make one toolchain have a dependency on another toolchain You can. See how e.g. sccaches uses clang or clang uses gcc. > > It feels like it would be better to have these jobs in a separate group than TL. > > > > I'm also tempted to tell you to move them in a separate yml file, but I'm not completely sold on this. > > Sure, I can put them wherever you like. What group should be in/symbol > should I use? Something that begins with T, and is not TL, TM or TW :) TMW, maybe? > > This is just wine, right? > > Yup. I could yum install it into the image but I don't think we want wine in > all our images.... I should have been more explicit: I meant "mingw32" in the name is not relevant. > > Why are these keys suddenly necessary? > > I added them because it seemed prudent to be futureproof. If a new gcc > release is signed by one of the other keys (which gcc says they may be) our > build will suddenly break. It may not be that difficult to figure it out and > fix it, but... it seemed more correct to accept any of the keys gcc says the > release may be signed by, instead of just the one that seems most commonly > used. Only two of those keys so far have been used to sign GCC tarballs we've used. I'd rather not add keys unconditionally if we don't need them. I doubt the other will be used to sign newer GCC releases anyways. And if they are, we can deal with that then ; we don't have to deal with it now. The only thing that can break is if *existing* releases are *resigned* with a different key, and I doubt that will happen. And if it does, it's better if we notice.

I know you already spent a lot of time on this, but before you spend some more time on it, a question came to me while under the shower: why exactly are we building mingw-gcc, wine, and nsis from source? Taking a step back, the main reason we build our own gcc and clang is that our base build docker image (mainly for binary compatibility reasons) is old. But we don't have to use the same docker image. So why not use, say, a docker image for debian stretch, freeze it on snapshot.debian.org and install the packages we need? nalexander did something like this for android, and could point you to the Dockerfile he's been using.

(I really should have thought of this before you went down this road in the first place :( ... hindsight is 20/20)

(In reply to Mike Hommey [:glandium] from comment #92) > I know you already spent a lot of time on this, but before you spend some > more time on it, a question came to me while under the shower: why exactly > are we building mingw-gcc, wine, and nsis from source? Taking a step back, > the main reason we build our own gcc and clang is that our base build docker > image (mainly for binary compatibility reasons) is old. But we don't have to > use the same docker image. So why not use, say, a docker image for debian > stretch, freeze it on snapshot.debian.org and install the packages we need? > nalexander did something like this for android, and could point you to the > Dockerfile he's been using. That could work for nsis and wine I suppose (and upx, which we need also but isn't available via the yum repos we are using)... But we follow mingw-w64 master very closely. Jacek has upstreamed at least half a dozen patches for me over the course of this project. Relying on their release schedule wouldn't work. Also, I like the idea of retaining the ability to apply our own patches on top of it, regardless of upstream status. (I think you might have mentioned that recently on a dev.platform thread...)

I'm splitting the warnings (that aren't compiler errors) out into Bug 1394433 to keep this one a little more manageable. (Should have done that in the beginning.)

Comment on attachment 8884418 [details] Bug 1330608 Add the MinGW32 toolchain build to Taskcluster https://reviewboard.mozilla.org/r/155334/#review177704 Okay I addressed all of the issues except for two (with comments below). You reviewed revision 17 if you want to diff to latest rev. (I may have a couple of typo-fixes post-18 because TC is busted right now and I can' get my jobs to complete.) Also - will you review the TaskCluster build job, or should I find someone else to review that? > The implicit contract is that toolchain artifacts extract to a directory with the stem name of the archive. That is, unpacking foo.tar.xz should create a foo directory. > > You should also be able to use tar -acf foo.tar.xz instead of first create a tar and then xz it. I fixed the tar acf thing; but there is a problem with stem name issue. When we build with winpthreads for MinGW, we need libwinpthread-1.dll to be in the same directory as the executables when we run them. This affects fxc2 for sure and I believe nsis also. The winpthread dll comes out of the gcc build. (Which occurs first.) I'd either need to do a post-tooltool step on the browser build machine to copy this dll out to those two directories (and I'm not sure where I'd shim that step in or how) - or I just build them, install them, and extract them out to the same gcc directory as the mingw compiler. > why not use download_and_check from build-gcc.sh? (possibly moving it to a separate file) I can, but to check first: build-gcc.sh lives in build/unix/build-gcc/ and build-mingw32-nsis.sh lives in taskcluster/scripts/misc/. Whether I use build-gcc.sh directly or abstract the functions to a separate file, I'll be including across vastly different directory hierarchies. Is that alright? (It's not that complicated of logic to duplicate...)

New tooltool manifest for rust 1.20.0 from bug 1396883. { "algorithm": "sha512", "visibility": "public", "filename": "rustc-x86_64-unknown-linux-gnu-mingw32-cross-repack.tar.xz", "version": "rustc 1.20.0 (f3d6973f4 2017-08-27) repack", "unpack": true, "digest": "9f1af5cf4febda716f5d89e9de2482814d62c456637fbe235469225f2cfdb3401cfc1b955a7225d17600332c88b093bccbef6f702f106d117bbb23786ea3fcbd", "size": 211950648 }

(In reply to Tom Ritter [:tjr] from comment #98) > Comment on attachment 8884418 [details] > Bug 1330608 Add the MinGW32 toolchain build to Taskcluster > > https://reviewboard.mozilla.org/r/155334/#review177704 > > Okay I addressed all of the issues except for two (with comments below). You > reviewed revision 17 if you want to diff to latest rev. (I may have a couple > of typo-fixes post-18 because TC is busted right now and I can' get my jobs > to complete.) > > Also - will you review the TaskCluster build job, or should I find someone > else to review that? > > > The implicit contract is that toolchain artifacts extract to a directory with the stem name of the archive. That is, unpacking foo.tar.xz should create a foo directory. > > > > You should also be able to use tar -acf foo.tar.xz instead of first create a tar and then xz it. > > I fixed the tar acf thing; but there is a problem with stem name issue. > > When we build with winpthreads for MinGW, we need libwinpthread-1.dll to be > in the same directory as the executables when we run them. This affects fxc2 > for sure and I believe nsis also. The winpthread dll comes out of the gcc > build. (Which occurs first.) I'd either need to do a post-tooltool step on > the browser build machine to copy this dll out to those two directories (and > I'm not sure where I'd shim that step in or how) - or I just build them, > install them, and extract them out to the same gcc directory as the mingw > compiler. If fxc2 needs the winpthreads dll, copy the winpthreads dll in the fxc2 package. > > why not use download_and_check from build-gcc.sh? (possibly moving it to a separate file) > > I can, but to check first: build-gcc.sh lives in build/unix/build-gcc/ and > build-mingw32-nsis.sh lives in taskcluster/scripts/misc/. Whether I use > build-gcc.sh directly or abstract the functions to a separate file, I'll be > including across vastly different directory hierarchies. Is that alright? > (It's not that complicated of logic to duplicate...) build-gcc.sh is already included from taskcluster/script/misc/ scripts.

Comment on attachment 8884418 [details] Bug 1330608 Add the MinGW32 toolchain build to Taskcluster https://reviewboard.mozilla.org/r/155334/#review183546 ::: build/unix/build-gcc/build-gcc.sh:127 (Diff revision 19) > tar caf $root_dir/gcc.tar.xz gcc/ > popd > } > + > +build_gcc_and_mingw() { > + export install_dir=$root_dir/tools/gcc/ Having /home/worker/workspace/build in the install dir is still suspicious. ::: taskcluster/ci/toolchain/linux.yml:224 (Diff revision 19) > + docker-image: {in-tree: desktop-build} > + max-run-time: 3600 > + run: > + using: toolchain-script > + script: build-upx.sh > + tooltool-downloads: public no need for this when you don't use a tooltool manifest. ::: taskcluster/scripts/misc/build-gcc-mingw32.sh:15 (Diff revision 19) > +root_dir=$HOME_DIR > +data_dir=$HOME_DIR/src/build/unix/build-gcc > + > +. $data_dir/build-gcc.sh > + > +gcc_version=5.4.0 Now that linux builds are using gcc 6, can that be updated? ::: taskcluster/scripts/misc/build-gcc-mingw32.sh:43 (Diff revision 19) > +7e3c02ff52f8540f6a85534f54158968417fd676001651c8289c705bd0228f36 isl-0.14.tar.bz2 > +ae79f8d41d8a86456b68607e9ca398d00f8b7342d1d83bcf4428178ac45380c7 mpc-0.8.2.tar.gz > +ca498c1c7a74dd37a576f353312d1e68d490978de4395fa28f1cbd46a364e658 mpfr-3.1.5.tar.bz2 > +EOF > + > +#5f835b04b5f7dd4f4d2dc96190ec1621b8d89f2dc6f638f9f8bc1b1014ba8cad gcc-5.2.0.tar.bz2 Please remove those commented-out lines. ::: taskcluster/scripts/misc/build-mingw32-fxc2.sh:20 (Diff revision 19) > + > +cd $WORKSPACE > + > +# -------------- > + > +git clone -n https://github.com/tomrittervg/fxc2.git maybe move this under the mozilla/ group on github? ::: taskcluster/scripts/misc/build-mingw32-fxc2.sh:33 (Diff revision 19) > +cp d3dcompiler_47.dll $INSTALL_DIR/i686-w64-mingw32/bin > + > +# -------------- > + > +cd $WORKSPACE > +tar caf fxc2.tar.xz gcc the stem name problem still applies.

Comment on attachment 8899589 [details] Bug 1330608 Add the MinGW32 browser build to Taskcluster https://reviewboard.mozilla.org/r/170882/#review183550 You'll probably want some releng people to review the mozharness config. ::: browser/config/tooltool-manifests/mingw32/releng.manifest:6 (Diff revision 4) > +[ > + { > + "size": 11189216, > + "digest": "18bc52b0599b1308b667e282abb45f47597bfc98a5140cfcab8da71dacf89dd76d0dee22a04ce26fe7ad1f04e2d6596991f9e5b01fd2aaaab5542965f596b0e6", > + "algorithm": "sha512", > + "filename": "gtk3.tar.xz", I doubt you need this. ::: taskcluster/ci/build/windows.yml:514 (Diff revision 4) > + platform: windows-mingw32-32/debug > + symbol: tc(B) > + tier: 2 > + worker-type: aws-provisioner-v1/gecko-{level}-b-linux > + worker: > + max-run-time: 72000 that's a lot. ::: testing/mozharness/configs/builds/releng_base_windows_32_mingw_builds.py:114 (Diff revision 4) > + 'yasm', 'wget', > + 'mpfr', # required for system compiler > + 'xorg-x11-font*', # fonts required for PGO > + 'imake', # required for makedepend!?! > + ### <-- from releng repo > + 'gcc45_0moz3', 'gcc454_0moz1', 'gcc472_0moz1', 'gcc473_0moz1', you can remove this line. ::: testing/mozharness/configs/builds/releng_base_windows_32_mingw_builds.py:119 (Diff revision 4) > + 'gcc45_0moz3', 'gcc454_0moz1', 'gcc472_0moz1', 'gcc473_0moz1', > + 'yasm', 'ccache', > + ### > + 'valgrind', > + ######## 32 bit specific ########### > + 'glibc-static.i686', 'libstdc++-static.i686', You probably can remove all those .i686 packages. ::: testing/mozharness/configs/builds/releng_base_windows_32_mingw_builds.py:154 (Diff revision 4) > + 'freetype-devel-2.3.11-6.el6_1.8.i686', > + 'freetype-2.3.11-6.el6_1.8.x86_64', > + ######## 32 bit specific ########### > + ], > + 'src_mozconfig': 'browser/config/mozconfigs/win32/mingw32', > + 'tooltool_manifest_src': "browser/config/tooltool-manifests/mingw32/releng.manifest", No need to put the tooltool manifest here. The taskcluster configuration takes precedence.

Comment on attachment 8884418 [details] Bug 1330608 Add the MinGW32 toolchain build to Taskcluster https://reviewboard.mozilla.org/r/155334/#review183546 > Having /home/worker/workspace/build in the install dir is still suspicious. I changed the build script to use a tmp directory for the root directory. I believe it still works, so this makes me think that the matching install and extraction directories is a conincidence and not something we're relying on. > Now that linux builds are using gcc 6, can that be updated? There are some bugs for gcc 6: 1389965 and 1393814 at least, with some more likely hidden behind those. I'm intending to get to those, but I would prefer not to block this on getting them resolved.

(In reply to Mike Hommey [:glandium] from comment #102) > If fxc2 needs the winpthreads dll, copy the winpthreads dll in the fxc2 > package. Yes, that makes sense. Done. > > > why not use download_and_check from build-gcc.sh? (possibly moving it to a separate file) > > > > I can, but to check first: build-gcc.sh lives in build/unix/build-gcc/ and > > build-mingw32-nsis.sh lives in taskcluster/scripts/misc/. Whether I use > > build-gcc.sh directly or abstract the functions to a separate file, I'll be > > including across vastly different directory hierarchies. Is that alright? > > (It's not that complicated of logic to duplicate...) > > build-gcc.sh is already included from taskcluster/script/misc/ scripts. Okay. I abstracted download_and_check into a separate file. Something else arose. This builds the browser successfully, but it does not run. I suspected it may have to do with my patch in Bug 1364560, but Georg from Tor pointed me at a msvcr problem. The tor ticket is https://bugs.torproject.org/9084 and it talks about Firefox not starting on WinXP because it pulls in msvcrt.dll; and msvcrt.dll on WinXP does not have functions used by Firefox (such as the _s secure functions). While hunting this down they determined that gcc is responsible for pulling in msvcrt.dll (including on newer versions of Windows, such as Windows 10). They correct this to link against msvcr100.dll which they ship with the browser. They fix this in four steps: 1) In https://gitweb.torproject.org/builders/tor-browser-bundle.git/tree/gitian/descriptors/windows/gitian-utils.yml#n92 (lines 92-99) they pause the gcc build in the middle and adjust gcc to link against msvcr100 before building the gcc libraries like libssp 2) In https://gitweb.torproject.org/builders/tor-browser-bundle.git/tree/gitian/descriptors/windows/gitian-firefox.yml#n49 (lines 49-72) they do the same thing - change the gcc spec file to use msvcr100, and add a build flag to the necessary environment variables to make it use that spec file 3) In https://gitweb.torproject.org/builders/tor-browser-bundle.git/tree/gitian/descriptors/windows/gitian-firefox.yml#n67 (line 67) they do an ugly hack on the nsprpub code to change a function name. 4) They ship msvcr100.dll and use it instead of any system dll that would be loaded I definetly don't intend to do #3, I'll figure out how to get that fixed in a different bug. But for the build job I am going to see if 1, 2, and 4 will produce a running build. Tor doesn't care for this hack. They would be very happy for this to be solved in a new way. But I am unsure what that would be. If anyone has suggestions we'd be happy to investigate them!

Comment on attachment 8884418 [details] Bug 1330608 Add the MinGW32 toolchain build to Taskcluster https://reviewboard.mozilla.org/r/155334/#review184828 ::: build/unix/build-gcc/7F74F97C103468EE5D750B583AB00996FC26A641.key:1 (Diff revision 20) > +-----BEGIN PGP PUBLIC KEY BLOCK----- Can you run this through https://github.com/glandium/pgpstrip ? ::: build/unix/build-gcc/build-gcc.sh:105 (Diff revision 20) > tar caf $root_dir/gcc.tar.xz gcc/ > popd > } > + > +build_gcc_and_mingw() { > + export install_dir=$root_dir/tools/gcc/ I should probably have been more explicit about what I meant. I didn't mean that specifically /home/worker/workspace/build was bad and that using a temporary directory would fix my concerns, but that $root_dir being in there at all is weird. OTOH, it works, and that's not used for tier-1 builds, so meh, let's land this already. Please file a followup, though.