Comment on attachment 8940941 [details] Bug 1427340 - Check differences between builds without changes. Wrong bug.

Some necessary explanation: - Only the third patch is to review and land. - The first patch duplicates android, linux and mac builds jobs, so there are two of each, and runs diffoscope on each pairs. - With all the work that led here, we now have very minimalistic diffs: https://treeherder.mozilla.org/#/jobs?repo=try&revision=fc8e4f28c63995a88d8bffc293c3999116e0f6da&filter-searchStr=diff The only exception is mac, where there are large diffs, due to the symbol tables we keep for profiling. There's an additional diff job on that push for mac, that I retriggered manually with a hack to strip the binaries before running diffoscope. And that led to ... no diff at all (which actually leads to the artifacts not existing, which can be surprising). - The second patch just reverts the first one. - The last patch does builds with all the new toolchains built on Debian, and compares the results with the results from that previous try push. https://treeherder.mozilla.org/#/jobs?repo=try&revision=c929e859fff3060b33e88567b4d6fef670871492 (there two there are two mac diffs, one with large changes due to symbol tables, and one hacked to strip first) - Looking around all those diffs show that everything is due to the differences in MOZ_BUILD_DATE and the try changeset, or to the intrinsic differences that can be observed in the first try. - A few random related observations: - MOZ_BUILD_DATE is statically stored in a whole lot of java classes on android. - There are non-obvious differences in mac binaries, because diffoscope doesn't know much about mach-o at the moment, and just ends up doing hexdump diffs. The ones that are clearly not coming from MOZ_BUILD_DATE are all preceded with 1b00·​0000·​1800·​0000 and are dyld commands for LC_UUID, which contain uuids that are generated from the content of the binary (LC_UUID is 0x1b, and 0x18 is the length of that header + the uuid)

Oh, and I disabled sccache on all those tries just in case it was hiding differences, and... it's not :)

With this new version of the last patch, I actually tricked a new try push to have the same MOZ_BUILD_DATE and try changeset info as the previous try that it's being compared to, leading to diffs as small as the ones from the first no-op try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=2fe3ebfa2e62148252cee8ad2f8768d6be07fb3c&filter-searchStr=diff

(In reply to Mike Hommey [:glandium] from comment #4) > Created attachment 8941351 [details] > Bug 1427340 - Build toolchains on a Debian-based docker image. > > ... except libdmg-hfsplus. RedHat decided to patch libbz2 to have a > different soname, so a binary built on Debian can't run on > RedHat/CentOS. Ironically, a binary built on RedHat/CentOS can run > on Debian. Is statically linking libbz2 into these tools too hard? (In reply to Mike Hommey [:glandium] from comment #7) > The only exception is mac, where there are large diffs, due to the symbol tables we keep for profiling. There's an additional diff job on that push for mac, that I retriggered manually with a hack to strip the binaries before running diffoscope. And that led to ... no diff at all (which actually leads to the artifacts not existing, which can be surprising). Related: https://github.com/rust-lang/rust/issues/47086 . I initially wondered why the Tor folks hadn't run into this, but I assume they've always been diffing stripped binaries. (In reply to Mike Hommey [:glandium] from comment #8) > Oh, and I disabled sccache on all those tries just in case it was hiding > differences, and... it's not :) \o/

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #11) > (In reply to Mike Hommey [:glandium] from comment #4) > > Created attachment 8941351 [details] > > Bug 1427340 - Build toolchains on a Debian-based docker image. > > > > ... except libdmg-hfsplus. RedHat decided to patch libbz2 to have a > > different soname, so a binary built on Debian can't run on > > RedHat/CentOS. Ironically, a binary built on RedHat/CentOS can run > > on Debian. > > Is statically linking libbz2 into these tools too hard? It would require changes to the CMakeLists.txt file in dmg, again. Because misusing cmake is apparently easy, and that's what dmg does. At this point, I went "meh". > (In reply to Mike Hommey [:glandium] from comment #7) > > The only exception is mac, where there are large diffs, due to the symbol tables we keep for profiling. There's an additional diff job on that push for mac, that I retriggered manually with a hack to strip the binaries before running diffoscope. And that led to ... no diff at all (which actually leads to the artifacts not existing, which can be surprising). > > Related: https://github.com/rust-lang/rust/issues/47086 . I initially > wondered why the Tor folks hadn't run into this, but I assume they've always > been diffing stripped binaries. I think what I've seen is not entirely due to that.

Comment on attachment 8941351 [details] Bug 1427340 - Build toolchains on a Debian-based docker image. https://reviewboard.mozilla.org/r/211658/#review218016 Shiny.

Can't have nice things... nsis now fails because sourceforce renewed their certificate last week and somehow the CA is not in ca-certificates in wheezy. And somehow I missed the fact that wine fails to build on that image. OTOH, all things considered, those toolchains are for the mingw32 builds, which we might as well move to a more modern image, based on stretch, like android-build, so let's keep them on desktop-build for now.

(even though, I'm told a ca-certificates update is due soon for wheezy)

Isn't ca-certificates entirely generated from our NSS certdata.txt? Presumably if you *really* wanted to fix it you could just fetch certdata.txt from hg.mo and generate ca-certificates fresh in the image builder task. :)

Pushed by mh@glandium.org: https://hg.mozilla.org/integration/mozilla-inbound/rev/538460b6e1fd Build toolchains on a Debian-based docker image. r=gps