Note that with 3-tier PGO, Linux builds should now be reproducible, provided one does a --enable-profile-use build using the same profile data as the Firefox build (the profile data is now available as an artifact), and uses the same build environment. This also assumes clang does reproducible decisions based on the profile data.

shouldn't this be easy, at least for a linux build? why not make "the operating system config and packages used on official infrastructure" publicly available?

They are publicly available. Not necessarily in a very convenient or easily discoverable way, but they are. This is why this bug is not closed while everything is theoretically in place to allow for it.

(In reply to Mike Hommey [:glandium] from comment #31)

They are publicly available. Not necessarily in a very convenient or easily discoverable way, but they are. This is why this bug is not closed while everything is theoretically in place to allow for it.

Excellent. Give me the link and the hash that will match a release Firefox build. I'll try it out.

This is the task that built the linux64 68.0 release (http://ftp.mozilla.org/pub/firefox/releases/68.0/linux-x86_64/en-US/firefox-68.0.tar.bz2):
https://tools.taskcluster.net/groups/HtPXIhzvRNa6DfzEOMSb2A/tasks/ed97SEKlQzm2wWedt11XIQ/details

You'll find there the links for the docker image and toolchains used for that build, as well as the profile data, and the script run in the docker image, along with all the environment variables. Differences you'd "obviously" get:

• signature files would either be missing (*.sig) or different (*.chk)
• the precomplete file would be different because of the .sig files.
• some API keys will be missing (this affects one file contained in omni.ja)

You can find, recursively, how each of the toolchains and docker images have been built.
Running the full script outside of taskcluster may fail. You may need to cherry-pick commands from the build log.

(In reply to Mike Hommey [:glandium] from comment #33)

This is the task that built the linux64 68.0 release (http://ftp.mozilla.org/pub/firefox/releases/68.0/linux-x86_64/en-US/firefox-68.0.tar.bz2):
https://tools.taskcluster.net/groups/HtPXIhzvRNa6DfzEOMSb2A/tasks/ed97SEKlQzm2wWedt11XIQ/details

You'll find there the links for the docker image and toolchains used for that build, as well as the profile data, and the script run in the docker image, along with all the environment variables. Differences you'd "obviously" get:

• signature files would either be missing (.sig) or different (.chk)
• the precomplete file would be different because of the .sig files.
• some API keys will be missing (this affects one file contained in omni.ja)

You can find, recursively, how each of the toolchains and docker images have been built.
Running the full script outside of taskcluster may fail. You may need to cherry-pick commands from the build log.

these conditions seem pretty odd. why isn't it easier? why can't I just build this from a shared cache almost instantly?

Which specific conditions?

(In reply to Mike Hommey [:glandium] from comment #35)

Which specific conditions?

Is this the default build for a release copy of Firefox on any Linux distro? Furthermore, what needs to happen to reproduce this build in similar ways on Windows and macOS?

Even if I accept the argument about some narrow Linux build... this bug is still open after six years. Why is that?

(In reply to Robert Sayre from comment #36)

(In reply to Mike Hommey [:glandium] from comment #35)

Which specific conditions?

Is this the default build for a release copy of Firefox on any Linux distro?

Distros build Firefox the way they want to build it. The simple fact of building Firefox from a source tarball rather than mercurial will introduce small differences. Building from a different directory with a different version of the compiler (or a different compiler altogether) will introduce large differences. So no, there is not one release copy of Firefox on any Linux distro. There's the one that Mozilla ships, and then there are the ones each distro builds. Welcome to the Linux world.

Furthermore, what needs to happen to reproduce this build in similar ways on Windows and macOS?

The same kind of thing, except they require SDKs that are not redistributable, so you'd have to procure them on your own. You can find the tasks used for Windows and macOS builds on https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&selectedJob=255019294&revision=353628fec415324ca6aa333ab6c47d447ecc128e&searchStr=shippable%2Cbuild

Even if I accept the argument about some narrow Linux build... this bug is still open after six years. Why is that?

Because it's not as simple as you seem to think it is, and because resources are scarce.

(In reply to Mike Hommey [:glandium] from comment #37)

(In reply to Robert Sayre from comment #36)

(In reply to Mike Hommey [:glandium] from comment #35)

Which specific conditions?

Is this the default build for a release copy of Firefox on any Linux distro?

Distros build Firefox the way they want to build it.

So, no.

Furthermore, what needs to happen to reproduce this build in similar ways on Windows and macOS?

The same kind of thing, except they require SDKs that are not redistributable,

So, no.

Even if I accept the argument about some narrow Linux build... this bug is still open after six years. Why is that?

Because it's not as simple as you seem to think it is, and because resources are scarce.

So, no.

(In reply to Mike Hommey [:glandium] from comment #37)

The same kind of thing, except they require SDKs that are not redistributable, so you'd have to procure them on your own. You can find the tasks used for Windows and macOS builds on https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&selectedJob=255019294&revision=353628fec415324ca6aa333ab6c47d447ecc128e&searchStr=shippable%2Cbuild

Although, the Windows builds have not switched to 3-tier PGO yet, so they are, in fact, not reproducible because their profile data is not published. This will be different for 69. MacOS builds don't do PGO at all yet, so they should be reproducible.

(In reply to Mike Hommey [:glandium] from comment #39)

(In reply to Mike Hommey [:glandium] from comment #37)

The same kind of thing, except they require SDKs that are not redistributable, so you'd have to procure them on your own. You can find the tasks used for Windows and macOS builds on https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&selectedJob=255019294&revision=353628fec415324ca6aa333ab6c47d447ecc128e&searchStr=shippable%2Cbuild

Although, the Windows builds have not switched to 3-tier PGO yet, so they are, in fact, not reproducible because their profile data is not published. This will be different for 69. MacOS builds don't do PGO at all yet, so they should be reproducible.

Maybe the easiest way to make some progress on this bug: let's set up other builds that mirror the Firefox release. I'm happy to pay for the Firefox Linux x64 (Ubuntu) release reproducibility check.

If you want to reproduce the Firefox Linux x64 release from Ubuntu, you need to talk to Ubuntu.

(In reply to Robert Sayre from comment #40)

(In reply to Mike Hommey [:glandium] from comment #39)

(In reply to Mike Hommey [:glandium] from comment #37)

The same kind of thing, except they require SDKs that are not redistributable, so you'd have to procure them on your own. You can find the tasks used for Windows and macOS builds on https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&selectedJob=255019294&revision=353628fec415324ca6aa333ab6c47d447ecc128e&searchStr=shippable%2Cbuild

Although, the Windows builds have not switched to 3-tier PGO yet, so they are, in fact, not reproducible because their profile data is not published. This will be different for 69. MacOS builds don't do PGO at all yet, so they should be reproducible.

Maybe the easiest way to make some progress on this bug: let's set up other builds that mirror the Firefox release. I'm happy to pay for the Firefox Linux x64 (Ubuntu) release reproducibility check.

(In reply to Mike Hommey [:glandium] from comment #41)

If you want to reproduce the Firefox Linux x64 release from Ubuntu, you need to talk to Ubuntu.

Just to check understanding: you're saying that Windows and macOS contain binaries that might change, and you can't reproducibly build any "Firefox" build that might appear in a common Linux distro.

No, I'm saying the Windows builds for 68 are not reproducible because the profile data is not published, but it will be for 69, and that Windows and macOS builds should be reproducible, but if you want to reproduce them, you must download the necessary SDKs (MacOS SDK/Windows 10 SDK) on your own, and not from Mozilla servers, because they can't be redistributed by Mozilla due to their license.

As for common Linux distros, Mozilla doesn't control how they build Firefox, so reproducibility of those builds depends on them making them reproducible. The Debian builds used to be reproducible but apparently aren't anymore... which could be caused by a reproducibility issue in the rust compiler.

(In reply to Mike Hommey [:glandium] from comment #43)

No, I'm saying the Windows builds for 68 are not reproducible because the profile data is not published, but it will be for 69, and that Windows and macOS builds should be reproducible, but if you want to reproduce them, you must download the necessary SDKs (MacOS SDK/Windows 10 SDK) on your own, and not from Mozilla servers, because they can't be redistributed by Mozilla due to their license.

As for common Linux distros, Mozilla doesn't control how they build Firefox, so reproducibility of those builds depends on them making them reproducible. The Debian builds used to be reproducible but apparently aren't anymore... which could be caused by a reproducibility issue in the rust compiler.

So, just to check understanding: the Debian builds used be reproducible, but no longer are, and that might be due to an issue in the Rust compiler?

Looking a little bit closer, the status on Debian is that they changed what reproducible builds test, and they now test building from different build paths, so that's why the Debian builds are marked as non-reproducible on https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/firefox.html. They are otherwise reproducible.

Turned out to be less problematic than I thought it would be for Linux: https://glandium.org/blog/?p=3923
I also confirmed that the Mac toolchain has some determinism issue that manifests itself in mach-o headers. I didn't check Windows, but now that I think of it, ISTR last time I tried there were linker-induced non-determinism. But that may have been when were still using MSVC.

(In reply to Mike Hommey [:glandium] from comment #46)

Turned out to be less problematic than I thought it would be for Linux: https://glandium.org/blog/?p=3923
I also confirmed that the Mac toolchain has some determinism issue that manifests itself in mach-o headers. I didn't check Windows, but now that I think of it, ISTR last time I tried there were linker-induced non-determinism. But that may have been when were still using MSVC.

I'm not sure what you're saying.

"the Mac toolchain has some determinism issue that manifests itself in mach-o headers."

what do you mean, exactly?

and you "didn't check Windows, but now that I think of it, ISTR last time I tried there were linker-induced non-determinism"

are you working on this bug?

(In reply to Robert Sayre from comment #47)

"the Mac toolchain has some determinism issue that manifests itself in mach-o headers."

what do you mean, exactly?

That building Firefox for Mac with the same environment, same tools, etc. generates executables and libraries that differ in their headers. Although there seems to be other differences in the XUL library, even with LTO and signing disabled. See https://taskcluster-artifacts.net/GQvevOGtS0yOcB1OGb7H_w/0/public/diff.html

and you "didn't check Windows, but now that I think of it, ISTR last time I tried there were linker-induced non-determinism"

are you working on this bug?

Not actively. But also note that this bug is about Linux.

(In reply to Mike Hommey [:glandium] from comment #48)

are you working on this bug?

Not actively. But also note that this bug is about Linux.

so, although you are arguing, you're not working on this bug, this bug is only about Linux, and this bug is assigned to no one.

this all sounds mozilla af, but I want to check the facts

Robert, are you volunteering to help? Contributions are welcome.

(btw, Mike is explaining what is the state of this bug - not arguing)

(In reply to Sylvestre Ledru [:sylvestre] from comment #50)

Robert, are you volunteering to help? Contributions are welcome.

(btw, Mike is explaining what is the state of this bug - not arguing)

Who should this bug be assigned to? I don't think I'm the right choice.

Well, why is this issue unassigned? I mean, I suspect it's assigned to no one because it's been deemed impossible. Not in a technical sense, that's actually easy. Anybody can build something with Bazel and Nix and get good reproducibility.

So, like, what's the real reason?

First, please start contributing for real to this bug instead of questioning or suggesting that we have an hidden agenda.
Otherwise, i will have to moderate this thread.

The real reason is what mike said. We cannot do everything. Lately, we decided to focus on performances, which involved pgo (with many other things) which is making the reproducible effort much harder.
In parallel, Firefox is one of the most complex piece of software, we depends on many tools (compilers, linker, source to source tools, libraries, etc) which makes this work much harder than for most software or libraries.

As you can on Mike blog post, it is possible but not trivial. Otherwise, we would have done it already.

(In reply to Sylvestre Ledru [:sylvestre] from comment #53)

First, please start contributing for real to this bug instead of questioning or suggesting that we have an hidden agenda.
Otherwise, i will have to moderate this thread.

Go for it. No one commented on this thread for years. One action that would make sense: assign the bug to a person.

I'll add this, and then I'm done "arguing". While this bug doesn't show progress and is not assigned, countless bugs over the years have contributed to being in a state where it is, in fact, now possible, albeit not entirely trivial, to do it. It was not possible at all a few days ago (if you don't count betas and nightlies for which it's only been possible for a few weeks). Like, all the dependencies of this bug that are closed, and also a bunch of others.

(In reply to Mike Hommey [:glandium] from comment #55)

I'll add this, and then I'm done "arguing". While this bug doesn't show progress and is not assigned, countless bugs over the years have contributed to being in a state where it is, in fact, now possible, albeit not entirely trivial, to do it. It was not possible at all a few days ago (if you don't count betas and nightlies for which it's only been possible for a few weeks). Like, all the dependencies of this bug that are closed, and also a bunch of others.

I'm confused. are you saying this bug should be assigned to you? Maybe Sylvestre?

(In reply to Mike Hommey [:glandium] from comment #46)

Turned out to be less problematic than I thought it would be for Linux: https://glandium.org/blog/?p=3923

Whoa! That's an amazing milestone reached on this topic!
Thank you for the blogpost with the steps!
Thank you very much for all the work on this to everyone involved over the years!