This is a tracking bug for implementing the three stages of PGO builds (building the instrumented build, generating the profile, and generating a build based on that profile) as separate tasks. This is a prerequisite for PGO'd builds for Android and macOS as well as verifying the reproducibility of builds we ship on other platforms. mshal and myself discussed some details of what this means for the build system, I'll be spinning individual pieces of that into dependent bugs.

This seems as good a place as any to comment on the potential performance overhead of compression. I anticipate that the artifact transferred from one task to another will be large. The overhead of zlib-based compression could be significant. I would strongly advise the use of zstandard compressed artifacts for exchange. Its single threaded compression/decompression is faster than zlib. And we can use multiple threads on the compression side to speed things up mostly linearly (slightly sacrificing compression ratio in the process). python-zstandard should be available in the Docker images and `import zstandard` should "just work." If the `fetch-content` script is used to obtain the artifact, it automatically recognizes the `.tar.zst` extension.

As an aside, catlee mentioned in the firefoxci meeting yesterday that he would be interested in the buildtimes after this change compared to the current ones. (I realize that we don't do PGO on Android or Mac currently) but my understanding that Linux will also change with this approach because the tasks will be split up.

The end-to-end times will absolutely be longer because we'll have setup overhead for 3 tasks instead of 1, and we'll be uploading and downloading artifacts in between them. Obviously gaining the ability to do PGO for platforms where we cross-compile is different, but I suspect this will basically always be net negative for build times on Linux. For Windows this (along with a bunch of other work) will unlock the ability for us to cross-compile Windows builds from Linux which has the potential to be a huge build time win.

Note that this actually gives us the occasion to create /some/ builds faster. As in, we could generate nightlies as just the last of the 3 tasks, picking the PGO data artifact from the normal PGO builds.

Is it possible that PGO data can be used from a build that came from different sources? Would this allow us to use the same PGO data for libraries that don't change often? In particular, our partial updates often include patches to things like libGLESv2.dll, or nss3.dll. I assume the binary differences to those libraries are caused by PGO, and not by source changes.

> I assume the binary differences to those libraries are caused by PGO, and not by source changes. IIRC, DLLs have timestamps embedded, and that causes differences in binaries that haven't been built at the exact same time. You should exclude that first.

> IIRC, DLLs have timestamps embedded, and that causes differences in binaries > that haven't been built at the exact same time. You should exclude that > first. I imagine those would be pretty small deltas though? If we see nontrivial amounts of change, PGO sounds like a better suspect.