This bug covers the taskgraph machinery we'll need to implement PGO builds as three separate tasks. The first is a regular build with (at least) MOZ_PROFILE_GENERATE set. Its output is the instrumented build. The second is a test-like task that will run the instrumented build via profileserver.py and upload merged.profdata (the profile) as well as jarlog/en-US.log (I'm not entirely sure this is relevant but it seems to be a part of the current flow). The third is a build that consumes the profile (sets MOZ_PROFILE_USE). These things should be relatively straightforward to invoke individually but we weren't sure of the details of how the tasks need to be set up. Mike volunteered to find appropriate guidance and look into this.

(In reply to Chris Manchester (:chmanchester) from comment #0) > I'm not entirely sure this is relevant but it seems to be a part of the current flow It is, but I though the file name was different. It's used to reorder the omni.ja files according to the order its contents are used.

Support for running profileserver.py multiple times was originally added in bug 663251 and subsequently removed in bug 672563. Since then we've been copying around the argument to profileserver.py from mozconfigs in buildbot-configs, to mozconfigs in mozilla-central, to client.mk, and ultimately to Makefile.in. It no longer does anything and can be removed. MozReview-Commit-ID: DsZkInTOyU Bug 1507334 - Add mozconfigs for 3-stage PGO; r?firefox-build-system-reviewers These two mozconfigs are used for the 1st and 3rd stages of PGO. 'profile-generate' is used for the 1st stage to generate a firefox binary with instrumentation enabled. Since it is only used for getting profile data, we don't run tests against it or need crashreporter symbol.s The 'profile-use' mozconfig is for the 3rd and final stage were we produce the fully optimized build. The profiling information and jarlog are input artifacts produced by the 2nd stage (running the profileserver). MozReview-Commit-ID: JGwSM1OqZFH Bug 1507334 - Taskcluster configurations for 3-tier PGO; r?firefox-build-system-reviewers Two new kinds are introduced - 'instrumented-build' and 'generate-profile'. The instrumented-build kind is almost identical to the build kind, except it can be used earlier in the task graph. The 3-tier PGO process becomes: instrumented-build -> generate-profile -> build The final build stage is identical to any other build, except it has the 'use-pgo' flag set to True in its task definition. This flag causes the transforms to add the instrumented-build and generate-profile tasks to the taskgraph. MozReview-Commit-ID: 9M5YXjDo4sc Bug 1507334 - Disable 3-tier Linux PGO until bug 1516114 is fixed; r?firefox-build-system-reviewers With 3-tier PGO enabled, we seem to hit bug 1516114 more frequently than we do with the standard all-in-one PGO. It probably makes sense for us to wait until that bug is fixed before fully switching it over, otherwise we will have to mess with retriggering parts of the taskgraph. The bulk of the 3-tier PGO work has been to support macOS and Android, so we can revisit Linux after those are completed. MozReview-Commit-ID: BEFZA32UDhn

Support for running profileserver.py multiple times was originally added in bug 663251 and subsequently removed in bug 672563. Since then we've been copying around the argument to profileserver.py from mozconfigs in buildbot-configs, to mozconfigs in mozilla-central, to client.mk, and ultimately to Makefile.in. It no longer does anything and can be removed.

These two mozconfigs are used for the 1st and 3rd stages of PGO. 'profile-generate' is used for the 1st stage to generate a firefox binary with instrumentation enabled. Since it is only used for getting profile data, we don't run tests against it or need crashreporter symbol.s The 'profile-use' mozconfig is for the 3rd and final stage were we produce the fully optimized build. The profiling information and jarlog are input artifacts produced by the 2nd stage (running the profileserver). Depends on D15748

Two new kinds are introduced - 'instrumented-build' and 'generate-profile'. The instrumented-build kind is almost identical to the build kind, except it can be used earlier in the task graph. The 3-tier PGO process becomes: instrumented-build -> generate-profile -> build The final build stage is identical to any other build, except it has the 'use-pgo' flag set to True in its task definition. This flag causes the transforms to add the instrumented-build and generate-profile tasks to the taskgraph. Depends on D15749

With 3-tier PGO enabled, we seem to hit bug 1516114 more frequently than we do with the standard all-in-one PGO. It probably makes sense for us to wait until that bug is fixed before fully switching it over, otherwise we will have to mess with retriggering parts of the taskgraph. The bulk of the 3-tier PGO work has been to support macOS and Android, so we can revisit Linux after those are completed. Depends on D15750

Pushed by mshal@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/1f487879b4d7 Remove '10' argument to profileserver.py; r=firefox-build-system-reviewers,chmanchester#firefox-build-system-reviewers https://hg.mozilla.org/integration/autoland/rev/6f12b99d521c Add mozconfigs for 3-stage PGO; r=firefox-build-system-reviewers,chmanchester#firefox-build-system-reviewers https://hg.mozilla.org/integration/autoland/rev/35396e93307e Taskcluster configurations for 3-tier PGO; r=tomprince,firefox-build-system-reviewers,chmanchester#firefox-build-system-reviewers https://hg.mozilla.org/integration/autoland/rev/6692901d43cf Disable 3-tier Linux PGO until bug 1516114 is fixed; r=firefox-build-system-reviewers,chmanchester#firefox-build-system-reviewers