We have detected a build metrics regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=45c8ab7db844100c8d887548d6bb04a960a8a75a&tochange=085342ba416f464742b068edd32f20fae8e18e87

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

32% build times linux64-shippable opt nightly taskcluster-c5d.4xlarge 2,793.25 -> 3,692.66
32% build times linux64-shippable opt nightly taskcluster-m5.4xlarge 2,931.93 -> 3,858.12
29% build times linux64-shippable opt nightly taskcluster-c5.4xlarge 2,834.58 -> 3,661.59
29% build times linux64-shippable opt nightly taskcluster-c4.4xlarge 3,275.84 -> 4,214.16
27% build times linux64-shippable opt nightly taskcluster-m4.4xlarge 3,451.40 -> 4,373.75
15% build times android-4-0-armv7-api16 pgo taskcluster-c5.4xlarge 1,847.63 -> 2,119.92
15% build times android-5-0-aarch64 pgo taskcluster-m5.4xlarge 1,872.54 -> 2,144.78
14% build times android-4-0-armv7-api16 pgo taskcluster-c4.4xlarge 2,140.93 -> 2,447.63
14% build times android-4-0-armv7-api16 pgo taskcluster-m5.4xlarge 1,922.05 -> 2,188.87
13% build times android-4-0-armv7-api16 pgo taskcluster-m4.4xlarge 2,358.81 -> 2,667.79
11% build times android-5-0-aarch64 pgo taskcluster-c5.4xlarge 1,819.33 -> 2,026.70
11% build times android-4-0-armv7-api16 pgo taskcluster-c5d.4xlarge 1,772.97 -> 1,969.16
10% build times linux32-shippable opt nightly taskcluster-c5d.4xlarge 5,345.07 -> 5,904.27
10% build times android-5-0-aarch64 pgo taskcluster-c4.4xlarge 2,130.69 -> 2,352.00
10% build times android-5-0-aarch64 pgo taskcluster-c5d.4xlarge 1,737.69 -> 1,916.05

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=21114

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Automated_Performance_Testing_and_Sheriffing/Build_Metrics

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

:froydnj I'm pretty sure about the Linux build time increases. However, I've yet to confirm the Android increases, given the noisy graphs & the dynamic instances that do those builds (having hard time collecting perf data from a specific instance).

Also note that this caused binary increases on Windows:

== Change summary for alert #21085 (as of Sat, 25 May 2019 00:48:46 GMT) ==

Regressions:

1% installer size windows2012-64-shippable opt nightly 72,089,536.71 -> 72,846,615.83
1% installer size windows2012-32-shippable opt nightly 69,398,810.29 -> 69,856,027.92

Improvements:

5% build times windows2012-32-shippable opt nightly taskcluster-c4.4xlarge 5,956.90 -> 5,648.71
4% build times windows2012-64-shippable opt nightly taskcluster-c4.4xlarge 6,398.66 -> 6,114.51

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=21085

Those Linux increases are pretty bad (and peculiarly, the Windows changes go in the opposite direction); I suppose we weren't really getting good performance information from the Linux and/or Android builds before (?).

These patches are required for other performance work, and there's no clear path to make the compiler faster.

(In reply to Nathan Froyd [:froydnj] from comment #3)

Those Linux increases are pretty bad (and peculiarly, the Windows changes go
in the opposite direction);

I think that result is due to 3-tier versus 1-tier PGO. (See also bug 1516114 comment 28.) IIUC, on Linux, Talos is now only tracking the final optimization build? Maybe the patch made that phase longer but the instrumentation phase is cheaper now?