From an email thread about build times and perf alerts

erahm's words
In the case of build times the numbers we're reporting seem incorrect (or at least unintuitive, the top-level number doesn't correspond to the sum of the "subtests"). For example this alert (https://treeherder.mozilla.org/perf.html#/alerts?id=24904&hideDwnToInv=0) shows a delta of ~10 minutes, but summing the reported values shows a delta of ~3 minutes. It might make sense to only track the subtests and uses a geomean to trigger alerts. We should file an additional bug to look into how we're reporting build times and determine if we're a) reporting correct values and b) reporting useful values.

That alert is from a change that inadvertently turned off sccache for those builds, so the regression should be in the compile tier. But when I click through to subtests from that alert the entry for compile corresponds to a much more recent, smaller alert because an alert for compile wasn't registered for this regression for some reason (although it's quite obvious in the graph).

Other than the post-build automation steps running in parallel making timings confusing I'm not aware of numbers that are inaccurate apart from noise or discoverability in perfherder.

That alert is weird. It says previous value: 2,081.11, new value: 2,646.78. But if you follow through to the subtests, then click back on "Back to all tests and platforms" to get to the corresponding general perfherder view for the same perf diff, and look for "windows2012-32" under "build times opt taskcluster-c5.4xlarge", you see base: 2,255.37, new: 2,445.16, which is a ~3 minutes difference that matches what's in the subtests view.

Whoops, this isn't the sccache problem at all. I was looking at the wrong alert.

I think what we're seeing is that the alert view takes into account multiple changes (12, according to https://wiki.mozilla.org/TestEngineering/Performance/Sheriffing). But the "perf diff" view refers to a specific range, which seems to default to the change in question and the one immediately prior.