sccache alerts are triggered too often due to low reggression threshold. This issue was raised in bug 1357718. Use 5% as the new threshold for these build_metrics alerts.

:ted, would 5% make sense here? Also any other build metrics that we should increase from 2% to something higher?

For requests_not_cacheable we should have the alert threshold be very high, like 100%. The metric being reported here is the number of compiler calls that sccache isn't able to cache. We run something like 3000 compiles on a normal build, and all of our builds have a small set of existing non-cacheable compiles, which is OK. The purpose of reporting this number is purely to make sure we don't accidentally start using some compiler options that make sccache unable to cache any of our compiles. I don't know about other build metrics offhand. If you have specific examples we can certainly look at them!

100% might be odd, how about 50% for this metric

That's probably fine. Admittedly this metric is not very similar to the Talos metrics that Perfherder was designed around.

rwood, need your review for perfherder specifics. ted, if you see anything odd in my changes, speak up, maybe cache_write_errors needs to be adjusted as well, etc.

Comment on attachment 8861096 [details] [diff] [review] increase alert threshold LGTM

Comment on attachment 8861096 [details] [diff] [review] increase alert threshold Review of attachment 8861096 [details] [diff] [review]: ----------------------------------------------------------------- ::: testing/mozharness/mozharness/mozilla/building/buildbase.py @@ +1910,5 @@ > > + yield { > + 'name': 'sccache cache_write_errors', > + 'value': stats['stats']['cache_write_errors'], > + 'extraOptions': self.perfherder_resource_options(), We could probably use a similar or even higher threshold here. This is almost always zero, but occasionally jumps up (presumably when we hit intermittent network issues connecting to S3): https://mzl.la/2owrAdl It's really only an issue if we start consistently getting a large number here.

Pushed by jmaher@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/23cfef4874a0 Increase regression threshold for sccache alerts. r=rwood,ted