In bug 1713360 we investigated a potential regression in checkerboarding with fission.

We learned that every time a checkerboard happens we submit an event with a magnitude.

This means that if a change causes a large number of new checkerboard events to happen with a small magnitude that change would cause the mean magnitude of checkerboard event to decrease. This would look like an improvement on telemetry.mozilla.org and be classified as such.

So in order to properly interpret checkerboarding severity we at minimum need to look at the number of events in addition to the (mean) magnitude. Or ideally, look at before/after histograms with an equal x and y axis scales.

It would be good to improve this so that one just need to look at one number to know if we improve or regressed. This would also allowed the automatic telemetry regression/improvement detection to work better.

kats pointed out that there is a checkerboard potential event, which fires any time that checkerboarding could have happened. Using these as a denominator, and actual checkerboard events as the numerator in some way (being a little bit vague here because I haven't thought it through in detail) could be a way to solve this.

(See also bug 1693636.)

I came across some previous analysis in bug 1504863 comment 2 that may possibly be useful.