Raptor has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=a5440adc58d0f86c7aa67c302168a6dfe87e7056

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

62% raptor-tp6-facebook-firefox loadtime linux64-shippable opt 426.81 -> 691.29
13% raptor-tp6-facebook-firefox linux64-shippable opt 303.73 -> 344.20

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=24150

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a Treeherder page showing the Raptor jobs in a pushlog format.

To learn more about the regressing test(s) or reproducing them, please see: https://wiki.mozilla.org/TestEngineering/Performance/Raptor

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/TestEngineering/Performance/Talos/RegressionBugsHandling

This is surprising. My patch adds an addref of a browsing context, a null check, and a null check of a bool field to 3 different media notification methods. I can't imagine that loading Facebook is so perf sensitive that this would cause a 62% regression.

I'm not an expert on reading these perfherder graphs, but this doesn't look like a valid regression. It looks like the value of the benchmark returned to its original value later that day. I think this test is just bimodal. If you look at the last 14 days, the loadtime value periodically pops up for a bit, then goes down again.

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/TestEngineering/Performance/Talos/RegressionBugsHandling

The change is to fix a crash (bug 1599026), so backing it out without replacement doesn't sound like a good idea. It would make the Firefox hit the crash again.

Andrew, would you able to run the raptor-tp6-facebook-firefox tests locally to compare the result with or without D54635, to see if the regression is true? If it's true, is there another way to fix bug 1599026?

The crash is Fission-only so it isn't a huge deal to reintroduce it. I don't want to spend any time investigating what looks like a bimodality issue in the test, so feel free to back it out. I can reland it in a few days one we confirm that Facebook loading time hasn't been cut in half or whatever by the backout.

Can this get closed as invalid? It looks like these benchmarks just spike up a bit occasionally. Looking at the results over the last 30 days, the values have returned to their previous levels.