Splitting this off from bug 1537694 - :igoldan, can you update the perfherder stuff to point to this bug?

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from bug 1537694 comment #7)

I also noticed a Talos regression:

== Change summary for alert #20026 (as of Thu, 21 Mar 2019 05:48:40 GMT) ==

Regressions:

6% startup_about_home_paint windows10-64-pgo-qr opt e10s stylo 676.08 -> 714.79

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=20026

These are the Gecko profiles for it:

before: https://perf-html.io/from-url/https%3A%2F%2Fqueue.taskcluster.net%2Fv1%2Ftask%2FSm_S39pSQUCNXX7KAA2Log%2Fruns%2F0%2Fartifacts%2Fpublic%2Ftest_info%2Fprofile_startup_about_home_paint.zip

after: https://perf-html.io/from-url/https%3A%2F%2Fqueue.taskcluster.net%2Fv1%2Ftask%2FHXvT1X7nRe6qmNh0QpOQwQ%2Fruns%2F0%2Fartifacts%2Fpublic%2Ftest_info%2Fprofile_startup_about_home_paint.zip

Done.

Felipe pointed out that in https://bugzilla.mozilla.org/show_bug.cgi?id=1531854#c1 he saw similar effects...

Pushed to try thrice:

1. baseline: https://treeherder.mozilla.org/#/jobs?repo=try&revision=eb6ff2e9c38c2ccd871f5f33491ed1870ed36156

2. backing out initializing the preloaded tab in the new window: https://treeherder.mozilla.org/#/jobs?repo=try&revision=d9eeb687553c734666ed021c4e8e3cc56cbfe88c

3. backing out all of bug 1353013: https://treeherder.mozilla.org/#/jobs?repo=try&revision=077a1a6911a64f185863ec53b0badef23130f79c

(In reply to :Gijs (he/him) from comment #3)

Pushed to try thrice:

1. baseline: https://treeherder.mozilla.org/#/jobs?repo=try&revision=eb6ff2e9c38c2ccd871f5f33491ed1870ed36156

2. backing out initializing the preloaded tab in the new window: https://treeherder.mozilla.org/#/jobs?repo=try&revision=d9eeb687553c734666ed021c4e8e3cc56cbfe88c

3. backing out all of bug 1353013: https://treeherder.mozilla.org/#/jobs?repo=try&revision=077a1a6911a64f185863ec53b0badef23130f79c

Even backing everything out doesn't return nearly as big a gap as appeared in the runs on autoland, and perf compare isn't confident that it's an improvement even with 15 runs of the job on each push. This test is pretty noisy...

My best guess is that we're somehow starting the load of the preloaded browser early some of the time and that interferes with things. This wasn't brought out in the profiles per mconley's analysis in bug 1537694 comment 21, but it's possible that the profiler interferes with things to the point where that run isn't representative for what happens without the profiler.

I guess we could delay preloading until after the initial about:home tab has loaded, if indeed we're loading about:home on startup, which hopefully would deal with that?

Then again, given that just not running the initial preload at all makes very little difference ( https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=eb6ff2e9c38c2ccd871f5f33491ed1870ed36156&newProject=try&newRevision=d9eeb687553c734666ed021c4e8e3cc56cbfe88c ), I wonder if it's even worth doing that, and if we should just accept the regression and move on with life... if there was something more serious going on I'd expect regressions on the other platforms, too.

I also can't reproduce any differences locally on my own windows build. Just noise.

Thoughts, Mike? Are there obvious causes/issues I'm missing here?

I think we've done our due diligence here. Sometimes these tests catch regressions that we can do something about. Sometimes they don't. I think this is one of the times that they don't. I think we need to just eat this one in the interest of better preloading behaviour.

Alright, let's call it.