Joel Maher ( :jmaher ) (UTC -8) Reporter
	Description • 8 years ago

Talos has detected a Firefox performance regression from push 020168373810. As author of one of the patches included in that push, we need your help to address this regression.

Summary of tests that regressed:

  damp summary windows7-32 opt e10s - 3.64% worse
  damp summary osx-10-10 opt e10s - 3.41% worse
  damp summary linux64 opt - 9.67% worse
  damp summary windows7-32 pgo - 9.23% worse
  damp summary linux64 pgo - 9.09% worse
  damp summary windows7-32 opt - 8.6% worse
  damp summary osx-10-10 opt - 8.72% worse
  damp summary windows8-64 opt - 9.4% worse
  damp summary windows8-64 pgo - 7.09% worse


You can find links to graphs and comparsion views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=1981

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/Buildbot/Talos/Tests

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/Buildbot/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling

	Joel Maher ( :jmaher ) (UTC -8) Reporter
	Comment 1 • 8 years ago

I had done a lot of retriggers, this is pretty much restricted to the 'damp' test.  Here is a compare view of the subtests for windows 7:
https://treeherder.mozilla.org/perf.html#/comparesubtest?originalProject=mozilla-inbound&originalRevision=0e7371939dd6&newProject=mozilla-inbound&newRevision=020168373810&originalSignature=966c66a8476bd1030d142a3277224d25e8b1bf1b&newSignature=966c66a8476bd1030d142a3277224d25e8b1bf1b&framework=1

:jryans, can you take a look a this bug and determine if we should fix this or accept it as a regression?

	J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow) Assignee
	Comment 2 • 8 years ago

It seems reasonable that e10s off does get a bit slower since the change in bug 1240907 means we are now using the message manager for that case (just like we always have with e10s on).

However, the top 2 slowdowns with e10s on are bit more perplexing:

damp summary windows7-32 opt e10s - 3.64% worse
damp summary osx-10-10 opt e10s - 3.41% worse

I'll keep investigating to see what I can find about this.

:jmaher, I am a bit perplexed by this link:

https://treeherder.mozilla.org/perf.html#/comparesubtest?originalProject=mozilla-inbound&originalRevision=0e7371939dd6&newProject=mozilla-inbound&newRevision=020168373810&originalSignature=0c0bb95cce8f2fc538367aeb786aba2a6381e9ad&newSignature=0c0bb95cce8f2fc538367aeb786aba2a6381e9ad&framework=1

It says "windows7-32 damp" at the top, which to me suggests "e10s off", but then the subtests each say "e10s".  Is e10s off or on for that result set?

	J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow) Assignee
	Comment 3 • 8 years ago

Ah, never mind, I guess I just clicked the wrong link from overall result page.

	J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow) Assignee
	Comment 4 • 8 years ago

:jmaher, is this the right way to run damp with and without e10s locally?

./mach talos-test -a damp
./mach talos-test -a damp --disable-e10s

	J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow) Assignee
	Comment 5 • 8 years ago

I do see some errors when running damp locally with the new changes that are not present at the base changeset.

I'll investigate this more tomorrow.

	Joel Maher ( :jmaher ) (UTC -8) Reporter
	Comment 6 • 8 years ago

:jryans, it sounds like you figured out the compare view, I filed bug 1289726 to clean that minor confusion up.

And for running talos via mach, we default to e10s mode and adding --disable-e10s will run without (on all tests, talos/mochitest/reftest/...)

	J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow) Assignee
	Comment 7 • 8 years ago

Continuing to attempt various fixes.  I can't seem to reproduce the perf difference locally, so I'm throwing various ideas at try to see what happens.

	Alison Shiue
	Comment 8 • 8 years ago

This regression patch 020168373810 has been merged into Aurora, so please make sure to uplift any fixes to Aurora, thank you.

https://treeherder.mozilla.org/perf.html#/alerts?id=2185

  damp summary linux64 pgo - 5.3% worse

	Joel Maher ( :jmaher ) (UTC -8) Reporter
	Comment 9 • 8 years ago

it is odd the windows regressions didn't show up on Aurora :(

I do see a 4.58% OSX regression (non e10s) that the alert tools didn't pick up.

	J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow) Assignee
	Comment 10 • 8 years ago

After more investigation, I have found a reason for the regressions in e10s on.  If you look at the results there, such as:

https://treeherder.mozilla.org/perf.html#/comparesubtest?framework=1&newProject=mozilla-inbound&newRevision=02016837381082d203059b4cc193f5f4ced3ef50&newSignature=0c0bb95cce8f2fc538367aeb786aba2a6381e9ad&originalProject=mozilla-inbound&originalRevision=0e7371939dd6&originalSignature=0c0bb95cce8f2fc538367aeb786aba2a6381e9ad

you see that only the simple page has regressions.  This is due to a bug in the DAMP test which loads the simple page over a chrome:// URL, which means it will use a non-remote browser even with e10s on.  I filed bug 1291809 to fix this issue in the test.

This means all the regressions make sense in the context of the patch that landed, since they were all non-remote browsers which are now forced to take the somewhat slower message manager path that has always been used with remote browsers.

Nothing else to do in this bug, but we should work on bug 1291809 to improve test correctness.