Hannes Verschore [:h4writer] Reporter
	Description • 9 years ago

A regression was reported on AWFY:
- slave: Mac OS X 10.10 (Mac Pro, browser)
- mode: Ion (=> this is the normal mode)

Regression(s)/Improvement(s):
- browsermark: Graphics 2.1: -6.49% (regression)
- browsermark: Graphics SVG 2.1: -17.96% (regression)

Recorded range:
- http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=f4f010e2f0b1&tochange=cef50aeb685e

More details: http://arewefastyet.com/regressions/#/regression/1792974
(Note: Clicking on the scores won't show a graph. We cannot publicly publish all results of Browsermark.)

	Hannes Verschore [:h4writer] Reporter
	Comment 1 • 9 years ago

Posted the steps to reproduce in private comment.

	Milan Sreckovic [:milan] (needinfo for best results)
	Comment 2 • 9 years ago

Lee, is this from your changes?

	Lee Salzman [:lsalzman]
	Comment 3 • 9 years ago

(In reply to Milan Sreckovic [:milan] from comment #3)
> Lee, is this from your changes?

My Cairo change landed on 7-29, so the timeline wouldn't fit this.

	Kartikaya Gupta (email:kats@mozilla.staktrace.com)
	Comment 4 • 9 years ago

I don't doubt that this was caused by APZ. Joel also started an email thread about a slew of talos regressions on Windows e10s PGO builds that point to APZ. It's something we'll need to look into.

	Hannes Verschore [:h4writer] Reporter
	Comment 5 • 9 years ago

Any update / progress on this bug? Regressions like these can mask other regressions.
Also the merge date is 10 August. In four days.

	Botond Ballo [:botond]
	Comment 6 • 9 years ago

APZ is Nightly-only for now, it is not riding the trains yet, so the regression would not make it to Aurora at the upcoming merge.

	Hannes Verschore [:h4writer] Reporter
	Comment 7 • 9 years ago

(In reply to Botond Ballo [:botond] from comment #7)
> APZ is Nightly-only for now, it is not riding the trains yet, so the
> regression would not make it to Aurora at the upcoming merge.

Good to know. But still keep in mind that this still isn't ideal for different reasons.

1) AWFY is only scanning Nightlies. As a result we don't have numbers for aurora/beta/release (yet?).

2) A regression can mask another regression. It happens often that another regression is introduced, but isn't visible until the first regression is solved. In that case we only see that when the first regression is fixed, we are still not at the original performance. In this case it is hard to deduce if the original regression isn't fully fixed or if we had a masked regression.

3) If a second regression happens and is visible, but gets fixed first and happens to give better performance than before the second regression, but worse than the first regression. In that case the fix improved performance, but once the first regression gets fixed, we don't know what the actual baseline should be. If we are happy with just having the same performance before the first regression, it might be that we actually didn't fixed the full regression, since we had an improvement in between.

Therefore I would suggest against keeping a regression in the tree ...

	Kartikaya Gupta (email:kats@mozilla.staktrace.com)
	Comment 8 • 9 years ago

I agree that it's bad to leave the regression in the tree. I can take a look at it this week, sorry for the delay. That being said, if it's possible to run AWFY with custom prefs it might be worth running with layers.async-pan-zoom.enabled set to false so that it continues to pick up other regressions and whatnot.

	Kartikaya Gupta (email:kats@mozilla.staktrace.com)
	Comment 9 • 9 years ago

Also since the graphics SVG test in browsermark is so hard to run in isolation I'm going to try and focus on the SVG tests that we have in talos that also regressed (tracked in bug 1189817) and hope that the fixes there take care of this. The talos tests are much easier to run in isolation/on-demand/with a profiler and so it should be easier to identify the problem.

	Hannes Verschore [:h4writer] Reporter
	Comment 10 • 9 years ago

On July 29th we regressed even more:
- browsermark: Graphics 2.1: -3.99% (regression)
- browsermark: Graphics SVG 2.1: -12.61% (regression)

Recorded range:
- http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=b65c048414f7&tochange=0d65dafbdce5

Given that range, I assume:
https://bugzilla.mozilla.org/show_bug.cgi?id=1186164

Gonna try to set "layers.async-pan-zoom.enabled" to false hopefully soonish. But in that case I need clear communication when this flag gets enabled in aurora and higher. To make sure I'm not measuring the wrong data.

	Kartikaya Gupta (email:kats@mozilla.staktrace.com)
	Comment 11 • 9 years ago

In that range bug 1180326 seems far more likely to me to be the culprit. Bug 1186164 should not have affected any desktop platforms, only B2G. And yes I will definitely let you know when the flag starts riding the trains.

	Kartikaya Gupta (email:kats@mozilla.staktrace.com)
	Comment 12 • 9 years ago

APZ will almost certainly ride the trains on 46. There have been a few talos regressions reported as well but in pretty much every case we've looked into so far the regression was a combination of the event regions code and the larger paint area, both of which are required for APZ correctness. Therefore we will probably have to just accept the regressions for now and incrementally improve things as we go forward. The apz-talos metabug tracks all the regressions as dependencies and will track improvements as blockers.

	Kartikaya Gupta (email:kats@mozilla.staktrace.com)
	Comment 13 • 8 years ago

Would you be able to check if APZ still produces a regression on this benchmark in Nightly? We have landed various improvements and although it likely won't be fixed entirely it should be a little better.

Preferably I'd like to know what the results are in these three configurations:
- e10s enabled, layers.async-pan-zoom.enabled true (default Nightly)
- e10s enabled, layers.async-pan-zoom.enabled false
- e10s enabled, layers.async-pan-zoom.enabled false, layout.event-regions.enabled true

Both of these pref changes require a browser restart.

	Hannes Verschore [:h4writer] Reporter
	Comment 14 • 8 years ago

Sure, I'll do that on Monday

	Hannes Verschore [:h4writer] Reporter
	Comment 15 • 8 years ago

The full numbers are in the attachment.

For browsermark svg I see:
(default Nightly): 6752
layers.async-pan-zoom.enabled false: 6771
layers.async-pan-zoom.enabled false, layout.event-regions.enabled true: 6762

Which all looks like in noise levels!
And the numbers are now back to what it was before July 2015

	Kartikaya Gupta (email:kats@mozilla.staktrace.com)
	Comment 16 • 8 years ago

Great, thanks! I'm going to close this bug as fixed then.