We have some data about talos tests which produce worse results with e10s than their non-e10s result. For the most part we tend to believe these results. However, we don't have a way to track these differences, we don't have automatic alerts on such differences, and currently it's hard for developers to assess fixes since we don't run talos e10s tests on try server. Some results I've collected recently using compare.py from the talos repository (at the beginning of each line: + is an improvement, - is a regression, and * by the change marks tests where higher is better): > $ python compare.py --compare-e10s --rev 42afc7ef5ccb --pgo --verbose --branch Firefox --platform Win7 --master-revision 42afc7ef5ccb > > --- [ non-e10s ] --- --- [ e10s ] --- > test geometric stddev points change geometric stddev points > mean mean > Win7: > > - tresize 15.1 +/- 0% (2#) [ +1.7%] 15.4 +/- 1% (2#) > - kraken 1529.3 +/- 0% (2#) [ +3.9%] 1589.3 +/- 0% (2#) > + v8_7 17798.4 +/- 3% (2#) [ +1.6%]* 18080.1 +/- 2% (2#) > + dromaeo_css 5815.2 +/- 3% (2#) [ +3.7%]* 6033.2 +/- 1% (2#) > - dromaeo_dom 1310.6 +/- 0% (2#) [ -0.5%]* 1304.5 +/- 0% (2#) > + a11yr 178.7 +/- 0% (2#) [ -0.2%] 178.5 +/- 2% (2#) > ++ ts_paint 797.7 +/- 1% (2#) [ -47.8%] 416.3 +/- 0% (2#) > + tpaint 155.3 +/- 1% (2#) [ -4.2%] 148.8 +/- 1% (2#) > ++ tsvgr_opacity 228.2 +/- 1% (2#) [ -56.5%] 99.2 +/- 0% (2#) > - tp5o 225.4 +/- 0% (2#) [ +5.3%] 237.3 +/- 1% (2#) > + tart 8.6 +/- 1% (2#) [ -1.0%] 8.5 +/- 0% (2#) > + tcanvasmark 5696.9 +/- 1% (2#) [ +0.6%]* 5732.0 +/- 0% (2#) > ++ tsvgx 199.1 +/- 0% (2#) [ -24.7%] 149.8 +/- 0% (2#) > + tscrollx 3.0 +/- 2% (2#) [ -0.2%] 3.0 +/- 0% (2#) > --- glterrain 5.1 +/- 0% (2#) [+268.9%] 18.9 +/- 2% (2#) > + cart 53.5 +/- 1% (2#) [ -1.2%] 52.8 +/- 1% (2#) > ++ tp5o_scroll 3.4 +/- 1% (2#) [ -13.0%] 3.0 +/- 0% (2#) > $ python compare.py --compare-e10s --rev 42afc7ef5ccb --pgo --verbose --branch Firefox --platform Linux --master-revision 42afc7ef5ccb > > --- [ non-e10s ] --- --- [ e10s ] --- > test geometric stddev points change geometric stddev points > mean mean > Linux: > ++ tresize 17.2 +/- 0% (2#) [ -25.1%] 12.9 +/- 0% (2#) > - kraken 1571.8 +/- 0% (2#) [ +2.2%] 1606.6 +/- 0% (2#) > + v8_7 19309.3 +/- 0% (2#) [ +0.5%]* 19399.8 +/- 0% (2#) > + dromaeo_css 5646.3 +/- 1% (2#) [ +3.9%]* 5866.8 +/- 0% (2#) > + dromaeo_dom 1129.1 +/- 1% (2#) [ +3.9%]* 1173.0 +/- 0% (2#) > - a11yr 241.5 +/- 1% (2#) [ +5.0%] 253.5 +/- 0% (2#) > ++ ts_paint 876.3 +/- 0% (2#) [ -50.6%] 432.6 +/- 1% (2#) > - tpaint 197.4 +/- 1% (2#) [ +5.2%] 207.6 +/- 2% (2#) > ++ tsvgr_opacity 218.3 +/- 13% (2#) [ -60.6%] 86.0 +/- 1% (2#) > -- tp5o 269.2 +/- 0% (2#) [ +21.8%] 328.0 +/- 1% (2#) > -- tart 6.2 +/- 0% (2#) [ +13.9%] 7.1 +/- 0% (2#) > -- tcanvasmark 8153.4 +/- 0% (2#) [ -15.6%]* 6877.7 +/- 0% (2#) > -- tsvgx 580.8 +/- 0% (2#) [ +10.2%] 639.7 +/- 0% (2#) > ++ tscrollx 9.1 +/- 0% (2#) [ -16.5%] 7.6 +/- 0% (2#) > + glterrain 22.6 +/- 0% (2#) [ -1.4%] 22.3 +/- 0% (2#) > - cart 42.0 +/- 1% (2#) [ +6.5%] 44.7 +/- 1% (2#) > ++ tp5o_scroll 8.8 +/- 1% (2#) [ -12.4%] 7.7 +/- 0% (2#) We should probably use this bug as a meta bug for such regressions, and for deciding to track and handle such regressions consistently.

Just adding some comparison numbers I have for about a month ago (revision ac277e615f8f from 2015-05-20): TL;DR: - Windows: most tests perform same or better on e10s, with the standing exception of WebGL which performs considerably worse. - Linux: mixed results, some better, some worse. - Compared to similar comparison from two months earlier (comment 0): - Windows: similar, but some of the improvements improved even more. - Linux: some of the regressions regressed more, the improvements are similar, maybe slightly better. Summary: Windows 7 32 (worst to best): ----------------------------- - WebGL 270 % worse - a11yr 7 % worse - tp5o 6 % worse - all other tests are within 2-3 % - tresize 11 % better - tp5o scroll 15 % better - tsvgx 21 % better - ts_paint 46 % better - svg_opacity 57 % better Linux 64 (worst to best): ------------------------- - a11yr 37 % worse - tpaint 30 % worse - tp5o 25 % worse - tsvgx 17 % worse - tresize 16 % worse - all other tests are within 2-3 % - WebGL 7 % better - tscrollx 8 % better - tp5o scroll 13 % better - ts_paint 50 % better - svg_opacity 70 % better

e10s number compared to non-e10s for revision 0093691d3715 from 2016-06-12 ("worse" means e10s is worse, "better" means e10s is better): Summary: Win7 (worst to best): --------------------- missing: kraken, v8_7, dromaeo_css, dromaeo_dom, sessionrestore, sessionrestore_no_auto_restore. tp5o 9% worse a11yr 8% worse except for missing results, the rest are within 2-3% either side. tp5o_scroll 13% better tsvgx 22% better tsvgr_opacity 54% better ts_paint 55% better Linux (worst to best): ---------------------- missing: sessionrestore, sessionrestore_no_auto_restore. a11yr 37% worse tpaint 32% worse tresize 29% worse tp5o 26% worse tsvgx 10% worse except for missing results, the rest are within 2-3% either side. dromaeo_css 4% better glterrain 4% better tp5o_scroll 6% better tscrollx 15% better ts_paint 57% better tsvgr_opacity 66% better --------------------------------------------------------------------------- Raw data: $ python compare.py --compare-e10s --rev 0093691d3715 --pgo --verbose --branch Firefox --platform Win7 --master-revision 0093691d3715 test Master gmean stddev points change gmean stddev points graph-url Win7: tresize 16.6 +/- 2% (6#) [ +0.9%] 16.7 +/- 1% (6#) kraken No results found v8_7 No results found dromaeo_css No results found dromaeo_dom No results found :( a11yr 178.0 +/- 1% (6#) [ +7.7%] 191.6 +/- 2% (6#) :) ts_paint 865.8 +/- 1% (6#) [ -55.4%] 385.9 +/- 2% (6#) tpaint 164.0 +/- 2% (6#) [ -1.6%] 161.4 +/- 2% (6#) :) tsvgr_opacity 225.2 +/- 2% (6#) [ -54.0%] 103.6 +/- 9% (6#) tp5n No data for platform :( tp5o 227.9 +/- 0% (6#) [ +8.5%] 247.3 +/- 1% (6#) :) tart 8.1 +/- 1% (6#) [ -1.4%] 7.9 +/- 1% (6#) :( tcanvasmark 5717.8 +/- 0% (6#) [ +1.0%] 5774.2 +/- 1% (6#) :) tsvgx 190.6 +/- 0% (6#) [ -21.6%] 149.3 +/- 1% (6#) :( tscrollx 3.0 +/- 2% (6#) [ +2.9%] 3.1 +/- 2% (6#) sessionrestore No data for platform sessionrestore_no_auto_restore No data for platform glterrain 5.0 +/- 2% (6#) [ -1.9%] 4.9 +/- 2% (6#) :( cart 51.4 +/- 1% (6#) [ +1.2%] 52.0 +/- 1% (6#) :) tp5o_scroll 3.3 +/- 3% (6#) [ -13.4%] 2.8 +/- 0% (6#) $ python compare.py --compare-e10s --rev 0093691d3715 --pgo --verbose --branch Firefox --platform Linux --master-revision 0093691d3715 test Master gmean stddev points change gmean stddev points graph-url Linux: :( tresize 16.0 +/- 2% (6#) [ +28.9%] 20.7 +/- 3% (6#) :( kraken 1628.2 +/- 0% (6#) [ +1.6%] 1653.5 +/- 0% (6#) :) v8_7 20930.8 +/- 2% (6#) [ +2.1%] 21378.8 +/- 1% (6#) :) dromaeo_css 5403.3 +/- 2% (6#) [ +3.7%] 5603.8 +/- 3% (6#) :) dromaeo_dom 1085.2 +/- 1% (6#) [ +1.9%] 1106.3 +/- 1% (6#) :( a11yr 247.6 +/- 1% (6#) [ +36.7%] 338.3 +/- 1% (6#) :) ts_paint 905.8 +/- 1% (6#) [ -56.7%] 392.3 +/- 3% (6#) :( tpaint 171.5 +/- 3% (6#) [ +31.9%] 226.3 +/- 2% (6#) :) tsvgr_opacity 264.9 +/- 1% (6#) [ -65.9%] 90.3 +/- 1% (6#) :( tp5o 269.0 +/- 0% (6#) [ +26.4%] 339.9 +/- 2% (6#) tart 6.7 +/- 0% (6#) [ +0.0%] 6.7 +/- 0% (6#) :) tcanvasmark 6921.9 +/- 1% (6#) [ -2.9%] 6721.9 +/- 1% (6#) :( tsvgx 599.8 +/- 0% (6#) [ +9.6%] 657.2 +/- 1% (6#) :) tscrollx 8.9 +/- 0% (6#) [ -14.7%] 7.6 +/- 1% (6#) sessionrestore No data for platform sessionrestore_no_auto_restore No data for platform :) glterrain 22.0 +/- 0% (6#) [ -4.4%] 21.1 +/- 0% (6#) cart 45.7 +/- 2% (6#) [ +0.7%] 46.1 +/- 2% (6#) :) tp5o_scroll 8.2 +/- 0% (6#) [ -6.1%] 7.7 +/- 0% (6#)

(In reply to Avi Halachmi (:avih) from comment #3) > e10s number compared to non-e10s for revision 0093691d3715 from 2016-06-12 > ("worse" means e10s is worse, "better" means e10s is better): In case it was missed, the most notable change is that WebGL is no longer slower on e10s on Windows. This was by far the biggest e10s perf issue on Windows. The rest are similar with relatively small changes.

filed a few new bugs, updated all the existing ones, here is a summary: using this build: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=aad95360a002&exclusion_profile=false&filter-searchStr=talos we have this set of results: https://pastebin.mozilla.org/8838353 and this is the summarization: linux 32/64 regressions >2%: tresize 11%/18% (bug 1179732) a11y 36%/37% (bug 1174771) tpaint 31%/33% (bug 1174770) tp5o 26%/28% (bug 1174776) svgx 9%/10% (bug 1179733) windows xp/7/8 regressions >2%: tresize 147%/--/-- (bug 1179732) a11y 12%/9%/7% (bug 1174771) tpaint 10%/--/3% (bug 1174770) tp5o 12%/9%/10% (bug 1174776) svgx 25%/--/-- (bug 1179733) tscrollx 592%/4%/-- (bug 1174772) tp5o_scroll 495%/--/-- (bug 1179735) osx 10.10 regressions >2%: a11y 25% (bug 1174771) tp5o 20% (bug 1174776)

(In reply to Joel Maher (:jmaher) from comment #5) > filed a few new bugs, updated all the existing ones, here is a summary: > > using this build: > https://treeherder.mozilla.org/#/jobs?repo=mozilla- > central&revision=aad95360a002&exclusion_profile=false&filter-searchStr=talos > > we have this set of results: > https://pastebin.mozilla.org/8838353 Would it be possible for you to gather DAMP results from a build as well?

oh right, damp will be in the next one. Thanks for pointing that out!

updated with data from this weekend (including damp): https://pastebin.mozilla.org/8839348 * one change is that previously compare.py was using the average, not the geometric mean- this is not using geometric mean (default graphs view, and alerting data)

ok, more data for folks who are tracking this (bonus- we have tps results!!!!): https://pastebin.mozilla.org/8840313 generated from this push: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=8e5c888d0d89&filter-searchStr=talos&exclusion_profile=false Maybe next week I will look at all the existing bugs and update them accordingly.

Great, thanks jmaher! Very interesting to see tps being neutral / win on Win 7 and Win 8...

As an added wrinkle, APZ has been turned on by default on Nightly - this is a variation that might affect our talos numbers. We should factor in the shipping estimate for APZ into any comparisons we do (if e10s is likely to ship before APZ, we need to do comparisons with APZ disabled - otherwise, enabled).

After some measurements and discussions in bug 1179735 I think this bug should be re-triaged. It seems that with APZ Display List creation is significantly slower when e10s is on, and that causes serious performance regressions. There is a debate about weather the regressing tests are measuring the right thing or not, but for example tp5o_scroll is regressing with 70% percentage right now (e10s vs not-e10s with APZ on) so at very least I want to gather more data to better understand the risk this comes with.

I'm going to move this comment over to the tp5o_scroll bug. Other open regression bugs attached here do not appear to be related to apz getting turned on.

Updated numbers from two days ago: Thanks to Joel, we have an up-to-date e10s vs non-e10s comparison for a recent revision (e1cf617a1f28). For each platform, the list is arranged from worst (top) to best (bottom) as far as e10s is concerned. The value at the middle of each line represents the diff in %, e.g. [ +20% ]. The list at each platform is divided to above 10% regression (top), between 10% regression and 10% improvement (middle), and above 10% improvement (bottom). Note that canvasmark and dromaeo are higher-is-better, so a positive percentage is an improvement (already taken into account at the sort). All the other tests are lower-is-better. The comparison is over 6-7 runs of each test, and the stddev of the runs is at the stddev column, appears as e.g. "+/- 5%" is 5% stddev (of the geometric mean). --------------------------- python compare.py --branch Firefox --masterbranch Firefox --master-revision e1cf617a1f28 --revision e1cf617a1f28 --compare-e10s --verbose --pgo non-e10s e10s test gmean stddev change gmean stddev Linux64 (worst to best e10s): -- tps 41.9 +/- 0% [ +71.0%] 71.7 +/- 2% -- tpaint 196.8 +/- 3% [ +46.0%] 287.3 +/- 3% -- cart 21.1 +/- 0% [ +13.3%] 23.9 +/- 0% -- tart 4.3 +/- 0% [ +10.9%] 4.8 +/- 0% -- a11yr 448.0 +/- 2% [ +8.6%] 486.5 +/- 2% -- tp5o 250.5 +/- 1% [ +3.2%] 258.5 +/- 1% -- kraken 1521.1 +/- 2% [ +1.2%] 1539.8 +/- 8% tresize 25.2 +/- 0% [ +1.2%] 25.5 +/- 0% glterrain 18.5 +/- 0% [ +0.4%] 18.6 +/- 0% ++ sessionrestore 1700.9 +/- 10% [ -1.5%] 1674.8 +/- 7% ++ tcanvasmark 8801.1 +/- 69% [ +2.2%] 8993.1 +/- 47% ++ dromaeo_css 6836.3 +/- 62% [ +2.4%] 6999.0 +/- 27% ++ damp 217.4 +/- 5% [ -3.7%] 209.5 +/- 2% ++ sessionrestore_NAR 712.6 +/- 5% [ -4.5%] 680.3 +/- 6% ++ tp5o_scroll 8.3 +/- 0% [ -17.2%] 6.9 +/- 0% ++ tscrollx 6.6 +/- 0% [ -28.2%] 4.7 +/- 0% ++ tsvgx 325.1 +/- 1% [ -41.5%] 190.1 +/- 1% ++ ts_paint 1023.1 +/- 3% [ -52.8%] 483.4 +/- 2% ++ tsvgr_opacity 371.6 +/- 4% [ -81.8%] 67.5 +/- 1% ++ tp5o_responsiveness 29.5 +/- 1% [ -86.2%] 4.1 +/- 0% Win7 (worst to best e10s): -- tp5o_scroll 2.9 +/- 0% [ +70.2%] 5.0 +/- 0% -- tps 46.0 +/- 0% [ +63.8%] 75.3 +/- 0% -- tpaint 185.1 +/- 2% [ +28.6%] 238.0 +/- 5% -- tart 3.8 +/- 0% [ +11.8%] 4.2 +/- 0% -- tresize 9.6 +/- 0% [ +7.3%] 10.3 +/- 0% -- tscrollx 2.8 +/- 0% [ +6.7%] 3.0 +/- 0% -- cart 23.4 +/- 0% [ +6.6%] 24.9 +/- 0% a11yr 407.3 +/- 6% [ +1.5%] 413.5 +/- 8% kraken 1603.3 +/- 10% [ +0.8%] 1616.4 +/- 12% damp 198.2 +/- 1% [ +0.2%] 198.6 +/- 1% sessionrestore 1544.0 +/- 14% [ -0.1%] 1541.7 +/- 18% dromaeo_css 6643.1 +/- 50% [ +0.1%] 6650.2 +/- 99% ++ tp5o 217.6 +/- 1% [ -1.1%] 215.2 +/- 1% ++ tcanvasmark 7779.8 +/- 36% [ +1.5%] 7893.0 +/- 36% ++ glterrain 5.2 +/- 0% [ -2.0%] 5.1 +/- 0% ++ sessionrestore_NAR 631.0 +/- 6% [ -2.6%] 614.5 +/- 6% ++ tsvgx 388.3 +/- 1% [ -50.4%] 192.4 +/- 1% ++ ts_paint 870.3 +/- 4% [ -55.8%] 385.1 +/- 3% ++ tp5o_responsiveness 39.7 +/- 1% [ -84.8%] 6.0 +/- 1% ++ tsvgr_opacity 505.7 +/- 3% [ -90.2%] 49.5 +/- 1% WinXP (worst to best e10s): -- tp5o_scroll 2.7 +/- 0% [+111.5%] 5.8 +/- 0% -- tps 42.2 +/- 0% [ +53.2%] 64.6 +/- 1% -- tscrollx 2.4 +/- 0% [ +51.1%] 3.6 +/- 0% -- tpaint 186.5 +/- 3% [ +31.9%] 246.0 +/- 7% -- tart 4.0 +/- 0% [ +12.0%] 4.5 +/- 0% -- tresize 9.9 +/- 0% [ +11.8%] 11.1 +/- 0% -- a11yr 367.2 +/- 8% [ +4.0%] 382.0 +/- 11% ++ sessionrestore 1441.1 +/- 6% [ -1.5%] 1419.0 +/- 9% ++ tcanvasmark 7737.3 +/- 51% [ +1.7%] 7869.8 +/- 72% ++ tp5o 219.4 +/- 2% [ -2.4%] 214.3 +/- 2% ++ sessionrestore_NAR 594.5 +/- 6% [ -6.7%] 554.4 +/- 5% ++ cart 34.1 +/- 1% [ -21.7%] 26.7 +/- 0% ++ tsvgx 396.6 +/- 3% [ -51.7%] 191.6 +/- 3% ++ ts_paint 826.4 +/- 12% [ -54.6%] 375.3 +/- 4% ++ glterrain 20.9 +/- 0% [ -64.8%] 7.4 +/- 0% ++ tsvgr_opacity 495.3 +/- 3% [ -89.8%] 50.3 +/- 1% ++ tp5o_responsiveness 43.9 +/- 1% [ -92.7%] 3.2 +/- 0% Win8 (worst to best e10s): -- tp5o_scroll 3.0 +/- 0% [ +46.6%] 4.4 +/- 0% -- tps 41.3 +/- 0% [ +38.3%] 57.1 +/- 1% -- tpaint 194.4 +/- 2% [ +28.4%] 249.7 +/- 0% -- tart 4.0 +/- 0% [ +7.9%] 4.4 +/- 0% -- cart 22.4 +/- 0% [ +4.0%] 23.3 +/- 0% a11yr 367.0 +/- 5% [ +0.6%] 369.0 +/- 2% sessionrestore 1565.9 +/- 2% [ -0.2%] 1563.0 +/- 4% tcanvasmark 8999.7 +/- 85% [ +0.2%] 9020.1 +/- 59% ++ damp 207.9 +/- 5% [ -2.5%] 202.6 +/- 1% ++ glterrain 4.8 +/- 0% [ -3.0%] 4.7 +/- 0% ++ sessionrestore_NAR 604.8 +/- 1% [ -4.1%] 580.2 +/- 2% ++ tp5o 215.7 +/- 1% [ -6.6%] 201.4 +/- 1% ++ tscrollx 2.7 +/- 0% [ -6.9%] 2.5 +/- 0% ++ tresize 11.0 +/- 0% [ -12.3%] 9.7 +/- 0% ++ tsvgx 120.4 +/- 1% [ -27.1%] 87.7 +/- 1% ++ ts_paint 801.3 +/- 2% [ -54.9%] 361.7 +/- 2% ++ tsvgr_opacity 270.1 +/- 1% [ -83.6%] 44.2 +/- 0% ++ tp5o_responsiveness 52.8 +/- 131% [ -91.8%] 4.3 +/- 0% OSX10.10: -- tps 66.3 +/- 1% [+104.8%] 135.9 +/- 2% -- tp5o_scroll 5.6 +/- 0% [ +45.6%] 8.1 +/- 0% -- tpaint 331.8 +/- 4% [ +30.3%] 432.5 +/- 15% -- tart 9.1 +/- 0% [ +8.7%] 9.9 +/- 0% -- damp 249.8 +/- 6% [ +5.4%] 263.2 +/- 3% -- cart 35.2 +/- 0% [ +5.2%] 37.0 +/- 0% -- glterrain 3.7 +/- 0% [ +3.5%] 3.8 +/- 0% -- tcanvasmark 6342.6 +/- 118% [ -2.8%] 6163.9 +/- 142% -- tresize 27.2 +/- 0% [ +2.6%] 27.9 +/- 0% a11yr 418.5 +/- 9% [ +2.0%] 426.9 +/- 3% tp5o 259.7 +/- 4% [ +1.5%] 263.5 +/- 2% sessionrestore 2314.0 +/- 70% [ -1.5%] 2279.1 +/- 29% sessionrestore_NAR 1060.6 +/- 32% [ -2.2%] 1037.8 +/- 28% ++ tscrollx 4.0 +/- 0% [ -16.6%] 3.4 +/- 0% ++ ts_paint 1356.3 +/- 17% [ -44.6%] 751.7 +/- 20% ++ tsvgx 367.6 +/- 2% [ -51.0%] 180.1 +/- 3% ++ tsvgr_opacity 514.4 +/- 6% [ -89.6%] 53.4 +/- 0%

We now have a dashboard you can use for tracking differences between non-e10s and e10s modes in Talos here: https://treeherder.allizom.org/perf.html#/e10s