Closed
Bug 563319
Opened 15 years ago
Closed 13 years ago
Test performance of Visual C++ 2010 build vs. current trunk nightly build
Categories
(Firefox Build System :: General, defect)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: ted, Assigned: matjk7)
References
Details
I'd like to switch the trunk nightly builds to VC++ 2010 (bug 563318), since it will fix some bugs with symbol dumping. I think it may also improve performance slightly, but that bears investigation. We should probably test some builds just to verify that it doesn't negatively impact performance, at least. We'll need to fix bug 563316 first to get an apples-to-apples comparison.
Assignee | ||
Updated•13 years ago
|
Assignee: nobody → matjk7
Status: NEW → ASSIGNED
Assignee | ||
Comment 1•13 years ago
|
||
First attempt at comparison: http://goo.gl/wgxSE
Ignoring what looks like noise, the differences are:
- Small win on tp4
- 10% tp4_memset regression on win7. On a second run this was only a 4.23% regression, but it still worries me that we are regressing at all.
- Small win on tp5_pbytes_paint
- Small win across all dromaeo benchmarks
- Small win on tsspider_* tests
- Significant (8.7%) win on tspaint on winxp, small regression (noise, most likely) on the same test on win7
- Significant wins on ts_places_* benchmarks except ts_places_generated_med_shutdown where both platforms regress a bit.
Apart from the weird result on tp4_memset this looks like a minor win on most benchmarks.
Fantastic. Thanks Matheus!
Reporter | ||
Comment 3•13 years ago
|
||
Thanks for running that comparison!
Reporter | ||
Comment 4•13 years ago
|
||
Interesting. So tp4_memset is the working set of the process, sampled during Tp4. Tp5_memset_paint (the replacement metric), doesn't seem to have budged, so I don't think this is an across-the-board regression.
Also, what's up with the *gigantic* variation number on the new tp4_memset run? ±33912841.22??? That's crazytown!
https://wiki.mozilla.org/Buildbot/Talos#Working_Set_.28tp5_memset.29
Comment 5•13 years ago
|
||
Looks like you avoided bug 696196 by building the original revision on try yourself. That's a good idea.
> Also, what's up with the *gigantic* variation number on the new tp4_memset run? ±33912841.22???
> That's crazytown!
When you call it 32mb, it doesn't sound so ginormous. :) Anyway, that test looks pretty noisy.
http://graphs-new.mozilla.org/graph.html#tests=[[45,1,12],[45,1,1]]&sel=none&displayrange=365&datatype=running
Matheus, I might kick more runs of these tests, since there's a large amount of noise on some of them. Then you'll be able to better quantify the various small wins.
If you really want to be sure, you may also want to kick a few new *builds*, just in case there's some randomness in the performance of the builds. Good luck using compare-talos there. :)
Assignee | ||
Comment 6•13 years ago
|
||
New builds with new results: http://goo.gl/lXKpG
Comparing with the first set of results:
- The tp4 win is definitely real.
- There might be a very minor tdhtml_paint regression.
- The 10% regression on tp4_memset was probably noise, but tp4_memset does regress a little.
- tp4_shutdown had some weird results on this run (21% regression on winxp and a 14% win on win7).
- tp5_paint might be a minor win too.
- The tp5_pbytes_paint result was probably noise.
- On this run tp5_shutdown_paint was a significant win (8-13%).
- The dromaeo benchmarks are quite noisy but dromaeo_dom is definitely a win.
- The tsspider_* tests show a consistent win.
- tspaint is noisy on win7 but the ~9% win on winxp is surprisingly consistent.
- The ts_places_* tests are quite noisy as well but show a overall win.
Overall only tdhtml_paint and tp4_memset show a consistent regression with everything else either staying the same or improving a little.
Status: ASSIGNED → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Updated•7 years ago
|
Product: Core → Firefox Build System
You need to log in
before you can comment on or make changes to this bug.
Description
•