Closed Bug 563319 Opened 15 years ago Closed 13 years ago

Test performance of Visual C++ 2010 build vs. current trunk nightly build

Categories

(Firefox Build System :: General, defect)

x86
Windows 7
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: ted, Assigned: matjk7)

References

Details

I'd like to switch the trunk nightly builds to VC++ 2010 (bug 563318), since it will fix some bugs with symbol dumping. I think it may also improve performance slightly, but that bears investigation. We should probably test some builds just to verify that it doesn't negatively impact performance, at least. We'll need to fix bug 563316 first to get an apples-to-apples comparison.
Depends on: 515492
No longer depends on: 563316
Assignee: nobody → matjk7
Status: NEW → ASSIGNED
First attempt at comparison: http://goo.gl/wgxSE Ignoring what looks like noise, the differences are: - Small win on tp4 - 10% tp4_memset regression on win7. On a second run this was only a 4.23% regression, but it still worries me that we are regressing at all. - Small win on tp5_pbytes_paint - Small win across all dromaeo benchmarks - Small win on tsspider_* tests - Significant (8.7%) win on tspaint on winxp, small regression (noise, most likely) on the same test on win7 - Significant wins on ts_places_* benchmarks except ts_places_generated_med_shutdown where both platforms regress a bit. Apart from the weird result on tp4_memset this looks like a minor win on most benchmarks.
Thanks for running that comparison!
Interesting. So tp4_memset is the working set of the process, sampled during Tp4. Tp5_memset_paint (the replacement metric), doesn't seem to have budged, so I don't think this is an across-the-board regression. Also, what's up with the *gigantic* variation number on the new tp4_memset run? ±33912841.22??? That's crazytown! https://wiki.mozilla.org/Buildbot/Talos#Working_Set_.28tp5_memset.29
Looks like you avoided bug 696196 by building the original revision on try yourself. That's a good idea. > Also, what's up with the *gigantic* variation number on the new tp4_memset run? ±33912841.22??? > That's crazytown! When you call it 32mb, it doesn't sound so ginormous. :) Anyway, that test looks pretty noisy. http://graphs-new.mozilla.org/graph.html#tests=[[45,1,12],[45,1,1]]&sel=none&displayrange=365&datatype=running Matheus, I might kick more runs of these tests, since there's a large amount of noise on some of them. Then you'll be able to better quantify the various small wins. If you really want to be sure, you may also want to kick a few new *builds*, just in case there's some randomness in the performance of the builds. Good luck using compare-talos there. :)
New builds with new results: http://goo.gl/lXKpG Comparing with the first set of results: - The tp4 win is definitely real. - There might be a very minor tdhtml_paint regression. - The 10% regression on tp4_memset was probably noise, but tp4_memset does regress a little. - tp4_shutdown had some weird results on this run (21% regression on winxp and a 14% win on win7). - tp5_paint might be a minor win too. - The tp5_pbytes_paint result was probably noise. - On this run tp5_shutdown_paint was a significant win (8-13%). - The dromaeo benchmarks are quite noisy but dromaeo_dom is definitely a win. - The tsspider_* tests show a consistent win. - tspaint is noisy on win7 but the ~9% win on winxp is surprisingly consistent. - The ts_places_* tests are quite noisy as well but show a overall win. Overall only tdhtml_paint and tp4_memset show a consistent regression with everything else either staying the same or improving a little.
Status: ASSIGNED → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.