Replace the currently used Tp3 pageset with one based upon the current alexa top 500, also attempt to get under 100 pages to decrease the cycle time of talos boxes without reducing the trustworthiness of the data collected.

That's one way to reduce Tp, I suppose .. ;p

(In reply to comment #2) > Replace the currently used Tp3 pageset with one based upon the current alexa > top 500, also attempt to get under 100 pages to decrease the cycle time of > talos boxes without reducing the trustworthiness of the data collected. Some Talos suites give a wide variety in performance results. To compensate, for those suites, we run the suite multiple times and then average the results. But this means running the entire test multiple times!! :-( 1) Can we find+fix why some pages caused that much variance? 2) If some pages by design will always have lots of variance, maybe we should consider removing those specific pages from the performance suites? If this was done, we could reduce the number of iterations of the complete set we have to run when running Talos. Reduced iterations == reduced time == happy! :-) /me pushes bug over to Alice, as this is one of her Q1 goals.

Grabbing a new set of the alexa top 100 (minus all the google duplicates). I'll clean it up and have it running on talos stage - from there we'll be able to determine if we can get as good regression tracking out of smaller set of pages. This is an initial easiest-thing-that-could-possibly-work solution, instead of attempting to pick and choose web pages just limit us to a total of 100 instead of 500.

As a reference for 'interesting' pages see bug 419105.

Whittled down the top 500 pages to 100 pages using a simple scheme: - remove copies of google (seem to have about 40 copies in the set from different locales) - remove broken pages (lost layout, missing images, broken links, redirects, popups, etc) - remove pages that are just simple login screens - favor pages with lots of content - favor pages in different languages, attempting to get the greatest variety in the final page set I'll be putting the new page set on all the Talos staging boxes and letting it run for a while. Hopefully they'll be a regression and we'll be able to observe if it appears in the new page set (the only time that I hope for a performance regression). Also, from my initial runs on linux the 100 page set takes approx. 20 minutes to run to completion. This is still doing 10 full cycles through the 100 pages.

Um. Hot.

Hitting a snag with the new pageset on tiger/leopard - seems to freeze up on some international pages. Will have to investigate if this is the pageset or the browser.

There was a Tp regression observed yesterday, it also shows up on the linux box using the new page set: http://graphs-stage.mozilla.org/graph.html#show=454230&sel=1238294359,1238613420 I think that this is good proof that the new page set collects the same sort of data as the old page set - but far more quickly.

Current status: - new pageset runs to completion on all 5 supported talos platforms as long as upgraded flash is installed. What we are waiting on for roll-out: - solution to getting new flash on the talos slaves, hopefully through bug 475383

(Mentioned this to Vlad today, actually, figured he'd appreciate the cc)

Will need to be landed with matching buildbot changes that will have pagesets downloaded per run into page_load_test/tp3 and page_load_test/tp4 directories, also with updates to download newer flash plugins into the base_profile directory.

Comment on attachment 384153 [details] [diff] [review] [Checked in]add tp4 test to graph server >diff -r b13b2f40f565 sql/data.sql >--- a/sql/data.sql Wed Jun 10 11:38:35 2009 -0700 >+++ b/sql/data.sql Fri Jun 19 13:59:21 2009 -0700 >@@ -278,6 +278,7 @@ > insert into pagesets values (NULL,"SVG"); > insert into pagesets values (NULL,"Dromaeo"); > insert into pagesets values (NULL,"SunSpider"); >+insert into pagesets values (NULL, "Tp February, 2009 (100 pages)"); Can we call this "Tp4 February, 2009 (100 pages)", or "Tp #4 February, 2009 (100 pages)"?

Make it so that we can download and unzip plugins and pagesets per-talos run.

Also using defaults for location of plugins/pagesets - each talos buildbot will have a local directory called 'zips' where these things can be stored.

Just for moz-central to start and do some baking.

Comment on attachment 384999 [details] [diff] [review] turn on tp4 on moz-central exlusively >+# How many tp4 tests per build to run, and whether to merge build requests >+BRANCHES['mozilla-central']['tp4'] = (1,True) This needs to be set for the other branches, and maybe should be called 'tp4_tests' to be consistent with the other flags. >diff -r 3bf03608f457 talos-pool/master.cfg >--- a/talos-pool/master.cfg Wed Jun 24 18:56:34 2009 +1200 >+++ b/talos-pool/master.cfg Wed Jun 24 16:00:53 2009 -0700 >@@ -172,6 +172,40 @@ > c['builders'].append(builder) > branch_builders.append(builder['name']) > >+ if tp4_tests: You need to be setting tp4_tests somewhere, so you probably need a line like tp4_tests, tp4_merge = branch_config['tp4_tests']

Comment on attachment 384998 [details] [diff] [review] same updates for those talos set ups not using buildbotcustom Is UnpackFile available to use in talos-staging, talos, and talos-try?

Ah, cut and paste errors. Should be fixed now.

Comment on attachment 384153 [details] [diff] [review] [Checked in]add tp4 test to graph server 836396688321 2009-06-24 16:29 -0700

Comment on attachment 384155 [details] [diff] [review] [Checked in]add tp4 test to talos /cvsroot/mozilla/testing/performance/talos/sample.config,v <-- sample.config new revision: 1.30; previous revision: 1.29 done Removing page_load_test/manifest.txt; /cvsroot/mozilla/testing/performance/talos/page_load_test/manifest.txt,v <-- manifest.txt new revision: delete; previous revision: 1.1 done RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/tp3.manifest,v done Checking in page_load_test/tp3.manifest; /cvsroot/mozilla/testing/performance/talos/page_load_test/tp3.manifest,v <-- tp3.manifest initial revision: 1.1 done RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/tp4.manifest,v done Checking in page_load_test/tp4.manifest; /cvsroot/mozilla/testing/performance/talos/page_load_test/tp4.manifest,v <-- tp4.manifest initial revision: 1.1 done

Comment on attachment 384997 [details] [diff] [review] [Checked in]fix for name of downloaded fast cycle manifest changeset: 353:567121bf725b

Comment on attachment 385465 [details] [diff] [review] [Checked in]turn on tp4 on moz-central exclusively (take 2) changeset: 1247:1f6fc287876d

Comment on attachment 385473 [details] [diff] [review] [Checked in]same updates for those talos set ups not using buildbotcustom (take 2) changeset: 1248:ce8bec434f9f

Bustage fix for space issues on fast talos boxes: changeset: 1249:f9b009d30b6a tag: tip user: Alice Nodelman <alice@mozilla.com> date: Tue Jun 30 19:24:00 2009 -0700 summary: Bug 473821 - create new tp pageset that is newer and smaller (p/r=anodelman) bustage fix for space problems on some talos boxes

Same bustage fix for try changeset: 1250:0f065cc25be4 tag: tip user: Alice Nodelman <alice@mozilla.com> date: Tue Jun 30 20:27:39 2009 -0700 summary: Bug 473821 - create new tp pageset that is newer and smaller (p/r=anodelman) bustage fix for downloading zip on try talos

Tp4 now up and running on moz-central.

Fixes behavior on machines where we are not downloading the pageset per-run.

Comment on attachment 386336 [details] [diff] [review] [Checked in]use installed pageset for running tp3 instead of downloaded Checking in tp3.manifest; /cvsroot/mozilla/testing/performance/talos/page_load_test/tp3.manifest,v <-- tp3.manifest new revision: 1.2; previous revision: 1.1 done

Comment on attachment 386368 [details] [diff] [review] don't download plugins.zip/pagesets/zip except for tp4 builders Please expand the tabs into spaces. Looks good otherwise!

Comment on attachment 386368 [details] [diff] [review] don't download plugins.zip/pagesets/zip except for tp4 builders >diff -r 0f065cc25be4 talos-pool/master.cfg >--- a/talos-pool/master.cfg Tue Jun 30 20:27:39 2009 -0700 >+++ b/talos-pool/master.cfg Wed Jul 01 14:22:23 2009 -0700 >@@ -117,6 +117,8 @@ > configOptions=TALOS_CONFIG_OPTIONS, > talosCmd=TALOS_CMD, > fetchSymbols=branch_config['fetch_symbols'], >+ plugins='', >+ pagesets='', I think this parameter is called 'pageset' in the Factory.

As bug#483902 is approved+landed, we'll also need to approval+land this other part of the tp4 setup on 1.9.1.x.

Bug 475383 is the one we need landed on the 1.9.1 branch.

Comment on attachment 385465 [details] [diff] [review] [Checked in]turn on tp4 on moz-central exclusively (take 2) Can you make a corresponding patch for talos-staging-pool?

Bug 475383 landed on mozilla-1.9.1 yesterday. Time to turn this on in the talos pool?

Tp4 for all of talos-staging-pool, along with Tracemonkey and Firefox3.5

Comment on attachment 390329 [details] [diff] [review] [Checked in]tp4 everywhere but 1.9.0 TP4 FTW!

Comment on attachment 390329 [details] [diff] [review] [Checked in]tp4 everywhere but 1.9.0 changeset: 1398:5c47b32a4364

All that is left here is rolling out to mobile devices, but tp4 is considered final and is in production so this bug is considered complete.