Rob Campbell [:rc] (:robcee) Reporter
	Description • 17 years ago

Before we can mothball the old perf test machines, we need to provide a rapidly-cycling talos box that can reproduce as closely as possible what the old machines provide.

The requirements are: 10-30m cycle times.

Pared down pageset for running Tp.

Other tests include: Ts, Tdhtml, and Txul.

	Johnathan Nightingale [:johnath]
	Comment 1 • 17 years ago

I've been spouting for a little while to whomever would listen that a pared down talos pageset should be possible, even one that tracks the "real" talos numbers.  My thinking is that if we can get our hands on a stats software package, we oughta be able to run a regression analysis on each of the ~400 pages, and find the N pages that, taken together, correlate well with real Tp over the history we currently have.

N could be 50, or 30, or 10 -- whatever tradeoff we want to make between rapid sniff tests and probability of catching regressions.  I'm thinking that we name a cycle-time, 10 minutes say, and keep running down the list of "next most useful page" until we start hitting that.

We'll want to be clear that this isn't the real Tp, that full Talos Tp results win all arguments.  Tp_snifftest?  Tquickcheck?  Something.

What are the odds that we could get some kind of straight DB dump that has historical scores for every page?  Maybe the CSV dumps from graph server are enough...

	Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)
	Comment 2 • 17 years ago

The CSV dumps would be enough, but you'd have a hard time going through enough historical data to get them.  To do this kind of analysis you really just want a copy of the database.

That said, I bet someone could spend an afternoon and manually identify pages that would be useful to have in the set.

	Mike Schroepfer
	Comment 3 • 17 years ago

Can we just get a mysql dump of the db Aravind?

	Aravind Gottipati [:aravind]
	Comment 4 • 17 years ago

(In reply to comment #3)
> Can we just get a mysql dump of the db Aravind?
> 

Yup, people.mozilla.com:/tmp/graphs_mozilla_org.2008.01.24.sql.gz

	alice nodelman [:alice] [:anode] Assignee
	Comment 5 • 17 years ago

From a meeting today with me, schrep, robcee & joduinn our current thinking is as follows:

- we want rapidly cycling talos boxes so that we can retire tinderbox perf test boxes
- if terms of the fastest route to get us to that point we need to have tests that run as quickly as the tinderbox perf tests do
- thus, to start, we will just set up a set of talos boxes (win/mac/linux) to run the same tests that the current tinderbox perf test boxes are running (tp with the historic pageset, ts, tdhtml, txul, ts)

This has the advantage of not having to do any statistical analysis of the new page set to determine an interesting sub set of pages, along with pretty much exactly replacing the machines that we want to retire.

We can still pursue the concept of a small scale talos machine that runs pared down versions of all the tests we have (including tgfx, tsvg, etc) - but that would considered a separate project.

	Johnathan Nightingale [:johnath]
	Comment 6 • 17 years ago

That's much easier, I agree.  :)

	Mike Schroepfer
	Comment 7 • 17 years ago

(In reply to comment #5)
> From a meeting today with me, schrep, robcee & joduinn our current thinking is
> as follows:

++ to this plan.   

	Mike Schroepfer
	Comment 8 • 17 years ago

If it is *super* easy it would be great to get a set of 1.8 boxes running the same tests - I'm curious on how we are doing on the old Tp pageset on branch vs. trunk.   

	Mikeal Rogers [:mikeal] (mrogers@)
	Comment 9 • 17 years ago

Created new task for mini talos run;

https://bugzilla.mozilla.org/show_bug.cgi?id=419105

	alice nodelman [:alice] [:anode] Assignee
	Comment 10 • 17 years ago

We now have both a set of trunk fast and branch fast machines reporting.  See Firefox & Mozilla1.8 waterfalls.

	(not currently active) Ted Mielczarek
	Comment 11 • 16 years ago

Mass move of Core:Testing bugs to mozilla.org:ReleaseEngineering. Filter on RelEngMassMove to ignore.