alice nodelman [:alice] [:anode] Reporter
	Description • 17 years ago

Right now we are depending on an ad hoc file name system to differentiate between different types of config files (per platforms, per machine, per talos version).  This is already confusing and will only get worse as we add more different types of machine configurations.

We need to standardize upon a directory structure/filename structure so that we can tell which config is being applied to which machine.

	bhearsum@mozilla.com (:bhearsum) Assignee
	Comment 1 • 17 years ago

How does this sound:
configs/$PLATFORM/$PAGELOADER/file
where $PLATFORM is 'mac', 'linux', or 'win32' and $PAGELOADER is 'js' or 'chrome'.
So, it would end up like this:
configs/linux/js/config.py
configs/linux/js/sample.config
configs/win32/chrome/config.py
configs/win32/chrome/sample.config
etc.

We also need a place to store scripts that are transferred to the slaves. I propose that we have a slave-scripts/ subdirectory for those.

	Rob Campbell [:rc] (:robcee)
	Comment 2 • 17 years ago

This is a good start. I'm worried about having differing config files for differently-purposed machines. I guess we could skip the download step for those slave instances, and store a local copy on the talos box. Or spin off a branched config dir, like, configs/win32-trunk-nocairo/ but that could get messy.

we'll likely get into the same issue with slave-scripts and the different platforms.

Maybe an extra midlevel directory to keep track of purpose/platform:

configs/trunk/linux
configs/trunk/win32
...

it's getting deep, but that's consistent with the mozconfig layout in build and the unittest machines.

	bhearsum@mozilla.com (:bhearsum) Assignee
	Comment 3 • 17 years ago

I knew I was forgetting something (branch/trunk). We do need to keep a distinction between the pageloaders somewhere since the configuration will be different for the two. We could do it one of two ways:
configs/$TREE-$PAGELOADER/$PLATFORM/config.py
etc.
or
configs/$TREE/$PLATFORM/$PAGELOADER-config.py
etc.

Where $TREE could be "1.9" or "1.8". (I think it's important to use numbers rather than names. The slaves report to the graph server with numbers. Also, when we have a 1.9.1 or 2.0 using trunk/branch could get confusing.)
$PLATFORM is linux/mac/win32 and $PAGELOADER is js/chrome.

As for the slave scripts, I don't see that directory getting very big...maybe it's enough to have a subdirectory per platform?

As for things like nocairo...are we planning to put them into CVS? My impression was that they would be relatively short term. Either way, if we want them in CVS we could create a subdirectory of $PLATFORM that describes the purpose. The configs without a subdirectory would be the "default", on-check-in configs.
So, it could look like this:
configs/$TREE/$PLATFORM/$PAGELOADER-config.py
configs/$TREE/$PLATFORM/$PAGELOADER-sample.config
configs/$TREE/$PLATFORM/$PURPOSE/$PAGELOADER-config.py
configs/$TREE/$PLATFORM/$PURPOSE/$PAGELOADER-sample.config

	bhearsum@mozilla.com (:bhearsum) Assignee
	Comment 4 • 17 years ago

I added "and update" to the title. As part of the organization patch we should merge in the latest changes from the talos tree.

	alice nodelman [:alice] [:anode] Reporter
	Comment 5 • 17 years ago

Are we going to be checking in more experimental style configs?  Say like the ones I've been using to test different amount of cycles through the web page set?  I would assume that these could just live on the buildbot master, but was wondering if that kind of config is being considered for this scheme.

	alice nodelman [:alice] [:anode] Reporter
	Comment 6 • 17 years ago

Also - how much pain are we creating for ourselves when the master config file changes?  Or is that just inevitable?

	bhearsum@mozilla.com (:bhearsum) Assignee
	Comment 7 • 17 years ago

(In reply to comment #5)
> Are we going to be checking in more experimental style configs?  Say like the
> ones I've been using to test different amount of cycles through the web page
> set?

Unless the config is being used on a Talos slave that will be in use for awhile (I don't know what the definition of "awhile" is yet), I don't think there's much point in checking them in. Experimental configs change so much that it's a pain to keep them up to date in CVS. Once something gets to a "stable" state (read: not going to change all the time) it should be checked in.

> I would assume that these could just live on the buildbot master, but was
> wondering if that kind of config is being considered for this scheme.

I think we should have a separate buildmaster for temporary/one-off things (like we do for baselining). The main 'perfmaster' _should_ be kept in line with CVS as much as possible. I think it's a good rule that when something is ready to move to perfmaster it is ready to be checked-in.

(In reply to comment #6)
> Also - how much pain are we creating for ourselves when the master config file
> changes?  Or is that just inevitable?

There's one reference to each config file in the master.cfg. Anytime these files paths/names change the master.cfg needs to be updated. It's a pretty painless thing, actually.

	bhearsum@mozilla.com (:bhearsum) Assignee
	Comment 8 • 17 years ago

This patch does a few things:
1) Adds the new builders that were moved to perfmaster last week.
2) Makes all builders automatically update themselves to the latest talos on every perf run.
3) Makes all builders use relative paths for filenames
4) Increases the Buildbot timeout for the perf run to 11000 (Talos timeout (10000) + 1000)
5) Removes all existing config.pys and sample.configs and replaces them with two new ones, sample.config.js and sample.config.pageloader that are used with the js framecycler and pageloader respectively.

I want to test this quite a bit before commiting it, some builders (qm-pxp* in paticular) may need a bit of massaging to come to terms with the update.

This shouldn't be commited until both the testing is done and the patch in bug 396232 is committed.

	bhearsum@mozilla.com (:bhearsum) Assignee
	Comment 10 • 17 years ago

I couldn't add generate-tpcomponent.py into that patch (cvsdo won't add directories for some reason...) so here it is. It needs to be put in scripts/generate-tpcomponent.py for things to work properly.

This is the same script as the one in bug 396095

	bhearsum@mozilla.com (:bhearsum) Assignee
	Comment 11 • 17 years ago

This version doesn't touch the qm-pxp* machines at all and leaves in the config files (sample.config.win32 and config-win32-pxp.py) that they use.

	bhearsum@mozilla.com (:bhearsum) Assignee
	Comment 12 • 17 years ago

I should note that this new patch makes me a lot less worried about breaking things when we update perfmaster since all of the machines it affects aren't really considered "production" (ie. they are on MozillaTest/graphs-stage)

	Rob Campbell [:rc] (:robcee)
	Comment 13 • 17 years ago

Comment on attachment 281498 [details] [diff] [review]
same as before, doesn't touch qm-pxp* machines

only nit is that the s() method for adding buildsteps to factories has been deprecated.

	alice nodelman [:alice] [:anode] Reporter
	Comment 14 • 17 years ago

Comment on attachment 281487 [details]
generate-tpcomponent.py

I r+ed this in bug 396095, so I consider this one obsolete (I guess).

	alice nodelman [:alice] [:anode] Reporter
	Comment 15 • 17 years ago

Comment on attachment 281498 [details] [diff] [review]
same as before, doesn't touch qm-pxp* machines

Looks like you removed the unix_counters:

+    win_counters : ['Working Set', 'Private Bytes', '% Processor Time']
+    unix_counters : []

You need to have the unix counters present so that the config file will work on mac/linux - the code is written in such a way to ensure that it won't be used on windows machines.

This is for both sample.config.js and sample.config.pageloader

	bhearsum@mozilla.com (:bhearsum) Assignee
	Comment 16 • 17 years ago

I removed the s() method of adding BuildSteps in the this version. Additionally, I noticed that I could removed one of the BuildFactorys due to the configuration files being the same. I also removed buildbot.process.step usage from master.cfg and perfrunner.py as it is also deprecated.

I ran this through checkconfig.py and it passed.

	alice nodelman [:alice] [:anode] Reporter
	Comment 17 • 17 years ago

Comment on attachment 281517 [details] [diff] [review]
functionally the same as before, avoids deprecated things

Same as comment 15.

	bhearsum@mozilla.com (:bhearsum) Assignee
	Comment 19 • 17 years ago

I swear, this one is right.

	alice nodelman [:alice] [:anode] Reporter
	Comment 20 • 17 years ago

Comment on attachment 281524 [details] [diff] [review]
same as before, but the right patch

The config files look correct now.

	bhearsum@mozilla.com (:bhearsum) Assignee
	Comment 21 • 17 years ago

This patch removes all of the unused scripts and files and supports auto-update Talos. These exact configurations are being used on perfmaster right now. All of the builders are green.

	bhearsum@mozilla.com (:bhearsum) Assignee
	Comment 22 • 17 years ago

The slaves have done 4-6 runs each and in all cases the numbers for each are
within a 10ms range.

http://graphs-stage.mozilla.org/#spst=range&spss=1190633105.5528572&spse=1190652641&spstart=1190505600&spend=1190652641&bpst=cursor&bpstart=1190633105.5528572&bpend=1190652641&m1tid=15423&m1bl=0&m1avg=0&m2tid=14699&m2bl=0&m2avg=0&m3tid=31975&m3bl=0&m3avg=0&m4tid=31975&m4bl=0&m4avg=0&m5tid=15613&m5bl=0&m5avg=0&m6tid=15677&m6bl=0&m6avg=0&m7tid=4021&m7bl=0&m7avg=0&m8tid=328&m8bl=0&m8avg=0

	alice nodelman [:alice] [:anode] Reporter
	Comment 23 • 17 years ago

Comment on attachment 282133 [details] [diff] [review]
auto-update perfmaster configs, final

The configs look correct to me, and the data collected on graphs-stage.m.o is within normal ranges.

	Rob Campbell [:rc] (:robcee)
	Comment 24 • 17 years ago

Comment on attachment 282133 [details] [diff] [review]
auto-update perfmaster configs, final

unecessary trailing commas in lines 72, 83, 94, ...

otherwise it looks fine. 1700 lines is a long patch to review. Hope this is tested.

	bhearsum@mozilla.com (:bhearsum) Assignee
	Comment 25 • 17 years ago

These configs are exactly what's been running on perfmaster since Monday morning.

	bhearsum@mozilla.com (:bhearsum) Assignee
	Comment 26 • 17 years ago

Checking in master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/master.cfg,v  <--  master.cfg
new revision: 1.6; previous revision: 1.5
done
Checking in perfrunner.py;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/perfrunner.py,v  <--  perfrunner.py
new revision: 1.4; previous revision: 1.3
done
Removing mountdmg.exp;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/mountdmg.exp,v  <--  mountdmg.exp
new revision: delete; previous revision: 1.1
done
Removing configs/config-linux.py;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/configs/config-linux.py,v  <--  config-linux.py
new revision: delete; previous revision: 1.2
done
Removing configs/config-mac.py;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/configs/config-mac.py,v  <--  config-mac.py
new revision: delete; previous revision: 1.1
done
Removing configs/config-new-pl.py;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/configs/config-new-pl.py,v  <--  config-new-pl.py
new revision: delete; previous revision: 1.1
done
Removing configs/config-win32.py;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/configs/config-win32.py,v  <--  config-win32.py
new revision: delete; previous revision: 1.1
done
Removing configs/hdiutil-expect.ex;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/configs/hdiutil-expect.ex,v  <--  hdiutil-expect.ex
new revision: delete; previous revision: 1.1
done
Removing configs/mountdmg.sh;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/configs/mountdmg.sh,v  <--  mountdmg.sh
new revision: delete; previous revision: 1.1
done
RCS file: /cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/configs/sample.config.js,v
done
Checking in configs/sample.config.js;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/configs/sample.config.js,v  <--  sample.config.js
initial revision: 1.1
done
Removing configs/sample.config.linux;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/configs/sample.config.linux,v  <--  sample.config.linux
new revision: delete; previous revision: 1.2
done
Removing configs/sample.config.mac;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/configs/sample.config.mac,v  <--  sample.config.mac
new revision: delete; previous revision: 1.1
done
Removing configs/sample.config.newpl.linux;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/configs/sample.config.newpl.linux,v  <--  sample.config.newpl.linux
new revision: delete; previous revision: 1.1
done
Removing configs/sample.config.newpl.win32;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/configs/sample.config.newpl.win32,v  <--  sample.config.newpl.win32
new revision: delete; previous revision: 1.2
done
RCS file: /cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/configs/sample.config.pageloader,v
done
Checking in configs/sample.config.pageloader;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/configs/sample.config.pageloader,v  <--  sample.config.pageloader
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/scripts/generate-tpcomponent.py,v
done
Checking in scripts/generate-tpcomponent.py;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/scripts/generate-tpcomponent.py,v  <--  generate-tpcomponent.py
initial revision: 1.1

	(not currently active) Ted Mielczarek
	Comment 27 • 16 years ago

Mass move of Core:Testing bugs to mozilla.org:ReleaseEngineering. Filter on RelEngMassMove to ignore.