The goal here is to move the Firefox builders on the 1.8 tree over to the release automation system (bug 410936). Initially, these machines will build in a continuous cycle, although we should have the option of easily moving to only building on checkin.

Comment on attachment 302981 [details] [diff] [review] [checked in] staging and production configs for 1.8 tree rollout This looks fine. Building every 5 minutes seems a bit excessive but I guess having a bunch of builds queued won't hurt (which I guess won't happen as much when we have a second buildslave here).

Comment on attachment 302981 [details] [diff] [review] [checked in] staging and production configs for 1.8 tree rollout Checking in production/master.cfg; /cvsroot/mozilla/tools/buildbot-configs/automation/production/master.cfg,v <-- master.cfg new revision: 1.16; previous revision: 1.15 done Checking in staging/master.cfg; /cvsroot/mozilla/tools/buildbot-configs/automation/staging/master.cfg,v <-- master.cfg new revision: 1.26; previous revision: 1.25 done

There isn't much that needs to be tweaked in here. Nick, I notice the staging win32 builder fails to upload symbols; should this be turned off in nightly mode?

(In reply to comment #4) > Created an attachment (id=303107) [details] > create nightly 1.8 bootstrap.cfg > > There isn't much that needs to be tweaked in here. > > Nick, I notice the staging win32 builder fails to upload symbols; should this > be turned off in nightly mode? Actually.. production as of right now is using the tagged tinderbox (bug 397554 is about figuring out the best way to do this, but I think we can safely start the initial "run in parallel with existing builders" rollout without this), so maybe we don't have to worry about this. This also makes me wonder if we should reconsider and have production nightly come from a tag and staging use HEAD.. it's more virtual paperwork, but it'd certainly keep production a lot more stable.

Comment on attachment 303140 [details] [diff] [review] do not assert if TinderConfig dir already exists in nightly mode sorry, this is handled in master.cfg, i was testing directly and forgot.

Did a little testing in production, and made the following changes: * stagingServer and externalStagingServer should be stage/ftp * support new vars defined in bug 397554 ** I tested this patch in order to set up Tinderbox, I think we should land it and take it for this * use MOZILLA_1_8_BRANCH_test ** we need to run these in parallel for a while, and we can't have them pushing to the default nightly location :/ besides that, TinderConfig step does what we need

Comment on attachment 303186 [details] [diff] [review] change path used for clobbers (should land on MOZILLA_1_8_BRANCH_test branch) What's the plan for nightly updates while both the existing and production boxes are running ? Having both sets of machines publishing info to aus2 doesn't seem like a good idea. Perhaps set $update_pushinfo to 0 on the prod boxes until any bugs are shaken out. If that sounds ok to you, then go ahead and make the change when you land this.

(In reply to comment #10) > (From update of attachment 303186 [details] [diff] [review]) > What's the plan for nightly updates while both the existing and production > boxes are running ? Having both sets of machines > publishing info to aus2 doesn't seem like a good idea. Perhaps set > $update_pushinfo to 0 on the prod boxes until any bugs are shaken out. If that > sounds ok to you, then go ahead and make the change when you land this. Yes, good catch.. thanks!

Comment on attachment 303185 [details] [diff] [review] more production like, and support new tinderbox setup patch >Index: fx-moz18-nightly-bootstrap.cfg >=================================================================== ... >+branchTag = MOZILLA_1_8_BRANCH_testing You talk about MOZILLA_1_8_BRANCH_test in the bug comments, choose one! >+from = bootstrap@mozilla.org >+to = build-announce@mozilla.com .org, here and in the other files > sshUser = cltbld >-sshServer = staging-1.8-master.build.mozilla.org >+sshServer = production-1.8-master.build.mozilla.org As we discussed on IRC, use stage.m.o here (and the tinder-config.pl patch puts the files in nightly/experimental) >+testsPhoneHome = 1 This is safe because $GraphNameOverride isn't set on any of the Fx/1.8 configs ?

Checking in linux/tinder-config.pl; /cvsroot/mozilla/tools/tinderbox-configs/firefox/linux/tinder-config.pl,v <-- tinder-config.pl new revision: 1.1.12.18.2.1; previous revision: 1.1.12.18 done Checking in macosx/tinder-config.pl; /cvsroot/mozilla/tools/tinderbox-configs/firefox/macosx/tinder-config.pl,v <-- tinder-config.pl new revision: 1.8.4.17.2.1; previous revision: 1.8.4.17 done Checking in win32/tinder-config.pl; /cvsroot/mozilla/tools/tinderbox-configs/firefox/win32/tinder-config.pl,v <-- tinder-config.pl new revision: 1.2.10.19.2.1; previous revision: 1.2.10.19 done

Checking in fx-moz18-nightly-bootstrap.cfg; /cvsroot/mozilla/tools/release/configs/fx-moz18-nightly-bootstrap.cfg,v <-- fx-moz18-nightly-bootstrap.cfg new revision: 1.8; previous revision: 1.7 done RCS file: /cvsroot/mozilla/tools/release/configs/fx-moz18-nightly-staging-bootstrap.cfg,v done

This should keep the MOZILLA_1_8_BRANCH and MOZILLA_1_8_BRANCH_test builders from clobbering eachother's builds or talkback symbols. It's still possible for it to happen of course (e.g. clobber fails once and fixed on the other's build hour, or clobber forced at the other's build hour), but we only need to put up with this for a week or so, then _test goes away and becomes production. 1.8 branch isn't super active so this isn't real likely, anyway.

(In reply to comment #0) > The goal here is to move the Firefox builders on the 1.8 tree over to the > release automation system (bug 410936). Initially, these machines will build in > a continuous cycle, although we should have the option of easily moving to only > building on checkin. > Er, that should be bug 401936 of course (set correctly in the blockers, which I can edit, ironically! grr)

Comment on attachment 303394 [details] [diff] [review] [checked in] MOZILLA_1_8_BRANCH_test should use a different build hour Checking in linux/tinder-config.pl; /cvsroot/mozilla/tools/tinderbox-configs/firefox/linux/tinder-config.pl,v <-- tinder-config.pl new revision: 1.1.12.18.2.2; previous revision: 1.1.12.18.2.1 done Checking in macosx/tinder-config.pl; /cvsroot/mozilla/tools/tinderbox-configs/firefox/macosx/tinder-config.pl,v <-- tinder-config.pl new revision: 1.8.4.17.2.2; previous revision: 1.8.4.17.2.1 done Checking in win32/tinder-config.pl; /cvsroot/mozilla/tools/tinderbox-configs/firefox/win32/tinder-config.pl,v <-- tinder-config.pl new revision: 1.2.10.19.2.2; previous revision: 1.2.10.19.2.1 done

Ok, I had to symlink in the startup-test.html for the Mac, but now all builders are green! I am going to go ahead and unhide and announce. I'll put a note on Tinderbox. The current plan is to run these for a week, we'll replace the old systems ASAP after Feb 25th.

This has been running for the past week with no problems (that I know of). Going to wait on 1.8 Talos migration before disabling the older machines and making this push to default nightly build and update locations.

(In reply to comment #19) > This has been running for the past week with no problems (that I know of). > > Going to wait on 1.8 Talos migration before disabling the older machines and > making this push to default nightly build and update locations. Talos is up, we're going to point the old perf machines to the new boxes for a while to make sure nothing funky is going on first.

As previously announced on Tinderbox and planet (http://roberthelmer.com/blog/?p=29), we're migrating nightly production to running on the same machines as release production. On the moz1.8 branch, we've been running the new nightlies in parallel with the "traditional" nightlies since Feb 15 2008, and are going to switchover live tomorrow. The new machines: * production-pacifica-vm * production-prometheus-vm * bm-xserve05 The old machines: * pacifica-vm * prometheus-vm * bm-xserve02 Starting tomorrow, the performance machines will begin following the new machines. The new machines will publish updates and nightly builds to the usual location, and the old machines will be disabled (but kept around for a while, just in case). If there is a reason that we should not proceed, or if you see any problems after the migration, please update this bug or email build@mozilla.org.

Comment on attachment 307304 [details] [diff] [review] [checked in] point perf machines at release boxes r+

Comment on attachment 307304 [details] [diff] [review] [checked in] point perf machines at release boxes Checking in firefox/linux/tinder-config.pl; /cvsroot/mozilla/tools/tinderbox-configs/firefox/linux/tinder-config.pl,v <-- tinder-config.pl new revision: 1.1.10.18; previous revision: 1.1.10.17 done Checking in firefox/win32/tinder-config.pl; /cvsroot/mozilla/tools/tinderbox-configs/firefox/win32/tinder-config.pl,v <-- tinder-config.pl new revision: 1.2.8.17; previous revision: 1.2.8.16 done

Comment on attachment 307515 [details] [diff] [review] [checked in] update monitor config for 1.8 change Checking in monitoring/Tier1_Mozilla1.8.txt; /cvsroot/mozilla/tools/tinderbox-configs/monitoring/Tier1_Mozilla1.8.txt,v <-- Tier1_Mozilla1.8.txt new revision: 1.5; previous revision: 1.4 done

Now that the Talos change in bug 421114 has landed, I went ahead and: * disabled tinderbox on pacifica-vm/prometheus-vm * disabled the Fx build on bm-xserve02 * set fx-1.8-nightly-bootstrap.cfg to use MOZILLA_1_8_BRANCH not MOZILLA_1_8_BRANCH_test (patch upcoming) I went ahead and hid the old Tinderboxes, and will hide the old perf reporters (they have the hostname in their ExtraName) as the new reporters come in.

Looks like the new boxes never got moved their configs moved from the MOZILLA_1_8_BRANCH_test branch to MOZILLA_1_8_BRANCH, so no perf testing is going on. We'll also need to remove the $ssh_key definitions since cltbld doesn't match the ffxbld key.

Comment on attachment 307600 [details] [diff] [review] [checked in] use MOZILLA_1_8_BRANCH for nightly bootrap.cfg [Note to future archaeologists, this is so the automation bumps the tinderbox configs from the right branch]

(In reply to comment #29) > Looks like the new boxes never got moved their configs moved from the > MOZILLA_1_8_BRANCH_test branch to MOZILLA_1_8_BRANCH, so no perf testing is > going on. We'll also need to remove the $ssh_key definitions since cltbld > doesn't match the ffxbld key. Hmm, the perf boxes use tinderbox-builds dir, so I don't think this is why Talos stopped testing (the test-only-tinderboxes are still going, for instance). However, you're right that the boxes needed their configs manually moved to the MOZILLA_1_8_BRANCH, otherwise nightlies and updates don't get published. Should we remove the $ssh_key definitions, or just explicitly put in ffxbld? Seems like we'd want the former, right? I'll force a clobber once we figure this out, to make sure nightlies go ok.

Comment on attachment 307600 [details] [diff] [review] [checked in] use MOZILLA_1_8_BRANCH for nightly bootrap.cfg Checking in fx-moz18-nightly-bootstrap.cfg; /cvsroot/mozilla/tools/release/configs/fx-moz18-nightly-bootstrap.cfg,v <-- fx-moz18-nightly-bootstrap.cfg new revision: 1.10; previous revision: 1.9 done

Comment on attachment 307723 [details] [diff] [review] [checked in] use ffxbld user for nightly/depend builders Checking in tinderbox-configs/firefox/win32/tinder-config.pl; /cvsroot/mozilla/tools/tinderbox-configs/firefox/win32/tinder-config.pl,v <-- tinder-config.pl new revision: 1.2.10.23; previous revision: 1.2.10.22 done Checking in tinderbox-configs/firefox/linux/tinder-config.pl; /cvsroot/mozilla/tools/tinderbox-configs/firefox/linux/tinder-config.pl,v <-- tinder-config.pl new revision: 1.1.12.22; previous revision: 1.1.12.21 done Checking in tinderbox-configs/firefox/macosx/tinder-config.pl; /cvsroot/mozilla/tools/tinderbox-configs/firefox/macosx/tinder-config.pl,v <-- tinder-config.pl new revision: 1.8.4.21; previous revision: 1.8.4.20 done Checking in release/configs/fx-moz18-nightly-bootstrap.cfg; /cvsroot/mozilla/tools/release/configs/fx-moz18-nightly-bootstrap.cfg,v <-- fx-moz18-nightly-bootstrap.cfg new revision: 1.11; previous revision: 1.10 done

I just clobbered these boxes, so we can make sure that we get good nightlies and updates.

Alright, looks like depend and clobber is working on all platforms.

There's a problem with pushing nightly update snippets to AUS, related to having a stale tinderbox checkout. We need rev1.141 of post-mozilla-rel.pl, which uses $ssh_key to push the update snippet (if specified). I've gone cvs up in /builds/tinderbox/mozilla/tools/tinderbox on the three production machines. Bug 397554 should help in the long term. Was surprised tinderbox wasn't on a tag like RELEASE_AUTOMATION_M7_1 though.