Comment on attachment 646175 [details] [diff] [review] buildbotconfig changes to support mountain lion I think the patch is good but I don't have time to review it before I head out. You also need a patch for buildbotcustom. If you do the following you'll see what I mean: ./setup-master.py bm bm04-tests1-macosx cd bm && buildbot checkconfig .

Comment on attachment 646175 [details] [diff] [review] buildbotconfig changes to support mountain lion Review of attachment 646175 [details] [diff] [review]: ----------------------------------------------------------------- r+ with the nits fixed. ::: mozilla-tests/config.py @@ +85,5 @@ > PLATFORMS['macosx64']['env_name'] = 'mac-perf' > PLATFORMS['macosx64']['leopard'] = {'name': "Rev3 MacOSX Leopard 10.5.8"} > PLATFORMS['macosx64']['snowleopard'] = {'name': "Rev4 MacOSX Snow Leopard 10.6"} > PLATFORMS['macosx64']['lion'] = {'name': "Rev4 MacOSX Lion 10.7"} > +PLATFORMS['macosx64']['mountainlion'] = {'name': "Rev5 MacOSX Lion 10.8"} Rev5 MacOSX Mountain Lion 10.8 ::: mozilla-tests/preproduction_config.py @@ +8,4 @@ > 'leopard': dict([("talos-r3-leopard-%03i" % x, {}) for x in range(1,67)]), > 'snowleopard': dict([("talos-r4-snow-%03i" % x, {}) for x in range(1,81) + range(82,85) + [86]]), > 'lion': dict([("talos-r4-lion-%03i" % x, {}) for x in range(1,83) + [84]]), > + 'mountainlion': dict([("talos-mtnlion-r5-%03i" % x, {}) for x in range(1,90)]), range(1,90) will give you 1-89 because range() is dumb. Do you mean range(1,91)? ::: mozilla-tests/production_config.py @@ +9,4 @@ > if x not in [7]]), # bug 655437 > 'snowleopard': dict([("talos-r4-snow-%03i" % x, {}) for x in range(4,10) + range(11,81) + [82,84]]), > 'lion': dict([("talos-r4-lion-%03i" % x, {}) for x in range(4,10) + range(11,83) + [84]]), > + 'mountainlion': dict([("talos-mtnlion-r5-%03i" % x, {}) for x in range(1,90)]), Same question about range() ::: mozilla-tests/staging_config.py @@ +8,4 @@ > 'leopard': dict([("talos-r3-leopard-%03i" % x, {}) for x in range(1,67)]), > 'snowleopard': dict([("talos-r4-snow-%03i" % x, {}) for x in range(1,81) + range(82,85) + [86]]), > 'lion': dict([("talos-r4-lion-%03i" % x, {}) for x in range(1,83) + [84]]), > + 'mountainlion': dict([("talos-mtnlion-r5-%03i" % x, {}) for x in range(1,90)]), Same question about range()

Comment on attachment 649765 [details] [diff] [review] buildbot-config changes to support mountain lion Backed out: http://hg.mozilla.org/build/buildbot-configs/rev/cb0cbaa5d5de

Rail tried to do a reconfig and it failed because these patches were missing. I didn't know there were changes that were required to the puppet masters to support a new platform.

I left the first patch live in buildbotcustom. The problem with the configs patch was bm21 failing checkconfig: KeyError: 'mountainlion' To make masters understand the new platform you need: 1) update http://hg.mozilla.org/build/puppet-manifests/file/tip/modules/buildmaster/templates/BuildSlaves-tests.py.erb and add new platform 2) update http://hg.mozilla.org/build/puppet-manifests/file/tip/secrets.pp.template and add new variable 3) update secretes.pp on master-puppet1 with real passwords

So once my puppet-manifest patch in comment #6 is approved I'd like to try another reconfig tomorrow :-) I'm also curious about how I didn't encounter this issue when running the tests on my dev master. What could have I done during testing to avoid this problem cropping up during the reconfig?

Comment on attachment 649765 [details] [diff] [review] buildbot-config changes to support mountain lion Checked in again, will try another reconfig this afternoon.

In production.

Didn't get a reply in #build & need to head home soon, so posting here instead: 21:49:05 - edmorley: nthomas, coop: please may you cancel the pending 10.8 builds that are on m-c + inbound (plus presumably elsewhere), since the mountain lion patch has been backed out so philor says they'll now be stuck pending 21:49:34 - edmorley: they are also showing up as OS X opt (formerly 10.5), tbpl needs a patch (but I can take a look at that tomorrow hopefully)

I deleted 1611 10.8 test jobs from the buildbot db, since these are new anyway. kmoir has been working on enabling the slaves, but we've hit a potential issue with 'sudo reboot' not working without auth for cltbld. That's required at the end of each job.

(In reply to Nick Thomas [:nthomas] from comment #12) > I deleted 1611 10.8 test jobs from the buildbot db Awesome, thank you :-)

I fixed the sudo reboot issue, and rebooted the other slaves early this evening. Now I'm working on some other issues after a long conversation with catlee in IRC.

Please may we kill off some more 10.8 jobs: Pending test(s) @ Aug 15 02:15:03 mac10.8 (477) 341 mozilla-inbound 28 ionmonkey 27 fx-team 27 build-system 27 birch 17 services-central 10 profiling Pending test(s) @ Aug 15 02:15:03 mac10.8 (342) 342 try

Ok, 1181 deleted. I think we should probably stop scheduling these jobs if we're blocked on setting up new masters in scl3, especially with IT all in GMT at the moment. That gives a chance to put some of these slaves on a staging master and chuck some more jobs at them to shake out any other bugs. eg, I don't see any mention of graph server sql additions here, so I think talos will have a problem posting perf results. See https://wiki.mozilla.org/ReleaseEngineering:GraphServer for clues. Perhaps we need a checklist for adding new machine classes ? AFAIK this stuff is all in the heads of old-timers.

I just deleted some more jobs. I'm writing a checklist of all the things that need to be done to a a new platform that I have encountered so far. I agree that we should probably back out these changes since obviously there are a lot of issues that need to be resolved that I wasn't aware of and we need new test masters implemented.

Until I can run more tests in the staging environment and we have a new test master up (bug 782870). If I understand correctly, only the buildbot-configs patch needs to be reverted to stop the builds from queueing?

Sorry to be a pain, but please may we have another round of 10.8 job deletion :-)

Ed, I just deleted some more jobs :-)

In production.

this didn't show up until I ran tests last night

Comment on attachment 655039 [details] [diff] [review] patch to add mountain lion key for jetpack builds Feel free to remove all trailing spaces from buildfarm/utils/run_jetpack.py ;)

with the new way of defining slave ranges in {production,preproduction,staging}_config.py.

Comment on attachment 655803 [details] [diff] [review] unbitrotted backout of http://hg.mozilla.org/build/buildbot-configs/rev/08fa3688ec70 The only differences here are the _config.py slave lists and unbitrotting.

closing

We only have builders on esr10, mozilla-release, mozilla-beta (dep and release automation). philor points out landing race with http://hg.mozilla.org/build/buildbot-configs/rev/613642b19123 is likely to be the issue.

I've verified this brings up to the same number of builders as we have for 10.7, across all branches.

Comment on attachment 655938 [details] [diff] [review] Add mountainlion for m-c + peers lgtm

Comment on attachment 655938 [details] [diff] [review] Add mountainlion for m-c + peers http://hg.mozilla.org/build/buildbot-configs/rev/20e2fcf03f23 This will add builders for aurora, despite the request in bug 786424. I'm figuring we need to green those tests up anyway.

Comment on attachment 655938 [details] [diff] [review] Add mountainlion for m-c + peers This made it to production today.

Closing.