We should finish moving all remaining minis to the colo. Lets start by figuring out which slaves are in MTV still and create batches to take them to the colo.

Based on conversations with jhford, it appears that these are the remaining machines: talos-rev2-tiger15 IN CNAME talos-rev2-tiger15.build.mtv1.mozilla.com. talos-rev2-x6408 IN CNAME talos-rev2-x648.build.mtv1.mozilla.com. talos-r3-w7-001 IN CNAME talos-r3-w7-001.build.mtv1.mozilla.com. talos-r3-w7-ref IN CNAME talos-r3-w7-ref.build.mtv1.mozilla.com. talos-r3-fed-ref IN CNAME talos-r3-fed-ref.build.mtv1.mozilla.com. talos-r3-fed64-ref IN CNAME talos-r3-fed64-ref.build.mtv1.mozilla.com. talos-r3-leopard-013 IN CNAME talos-r3-leopard-013.build.mtv1.mozilla.com. talos-r3-leopard-014 IN CNAME talos-r3-leopard-14.build.mtv1.mozilla.com. talos-r3-leopard-016 IN CNAME talos-r3-leopard-16.build.mtv1.mozilla.com. talos-r3-leopard-017 IN CNAME talos-r3-leopard-17.build.mtv1.mozilla.com. talos-r3-leopard-019 IN CNAME talos-r3-leopard-19.build.mtv1.mozilla.com. talos-r3-w7-002 IN CNAME talos-r3-w7-2.build.mtv1.mozilla.com. talos-r3-w7-003 IN CNAME talos-r3-w7-3.build.mtv1.mozilla.com. talos-r3-w7-004 IN CNAME talos-r3-w7-4.build.mtv1.mozilla.com. talos-r3-w7-006 IN CNAME talos-r3-w7-6.build.mtv1.mozilla.com. talos-r3-w7-007 IN CNAME talos-r3-w7-7.build.mtv1.mozilla.com. talos-r3-w7-008 IN CNAME talos-r3-w7-8.build.mtv1.mozilla.com. talos-r3-w7-009 IN CNAME talos-r3-w7-9.build.mtv1.mozilla.com. talos-r3-w7-010 IN CNAME talos-r3-w7-10.build.mtv1.mozilla.com. talos-r3-w7-012 IN CNAME talos-r3-w7-12.build.mtv1.mozilla.com. talos-r3-w7-013 IN CNAME talos-r3-w7-13.build.mtv1.mozilla.com. talos-r3-fed-020 IN CNAME talos-r3-fed-20.build.mtv1.mozilla.com. talos-r3-fed-009 IN CNAME talos-r3-fed-009.build.mtv1.mozilla.com. talos-r3-fed-010 IN CNAME talos-r3-fed-10.build.mtv1.mozilla.com. talos-r3-fed-018 IN CNAME talos-r3-fed-18.build.mtv1.mozilla.com. talos-r3-fed-019 IN CNAME talos-r3-fed-19.build.mtv1.mozilla.com. talos-r3-leopard-027 IN CNAME talos-r3-leopard-27.build.mtv1.mozilla.com. talos-r3-leopard-028 IN CNAME talos-r3-leopard-28.build.mtv1.mozilla.com. talos-r3-leopard-029 IN CNAME talos-r3-leopard-29.build.mtv1.mozilla.com. talos-r3-leopard-030 IN CNAME talos-r3-leopard-30.build.mtv1.mozilla.com. talos-r3-leopard-031 IN CNAME talos-r3-leopard-31.build.mtv1.mozilla.com. talos-r3-leopard-032 IN CNAME talos-r3-leopard-32.build.mtv1.mozilla.com. talos-r3-leopard-033 IN CNAME talos-r3-leopard-33.build.mtv1.mozilla.com. talos-r3-leopard-034 IN CNAME talos-r3-leopard-34.build.mtv1.mozilla.com. talos-r3-leopard-035 IN CNAME talos-r3-leopard-35.build.mtv1.mozilla.com. talos-r3-leopard-036 IN CNAME talos-r3-leopard-36.build.mtv1.mozilla.com. talos-r3-leopard-037 IN CNAME talos-r3-leopard-37.build.mtv1.mozilla.com. talos-r3-leopard-038 IN CNAME talos-r3-leopard-38.build.mtv1.mozilla.com. talos-r3-leopard-039 IN CNAME talos-r3-leopard-39.build.mtv1.mozilla.com. talos-r3-leopard-040 IN CNAME talos-r3-leopard-40.build.mtv1.mozilla.com. talos-r3-fed-029 IN CNAME talos-r3-fed-29.build.mtv1.mozilla.com. talos-r3-fed-030 IN CNAME talos-r3-fed-30.build.mtv1.mozilla.com. talos-r3-fed-034 IN CNAME talos-r3-fed-34.build.mtv1.mozilla.com. talos-r3-fed-035 IN CNAME talos-r3-fed-35.build.mtv1.mozilla.com. talos-r3-fed-036 IN CNAME talos-r3-fed-36.build.mtv1.mozilla.com. talos-r3-fed-037 IN CNAME talos-r3-fed-37.build.mtv1.mozilla.com. talos-r3-fed-038 IN CNAME talos-r3-fed-38.build.mtv1.mozilla.com. talos-r3-snow-013 IN CNAME talos-r3-snow-13.build.mtv1.mozilla.com. talos-r3-snow-014 IN CNAME talos-r3-snow-14.build.mtv1.mozilla.com. talos-r3-snow-015 IN CNAME talos-r3-snow-15.build.mtv1.mozilla.com. talos-r3-snow-016 IN CNAME talos-r3-snow-16.build.mtv1.mozilla.com. talos-r3-snow-018 IN CNAME talos-r3-snow-18.build.mtv1.mozilla.com. talos-r3-snow-020 IN CNAME talos-r3-snow-20.build.mtv1.mozilla.com. talos-r3-leopard-ref IN CNAME talos-r3-leopard-ref.build.mtv1.mozilla.com. talos-r3-snow-ref IN CNAME talos-r3-snow-ref.build.mtv1.mozilla.com. How shall we batch these?

The rev2 hosts are offline, ignore them.

Proposing two batches: Batch 1: talos-r3-fed-009,010,018,019,020,029 (6) talos-r3-leopard-013,014,016,017,019,027,028,029,030,031 (10) talos-r3-snow-013,014,015 (3) talos-r3-w7-001,002,003,004,006,007 (6) Batch 2: talos-r3-fed-030,034,035,036,037,038,ref (7) talos-r3-fed64-ref (1) talos-r3-leopard-032,033,034,035,036,037,038,039,040,ref (10) talos-r3-snow-016,018,020,ref (4) talos-r3-w7-008,009,010,012,013,ref (6) I'll have updated zone files ready to commit ahead of the move, and will physically tag the machines and power cords with color codes so we don't yank the wrong cables.

Doing a quick survey, all of the machines that I found were on the 2nd floor server room. Machines I can't find: All five -ref machines, talos-r3-fed-010 Machines I found and wasn't expecting: talos-r3-w7-14, leopard-003 Any pointers?

Some notes, for lack of a better place: fed-010 is live in scl1 w7-14 and leopard-003 are live in mtv1 Should I fix up DNS for the current state? The ref machines are on the 3rd floor, along with two machines that don't have hostname labels, but by the asset tags are in inventory w7-ref and w764-ref (machines labelled as such are sitting next to them) Not sure how to unravel those yet. Given that we have a relatively small number of machines for each OS, could we do all of this in one batch?

(In reply to comment #5) > Some notes, for lack of a better place: > > fed-010 is live in scl1 > w7-14 and leopard-003 are live in mtv1 > > Should I fix up DNS for the current state? Yes, please. > > The ref machines are on the 3rd floor, along with two machines that don't have > hostname labels, but by the asset tags are in inventory w7-ref and w764-ref > (machines labelled as such are sitting next to them) > > Not sure how to unravel those yet. Lets leave them alone for now, i wonder if there were two reference images created to test a change. who knows > Given that we have a relatively small number of machines for each OS, could we > do all of this in one batch? If we need to do this before I can fix the slaves left over from Friday, I'd like to keep it at two batches.

(In reply to comment #6) > (In reply to comment #5) > > Some notes, for lack of a better place: > > > > fed-010 is live in scl1 > > w7-14 and leopard-003 are live in mtv1 > > > > Should I fix up DNS for the current state? > > Yes, please. Done, fed-010, w7-14 and leopard-003 should now be accessible from build.m.o hostnames. > If we need to do this before I can fix the slaves left over from Friday, I'd > like to keep it at two batches. I'm unaware of any time pressures from my side. If waiting a couple of days for the changes from last week to get ironed out lets me to the move in one batch, that's a win for me. (Power in the 2nd floor server room is a *mess*... if we can power that all down at once, we can avoid a lot of possibility for error)

(In reply to comment #7) > Done, fed-010, w7-14 and leopard-003 should now be accessible from build.m.o > hostnames. Thanks :D > I'm unaware of any time pressures from my side. If waiting a couple of days for > the changes from last week to get ironed out lets me to the move in one batch, > that's a win for me. (Power in the 2nd floor server room is a *mess*... if we > can power that all down at once, we can avoid a lot of possibility for error) Yah, I agree. I hope to get the machines ironed out today. Would thursday work for you?

Sounds great, let's plan for Thursday.

talos-r3-leopard-013 is pointing to talos-r3-leopard-013.build.mtv1 when it should be pointing to leopard-13.build.mtv1 This is temporary and depending on how many slaves are having this problem, might not be worth fixing before the move

(In reply to comment #10) > talos-r3-leopard-013 is pointing to talos-r3-leopard-013.build.mtv1 when it > should be pointing to leopard-13.build.mtv1 to be more specific, it should be pointing to talos-r3-leopard-13.build.mtv1.mozilla.com > This is temporary and depending on how many slaves are having this problem, > might not be worth fixing since the plan is to move them tomorrow

leopard-[0]13 was one of the machines that was allegedly moved, but I found on the 2nd floor. Fixed. Inconsistent numbering makes me sad.

Ref machines are fine, btw...transcription error on my part. Sounds like we can move those at any time, so I'm going to ignore them for now.

(In reply to comment #12) > leopard-[0]13 was one of the machines that was allegedly moved, but I found on > the 2nd floor. Um, ignore the above. Still fixed, though.

Revised move list: (NB: These are post-move CNAMES, /not/ what's currently in DNS) +talos-r3-fed-009 IN CNAME talos-r3-fed-009.build.scl1.mozilla.com. +talos-r3-fed-018 IN CNAME talos-r3-fed-018.build.scl1.mozilla.com. +talos-r3-fed-019 IN CNAME talos-r3-fed-019.build.scl1.mozilla.com. +talos-r3-fed-020 IN CNAME talos-r3-fed-020.build.scl1.mozilla.com. +talos-r3-fed-029 IN CNAME talos-r3-fed-029.build.scl1.mozilla.com. +talos-r3-fed-030 IN CNAME talos-r3-fed-030.build.scl1.mozilla.com. +talos-r3-fed-034 IN CNAME talos-r3-fed-034.build.scl1.mozilla.com. +talos-r3-fed-035 IN CNAME talos-r3-fed-035.build.scl1.mozilla.com. +talos-r3-fed-036 IN CNAME talos-r3-fed-036.build.scl1.mozilla.com. +talos-r3-fed-037 IN CNAME talos-r3-fed-037.build.scl1.mozilla.com. +talos-r3-fed-038 IN CNAME talos-r3-fed-038.build.scl1.mozilla.com. +talos-r3-leopard-003 IN CNAME talos-r3-leopard-003.build.scl1.mozilla.com. +talos-r3-leopard-013 IN CNAME talos-r3-leopard-013.build.scl1.mozilla.com. +talos-r3-leopard-014 IN CNAME talos-r3-leopard-014.build.scl1.mozilla.com. +talos-r3-leopard-016 IN CNAME talos-r3-leopard-016.build.scl1.mozilla.com. +talos-r3-leopard-017 IN CNAME talos-r3-leopard-017.build.scl1.mozilla.com. +talos-r3-leopard-019 IN CNAME talos-r3-leopard-019.build.scl1.mozilla.com. +talos-r3-leopard-027 IN CNAME talos-r3-leopard-027.build.scl1.mozilla.com. +talos-r3-leopard-028 IN CNAME talos-r3-leopard-028.build.scl1.mozilla.com. +talos-r3-leopard-029 IN CNAME talos-r3-leopard-029.build.scl1.mozilla.com. +talos-r3-leopard-030 IN CNAME talos-r3-leopard-030.build.scl1.mozilla.com. +talos-r3-leopard-031 IN CNAME talos-r3-leopard-031.build.scl1.mozilla.com. +talos-r3-leopard-032 IN CNAME talos-r3-leopard-032.build.scl1.mozilla.com. +talos-r3-leopard-033 IN CNAME talos-r3-leopard-033.build.scl1.mozilla.com. +talos-r3-leopard-034 IN CNAME talos-r3-leopard-034.build.scl1.mozilla.com. +talos-r3-leopard-035 IN CNAME talos-r3-leopard-035.build.scl1.mozilla.com. +talos-r3-leopard-036 IN CNAME talos-r3-leopard-036.build.scl1.mozilla.com. +talos-r3-leopard-037 IN CNAME talos-r3-leopard-037.build.scl1.mozilla.com. +talos-r3-leopard-038 IN CNAME talos-r3-leopard-038.build.scl1.mozilla.com. +talos-r3-leopard-039 IN CNAME talos-r3-leopard-039.build.scl1.mozilla.com. +talos-r3-leopard-040 IN CNAME talos-r3-leopard-040.build.scl1.mozilla.com. +talos-r3-snow-013 IN CNAME talos-r3-snow-013.build.scl1.mozilla.com. +talos-r3-snow-014 IN CNAME talos-r3-snow-014.build.scl1.mozilla.com. +talos-r3-snow-015 IN CNAME talos-r3-snow-015.build.scl1.mozilla.com. +talos-r3-snow-016 IN CNAME talos-r3-snow-016.build.scl1.mozilla.com. +talos-r3-snow-018 IN CNAME talos-r3-snow-018.build.scl1.mozilla.com. +talos-r3-snow-020 IN CNAME talos-r3-snow-020.build.scl1.mozilla.com. +talos-r3-w7-001 IN CNAME talos-r3-w7-001.build.scl1.mozilla.com. +talos-r3-w7-002 IN CNAME talos-r3-w7-002.build.scl1.mozilla.com. +talos-r3-w7-003 IN CNAME talos-r3-w7-003.build.scl1.mozilla.com. +talos-r3-w7-004 IN CNAME talos-r3-w7-004.build.scl1.mozilla.com. +talos-r3-w7-006 IN CNAME talos-r3-w7-006.build.scl1.mozilla.com. +talos-r3-w7-007 IN CNAME talos-r3-w7-007.build.scl1.mozilla.com. +talos-r3-w7-008 IN CNAME talos-r3-w7-008.build.scl1.mozilla.com. +talos-r3-w7-009 IN CNAME talos-r3-w7-009.build.scl1.mozilla.com. +talos-r3-w7-010 IN CNAME talos-r3-w7-010.build.scl1.mozilla.com. +talos-r3-w7-012 IN CNAME talos-r3-w7-012.build.scl1.mozilla.com. +talos-r3-w7-013 IN CNAME talos-r3-w7-013.build.scl1.mozilla.com. +talos-r3-w7-014 IN CNAME talos-r3-w7-014.build.scl1.mozilla.com.

Ok, it looks like the majority of slaves that are not reporting to buildbot are either can't be pinged or are at mountain view. Because of this, I think that the best choice is to move all remaining minis in mountain view to scl. I will start shutting these slaves down once I get confirmation that we are going ahead with the move tomorrow morning.

Sorry, didn't see this until this morning, and I don't think well have time to get this done right before I need to be back in MV this afternoon. Let's plan for tomorrow (Friday) morning to move these to scl1.

talos-r3-fed-009 -- not pingable talos-r3-fed-018 -- not pingable talos-r3-fed-019 -- not pingable talos-r3-fed-020 -- not pingable talos-r3-fed-029 -- off talos-r3-fed-030 -- not pingable talos-r3-fed-034 -- off talos-r3-fed-035 -- not pingable talos-r3-fed-036 -- not pingable talos-r3-fed-037 -- off talos-r3-fed-038 -- off talos-r3-leopard-003 -- turned off, weird state talos-r3-leopard-013 -- renamed buildbot.tac, still needs puppet work talos-r3-leopard-014 -- renamed buildbot.tac, still needs puppet work talos-r3-leopard-016 -- not pingable talos-r3-leopard-017 -- renamed buildbot.tac, still needs puppet work talos-r3-leopard-019 -- renamed buildbot.tac, still needs puppet work talos-r3-leopard-027 -- renamed buildbot.tac, still needs puppet work talos-r3-leopard-028 -- renamed buildbot.tac, still needs puppet work talos-r3-leopard-029 -- renamed buildbot.tac, still needs puppet work talos-r3-leopard-030 -- renamed buildbot.tac, still needs puppet work talos-r3-leopard-031 -- renamed buildbot.tac, still needs puppet work talos-r3-leopard-032 -- renamed buildbot.tac, still needs puppet work talos-r3-leopard-033 -- renamed buildbot.tac, still needs puppet work talos-r3-leopard-034 -- renamed buildbot.tac, still needs puppet work talos-r3-leopard-035 -- renamed buildbot.tac, still needs puppet work talos-r3-leopard-036 -- renamed buildbot.tac, still needs puppet work talos-r3-leopard-037 -- renamed buildbot.tac, still needs puppet work talos-r3-leopard-038 -- renamed buildbot.tac, still needs puppet work talos-r3-leopard-039 -- renamed buildbot.tac, still needs puppet work talos-r3-leopard-040 -- renamed buildbot.tac, still needs puppet work talos-r3-snow-013 -- renamed buildbot.tac, still needs puppet work talos-r3-snow-014 -- not pingable talos-r3-snow-015 -- renamed buildbot.tac, still needs puppet work talos-r3-snow-016 -- renamed buildbot.tac, still needs puppet work talos-r3-snow-018 -- renamed buildbot.tac, still needs puppet work talos-r3-snow-020 -- renamed buildbot.tac, still needs puppet work talos-r3-w7-001 -- buildbot was already off talos-r3-w7-002 -- buildbot was already off talos-r3-w7-003 -- buildbot was already off talos-r3-w7-004 -- buildbot was already off talos-r3-w7-006 -- buildbot was already off talos-r3-w7-007 -- buildbot was already off talos-r3-w7-008 -- buildbot was already off talos-r3-w7-009 -- buildbot was already off talos-r3-w7-010 -- buildbot was already off talos-r3-w7-012 -- buildbot was already off talos-r3-w7-013 -- buildbot was already off talos-r3-w7-014 -- buildbot was already off

(In reply to comment #18) > talos-r3-snow-013 -- renamed buildbot.tac, still needs puppet work > talos-r3-snow-014 -- not pingable > talos-r3-snow-015 -- renamed buildbot.tac, still needs puppet work > talos-r3-snow-016 -- renamed buildbot.tac, still needs puppet work > talos-r3-snow-018 -- renamed buildbot.tac, still needs puppet work > talos-r3-snow-020 -- renamed buildbot.tac, still needs puppet work accidentally turned these off before making the required puppet changes. Need to boot them to clear the ssl keys and change the puppet master.

(In reply to comment #18) > talos-r3-fed-009 -- not pingable > talos-r3-fed-018 -- not pingable > talos-r3-fed-019 -- not pingable > talos-r3-fed-020 -- not pingable > talos-r3-fed-029 -- off > talos-r3-fed-030 -- not pingable > talos-r3-fed-034 -- off > talos-r3-fed-035 -- not pingable > talos-r3-fed-036 -- not pingable > talos-r3-fed-037 -- off > talos-r3-fed-038 -- off > > talos-r3-leopard-003 -- turned off, weird state > talos-r3-leopard-013 -- renamed buildbot.tac, still needs puppet work > talos-r3-leopard-014 -- renamed buildbot.tac, still needs puppet work > talos-r3-leopard-016 -- not pingable > talos-r3-leopard-017 -- renamed buildbot.tac, still needs puppet work > talos-r3-leopard-019 -- renamed buildbot.tac, still needs puppet work > talos-r3-leopard-027 -- renamed buildbot.tac, still needs puppet work > talos-r3-leopard-028 -- renamed buildbot.tac, still needs puppet work > talos-r3-leopard-029 -- renamed buildbot.tac, still needs puppet work > talos-r3-leopard-030 -- renamed buildbot.tac, still needs puppet work > talos-r3-leopard-031 -- renamed buildbot.tac, still needs puppet work > talos-r3-leopard-032 -- renamed buildbot.tac, still needs puppet work > talos-r3-leopard-033 -- renamed buildbot.tac, still needs puppet work > talos-r3-leopard-034 -- renamed buildbot.tac, still needs puppet work > talos-r3-leopard-035 -- renamed buildbot.tac, still needs puppet work > talos-r3-leopard-036 -- renamed buildbot.tac, still needs puppet work > talos-r3-leopard-037 -- renamed buildbot.tac, still needs puppet work > talos-r3-leopard-038 -- renamed buildbot.tac, still needs puppet work > talos-r3-leopard-039 -- renamed buildbot.tac, still needs puppet work > talos-r3-leopard-040 -- renamed buildbot.tac, still needs puppet work > > talos-r3-snow-013 -- renamed buildbot.tac, still needs puppet work > talos-r3-snow-014 -- not pingable > talos-r3-snow-015 -- renamed buildbot.tac, still needs puppet work > talos-r3-snow-016 -- renamed buildbot.tac, still needs puppet work > talos-r3-snow-018 -- renamed buildbot.tac, still needs puppet work > talos-r3-snow-020 -- renamed buildbot.tac, still needs puppet work > > talos-r3-w7-001 -- buildbot was already off > talos-r3-w7-002 -- buildbot was already off > talos-r3-w7-003 -- buildbot was already off > talos-r3-w7-004 -- buildbot was already off > talos-r3-w7-006 -- buildbot was already off > talos-r3-w7-007 -- buildbot was already off > talos-r3-w7-008 -- buildbot was already off > talos-r3-w7-009 -- buildbot was already off > talos-r3-w7-010 -- buildbot was already off > talos-r3-w7-012 -- buildbot was already off > talos-r3-w7-013 -- buildbot was already off > talos-r3-w7-014 -- buildbot was already off all slaves either off or not pingable. After a reboot (thanks Zandr!), I was able to prepare talos-r3-snow-014. The fedora slaves that weren't pingable are going to be brought up *without* network to ensure that they boot and to change the required puppet settings before attempting to connect to the scl puppet. Lets take these machines apart.

Just pushed the DNS changes so the build.mozilla.org CNAMEs now point at scl1 A records.

> > talos-r3-leopard-003 -- turned off, weird state Don't know what the state of this staging slave is > > talos-r3-leopard-013 -- renamed buildbot.tac, still needs puppet work > > talos-r3-leopard-014 -- renamed buildbot.tac, still needs puppet work > > talos-r3-leopard-016 -- not pingable Probably needs reimaging > > talos-r3-leopard-017 -- renamed buildbot.tac, still needs puppet work > > talos-r3-leopard-019 -- renamed buildbot.tac, still needs puppet work > > talos-r3-leopard-027 -- renamed buildbot.tac, still needs puppet work > > talos-r3-leopard-028 -- renamed buildbot.tac, still needs puppet work > > talos-r3-leopard-029 -- renamed buildbot.tac, still needs puppet work > > talos-r3-leopard-030 -- renamed buildbot.tac, still needs puppet work > > talos-r3-leopard-031 -- renamed buildbot.tac, still needs puppet work > > talos-r3-leopard-032 -- renamed buildbot.tac, still needs puppet work > > talos-r3-leopard-033 -- renamed buildbot.tac, still needs puppet work > > talos-r3-leopard-034 -- renamed buildbot.tac, still needs puppet work > > talos-r3-leopard-035 -- renamed buildbot.tac, still needs puppet work > > talos-r3-leopard-036 -- renamed buildbot.tac, still needs puppet work > > talos-r3-leopard-037 -- renamed buildbot.tac, still needs puppet work > > talos-r3-leopard-038 -- renamed buildbot.tac, still needs puppet work > > talos-r3-leopard-039 -- renamed buildbot.tac, still needs puppet work > > talos-r3-leopard-040 -- renamed buildbot.tac, still needs puppet work All other leopard slaves back online and running jobs

leopard 36,37,38,40 turned off to avoid overloading pdu

> talos-r3-snow-013 -- renamed buildbot.tac, still needs puppet work > talos-r3-snow-014 -- not pingable > talos-r3-snow-015 -- renamed buildbot.tac, still needs puppet work > talos-r3-snow-016 -- renamed buildbot.tac, still needs puppet work > talos-r3-snow-018 -- renamed buildbot.tac, still needs puppet work > talos-r3-snow-020 -- renamed buildbot.tac, still needs puppet work All online and accounted for

(In reply to comment #23) > leopard 36,37,38,40 turned off to avoid overloading pdu these were on the wrong pdu, instead we took down some of the offline fedora64 slaves.

fedora 29, 34, 37,38 are all connected to the buildbot master

Declaring victory here. There's some cleanup to be done at a future downtime, but the machines are moved with power and net.