In the light of branching coming up for Mozilla 1.9.1 soon, and for SeaMonkey probably some time between now and some point in Q1 of 2009, as well as requests for a debug build/test box and x86_64 Linux builds, and even trunk deprecation of Tiger possibly coming up soon, I'd like to request a number of machines for SeaMonkey. In the ideal case, my vision is to get the following machines: - 2 VMs with the Linux centos5 reference image, going into the current build/test pools for now, to have the resources for both trunk/branch once we branch SeaMonkey 2 - 2 VMs with the Win2k3 reference image, going into the current build/test pools for now, to have the resources for both trunk/branch once we branch SeaMonkey 2 - 1 VM with a Linux x86_64 system (do we have a refimage there?), for doing 64bit builds - 1 VMs with the Linux centos5 reference image, for building debug, see bug 452147 - 2 Mac machines (min or xserve, whatever IT is more comfortable with), those should be Intel-based machines with Leopard installed, going into the current build/test pools for now, to have the resources for both trunk/branch once we branch SeaMonkey 2 This results in 6 VMs and 2 Macs, which is more machines than we even have right now, actually. I suspect this may raise some eyebrows, given we're a small project for a pretty niche product, but then, we never had a branch to maintain next to trunk apart from what I've been running on private machines. If money is a concern, maybe needing an extra ESX server to be added, or for the Macs, we have some amount of SeaMonkey-directed donations waiting at MoFo to be used, which we could possibly throw into this - but those possible concerns also why I'm filing this with enough time before our actual branching, so everything can be planned without too much pressure.

Passing over to community giving.

The community ESX host in Amsterdam could probably take a few more VMs but it'll be CPU constrained. CPU is over 75% busy. To handle the 6 VMs would very likely require another ESX server (with storage). The Macs could either be two Mac Minis or a single Xserve running some number of VMs (which depends on what other community needs might come up).

It would be helpful to get some cost quotes on these machines. Community Giving will need to evaluate the cost to help make the case for leverage. MRZ: is it easy to provide a cost for a single Xserve and how many VMs can run here using an "average" amount of the Xserve resources? Mac Minis, I am guessing, could run about $600-800 apiece, no?

The ESX hardware is ballpark $8k + software license = $12k. We typically budget about 20 VMs per ESX host. Xserve is probably around $6k + Parallels server. We don't have anything like this in production yet so I haven't any idea how many VMs it could handle.

If we finally have means to virtualize Mac machines, we could potentially move boxes there that currently run on separate hardware, like the PPC Xserve or mini we're already using right now for SeaMonkey machines.

> Xserve is probably around $6k + Parallels server. We don't have anything like > this in production yet so I haven't any idea how many VMs it could handle. Doesn't MoMo use an Xserve for their Mac buildbots? cc: gozer

Yes, VMWare Fusion on the Apple Xserve supports Mac OS X 10.5

Seth, how is budget situation?

I am going to suggest the following: Start with 2 VMs (one linux, one windows) and one mac mini. If MRZ would like to discuss the Xserve with me, that is fine. One way or another we will get you three machines to start. After we branch, we will revisit getting the testing machine. Finally, if SeaMonkey wants to be one of the test pilots for Mac OS virtualization, then I will let MRZ determine the best setup. Not sure if that is possible right now for IT, but KaiRo has suggested that SeaMonkey would like to do this experiment.

seth, any timeframes on all this from your side?

mrz and i chatted about this. it seems that we'll go with an xserve so SeaMonkey folks can be one of the groups to test the OS virtualization. mrz can you confirm this? is your team ready to work on this?

Fell off my plate, grabbing.

Hardware is on order, will update with ETA when I get it.

ETA on this XServe is 3/19 via Fed EX # 110911170297076

Was too aggressive on my ETA. That ETA reflected hardware delivery, not when they'd be online (we've been blocked on two 10GE cross-connects). Let's shoot for this Friday.

The server is installed xserve 10.5. I will need a copy of Parallel server license to complete the install.

Please proceed and bill the $1248.75 to Community Giving.

I got the license key from Seth.

I'm trying to figure out a way to import the current build ESX templates into Parallels server. If this doesn't work, then we will have to build these as fresh installs.

Update - the build scrubbed Windows image converted okay. The CentOS one failed twice. Might need to do a fresh image (isn't clear if it's an issue with Parallels supporting CentOS5.0 or something else). Do you -need- the build reference image? Could you make a new SeaMonkey ref image? Can you suggest hostnames for these VMs?

I can't get the CentOS ref image to import and we don't have an x86_64 ref image that I'm aware of. Can I install a fresh CentOS5.3 image and let you customize both and make those the ref images?

(In reply to comment #21) > I can't get the CentOS ref image to import and we don't have an x86_64 ref > image that I'm aware of. I'd really like to be as near as possible to the config also used by the Firefox pools, and esp. to our existing refplatform VMs as all the machines of one platform should go into one generic pool in the future (just like Firefox does) to make maintenance as easy as possible (and all generating completely exchangeable binaries).

That means a fresh install of CentOS 5.0, both 32 and 64bit and you can install whatever tools are needed before calling them ref images. My concern is that 5.0 is pretty old, probably filled with bugs. Did you have hostname suggestions?

For 64bit, I think we can safely go with 5.3, but for 32bit, my main concern is that binaries produced are exchangeable with binaries produced by the refplatform VMs, so the machines can run in a generic pool. I don't have hostname suggestions, their target is to serve as slaves for a build pool, can we make us have a similar naming scheme as Firefox slaves have?

OK, Ben Hearsum tells me their 64bit machine is CentOS 5.0 as well, due to being as close to the refplatform as possible, and given that I want to be as near as possible to Firefox reference stuff, it's probably best to go with 5.0 for all the machines then. I guess that one 32bit VM set up with that would be best for the moment though if we can easily clone it to the second one when I have gone through all the steps to install and configure everything to make it parallel the refplatform. I wonder if that strategy would work nicely for the Macs as well. ;-) With this, the question comes up if we want to keep around such reference images of Linux and Mac on Parallels for possible further cloning (the build team might be interested).

I grabbed the pre-build VM image from https://wiki.mozilla.org/ReferencePlatforms/Linux-Public and was able to get that imported. So I have a win2k3-sp2 ref image and now a CentOS5.0 ref image. Ben says he has an updated ref image though so I'll import that one as well. Your current Linux VMs are based off the 01/2008 image.

(In reply to comment #26) > I grabbed the pre-build VM image from > https://wiki.mozilla.org/ReferencePlatforms/Linux-Public and was able to get > that imported. > > So I have a win2k3-sp2 ref image and now a CentOS5.0 ref image. Ben says he > has an updated ref image though so I'll import that one as well. Your current > Linux VMs are based off the 01/2008 image. It's on ftp.m.o now: ftp://ftp.mozilla.org/pub/mozilla/VMs/CentOS5-ReferencePlatform.tar.bz2

However that new VM differs from the old, it has the same import problems. For now the only ref Linux VM I've been able to import is the one from 01/2008.

I now have: 1. Fresh install OSX10.5 Server 2. CentOS 5.0 i386 Ref image (01/2008) 3. Windows 2003 SP2 / VC8 Ref Image At this point I'll need to clone the OSX10.5 Server image and let you customize it for your build purposes and then declare that as a ref image. There is no x86_64 CentOS ref image. Do you want a fresh install or postpone on that one? The other two platforms can be imaged quickly.

Login info emailed out of band. The OSX hosts aren't yet up. > In the ideal case, my vision is to get the following machines: > - 2 VMs with the Linux centos5 reference image, cb-seamonkey-linux-01 / 63.245.210.32 cb-seamonkey-linux-02 / 63.245.210.33 > - 2 VMs with the Win2k3 reference image, cb-seamonkey-win32-01 / 63.245.210.34 cb-seamonkey-win32-02 / 63.245.210.35 > - 1 VMs with the Linux centos5 reference image, cb-seamonkey-linuxdebug-01 / 63.245.210.36 > - 2 Mac machines (min or xserve, whatever IT is more comfortable with), cb-seamonkey-osx-01 / 63.245.210.37 cb-seamonkey-osx-02 / 63.245.210.38

(In reply to comment #29) > I now have: > > 1. Fresh install OSX10.5 Server > 2. CentOS 5.0 i386 Ref image (01/2008) > 3. Windows 2003 SP2 / VC8 Ref Image Thanks a lot! > There is no x86_64 CentOS ref image. Do you want a fresh install or postpone > on that one? Yes, a fresh install, please. From what I hear, there's only one other machine with x86_64 as of now.

Hmm, and I can't reach the machines from cb-jumphost01 right now. :(

(In reply to comment #32) > Hmm, and I can't reach the machines from cb-jumphost01 right now. :( Netmask on that network changed but wasn't changed on cb-jumphost01. I've changed it now and you should be good to go.

Thanks, I can reach and log into the Linux and Windows VMs now and have brought up the Linux ones to the state of the current refplatform except for scratchbox. What I didn't realize at first is that we also need extra disks for builds on the machines, from what I know, a 30 GB disk for every machine should do fine, would that be possible?

Parallels Server doesn't let me hot-add hard drives. Can you shut those machines down so I can add?

The two OSX servers are up. Login info emailed out of band.

(In reply to comment #35) > Parallels Server doesn't let me hot-add hard drives. Can you shut those > machines down so I can add? I did halt the Linux machines, I can't find the knob though to shut down the Windows ones.

> What I didn't realize at first is that we also need extra disks for builds on > the machines, from what I know, a 30 GB disk for every machine should do fine, > would that be possible? Added for each platform. The OSX images started off with a large disk (64GB) than the others so I didn't add anything there. At this point all you need is a 64bit CentOS 5.0 install. And don't update the OSX image - it'll stop booting (still debugging that). Parallels Server doesn't understand Vlans. I've ordered a dual-port ethernet card and will need to take a downtime to install it.

Found the fix, updating the Server software...

Okay, you can update to 10.5.6. You should not update to 10.5.7 until we've confirmed you won't hose your VM. When you have the OSX VMs configured shut one down and let me know so I can clone a ref image from it.

Am building the x86_64 CentOS 5.0 VM off the kickstart @ https://bug419924.bugzilla.mozilla.org/attachment.cgi?id=306233 .

(In reply to comment #41) > Am building the x86_64 CentOS 5.0 VM off the kickstart @ > https://bug419924.bugzilla.mozilla.org/attachment.cgi?id=306233 . That should give me the same base system as in https://wiki.mozilla.org/ReferencePlatforms/Linux-CentOS-5.0 right? If so, probably a good idea to link that kickstart from that doc :)

cb-seamonkey-linux64-01 / 63.245.209.39 (login is as the other linux hosts) I attached two additional drives to match the 32bit Linux hosts but left it to you to fdisk & mount. When you have this VM configured, file a new bug to take clone of it as a 64bit Ref image. I'm calling this one closed. If anything breaks, file new bugs please.

(In reply to comment #40) > When you have the OSX VMs configured shut one down and let me know so I can > clone a ref image from it. cb-seamonkey-osx-02 has been shut down for ref imaging, the setup and remaining steps for post-imaging are documented at https://wiki.mozilla.org/User:KaiRo/ParallelsMacOSX (doc can moved to a more "official" place on the wiki as wanted, when doing that, will redirect the current place accordingly).

(In reply to comment #30) > > - 2 VMs with the Linux centos5 reference image, > > cb-seamonkey-linux-01 / 63.245.210.32 > cb-seamonkey-linux-02 / 63.245.210.33 > > > - 1 VMs with the Linux centos5 reference image, > > cb-seamonkey-linuxdebug-01 / 63.245.210.36 As well rename this one to 'cb-seamonkey-linux-03' as they're used in a pool of 3 actually.

(In reply to comment #45) > > cb-seamonkey-linuxdebug-01 / 63.245.210.36 > > As well rename this one to 'cb-seamonkey-linux-03' as they're used in a pool of > 3 actually. Nah, doesn't matter, it's already running and set up, and the internal machine names don't matter much.

(In reply to comment #43) > cb-seamonkey-linux64-01 / 63.245.209.39 > > (login is as the other linux hosts) > > I attached two additional drives to match the 32bit Linux hosts but left it to > you to fdisk & mount. When you have this VM configured, file a new bug to take > clone of it as a 64bit Ref image. Hmm, I somehow overlooked that we want to do a ref image of it when setting it up, so it has the seabld user and SeaMonkey builder passwords now. Should I neutralize this somehow before we take the ref image?

Up to you - you're likely the only user of this ref image in the near term.

OK, I changed to root password to what the Mac ref image has and stripped the ssh keys from the seabld user, I guess that's generic enough that anyone can consume the image. Filed bug 492527 for imaging.

verifying, we have all those shiny new VMs, thanks a lot to everyone involved!