Closed Bug 386779 Opened 17 years ago Closed 17 years ago

Add additional performance machines for talos

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Platform:
All
Other
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: mtschrep, Assigned: mrz)

References

(
URL
)

Details

Due to both concerns over stability and the need to have additional machines available for running performance tests we'd like to augment the 10 talos machines with some additional windows/linux capable x86 boxes.

Specs:

Capable of running Windows XP and Linux
At least 1GB RAM
Single-core if possible (duo-core ok if there is no choice) 
Stable server-class hardware

E.g. 

http://h71016.www7.hp.com/dstore/MiddleFrame.asp?page=config&ProductLineId=431&FamilyId=2424&BaseId=19129&oi=E9CED&BEID=19701&SBLID=
or
http://h71016.www7.hp.com/dstore/MiddleFrame.asp?page=config&ProductLineId=431&FamilyId=2424&BaseId=19130&oi=E9CED&BEID=19701&SBLID=
or
any spare DL360's we have are probably fine.   

We'd want about 4 windows boxes to start.   Since we have slow-as-heck machines already in the Talos cluster the slighter faster machines may be more useful here (to get for example a larger number of Tp runs in during a cycle time).  But I'll leave it to Robcc/Vlad to comment there...
Blocks: 386081
Talked with schrep - dl140, single core, 1gb mem, single 80gb drive is what they are after - working on the order.
Assignee: server-ops → justin
We get an eta and count on these?
4x, late this week or early next week.  I'll see if I can get a more hard date.
K - are you clear on Software requirements?  All should be loaded with same windows build as the other Talos Machines.  I'd like to get this up and running soon after they come in.


verified early-mid next week - we'll be at oscon, but we can get it up right when we get back.  I know nothing about what software is needed - could someone document what you'd like installed in this bug?
Boxes are here and will be online tomorrow - what OS do you need on these?
Assignee: justin → mrz
All four the same as the 5 windows talos boxes.  which I believe is WinXP  - Robcc can clarify..
Specs in the URL field
Assignee: mrz → server-ops
URL: http://wiki.mozilla.org/ReferencePlat...
Assignee: server-ops → mrz
Something that was overlooked here was the OS you guys wanted to run.  HP doesn't support Windows XP on the DL140s.  HP doesn't have NIC or SATA drivers which may be moot because the XP installer blue screens before it fails to find any storage to install on.  

Any chance win2k3 standard is close enough to XP?
Well I guess it's gonna have to be.  can you get as close to the above specs just with Win2k3?

best,

Schrep
win2k3 will have to do. Hopefully our new timer resolution patch runs properly on that environment (I expect it should).
I'm trying to get these built... need to track down a USB floppy drive (Linux is so much easier).  

I should have this going today and either all wrapped up either today or tonight (or worse case, tomorrow from OSCON).
I have the first of these boxes up - qm-talos03/10.2.73.38 - running an UNPATCHED version of Windows 2003 Server Standard SP1.  Since this doesn't match the OS for the XP ref platform, I wasn't sure if you wanted any MSFT updates.  If you let me know, the rest will be built accordingly.

I'd like someone to check this box out and let me know if it works as an XP substitute.  

ps. If you want a different hostname, let me know - qm-pxp06 didn't make sense though.
qm-talos04 is up.  Alice has login info.  Unpatched :)
I'll need the additional font support installed on these machines (Regional &
Language Options -> Languages -> Supplemental Language Support) for viewing
international web pages.
Also - I can only seem to get a screen resolution of 1024 x 768.  Ideally I'd like  1280 x 1024 to match up with the other test boxes.  

We have a history of having to fight pretty hard to get screen resolutions to work, so if this is as high as it gets I'm willing to live with it.
This is the same issue you had with the XP Celeron boxes and I had to put KVMs on each one (someone had a technical explanation for this).

Can you check with qm-talos04?  It currently has an external KVM attached.  If it works then I'll need to see about buying more KVM dongles.
On 04 I'm seeing the same 1024 x 768, and oddly enough I can only get 16 bit color quality instead of 32.
I think we'll want to see some sample performance numbers from these machines before judging video resolution/etc; as far as video is concerned, 1280x1024 would be greatly preferable, because otherwise the actual window size will be fairly small.  Also, the video chip in these boxes is an 8MB ATI RAGE XL -- which is, unfortunately, nothing like any real-world user video cards, but I have no idea what performance characteristics it has.

Rendering is only a part of the overall pageload performance (I think around 15-20%?), so as long as there's enough sensitivity and consistency in the rest, then they should be fine.

For the record, the lowest common user denominator, video wise, should be an intel GMA 900 or 950, which is the onboard video with the G variants of the intel 915/935/945 chipsets.
We've had a lot change in or lack of spec on these - could we spend some time at the next perf meeting defining exactly what you need and I can go spec out 4 machines that meet all the reqs?  I can re-purpose these 140's if need be.

I've updated the information about what needs to be installed on these boxes for them to become working Talos machines:

http://wiki.mozilla.org/ReferencePlatforms/Test/WinXP

Unfortunately, I can't help you with the machine requirements themselves.
According to the specs:

http://h18004.www1.hp.com/products/quickspecs/12509_na/12509_na.html

#  Additional Features:
    * Support for:
          o NVIDIA Quadro FX3500 256MB PCIe graphics card
          o NVIDIA Quadro FX1500 256MB PCIe graphics card

But they are $600!  

To answer your question Justin ideally we'd have:
  Support for WindowsXP
  1280x1024x32bit video with a "consumer" video card like built-in GMA or common ATI/Nvidia cards

That doesn't seem to be common on servers.  The Dell servers have an embedded AT ES1000 with 16MB of RAM.   So that might do the resolution.  Other option is just go back to desktops in the darn server room. E.g. SFF OptiPlex 745 have GMA graphics and are cheap.  

Right now I just want a couple of extra machines for the Talos team that can do 1280x1024x32bit video. 



Perhaps we just give up on the server solution:

http://www.cdw.com/shop/products/default.aspx?EDC=1217966

1) MRZ would you be willing to do a zip-tie mount ala the Mini's?  These guys are 12x12x3" so we'd be able to get 6 or so in a 8U if we flip them sideways.
2) Vlad/Alice does this look like a reasonable machine specs to test on (GMA 3000, 1GB RAM)? 

Drivers are available for these for XP.

Sorry for the thrash here...
GMA 3000 would be fine, it's basically a GMA 950 minus some video hardware.

In case it's any easier, Mac Minis are actually great machines to test on for all three platforms; they're very generic "user-level hardware".  They're probably not any easier from an IT perspective, but it would mean just supporting 1 testing box type that's fairly generically swappable...
(In reply to comment #24)
> GMA 3000 would be fine, it's basically a GMA 950 minus some video hardware.
> 
> In case it's any easier, Mac Minis are actually great machines to test on for
> all three platforms; they're very generic "user-level hardware".  They're
> probably not any easier from an IT perspective, but it would mean just
> supporting 1 testing box type that's fairly generically swappable...
> 

Right.  Now that's a plan.   Can we order up a raft of the low-end Mini's bumped to 1GB of Ram and get them racked up and loaded with WinXP?  We'd like:
    8 for perf work
    4 for cairo buildbot (vlad knows what this is)

Mrz/Justin thanks again for your help on this.   
(In reply to comment #25)
> Right.  Now that's a plan.   Can we order up a raft of the low-end Mini's
> bumped to 1GB of Ram and get them racked up and loaded with WinXP?  We'd like:

Wait, not all with WinXP, no?  We probably want:

8 perf minis:
  2 winxp
  2 macosx
  2 linux
  2 .. spare?

4 cairo minis:
  1 winxp
  1 macosx
  2 linux

hm, 2 spare should probably be 2 vista (business, I guess, maybe home, or home premium; guess it doesn't really matter much)
We are testing mini's today to verify driver support and correct video resolution - I'll put the order in tomorrow and have them here and racked by the end of this week.  We'll follow http://wiki.mozilla.org/ReferencePlatforms/Test/WinXP for the XP build - is there something similar to that for OS X, vista and linux?
Assignee: mrz → server-ops
We are testing mini's today to verify driver support and correct video resolution - I'll put the order in tomorrow and have them here and racked by the end of this week.  We'll follow http://wiki.mozilla.org/ReferencePlatforms/Test/WinXP for the XP build - is there something similar to that for OS X, vista and linux?
Assignee: server-ops → mrz
For Linux, I think the Ubuntu Feisty setup is what we want... I thought that was documented somewhere.

Not sure about OSX or Vista, though for Vista, it's going to want to be pretty similar to XP (that is, just install the Boot Camp drivers).  What's our OS update policy on these machines?  Do we install the OS updates, or just whatever came on the CD?  I'd think we should install them, just because our normal users will be doing so, but it's ok if we install them only every few months maybe?
You(In reply to comment #29)
> We are testing mini's today to verify driver support and correct video
> resolution - I'll put the order in tomorrow and have them here and racked by
> the end of this week.  We'll follow
> http://wiki.mozilla.org/ReferencePlatforms/Test/WinXP for the XP build - is
> there something similar to that for OS X, vista and linux?
> 

that's the right Specs for XP - can you make sure to install everything on that list.  Dunno if you can build one and clone the rest.  to be sure on specs we want 1GB ram, slower processor.  Also note we already have 5 mac minis for Mac testing.  So let's build 4 XP first, 2 Vista, 2 Linux.  Also - if you think this model is going to be discounted soon we should buy some spares since they are so cheap..

To answer vlad's question we should start these guys out update to date and then disable updates so we don't introduce the possibility of perf changes...

(In reply to comment #31)
> To answer vlad's question we should start these guys out update to date and
> then disable updates so we don't introduce the possibility of perf changes...

I dunno -- these are supposed to be testing user perf, right?  So if an OS update changes perf characteristics, we'd want to know...


(In reply to comment #32)
> (In reply to comment #31)
> > To answer vlad's question we should start these guys out update to date and
> > then disable updates so we don't introduce the possibility of perf changes...
> 
> I dunno -- these are supposed to be testing user perf, right?  So if an OS
> update changes perf characteristics, we'd want to know...
> 

Right but in a controlled (e.g. scheduled) manner so we know it's the patch an not a real regression.
(In reply to comment #33)
> Right but in a controlled (e.g. scheduled) manner so we know it's the patch an
> not a real regression.

Ah right, yep, no automatic updates.


machines have arrived - imaging today.
it'll be interesting to see if we can even run Talos in all its glory on Vista. From webserver to python and win32 extensions, it could be a bit of a crapshoot.
I need clarification on what to hand over to you guys.  From comment 31, looks like out of the 12 I have up:

4 XP
2 Vista
2 Linux (which distro?)
(-no- OSX)

and from Bug 389991,
1 CentOS
1 Ubuntu
1 OSX
1 XP

Is that right?

Tracking allocation here (i'm counting the existing 5 minis for mac talos): 

http://spreadsheets.google.com/ccc?key=pq20lsXjZGAwGArb8UO3GWA&hl=en#

Assuming these are easier to do in bulk let's do:

10 WinXP 
4 Vista 
6 Linux
1 Mac

Platforms are documented here:

http://wiki.mozilla.org/ReferencePlatforms#Performance_Test_Reference_Platforms

I have three of you boxes up - 

qm-mini-xp01 / 10.2.73.36
qm-mini-xp02 / 10.2.73.37
qm-pmac05 / 10.2.73.19

There's a lot of people on this bug - not sure who needs to know login info.  Ping me offline and I'll let you know what logins are.  

I'll have the rest of the XP boxes up Monday morning and the Vista ones and maybe the Ubunto ones Monday afternoon.
qm-mini-xp01 - 10 are online.  10.2.73.26 - 10.2.73.45 .
Awesome!  Can you update the spreadsheet (Mrz) with the current IP/hostname info.  Alice/Robcc as you use them for new stuff if you could update as well (with branch/task) so we have a good inventory of what these are being used for.   
Does that include the cairo bits or no?  If not, I'd be happy to do the setup myself in the background during the onsite, since I won't be getting much work done that I need to concentrate for...
That's a good question - I think so - schrep?

I have reed building one centos that I'll get up Tuesday or so.
Okay, qm-mini-vista01-03 are online as well as qm-mini-ubuntu01 & 02.  To make this easier and per schrep in comment #41, I've uploaded a revised spreadsheet.  The second sheet is from the actual colo inventory.  I've added a couple columns for this (and removed some that would be noise to you).  

If it's green, it's up.  If it's gold, it's not yet up.  

I'm hoping one of you can fill in branch & task.
All the Minis are online and the spreadsheet has been updated to reflect that.  Calling this FIXED.
Status: NEW → RESOLVED
Closed: 17 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.