Sadly, I didn't say how it was broken in bug 716326, so I don't know whether that was also like https://tbpl.mozilla.org/php/getParsedLog.php?id=9433872&tree=Mozilla-Inbound where all of rm -rf tools and then rm -rf build says "rm: tools/.hg/00changelog.i: Invalid argument" about every file, and then it craps out trying to download the build, with "Cannot write to `firefox-13.0a1.en-US.mac.dmg' (Invalid argument)." but, I sort of think it was, and it probably needs its disk looked at, rather than just another reimage. Maybe. Anyway, it's once again chewing up jobs like crazy because it only takes a couple of seconds to fail to rm and then fail to save a downloaded build.

Disabled in slavealloc and on the buildbot master. It's refusing ssh access so will needs some hands on help.

Putting it back in the pool. Will monitor.

I disabled it per Philor's request in IRC and didn't realize it already had a bug (teach me to not check slavealloc *before* filing bug...)

decomission?

(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #5) > decomission? rev4 machines are (essentially) brand-new, so I certainly hope not. Please file a server ops bug to get it fixed.

Handing back to buildduty now that hands-on has been requested.

https://tbpl.mozilla.org/php/getParsedLog.php?id=11156650&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=11156669&tree=Mozilla-Inbound rm: tools/.hg: Device not configured rm: tools/.hgignore: Invalid argument rm: tools/.hgtags: Invalid argument rm: tools/.pylintrc: Invalid argument rm: tools/breakpad: Device not configured rm: tools/buildbot-helpers: Device not configured rm: tools/buildfarm: Device not configured rm: tools/cdmaker: Device not configured rm: tools/clobberer: Device not configured rm: tools/graphserver_webapp: Device not configured rm: tools/lib: Device not configured rm: tools/MANIFEST.in: Invalid argument rm: tools/misc: Device not configured rm: tools/release: Device not configured rm: tools/scripts: Device not configured rm: tools/setup.py: Invalid argument rm: tools/stage: Device not configured rm: tools/sut_tools: Device not configured rm: tools/trychooser: Device not configured rm: tools: Directory not empty

https://tbpl.mozilla.org/php/getParsedLog.php?id=11156809&tree=Birch https://tbpl.mozilla.org/php/getParsedLog.php?id=11156805&tree=Birch https://tbpl.mozilla.org/php/getParsedLog.php?id=11156825&tree=Birch rm: tools/.hg: Device not configured rm: tools/.hgignore: Invalid argument rm: tools/.hgtags: Invalid argument rm: tools/.pylintrc: Invalid argument rm: tools/breakpad: Device not configured rm: tools/buildbot-helpers: Device not configured rm: tools/buildfarm: Device not configured rm: tools/cdmaker: Device not configured rm: tools/clobberer: Device not configured rm: tools/graphserver_webapp: Device not configured rm: tools/lib: Device not configured rm: tools/MANIFEST.in: Invalid argument rm: tools/misc: Device not configured rm: tools/release: Device not configured rm: tools/scripts: Device not configured rm: tools/setup.py: Invalid argument rm: tools/stage: Device not configured rm: tools/sut_tools: Device not configured rm: tools/trychooser: Device not configured rm: tools: Directory not empty

https://tbpl.mozilla.org/php/getParsedLog.php?id=11156872&tree=Birch https://tbpl.mozilla.org/php/getParsedLog.php?id=11156764&tree=Birch Connecting to build.mozilla.org|10.2.74.128|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1,169 (1.1K) [application/x-sh] installdmg.sh: Invalid argument Cannot write to `installdmg.sh' (Invalid argument). program finished with exit code 1

https://tbpl.mozilla.org/php/getParsedLog.php?id=11157344&tree=Birch https://tbpl.mozilla.org/php/getParsedLog.php?id=11157301&tree=Firefox

https://tbpl.mozilla.org/php/getParsedLog.php?id=11157557&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=11157563&tree=Mozilla-Inbound

https://tbpl.mozilla.org/php/getParsedLog.php?id=11157755&tree=Firefox https://tbpl.mozilla.org/php/getParsedLog.php?id=11157899&tree=Firefox

https://tbpl.mozilla.org/php/getParsedLog.php?id=11159447&tree=Birch https://tbpl.mozilla.org/php/getParsedLog.php?id=11159436&tree=Birch https://tbpl.mozilla.org/php/getParsedLog.php?id=11159396&tree=Birch

https://tbpl.mozilla.org/php/getParsedLog.php?id=11159703&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=11159623&tree=Mozilla-Inbound

disabled in slavealloc and ssh'ing in to make sure it's offline

something is wrong with the harddrive on this host - the above activity points to a bad or full drive and df -h shows: talos-r4-snow-007:~ cltbld$ df -h Filesystem Size Used Avail Capacity Mounted on /dev/disk0s2 298Gi 8.3Gi 289Gi 3% / devfs 106Ki 106Ki 0Bi 100% /dev map -hosts 0Bi 0Bi 0Bi 100% /net map auto_home 0Bi 0Bi 0Bi 100% /home please pull this unit and check it's harddrive

this slave has returned from repairs with a new hdd. Re-imaged and back in SCL1.

Back in production

Needs a reboot, and to be added back to nagios.

Rebooted via PDU after nagios reported it down.

Attempting SSH reboot...Failed. Attempting PDU reboot...Failed. Filed IT bug for reboot (bug 1037831)