Nick Thomas [:nthomas] (UTC+12) Reporter
	Description • 12 years ago

Needs a reboot.

	Jake Watkins [:dividehex]
	Comment 1 • 12 years ago

This error was being repeated on the screen

"disk0s2: media is not present."

The system came back online after a reboot

	bhearsum@mozilla.com (:bhearsum)
	Comment 2 • 12 years ago

back in production

	Ed Morley [:emorley]
	Comment 3 • 12 years ago

I think this slave has RAM issues or something:
Bug 784328
Bug 784323
Bug 781820
Bug 781816
Bug 781150
Bug 781148
Bug 780889

I've been merrily filing bugs for these new crashes - but I'm starting to suspect the slave now, since 7 out of the 21 new crashes in the last two weeks have come from that slave alone - and those failures (above) have not been seen on any other machine.

Please can we take this machine out of production and run a memory diag or something? :-)

	Phil Ringnalda (:philor)
	Comment 4 • 12 years ago

https://tbpl.mozilla.org/php/getParsedLog.php?id=14615019&tree=Mozilla-Inbound

	Nick Thomas [:nthomas] (UTC+12) Reporter
	Comment 5 • 12 years ago

Disabled in slavealloc.

	Jordan Lund (:jlund)
	Comment 6 • 12 years ago

hardware diagnostics were run twice with no errors.

	bhearsum@mozilla.com (:bhearsum)
	Comment 7 • 12 years ago

Looks like we didn't try to re-image this machine yet. Let's do that, and then kill it with fire if it doesn't work.

	bhearsum@mozilla.com (:bhearsum)
	Comment 8 • 12 years ago

Running tests fine again in production.

	Phil Ringnalda (:philor)
	Comment 9 • 12 years ago

The problem is, other than the now-burned edmorley, the rest of us aren't likely to file on a random-looking crash like https://tbpl.mozilla.org/php/getParsedLog.php?id=16767443&tree=Mozilla-Inbound, we'll just blow it off.

	Phil Ringnalda (:philor)
	Comment 10 • 12 years ago

https://tbpl.mozilla.org/php/getParsedLog.php?id=16792815&tree=Fx-Team (a talos run with a strange and sudden "process killed by signal 11")

	Phil Ringnalda (:philor)
	Comment 11 • 12 years ago

https://tbpl.mozilla.org/php/getParsedLog.php?id=16800234&tree=Mozilla-Inbound with exactly the sort of "this is the only slave that ever has or ever will hit it" memory corruption GC crash that started the whole "pull busted slaves and run diagnostics on them" thing.

	Kim Moir [:kmoir] ET
	Comment 12 • 12 years ago

disabled in slavealloc

	Armen [:armenzg]
	Comment 13 • 12 years ago

Setup needed.

	bhearsum@mozilla.com (:bhearsum)
	Comment 14 • 12 years ago

Well, after this machine sat idle for nearly two months I've done post-imaging setup and it's back in production.

	Phil Ringnalda (:philor)
	Comment 15 • 12 years ago

https://tbpl.mozilla.org/php/getParsedLog.php?id=18649267&tree=Mozilla-Inbound is it crashing with a minidump so malformed, minidump_stackwalk just stared at it saying "wtf is that? is that an AMD64 crash? wtf?"

	Phil Ringnalda (:philor)
	Comment 16 • 12 years ago

https://tbpl.mozilla.org/php/getParsedLog.php?id=18654576&tree=Mozilla-Aurora and
https://tbpl.mozilla.org/php/getParsedLog.php?id=18652897&tree=Mozilla-Inbound are exactly the sort of... oh, I already typed that in comment 11.

Please apply the comment 7 solution and decomm it - it is clearly and unquestionably busted, and we apparently lack sufficient diagnostics to even guess what parts to start replacing.

	bhearsum@mozilla.com (:bhearsum)
	Comment 17 • 12 years ago

(In reply to Phil Ringnalda (:philor) from comment #16)
> https://tbpl.mozilla.org/php/getParsedLog.php?id=18654576&tree=Mozilla-
> Aurora and
> https://tbpl.mozilla.org/php/getParsedLog.php?id=18652897&tree=Mozilla-
> Inbound are exactly the sort of... oh, I already typed that in comment 11.
> 
> Please apply the comment 7 solution and decomm it - it is clearly and
> unquestionably busted, and we apparently lack sufficient diagnostics to even
> guess what parts to start replacing.

We've been looking at repairing the r4 machines lately, so we'll give that a try first.

	Justin Wood (:Callek)
	Comment 18 • 11 years ago

Diagnostics found corrupt files, was reimaged and brought back to life just now.

	Ryan VanderMeulen [:RyanVM]
	Comment 19 • 11 years ago

And we're seeing the same failures on it that we were seeing before in bug 781816.
https://tbpl.mozilla.org/?tree=Mozilla-Aurora

	Ryan VanderMeulen [:RyanVM]
	Comment 20 • 11 years ago

https://tbpl.mozilla.org/php/getParsedLog.php?id=19465580&tree=Mozilla-Aurora

	Ed Morley [:emorley]
	Comment 21 • 11 years ago

Disabled in slavealloc.

	Armen [:armenzg]
	Comment 22 • 11 years ago

Diagnostics does not help.

Change memory, re-image (bug 864979) and try again.

If not, decommission.

	Aki Sasaki (not active)
	Comment 23 • 11 years ago

Already puppetized.
Updated the old password + autologin.
Back in the pool, for good or ill.

	Phil Ringnalda (:philor)
	Comment 24 • 11 years ago

Yeah, ill. It did one try reftest job, finished with an exception, and has since "done" 393 jobs all saying "device not configured" the first time it tries to write anything to disk, setting retry on everything else and burning talos.

	Nick Thomas [:nthomas] (UTC+12) Reporter
	Comment 25 • 11 years ago

Rebooted and disabled in slavealloc to stop the rot. Decommission time ?

	Chris Cooper [:coop] (he/him)
	Comment 26 • 11 years ago

Sent for decomm in bug 885875.