Armen [:armenzg] Reporter
	Description • 16 years ago

The windows nightly that I triggered around few hours ago in staging failed in slave 15.
command timed out: 1200 seconds without output

Log:
http://staging-master.build.mozilla.org:8010/builders/WINNT%205.2%20mozilla-central%20nightly/builds/173/steps/shell_16/logs/stdio

	John O'Duinn [:joduinn] (please use "needinfo?" flag)
	Comment 1 • 16 years ago

Gragging this, as I must have missed something in the setup.

	Nick Thomas [:nthomas] (UTC+12)
	Comment 2 • 16 years ago

|ssh -l cltbld staging-stage.build.mozilla.org| works fine on the msys prompt, as does the mkdir command in the log. I did notice that staging-stage was slow to respond though, I wonder which storage it's on.

	Nick Thomas [:nthomas] (UTC+12)
	Comment 3 • 16 years ago

staging-stage is on d-sata-build-002, which is slower than fcal. We should enumerate the jobs staging-stage is doing and consider moving it to faster storage.

	John O'Duinn [:joduinn] (please use "needinfo?" flag)
	Comment 4 • 16 years ago

1) I also confirmed what nthomas mentioned in comment#2 below. Looks like setup of ssh keys on moz2-win32-slave15 are fine. Adjusting summary to match.

2) Slow staging-stage should be impacting other staging machines also, but we've not see this before or since, afaict. I wonder if there were other problems with the ESX servers at the time causing everything to run slowly? Putting back in the pool to watch and see if this happens again.

	bhearsum@mozilla.com (:bhearsum)
	Comment 5 • 16 years ago

Are you sure this was caused from slow staging-stage? The log is gone now, but I suspect that it was an ssh key problem (despite comment #4).

	John O'Duinn [:joduinn] (please use "needinfo?" flag)
	Comment 6 • 16 years ago

Tweaking summary, as moz2-win32-slave15 was confirmed ok in comment#2 and comment#4, and is now in production. 

Going to leave this bug open for a little longer to see if it happens again, with any other machine.

	bhearsum@mozilla.com (:bhearsum)
	Comment 7 • 16 years ago

So, I saw this happening again today, on moz2-win32-slave03. As it turns out, the staging-stage.b.m.o host key was never accepted when this was moved back to staging. I logged in, accepted it. I also logged into moz2-win32-slave15 and indeed, the host key was not accepted there, either.

With the host key accepted on moz2-win32-slave03 builds are uploading fine now.