Starting the early morning of January 18th, it's done just over 200 runs without ever hitting anything other than exception or retry.

I called './stop_cp.sh tegra-100' on foopy12 to (hopefully) prevent it doing more jobs.

According to the tegra dashboard and buildbot tegra-100 is up and running.

Needs a reimage. |Unable to properly uninstall org.mozilla.fennec]| Joel claims it doesn't technically have fennec "installed" but its files are still present.

I marked as offline-* and ran stop_cp on foopy12

Re-imaged and initialized.

Up on foopy12

SUTAgent not present; time for recovery

Ran ./stop_cp.sh

Brought back to life

Last job today @ Dec 10 07:33 -- even powercycling its not coming back up as pingable, please reimage.

--- tegra-100.build.mtv1.mozilla.com ping statistics --- 7 packets transmitted, 7 packets received, 0.0% packet loss

reopen as this is a releng bug (dcops one is the tegra-recovery which blocks this). Ashlee did you [dcops] do anything here, or did it recover itself somehow?

(In reply to Justin Wood (:Callek) from comment #12) > Ashlee did you [dcops] do anything here, or did it recover itself somehow? Just got answer on IRC that it was reimaged. We'll still need to spin it back up though.

Its last successful job was on April 23rd, which is awful even by the lax standards of tegras.

You can see some of its handiwork in bug 690176, when we ignore the "DMError: Non-zero return code for command: ['/system/bin/logcat', '-c'] (output: 'link_image[2046]: failed to link su" and just call it initialization timed out, though mostly it just likes to set retry.

I stopped it.

Sending this slave to recovery -->Automated message.

Successfully recovered.

2014-01-16 10:42:40 tegra-100 p online active OFFLINE :: error.flg [Automation Error: Unable to connect to device after 5 attempts] pdu reboot didn't help

SD card reimaged/flashed.

Passing tests again... Seems ok.

Attempting SSH reboot...Failed. Attempting PDU reboot...Failed. Filed IT bug for reboot (bug 982004)

looks like it's in prod and running green :)

Attempting SSH reboot...Failed. Attempting PDU reboot...Failed. Filed IT bug for reboot (bug 995105)

Attempting SSH reboot...Failed. Attempting PDU reboot...Failed. Filed IT bug for reboot (bug 998120)

replaced SD card, flashed and imaged tegra. [vle@admin1a.private.scl3 ~]$ telnet tegra-100.tegra.releng.scl3.mozilla.com 20701 Trying 10.26.85.76... Connected to tegra-100.tegra.releng.scl3.mozilla.com. Escape character is '^]'. $>^]q

Attempting SSH reboot...Failed. Attempting PDU reboot...Failed. Filed IT bug for reboot (bug 999233)

replaced SD card, flashed and imaged tegra. [vle@admin1a.private.scl3 ~]$ telnet tegra-100.tegra.releng.scl3.mozilla.com 20701 Trying 10.26.85.76... Connected to tegra-100.tegra.releng.scl3.mozilla.com. Escape character is '^]'. $>^]q

Attempting SSH reboot...Failed. Attempting PDU reboot...Failed. Filed IT bug for reboot (bug 1003404)

formatted, flashed and reimaged. [vle@admin1a.private.scl3 ~]$ telnet tegra-100.tegra.releng.scl3.mozilla.com 20701 Trying 10.26.85.76... Connected to tegra-100.tegra.releng.scl3.mozilla.com. Escape character is '^]'. $>^]q

Attempting SSH reboot...Failed. Attempting PDU reboot...Failed. Filed IT bug for reboot (bug 1008638)

formatted, flashed and reimaged. [vle@admin1a.private.scl3 ~]$ telnet tegra-100.tegra.releng.scl3.mozilla.com 20701 Trying 10.26.85.76... Connected to tegra-100.tegra.releng.scl3.mozilla.com. Escape character is '^]'. $>^]q

Dying with failures like this: https://tbpl.mozilla.org/php/getParsedLog.php?id=40570696&tree=Mozilla-Inbound Disabled in slavealloc.

formatted SD card, flashed and reimaged tegra. vle@vle-10516 ~ $ telnet tegra-100.tegra.releng.scl3.mozilla.com 20701 Trying 10.26.85.76... Connected to tegra-100.tegra.releng.scl3.mozilla.com. Escape character is '^]'. $>^] telnet> q