At least, I think hitting the buildbot timeout is the reason that in the post bug 505512 world http://tinderbox.mozilla.org/showlog.cgi?log=TraceMonkey/1286215375.1286218504.15064.gz&fulltext=1 was red rather than orange.

Looks like in my bug 505512 patch I changed the behaviour of the unittest steps' evaluateCommand. They used to take any non SUCCESS status from ShellCommand and turn it into WARNINGS -- I changed that in such a way that FAILURE, EXCEPTION, and RETRY overrode WARNINGS. In this new, purple, world, we want EXCEPTION and RETRY to make it through, but FAILURE (generally, return code != 0), should be treated as WARNINGS for unittests. I'm going to run this through staging, but how does it look, Rail?

Comment on attachment 481216 [details] [diff] [review] override ShellCommand setting FAILURE Thanks for the detailed comments, it helped me to understand the logic faster. :) Looks good.

I repro'ed the failure in staging by killing firefox.exe as a test was running. After applying this patch and doing the same thing, the overall build was orange instead of red. This is ready to land IMHO.

Comment on attachment 481216 [details] [diff] [review] override ShellCommand setting FAILURE http://hg.mozilla.org/build/buildbotcustom/rev/7147fc9c7c69

This should be fixed going forward.