I've seen two occurences of this problem; one with a mochitest from bug 471281 and the other with a crashtest from bug 492186. Details on the mochitest case: http://hg.mozilla.org/mozilla-central/rev/b116b49459f8 checked in a fix to arcTo stuff and changed the tests. These tests passed on the normal Firefox unittest box, but failed on the mochitest box: http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1245263253.1245264816.16573.gz http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-unittest/1245261825/firefox-3.6a1pre.en-US.mac.dmg The next run passed: http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1245264054.1245266769.19988.gz http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-unittest/1245262441/firefox-3.6a1pre.en-US.mac.dmg And the next run failed again: http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1245265900.1245267488.21136.gz http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-unittest/1245262650/firefox-3.6a1pre.en-US.mac.dmg Then the patch and the test changes were backed out: http://hg.mozilla.org/mozilla-central/rev/ac55c0c1c34a and the machine was green for a while... until http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1245279898.1245281588.18871.gz http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-unittest/1245278138/firefox-3.6a1pre.en-US.mac.dmg failed when the disabled tests started to pass unexpectedly. All these things only happened on the Mac boxes, until http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1245335889.1245338355.31941.gz http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-unittest/1245333067/firefox-3.6a1pre.en-US.linux-i686.tar.bz2 reported the same "unexpected pass" on a Linux box. Details on the crashtest case: http://hg.mozilla.org/mozilla-central/rev/7fcb443a08ce fixed a crash and added a crashtest. This crashtest passed on the normal box, but failed on the Mac everythingelse box: http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1245278334.1245279720.15019.gz http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-unittest/1245275333/firefox-3.6a1pre.en-US.mac.dmg

I think it's unlikely that flaky tests are responsible for this phenomenon: - The failing tests passed consistently on the unittest boxes on the Firefox tree. - I've never seen the "unexpected pass" on the arcTo test before; if the arcTo tests were flaky, they wouldn't have started to act wonky only after the backout. - jrmuizel says that intermittent failures on the arcTo test don't make sense.

The arcTo tests that fail are exactly the ones where the expected result was changed. In these cases the failures seem like they would come from running old builds on a new test suite.

catlee was able to determine that automation.py was using the original DIST_BIN in this call to environment(): http://mxr.mozilla.org/mozilla-central/source/build/automation.py.in#588 Which means that sometimes builds would find a build in the original build path and load libxul et. al from there, giving you a build that functions like that other build. Badness ensues.

Comment on attachment 383970 [details] [diff] [review] Set xrePath when creating the environment for the main process What a stupid mistake!

Pushed to m-c: http://hg.mozilla.org/mozilla-central/rev/a4a15aa37fa5 We should land this on 1.9.1 too.

I hate to say it, but there has been another unexplained failure, this time it's the reftest from bug 481614: http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1245377401.1245379142.11871.gz See also bug 481614 comment 15 / 16.

This is still happening on at least reftests.

Comment on attachment 384099 [details] [diff] [review] Pass in xrePath to runApp for reftests Another good catch, thanks!

Pushed the second patch to m-c: http://hg.mozilla.org/mozilla-central/rev/383949cfb7dc

Pushed both patches to 1.9.1: http://hg.mozilla.org/releases/mozilla-1.9.1/rev/814fd57632f9 http://hg.mozilla.org/releases/mozilla-1.9.1/rev/5d021be5fac5