All of these builds have failures yet half of them show up in green: https://tbpl.mozilla.org/?tree=Jetpack&rev=4a4077dc6705

(You're probably already aware of this, but just in case... :-)) TBPL uses the result state specified by buildbot; buildbot finished this step with: { 8789 of 8789 tests passed. Total time: 492.354662 seconds Program terminated successfully. Total time for all tests: 882.965177 seconds All tests were successful. Ship it! TinderboxPrint:Jetpack<br/>9090/1 program finished with exit code 0 elapsedTime=921.447754 ========= Finished 'python scripts/buildfarm/utils/run_jetpack.py ...' (results: 0, elapsed: 15 mins, 22 secs) (at 2013-09-05 08:48:02.650783) ========= } So looks like: a) run_jetpack.py is miss-summarising test pass/fail counts [1], since zero are listing as failing above. b) By inspection, it doesn't seem like run_jetpack.py takes into account the failed test count from (a), and instead only uses exit code of the called process [2]. c) In this particular case, the jetpack test runner that run_jetpack.py calls, isn't exiting with a non-zero exit code. [1] https://hg.mozilla.org/build/tools/file/bb01b4c812f0/buildfarm/utils/run_jetpack.py#l70 [2] https://hg.mozilla.org/build/tools/file/bb01b4c812f0/buildfarm/utils/run_jetpack.py#l276

(Tinderbox is no more)

(In reply to Ed Morley [:edmorley UTC+1] from comment #1) > a) run_jetpack.py is miss-summarising test pass/fail counts [1], since zero > are listing as failing above. > b) By inspection, it doesn't seem like run_jetpack.py takes into account the > failed test count from (a), and instead only uses exit code of the called > process [2]. > c) In this particular case, the jetpack test runner that run_jetpack.py > calls, isn't exiting with a non-zero exit code. d) summarizeJetpackTestLog() isn't taking into account the cfx tests (not sure if this is expected or not). The regexp used is r"(\d+) of (\d+) tests passed"... { Testing cfx... .................................................................................... ---------------------------------------------------------------------- Ran 84 tests in 26.984s }

Hmm I misread "8789 of 8789 tests passed." to be the overall summary, whereas that is also from the called process. summarizeJetpackTestLog() actually only outputs the 'TinderboxPrint:Jetpack<br/>9090/1', which has correctly identified the failure. Therefore (a) isn't a problem, though (b) to (d) still are.