Closed
Bug 913169
Opened 11 years ago
Closed 7 years ago
Jobs in automation sometimes show up as successful even when tests have failed
Categories
(Add-on SDK Graveyard :: General, defect)
Add-on SDK Graveyard
General
Tracking
(Not tracked)
RESOLVED
INCOMPLETE
People
(Reporter: mossop, Unassigned)
References
Details
All of these builds have failures yet half of them show up in green: https://tbpl.mozilla.org/?tree=Jetpack&rev=4a4077dc6705
Comment 1•11 years ago
|
||
(You're probably already aware of this, but just in case... :-))
TBPL uses the result state specified by buildbot; buildbot finished this step with:
{
8789 of 8789 tests passed.
Total time: 492.354662 seconds
Program terminated successfully.
Total time for all tests: 882.965177 seconds
All tests were successful. Ship it!
TinderboxPrint:Jetpack<br/>9090/1
program finished with exit code 0
elapsedTime=921.447754
========= Finished 'python scripts/buildfarm/utils/run_jetpack.py ...' (results: 0, elapsed: 15 mins, 22 secs) (at 2013-09-05 08:48:02.650783) =========
}
So looks like:
a) run_jetpack.py is miss-summarising test pass/fail counts [1], since zero are listing as failing above.
b) By inspection, it doesn't seem like run_jetpack.py takes into account the failed test count from (a), and instead only uses exit code of the called process [2].
c) In this particular case, the jetpack test runner that run_jetpack.py calls, isn't exiting with a non-zero exit code.
[1] https://hg.mozilla.org/build/tools/file/bb01b4c812f0/buildfarm/utils/run_jetpack.py#l70
[2] https://hg.mozilla.org/build/tools/file/bb01b4c812f0/buildfarm/utils/run_jetpack.py#l276
Comment 2•11 years ago
|
||
(Tinderbox is no more)
Summary: Tinderbox builds sometimes show up as successful when they have failed → Jobs in automation sometimes show up as successful even when tests have failed
Comment 3•11 years ago
|
||
(In reply to Ed Morley [:edmorley UTC+1] from comment #1)
> a) run_jetpack.py is miss-summarising test pass/fail counts [1], since zero
> are listing as failing above.
> b) By inspection, it doesn't seem like run_jetpack.py takes into account the
> failed test count from (a), and instead only uses exit code of the called
> process [2].
> c) In this particular case, the jetpack test runner that run_jetpack.py
> calls, isn't exiting with a non-zero exit code.
d) summarizeJetpackTestLog() isn't taking into account the cfx tests (not sure if this is expected or not). The regexp used is r"(\d+) of (\d+) tests passed"...
{
Testing cfx...
....................................................................................
----------------------------------------------------------------------
Ran 84 tests in 26.984s
}
Comment 4•11 years ago
|
||
Hmm I misread "8789 of 8789 tests passed." to be the overall summary, whereas that is also from the called process. summarizeJetpackTestLog() actually only outputs the 'TinderboxPrint:Jetpack<br/>9090/1', which has correctly identified the failure. Therefore (a) isn't a problem, though (b) to (d) still are.
Priority: -- → P1
Assignee: nobody → dtownsend+bugmail
Reporter | ||
Updated•11 years ago
|
Assignee: dtownsend+bugmail → nobody
Updated•10 years ago
|
Blocks: sdk-test-issues
Updated•9 years ago
|
Priority: P1 → --
Comment 6•7 years ago
|
||
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → INCOMPLETE
You need to log in
before you can comment on or make changes to this bug.
Description
•