Open
Bug 1460901
Opened 7 years ago
Updated 2 years ago
./mach reftest --verify doesn't work properly, ignores actual failures
Categories
(Testing :: General, defect, P3)
Tracking
(Not tracked)
NEW
People
(Reporter: kats, Unassigned)
References
(Blocks 1 open bug)
Details
Attachments
(1 file)
(deleted),
text/plain
|
Details |
I ran this command:
MOZ_WEBRENDER=1 ./mach reftest layout/reftests/transform-3d/animate-preserve3d-child.html --verify 2>&1 | tee log
with some local patches (that affect the reftest, but not mach or the mochitest harness).
As the output was going by I saw what looked like a failure. But at the end of the run, the test verification summary showed everything passed. Good thing I captured all the output into a logfile, because I double-checked, and sure enough, there were some failures during the runs that the harness decided to ignore for no apparent reason:
$ grep "UNEXPECTED" log
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/kats/zspace/mozilla-w2/layout/reftests/transform-3d/animate-preserve3d-child.html == file:///c:/Users/kats/zspace/mozilla-w2/layout/reftests/transform-3d/animate-preserve3d-ref.html | image comparison, max difference: 255, number of differing pixels: 18228
REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/kats/zspace/mozilla-w2/layout/reftests/transform-3d/animate-preserve3d-child.html == file:///c:/Users/kats/zspace/mozilla-w2/layout/reftests/transform-3d/animate-preserve3d-ref.html | image comparison, max difference: 255, number of differing pixels: 18228
The full log is attached.
Reporter | ||
Updated•7 years ago
|
OS: Unspecified → Windows 10
Hardware: Unspecified → x86_64
Version: 59 Branch → Trunk
Reporter | ||
Comment 1•7 years ago
|
||
Not sure why I put this in mochitest, since this was a reftest run.
Component: Mochitest → General
Summary: ./mach test --verify doesn't work properly, ignores actual failures → ./mach reftest --verify doesn't work properly, ignores actual failures
Updated•7 years ago
|
Assignee: nobody → gbrown
Updated•7 years ago
|
Priority: -- → P2
Updated•6 years ago
|
Blocks: test-verify
Updated•6 years ago
|
Assignee: gbrown → nobody
Priority: P2 → P3
Updated•2 years ago
|
Severity: normal → S3
You need to log in
before you can comment on or make changes to this bug.
Description
•