Open Bug 1460901 Opened 7 years ago Updated 2 years ago

./mach reftest --verify doesn't work properly, ignores actual failures

Categories

(Testing :: General, defect, P3)

x86_64
Windows 10
defect

Tracking

(Not tracked)

People

(Reporter: kats, Unassigned)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

Attached file log (deleted) —
I ran this command: MOZ_WEBRENDER=1 ./mach reftest layout/reftests/transform-3d/animate-preserve3d-child.html --verify 2>&1 | tee log with some local patches (that affect the reftest, but not mach or the mochitest harness). As the output was going by I saw what looked like a failure. But at the end of the run, the test verification summary showed everything passed. Good thing I captured all the output into a logfile, because I double-checked, and sure enough, there were some failures during the runs that the harness decided to ignore for no apparent reason: $ grep "UNEXPECTED" log REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/kats/zspace/mozilla-w2/layout/reftests/transform-3d/animate-preserve3d-child.html == file:///c:/Users/kats/zspace/mozilla-w2/layout/reftests/transform-3d/animate-preserve3d-ref.html | image comparison, max difference: 255, number of differing pixels: 18228 REFTEST TEST-UNEXPECTED-FAIL | file:///c:/Users/kats/zspace/mozilla-w2/layout/reftests/transform-3d/animate-preserve3d-child.html == file:///c:/Users/kats/zspace/mozilla-w2/layout/reftests/transform-3d/animate-preserve3d-ref.html | image comparison, max difference: 255, number of differing pixels: 18228 The full log is attached.
OS: Unspecified → Windows 10
Hardware: Unspecified → x86_64
Version: 59 Branch → Trunk
Not sure why I put this in mochitest, since this was a reftest run.
Component: Mochitest → General
Summary: ./mach test --verify doesn't work properly, ignores actual failures → ./mach reftest --verify doesn't work properly, ignores actual failures
Assignee: nobody → gbrown
Priority: -- → P2
Assignee: gbrown → nobody
Priority: P2 → P3
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: