Closed
Bug 1401997
Opened 7 years ago
Closed 7 years ago
Report the reftest difference for fails-if tests that do fail
Categories
(Testing :: Reftest, enhancement)
Tracking
(firefox58 fixed)
RESOLVED
FIXED
mozilla57
| Tracking | Status | |
|---|---|---|
| firefox58 | --- | fixed |
People
(Reporter: kats, Assigned: kats)
Details
Attachments
(1 file)
Right now, when reftests are run, we report the pixel differences for tests marked fuzzy, and for tests that result in UNEXPECTED-FAIL. I think we should just always report it.
In particular, it's useful to see if tests marked as fails[-if] are actually very close to passing (e.g. they have a 1-pixel difference). In such cases the harness will treat the test as TEST-KNOWN-FAIL which is fine but if the difference is small the test could just as well be annotated as fuzzy-if and be made passing.
| Comment hidden (mozreview-request) |
|
||
Comment 2•7 years ago
|
||
| mozreview-review | ||
Comment on attachment 8910786 [details]
Bug 1401997 - Always report pixel difference counts for reftests.
https://reviewboard.mozilla.org/r/182246/#review187586
this seems very useful :)
Attachment #8910786 -
Flags: review?(jmaher) → review+
Pushed by kgupta@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/2b87ee06ac04
Always report pixel difference counts for reftests. r=jmaher
|
||
Comment 4•7 years ago
|
||
| bugherder | ||
Status: NEW → RESOLVED
Closed: 7 years ago
status-firefox58:
--- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla57
You need to log in
before you can comment on or make changes to this bug.
Description
•