In bug 1353461 we are in the process of switching reftest to "run-by-manifest" mode. This means that between every manifest of tests we will restart Firefox. In general this leads to improved stability, though in a few cases existing tests that happened to depend on side-effects of previous tests can start failing. This is one of those cases. This one looks like a previously filed intermittent (bug 1407479), so I suspect that the change to run-by-manifest simply made it more reproducible. Here's a push with the error: https://treeherder.mozilla.org/#/jobs?repo=try&revision=e796b66e98246f231ded81e88fbb81ca8dfe183d Example log: https://treeherder.mozilla.org/logviewer.html#?job_id=163443226&repo=try&lineNumber=11737 We'll need to fix this, or disable the test on Linux QR before we can land run-by-manifest.

Kats may be interested in this.

If it's just the one test and it's permafailing reliably, then please go ahead and mark it fails-if(webrender). Looks like it already has a fuzzy-if annotation for webrender, you can replace that with fails-if. I'll update my spreadsheet of WR failures to include this.

FWIW: looking at the screenshot, the "failure" here is kind of borderline between "fuzzy" & "fails". (The characters are rasterized a bit differently in the testcase vs. reference -- some a bit wider, some with different serifs. It looks like there's no antialiasing, too, so every differing pixel is black-or-white -- hence the max-difference of 255. Anyway - agreed that we should just update the annotation, to either "fails-if" or extremely-permissive "fuzzy-if", conditioned on webrender.

Thanks, I'll land the annotation as part of the series in bug 1353461 along with a comment pointing back here.

[ Triage 2018/02/21: P3 ]