In OkVeANnWSf2bQeDwFmQgIA, the raw.log and errorsummary.log show /svg/animations/correct-events-for-short-animations-with-syncbases.html as status=FAIL expected=PASS, but in the full log there's only TEST-KNOWN-INTERMITTENT-FAIL | /svg/animations/correct-events-for-short-animations-with-syncbases.html
https://firefox-ci-tc.services.mozilla.com/tasks/OkVeANnWSf2bQeDwFmQgIA
raw.log: https://firefoxci.taskcluster-artifacts.net/OkVeANnWSf2bQeDwFmQgIA/0/public/test_info//wpt_raw.log
full log: https://firefoxci.taskcluster-artifacts.net/OkVeANnWSf2bQeDwFmQgIA/0/public/logs/live_backing.log

In the raw.log, we can see known_intermittent: ["FAIL"], which means this test is known to intermittently fail.

The task itself is green, which means it's green on Treeherder, which means the failure is hidden to developers and sheriffs and so is not classified as intermittent.

In order to check if a group failed, we check if its status is equal or different to its expected status. In this case they are different, so we consider it a failure (and a regression of the push), even though it actually is very likely intermittent.

James says the long term plan is to build infra to detect where a test goes from intermittent to perma, or maybe look for changes in the rate.

Two possible short term solutions to avoid considering these as failures could be:
I) Report known_intermittent: ["FAIL"] in the errorsummary.log file too;
II) Report expected = ["FAIL", "PASS"] for such tests, and check status in expected rather than status == expected.

Depends on D80264

https://hg.mozilla.org/mozilla-central/rev/ae500dbf197b
https://hg.mozilla.org/mozilla-central/rev/279fd282a0ae