If you run the webauthn/ web platform tests in a debug build, it hits a fatal assertion in shutdown and crash, but the job is still green.

Here's an example log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=222252847&repo=mozilla-central&lineNumber=28226

Right now it is okay to ignore this failure because there are leaks to be fixed, but it seems bad that a crash is ignored, unless it is whitelisted in some way that I can't see.

I'm actually not sure why the crash isn't turning the job orange in that case, but I think some recent updates that haven't landed yet made that crash show up, upon which I had to make us not fail the job for crashes in more cases. The problem is that we don't have any way to explictly ignore crashes not associated with a test (e.g. shutdown crashes). That means any test addition or modification causing such a crash would end up blocking all other imports until it's fixed, which isn't acceptable (this is basically the situation with the current set of updates due to harness changes).

It's possible we should have some other way of allowing crashes on a per-directory basis. Now that we have the leak stuff, a lot more of the infrastructure for that exists, so it wouldn't be too hard to build I think.

Thanks for the explanation. So if it was crashing during a test, then there would be an orange. That's good.

I don't know this is a separate issue or not, but the "tab missing output line for total leaks!" should also be a failure. That does seem like the same basic problem of a failure not associated with a test.

Can tests be disabled? Right now, failing tests seem to still run, but if you add a test and then there are crashes maybe the new test could not be run? I suppose computing a minimal set of tests not to run might be a pain.

I'll just change this to be about the "tab missing output line for total leaks!" failure being ignored. It feels like that could be whitelisted as part of the same leak infrastructure that already exists.