If we get a NoSuchWindow error from marionette, then something happend that caused the window to disappear, which we can interpret as a crash. This may not be true if we accidentially navigated the window to file:// or something else that causes a remoteness change, but those are basically marionette bugs so in the long term this should be correct.

This means that if closing the windows is problematic it affects the status of the current test not the next one. Depends on D16830

This behaviour was leading to intermittent failures with shutdown crashes, as according to timing we would sometimes get a crash dump file and sometimes not (or we would sometimes get a shutdown crash and sometimes not). Until we either have the ability to allow certain crashes or the ability to set multiple expected statuses, don't change the test status when we find a process crashed (the crash will still be logged). Depends on D16831

It can happen that a tab has an expected crash in wpt. As a result we won't get a valid leak file for that content process. But this shouldn't cause the tests to fail. In the absence of a more precise mechanism we just disable that check for the tab process, but it would be an improvement to know which processes crashed and ensure that only those ended up with unexpected leak files. Depends on D16832

Instead of printing one log message per line about the fact the output was unstructured, just print one per group of unstructured lines. Depends on D16833

Sometimes it's unclear which of the errors are leading to a failure. For example wpt sometimes has PROCESS-CRASH lines that don't contribute to a failure. To make this easier to understand record the reason for the failure in the log so it appears in the summaries. Depends on D16834

For shutdown crashes in wpt wse need to ignore the crashes for now until we have a way of allowing specific ones. So just have a global flag that permits this. Depends on D16835

This works around the fact that we sometimes get an IOError trying to read a dump file on Windows. Depends on D16836

Depends on D16837

Depends on D16838

Pushed by james@hoppipolla.co.uk: https://hg.mozilla.org/integration/mozilla-inbound/rev/a1544d1ae5be Interpret the window being missing as a crash, r=ato https://hg.mozilla.org/integration/mozilla-inbound/rev/2c82daad929f Close windows after test finishes rather than before next test, r=ato https://hg.mozilla.org/integration/mozilla-inbound/rev/8d89fc8ca162 Don't set status to crash if we find a crash dump, r=ato https://hg.mozilla.org/integration/mozilla-inbound/rev/a2e2dd4e4539 Ignore missing leak files for tab process in wpt, r=ato https://hg.mozilla.org/integration/mozilla-inbound/rev/f7f763f7c5b4 Handle unstructured lines in mozlog output better in mozharness, r=ahal https://hg.mozilla.org/integration/mozilla-inbound/rev/765ec7f39243 Give mozharness output to indicate why a job failed, r=ahal https://hg.mozilla.org/integration/mozilla-inbound/rev/add585284f4e Allow ignoring process_crash in determing mozharness job status, r=ahal https://hg.mozilla.org/integration/mozilla-inbound/rev/b6e31c3bfb6e Handle case where mozcrash can't read the dump file, r=ato https://hg.mozilla.org/integration/mozilla-inbound/rev/c1a9c866d74b Improve logging for crashes, r=ato https://hg.mozilla.org/integration/mozilla-inbound/rev/3d96125d18d7 Provide a helpful error message when we get a JS Document Unloaded exception, r=ato

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/15077 for changes under testing/web-platform/tests