After today's merge of 50 to mozilla-beta, ASan gtest is permafailing like https://treeherder.mozilla.org/logviewer.html#?job_id=1611597&repo=mozilla-beta#L6662 19:30:33 INFO - TEST-START | GeckoMediaPlugins.GMPTestCodec 19:30:33 INFO - [1970] WARNING: Failed to launch geckomediaplugin subprocess: file /builds/slave/m-beta-l64-asan-00000000000000/build/src/ipc/glue/GeckoChildProcessHost.cpp, line 570 19:30:33 INFO - [1970] WARNING: pipe error (9): Connection reset by peer: file /builds/slave/m-beta-l64-asan-00000000000000/build/src/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 320 19:35:34 INFO - gtest TEST-UNEXPECTED-FAIL | gtest | timed out after 1200 seconds 19:35:34 ERROR - Return code: 1 One of several things keeping mozilla-beta closed after the merge.

Did something change on the configuration of the ASAN machines, like a new kernel version for example?

These tests pass fine on Aurora. The logs on Aurora look very different from the logs on Beta. It looks like TaskCluster is being used on Aurora but not so much on Beta. My guess is there's some key difference in the environment between the Aurora and Beta machines that's causing this.

They passed on aurora when they ran on buildbot machines on this same code when it lived there, 28 hours ago, https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=c174d0c3195371011b3bf8044512c06c26f5c741&selectedJob=3584994

(In reply to Chris Pearce (:cpearce) from comment #2) > These tests pass fine on Aurora. The logs on Aurora look very different from > the logs on Beta. It looks like TaskCluster is being used on Aurora but not > so much on Beta. My guess is there's some key difference in the environment > between the Aurora and Beta machines that's causing this. catlee, do you know who could help here ?

Or, since Taskcluster does run ASan as tier-2 on beta, https://treeherder.mozilla.org/logviewer.html#?job_id=1610241&repo=mozilla-beta#L6775 is them failing the exact same way as the buildbot ones, on beta.

Nobody from releng can, because it's not releng: the difference is in RELEASE_BUILD defines somewhere, not the hardware it's running on.

Oh, out-freakin-standing! It has actually been broken for as long as 50 was on aurora, since https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=70ee99f185b48cb59a4ec701282bf6a3fd6ce3e3&filter-searchStr=asan%20gtest was the first buildable revision there. So, pretty clear we don't care, and are quite happy to just star near-permaorange.

Actually, I need this bug to be open, so it can be a dependency for my bug about hiding ASan gtest on release branches, since when 51 hits beta this will continue to fail around half the time, but some new bustage will make the rest of the runs OOM.