If 'make check' is enabled in cross-OSX builds (bug 1340698), testing/xpcshell/selftest.py tries to run despite being listed with 'skip-if = os == "mac"' in the manifest (it is skipped due to bug 1255588). I'm not sure why this isn't skipped in a cross-build, since mozinfo.json still defines os as "mac" in that case.

ahal and I were talking about selftest.py. It shouldn't be incredibly hard to run as a standalone job, it just needs access to the build + the xpcshell test package. My thought was just to stick selftest.py in the xpcshell test package and then run it like it was any other xpcshell job, but ahal thought that would be complicated because of the way the mozharness scripts are written. His idea was to run it as one of the new Python test tasks that the mozbase tests are using, but it would need to also depend on the build task.

I'm about to land the ability to depend on a build task in bug 1336559. Should be in by early next week. I haven't tested pulling these tasks out on OSX yet though. It might *just work* or there might be some taskcluster changes required.

The weird part to me was that they run at all on a cross build with 'make check' enabled, since it is supposed to be disabled on mac. But since we're not going to enable 'make check' there anyway I don't think this will block us. Still, it's great that we'll have a solution ready in the event that we update our builders to 10.9+ and want to re-enable selftest.py.

I don't think the `python-test` command is loading the generated mozinfo.json file at all, so it's getting the mozinfo defaults, where `os` is the host OS: https://dxr.mozilla.org/mozilla-central/rev/5069348353f8fc1121e632e3208da33900627214/python/mach_commands.py#50

I did a bit of digging into what would be involved with re-using run-task for this and posted my findings here: https://bugzilla.mozilla.org/show_bug.cgi?id=1337903#c2 I think jgriffin mentioned that we won't have any buildbot hardware in the new data centre. And as far as I can tell, the in-tree taskcluster config support for the "test kind" (traditional mozharness jobs) hasn't been implemented on OSX either. So even going the mozharness route would require fiddling with taskcluster configs.. Given that, I'd recommend just focusing on the "run-task" route.

Since we're not planning to turn 'make check' on in cross-OSX builds, this no longer blocks the OSX Taskcluster work. However, it is still useful to pull this out into a separate task for performance reasons, so I'm renaming this bug.

ahal has some patches in bug 1048446 that will be useful for this as well (specifically, downloading & extracting the packages for these kinds of tests, and a better way of conditionally including the test manifest). So once those are done this bug should be pretty straightforward.