As philor correctly pointed out, make check is a terrible place for tests which suffer from any degree of intermittency, as failed builds that get retriggered waste a lot of machine resources. Since the mozbase tests do suffer some intermittent oranges, and that doesn't seem likely to completely disappear, we should move the tests out of make check and into a separate test job. This would involve making a mozharness script and then getting them scheduled everywhere, plus some minor TBPL changes to add a new test job name. None of this should be too difficult.

I don't think this is hugely valuable, honestly. This is sort of a stress test of the mozbase tests, since they've never been run this frequently in automation in the past. I think we should table this for a while and revisit it if it continues to be a problem. "make check" has been historically where we have all of our harness/framework-type tests. Running a separate test suite that doesn't actually test the product being built would be a departure from what we currently do.

Yeah, everybody who never ever watches the tree keeps saying that this is just fine. Everybody who does watch the tree keeps saying it is not fine. Wonder who is right?

FWIW which is essentially nothing since it won't be done I would advocate moving all[*] of the make check tests to their own target. I would also advocate making adding a new test suite (I guess we'd call these a TBPL suite in current nomenclature) an intent-driven activity: that is to say there should be instructions as to how to add a test suite and whatever other rules and guidelines go with that. For instance (and only as a random specific example amongst many), I don't know if it is ever stated that tests shouldn't touch (wild) net; however, again AIUI, this is policy for "official" (that is, TBPL-displayed and cause for backout) tests. I would suppose that these instructions would be on https://developer.mozilla.org/en-US/docs/Mozilla/QA/Automated_testing or linked to therefrom, but AFAIK there is no such set of instructions. I hypothesize that the reaction to this will be some subset of "no reason to be over-regulatory", "not enough man hours to tackle this issue", "not a priority", or "we haven't needed this before". It would take a considerable amount of time and assessment of how we test vs how we want to test to move towards a system where these issues are resolvable, let alone, as always, things are in flux. -- [*] perhaps some of them do make sense to be with the build; however, I would guess not very many. I have not audited.

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #1) > > "make check" has been historically where we have all of our > harness/framework-type tests. Running a separate test suite that doesn't > actually test the product being built would be a departure from what we > currently do. I think this distinction will be going away. Eventually, we'll want to run at least some of the 'make check' tests on B2G, and we can't do that during build jobs, since many of them need to be run on device (jit tests, for example). Since "product tests" do not get blocked on the result of "make check", I don't see a compelling reason to require that all "non product" tests be run in the build job; I think TBPL could easily be tweaked to make that distinction in other ways. The main disadvantage of breaking the tests out unnecessarily is the additional load on test slaves, along with the additional setup/teardown time new jobs add. We could probably try to calculate a breakeven point in terms of orange frequency; a much better solution would be to modify the buildbot scheduling involved in a retrigger, so that build retriggers would not automatically kick off a full suite of tests. But this is buildbot work and I don't know how tricky that would be. In the short term, we could elect to aggressively disable-fix-re-enable tests that cause oranges, in hope of keeping our orange to a bare minimum, and see if that's sufficient.