After seeing Mossop's suggestion on bug 445611, and also noticing that most of the delay in waiting for unit test results on the Firefox tree is due to the unit tests not having a pool of slaves to pull from, I'd like to make a proposal: We should combine the unit test and depend builders, so that the builders do a normal build (with the default of --enable-tests), package and upload it, and then run the unit tests. Currently the build machines build the latest checkins, then upload those builds for the Talos machines to test. The unit test machines build their own builds, and then run the tests on them. There's been talk about making it possible to run tests on an arbitrary build, but that doesn't help the build time much, it just makes the testing machine wait on the build machine (much like with Talos currently). I don't see any reason that the build machines can't just run the tests themselves. We'd drop the unit test machines as a separate builder, and the build machine (with their nice big pool of slaves) would substitute. With more slaves running unit tests, it should be much easier to pinpoint regressions. Some potential downsides: * Each build will take more time, since it takes ~40 minutes to run the tests * Tinderbox doesn't actually show you any useful status until the entire build + test cycle has completed, so it's difficult to tell if your patch has successfully built. * We would probably need to split the Win32 builder in two, one with PGO and one without, since the unittest builder currently builds without (for cycle time reasons). The upside to this is that we'd get a fast(er) cycling win32 build box, but still have the slower PGO box for accurate perf measurement.

Dependent on bug 460282, since we should make sure we're not shipping extra stuff (like reftest) if we package up builds with --enable-tests.

One way to alleviate the downside is for the buildbot to notify tinderbox twice for each "build": once for the build-proper and once for the unit tests which are run subsequently. If this sounds like it would be useful, I'd be happy to try and prepare a patch to the tinderbox notifier.

(In reply to comment #2) > One way to alleviate the downside is for the buildbot to notify tinderbox twice > for each "build": once for the build-proper and once for the unit tests which > are run subsequently. If this sounds like it would be useful, I'd be happy to > try and prepare a patch to the tinderbox notifier. We might be able to do it this way, too: https://bugzilla.mozilla.org/show_bug.cgi?id=342249 (add the ability for tinderbox to append to the log).

This could also make fixing bug 372581 much easier.

Erm.. we run unit tests on three different machines per target, to have a way of detecting spurious oranges. But we only build on one machine. Doing it this way would mean that we would only be doing one unit test run, which isn't good. What's wrong with what's proposed in bug 421611? The build box should build with --enable-tests and we should ship the pageloader/reftest drivers in the hourlies (and probably nightlies), and then it should do an additional pass to package up the _testing directory and any deps. That seems much simpler than this.

I disagree that it's simpler, but you do have a point about the multiple-unittest-slaves. We could probably package and upload the test harness+tests for mochitest without a lot of effort. Reftest is a bit harder, we need to package the reftest extension, plus the tests, which just live in the srcdir. This means you'd have to package all the reftest.list files (which have relative include paths) and the test files they mention, and probably all the dependent files that aren't mentioned in the manifests but the reftests need anyway (like image files).

Regardless, I have discussed this all in detail on bug 421611.

We want to be able to run unittest, without rebuilding, on any slave - regardless of whether that slave was the slave used to generate the build being tested. bug#421611 is tracking that project. Therefore marking this bug as WONTFIX.