+++ This bug was initially created as a clone of Bug #494676 +++ This should be as trivial as porting the configuration patch (Bv1).

I know we need it for bug 485032, but I'm curious as to why you think SM & TB need this *now*.

(In reply to comment #1) > I know we need it for bug 485032, but I'm curious as to why you think SM & TB > need this *now*. Ok, maybe I shouldn't have stressed the now, but I'm still curious.

Because it allows anyone to download (a known build) and run tests locally, without needing to (have a) build (environment): quite useful to check everything is fine or to try and reproduce a (reported) failure. No hurry, much less for TB than SM at least, but it seems easy enough that the sooner would be the better...

Right, thinking about this a bit more if we pull in the code and port the build config, then we'll have two extra commands run on the unit test boxes (or maybe more, but I think these would be the vital ones): make package make package-tests The unit test boxes are run as non-static builds therefore as soon as they hit make package they will fail because we've got a #pramga in there that says we must have --enable-static --disable-shared to generate a package. We do need both the package and package-tests because bug 485032 will want to download the two builds generated from the same box. So I think this therefore means we're going to have to manage a packages file on windows which has ifdefs for shared/static. Note that SeaMonkey will also have to do this as well for its builds when it moves to static. Shame we're not on libxul yet.

Ok, let's see if we can get this going on the unit test boxes. We want to end up running make package and make package tests on the unit boxes, and have them upload the results to something like Firefox, e.g. ftp://ftp.mozilla.org/pub/firefox/tinderbox-builds/mozilla-1.9.1-linux-unittest/ Like Firefox we only need the last 12 builds or so. Porting the configuration from bug 494676 (and making sure we've picked up the other patches) should hopefully be enough.

Once bug 599305 lands, and bug 497353 completes, we'll then be able to do this. The bonus is that this will save us some build time on the production builders as we won't need to do separate dep and unit test builds. Here's a rough plan of what we need to do: - Finish landing and getting working the previously mentioned bugs. - Set "ac_add_options --enable-tests" in the mozconfigs for dep/nightly builds (we should check that it doesn't break nightlies). - Enable the "make package-tests" step for the dep/nightly builds. - Enable the "make check" step for those dep builds (but not on nightly builds) - Create separate builders in buildbot-configs for actually downloading and running the packaged tests and get them running. - Turn off the existing check test builders. If possible I'd like to run the package-test builders and the existing check test builders at the same time for a few hours at least.

I think this also depends on bug 594281 - iirc MoCo have two pools of builders - one for build, one for unit tests, and hence effectively get around this problem. We haven't got that, so we need to resolve that here before we enable this on production.

This enables tests for our dep/nightly builds - the same way as Firefox does it.

Comment on attachment 479106 [details] [diff] [review] [checked in] Enable building of tests on nightly builds Checked in: http://hg.mozilla.org/build/buildbot-configs/rev/8797216b1038

(In reply to comment #6) > Once bug 599305 lands, and bug 497353 completes, we'll then be able to do this. > The bonus is that this will save us some build time on the production builders > as we won't need to do separate dep and unit test builds. > > Here's a rough plan of what we need to do: > > [...] > - Enable the "make package-tests" step for the dep/nightly builds. If you look in misc.py, there is a generateBranchObjects method, it's huge, but it's what Firefox uses to generate all their stuff for each branch. Looking in there for the NightlyBuildFactory invocation, i can see from its args that what we are interested in is: # This probalby runs 'make pakcage-tests' , not sure about what is responsible for uploading them packageTests=packageTests, # These 2 control which buildbot masters should recieve notifications of builds completion # to trigger the opt test builds # The branch is used to isloate these opt builders, so in our case, we probably want to use # branches like comm-central-linux-opt-unittest or something, just re-use what try gets. unittestMasters=config['unittest_masters'], unittestBranch=unittestBranch, # Not sure if this is relevant, but I wouldn't be surprised if it has to do with where # to upload things, will need digging. tinderboxBuildsDir=tinderboxBuildsDir, So, as a first step, we could just turn on packageTests on comm-central and see if that builds the package tests bundle and uploads it correctly. It's simple and a good place to start.

(In reply to comment #6) > Once bug 599305 lands, and bug 497353 completes, we'll then be able to do this. > The bonus is that this will save us some build time on the production builders > as we won't need to do separate dep and unit test builds. > > Here's a rough plan of what we need to do: > > [...] > - Enable the "make check" step for those dep builds (but not on nightly builds) Why not on nightly builds, might as well, if we do it for dep, should also do it on nightly.

Fix config error in previous patch. (comm-central-trunk vs comm-central)

(In reply to comment #6) > Once bug 599305 lands, and bug 497353 completes, we'll then be able to do this. > The bonus is that this will save us some build time on the production builders > as we won't need to do separate dep and unit test builds. > > Here's a rough plan of what we need to do: > > [...] > - Create separate builders in buildbot-configs for actually downloading and > running the packaged tests and get them running. That's the UnittestPackagedBuildFactory() that does that. Basically, once you have a unittest branch that gets the sendchanges from the slaves when regular builds have completed, you want to create a builder with that factory, and stick it on that branch via an ordinary Scheduler on that fake branch.

(In reply to comment #10) > If you look in misc.py, there is a generateBranchObjects method, it's huge, but > it's what > Firefox uses to generate all their stuff for each branch. There's also a generateCCBranchObjects method that in theory both Thunderbird and SeaMonkey could/should use, but I don't think you do yet. ;-)

(In reply to comment #11) > > - Enable the "make check" step for those dep builds (but not on nightly builds) > > Why not on nightly builds, might as well, if we do it for dep, should also do > it on nightly. At the moment, I think Firefox doesn't run tests on nightly builds, I'm not sure we need to either - if they fail then the tree will be red/orange for half a day and we won't really want to respin a nightly + l10n repacks to get the tree green just for a unit test orange/fail.

{ /pub/firefox/nightly/2010/09/2010-09-29-04-mozilla-central firefox-4.0b7pre.en-US.win32.tests.zip } FF packages and uploads tests. I guess that's what (SM &) TB should do too.

(In reply to comment #16) > FF packages and uploads tests. I guess that's what (SM &) TB should do too. Serge, thanks but that's not what we're discussing at the moment. We are working through this bit by bit and we'll get there in a few days.

Just had a thought, for the packaged unitests builders, do we want to have a single builder per platform that runs check/xpcshell/mozmill in sequence, or break those down into 3 separate builders for each test suite like Firefox does. I like the 2nd option better, it's what Firefox does, and can provide faster turnaround time if there is enough capacity.

(In reply to comment #18) > Just had a thought, for the packaged unitests builders, do we want to have a > single builder per platform that runs check/xpcshell/mozmill in sequence, or > break those down into 3 separate builders for each test suite like Firefox > does. check has to be run on the builder doing the actual build > I like the 2nd option better, it's what Firefox does, and can provide faster > turnaround time if there is enough capacity. Yes, that's fine, just like try server.

(In reply to comment #19) > (In reply to comment #18) > > Just had a thought, for the packaged unitests builders, do we want to have a > > single builder per platform that runs check/xpcshell/mozmill in sequence, or > > break those down into 3 separate builders for each test suite like Firefox > > does. > > check has to be run on the builder doing the actual build Yes, but it also can be run as a packaged test run, or we could skip it there, being somewhat redundant and keep only xpcshell and mozmill as separate builders. What do you think ? > > I like the 2nd option better, it's what Firefox does, and can provide faster > > turnaround time if there is enough capacity. > > Yes, that's fine, just like try server.

(In reply to comment #20) > (In reply to comment #19) > > (In reply to comment #18) > > > Just had a thought, for the packaged unitests builders, do we want to have a > > > single builder per platform that runs check/xpcshell/mozmill in sequence, or > > > break those down into 3 separate builders for each test suite like Firefox > > > does. > > > > check has to be run on the builder doing the actual build > > Yes, but it also can be run as a packaged test run, or we could skip it there, > being somewhat redundant and keep only xpcshell and mozmill as separate > builders. What do you think ? I'd be surprised if "make check" can be run as a packaged test run, given that it generally needs all the makefiles and there's no packaging step for it in make package-tests.

(In reply to comment #20) > Yes, but it also can be run as a packaged test run AFAIK nobody has ever run those as packaged tests, and the stuff we share with Firefox don't have explicit support for that.

First step checked in: enable packageTests: http://hg.mozilla.org/build/buildbot-configs/rev/afea27c85814

(In reply to comment #16) > FF packages and uploads tests. I guess that's what (SM &) TB should do too. (Ftr, bug 545172 fixed this for SM.)

Sendchange are now being invoked on dep builds (nothing is listening) comparing with ssh://hg.mozilla.org/build/buildbot-configs searching for changes changeset: 3083:0a031977c30f tag: tip user: Philippe M. Chiasson <gozer@mozillamessaging.com> date: Wed Oct 06 16:39:11 2010 -0400 summary: Bug 495299 - Enable sendchange for TB dep builds on trunk

And xpcshell packaged tests are now enabled on comm-central. changeset: 3134:b8f52e44afcd user: Philippe M. Chiasson <gozer@mozillamessaging.com> date: Fri Oct 15 13:12:17 2010 -0400 summary: Bug 495299 - Enable packaged xpcshell tests for Thunderbird comm-central. r=Standard8 mozmill will need some buildbotcustom patches from try to land upstream

John, the MozMill packaged builders have been stable all day. I'm therefore happy that we can drop the old non-packaged unit test builds. Can you drop the "* comm-central check" builders from the buildbot configuration? I'd like to get the builder time back again :-) These are already hidden on tinderbox view, so you can kill those builds if they are running when you do this. Don't forget the two lines we added here: http://hg.mozilla.org/build/buildbot-configs/file/8f8f68b85baa/thunderbird/config.py#l401

Thanks, Mark. Done and deployed: http://hg.mozilla.org/build/buildbot-configs/rev/09e499af4973