Until now, we've always run unittests on the same OS that we build on. This is because of an original unittest requirement of doing a build on the machine immediately before running any unittests. This meant that we could only run unittests on osx10.5, win2k3 and centos5, even though these are not typical user desktop OS. It would be better if we ran unittests on more typical user desktop OS. We already have Talos perf tests running on osx10.4, osx10.5, WinXP, Win7, fedora12, fedora12x64, and soon also osx10.6. Running unittests on those OS would be more representative of typical users. One approach would have been to build out a separate pool of machines imaged with these same 7 identical OSs. However, that would require maintaining two different sets of 7 ref images, and also reducing our burst capability across the divided pools. Instead we decided to increase the pool-o-slaves to handle extra load and then run unittests on the same pool-o-slaves that already run Talos. This would be easier to maintain, scale better and handle burst loads better. Some pre-req projects were: 1) separate out unittests from the builds. This allows us to run unittests on a pre-existing build, separate from the machine that actually generated the build. Details in bug#383136. 2) unify account/password/network configurations of talos machines to line up with what are used by the build+unittest machines. Details in bug#537065. 3) rebase talos machines on newer still-available hardware. This allowed us to build out a larger pool of identically spec'd slaves. Its unclear if this larger pool-o-slaves increased enough to handle the extra load of the newly added talos OS, and *also* handle the load of running unittests on those OS. However, because we're now using still-available hardware, we can further grow the pool-o-slaves if needed. Details in bug#537065 Whats left to do? This bug is to track: - enabling running-unittests-on-preexisting-build on each talos OS - maybe each OS would be different dep bug? - disabling unittests on the build machines

It's hard for me to understand how "run unit tests on different OSes" is a duplicate of "keep fixed performance characteristics for the unit test systems", even having read this bug a few times now. Could someone point me at the context that explains how they're related?

If we move our tests to hardware, VM settings will no longer be applicable ?

Oh, OK -- one really has to read between the lines to see that part, I guess! Thank you, I didn't manage to make that connection on my previous readings.

I am setting bug 548768 as blocking because I was unable to get gnuwin32's tar working on the slaves. We don't want Cygwin and I was unable to get gnuwin32's tar or bzip working on the windows 7 test machine.

Set a dependency on talos staging, as it needs to be in good shape to test this stuff.

(In reply to comment #0) ... > This bug is to track: > - enabling running-unittests-on-preexisting-build on each talos OS - maybe each > OS would be different dep bug? - running unittests manually first, on each OS, is in each of the dep bugs. - patch master.cfg to run unittests on minis - patch TBPL to handle unittests on different OS (bug#532560) - run concurrently and watch for variances/problems. Also watch for wait time/load problems. > - disabling unittests on the build machines I think thats it, but chime in if I missed anything?

Have taken rev3 boxes offline to be used a proof of concept (ie, manual set up only of unit test on talos): - talos-r3-w7-001 - talos-r3-fed-001 - talos-r3-fed64-001 - talos-r3-snow-001 - talos-r3-leopard-001

Alice is going to run point on this.

(In reply to comment #9) > - talos-r3-fed-001 > - talos-r3-fed64-001 These have been put back to the pool. No tool chain changes have been done to them. Nuke ~/armenzg-unittests if needed/wanted.

Update: - armenzg: unit test require no tool chain changes on fedora, will move on to looking at automating the process with a factory that inherits from TalosFactory, also look into getting sendchanges for packaged unit tests to talos-staging-master02 - lsblakk: possibly require buildbotcustom directory on mac machines, will have to investigate further as automation comes up - jhford: w7 blocked on packaged unit tests as zips, starting on xp - alice: bring up new talos staging using borrowed production rev3 machines coming up: - alice: need to hand off before going on long weekend so that we don't get blocked

(In reply to comment #12) > Update: > > - armenzg: unit test require no tool chain changes on fedora, will move on to > looking at automating the process with a factory that inherits from > TalosFactory, also look into getting sendchanges for packaged unit tests to > talos-staging-master02 Why not use UnittestPackagedBuildFactory? I expect it could be used with only minor modifications.

since we are changing to zipped tests, it might be worthwhile to verify that unzip works as expected on all platforms.

(In reply to comment #12) > - armenzg: ... getting sendchanges for packaged unit tests to > talos-staging-master02 The sendchange-unittest currently gets send to the localhost: master: localhost:9010 branch: mozilla-central-linux-opt-unittest revision: b3e305eb173c comments: user: sendchange-unittest files: ['http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1268302881/firefox-3.7a3pre.en-US.linux-i686.tar.bz2'] We should add another one for staging-master02 (sending to both places until we do the flip). Something like: master: talos-staging-master02.build.mozilla.org:9010 branch: mozilla-central-linux-opt-unittest The code in question is: http://hg.mozilla.org/build/buildbotcustom/file/tip/process/factory.py#l1322 We will also have to do this change for debug builds.

This sendchanges happen as soon as the build (and other stuff) gets uploaded. This will activate the sendchange for normal and debug builds.

Lukas this gets the job done with UnittestPackaged factories. As catlee mentioned this factory gets everything done. This patch adds the builders, the schedulers and the factories for the test builders for Linux. I did not have time to do more and nothing with regards the debug test builders. This is the sendchange I used locally: buildbot sendchange --master='localhost:9010' --branch=mozilla-central-linux-opt-unittest --revision=85fe77e1b558 --comment='' --user=sendchange-unittest 'http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1268329194/firefox-3.7a3pre.en-US.linux-i686.tar.bz2'

For the record, the previous patch added the test builders for Fedora 12 and my VM started doing packaged unit tests. The patch can be easily modified to make it available for all other platforms and adding the debug test builders. The most tedious thing will be to put just the right info in config.py As I mentioned on IRC and on my previous comment that my only concern is to try to hack TalosUnittestFactory when UnittestPackagedUnittestFactory already gets the job done.

Pro: tests packages by default, override when we do not want them Con: disk space usage for things that don't run packaged unittests. aiui, that is the current production try server. I am going to test this in sm01 right now.

Comment on attachment 432156 [details] [diff] [review] default packageTests factory argument to be true Ok, tested this in staging and i had the packages upload to: http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1268425908/firefox-3.7a3pre.en-US.linux-x86_64.tar.bz2 http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1268425908/firefox-3.7a3pre.en-US.linux-x86_64.tests.tar.bz2 http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1268425908/firefox-3.7a3pre.en-US.linux-x86_64.txt http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1268425908/firefox-3.7a3pre.en-US.langpack.xpi It failed to upload to AUS because it has the wrong private keys.

Comment on attachment 431932 [details] [diff] [review] sendchange-unitests being sent to talos-staging-master02 I ran this in staging at the same time as my patch and i did verify that the sendchange went through. It showed on talos-staging-master02 as Changed by: sendchange Changed at: Fri 12 Mar 2010 17:49:01 Branch: mozilla-central-linux64 Revision: a9b980fcb73a Changed files: * http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1268444163/firefox-3.7a3pre.en-US.linux-x86_64.tar.bz2 Comments: Properties:

Comment on attachment 432156 [details] [diff] [review] default packageTests factory argument to be true Yes, we want this! *stamp*

Comment on attachment 431932 [details] [diff] [review] sendchange-unitests being sent to talos-staging-master02 >diff --git a/mozilla2-staging/config.py b/mozilla2-staging/config.py >--- a/mozilla2-staging/config.py >+++ b/mozilla2-staging/config.py >@@ -56,17 +56,18 @@ GLOBAL_VARS = { > 'talos_masters': [ > ('talos-staging-master02.build.mozilla.org:9010', False), > ('talos-staging-master.mozilla.org:9010', False), > ('talos-master02.build.mozilla.org:9010', False), > ], > # List of unittest masters to notify of new builds to test, > # and if a failure to notify the master should result in a warning > 'unittest_masters': [('localhost:9010', True, 0), >- ('localhost:9011', True, 0)], >+ ('localhost:9011', True, 0), >+ ('talos-staging-master02.build.mozilla.org:9010', True, 0),], nit: remove extra comma, to get " 0)]," r+ with this fix. > 'unittest_suites': [ > ('mochitests', ['mochitest-plain']), > ('mochitest-other', ['mochitest-chrome', 'mochitest-browser-chrome', > 'mochitest-a11y']), > ('reftest', ['reftest']), > ('crashtest', ['crashtest']), > ('xpcshell', ['xpcshell']), > ], >diff --git a/mozilla2/config.py b/mozilla2/config.py >--- a/mozilla2/config.py >+++ b/mozilla2/config.py >@@ -57,17 +57,18 @@ GLOBAL_VARS = { > ('talos-master.mozilla.org:9010', True), > ('talos-master.mozilla.org:9012', False), > ('talos-staging-master.mozilla.org:9010', False), > ('talos-staging-master02.build.mozilla.org:9010', False), > ('talos-master02.build.mozilla.org:9010', False), > ], > # List of unittest masters to notify of new builds to test, > # and if a failure to notify the master should result in a warning >- 'unittest_masters': [('localhost:9010', False, 0)], >+ 'unittest_masters': [('localhost:9010', False, 0), >+ ('talos-staging-master02.build.mozilla.org:9010', False, 0)], > 'unittest_suites': [ > ('mochitests', ['mochitest-plain']), > ('mochitest-other', ['mochitest-chrome', 'mochitest-browser-chrome', > 'mochitest-a11y']), > ('reftest', ['reftest']), > ('crashtest', ['crashtest']), > ('xpcshell', ['xpcshell']), > ],

ok, this bug has been added to https://wiki.mozilla.org/ReleaseEngineering:Maintenance for landing when we are able to.

Comment on attachment 432156 [details] [diff] [review] default packageTests factory argument to be true This patch is incorrect. If you look at: http://hg.mozilla.org/build/buildbotcustom/file/tip/misc.py#l598 > elif pf.get('enable_opt_unittests'): > packageTests = True You will see that you are enabling optimized unittests in all branches and all platforms. We should only turn optimized unitests were needed. Instead try adding "packageTests" in the platform you want: http://hg.mozilla.org/build/buildbotcustom/file/tip/misc.py#l603 > packageTests = pf.get('packageTests', packageTests) This should have the same effect but without triggering packaged unittests. I have filed bug 552430 to keep track of the work involved on uploading linux 64 tests for staging testing of unit tests on talos for Linux 64.

(In reply to comment #25) > (From update of attachment 432156 [details] [diff] [review]) > This patch is incorrect. > If you look at: > http://hg.mozilla.org/build/buildbotcustom/file/tip/misc.py#l598 > > elif pf.get('enable_opt_unittests'): > > packageTests = True > You will see that you are enabling optimized unittests in all branches and all > platforms. We should only turn optimized unitests were needed. Actually, the way I read that is if you have enabled optimized unit tests, then you will upload the packaged test file, not the reverse (i.e. opt_unittest depends on having packaged tests uploaded). The packageTests factory argument just enables the make package-tests step from my reading of the code (http://hg.mozilla.org/build/buildbotcustom/file/aed4cb0172ca/process/factory.py#l824) > Instead try adding "packageTests" in the platform you want: > http://hg.mozilla.org/build/buildbotcustom/file/tip/misc.py#l603 > > packageTests = pf.get('packageTests', packageTests) Yes, we could do that. As the purpose this bug is to stop running unit tests on build machines, we will need to have packaged unit tests for all platforms on all branches unless we want to keep certain branches/platforms on build machine unit tests. This seems like the most appropriate default under the circumstances. > This should have the same effect but without triggering packaged unittests. > > I have filed bug 552430 to keep track of the work involved on uploading linux > 64 tests for staging testing of unit tests on talos for Linux 64. sure, that is fine. I will put new things and move the conversation over there.

In testing this on talos-staging, I've installed mercurial 1.3.1 on talos-r3-fed-001 with: yum install mercurial

Status update: http://hg.mozilla.org/users/lsblakk_mozilla.com/buildbotcustom/rev/7af72111c5d0 http://hg.mozilla.org/users/lsblakk_mozilla.com/buildbot-configs/rev/128ec338de4b I'm getting the builders created, with the right slaves attached - now just need to get the sendchanges triggering the right builders eg: 'macosx' triggers leopard/snowleopard builders.

for the geriatric machines I created a dictionary to map build factory platforms to test running platforms. that logic is in nightlybuildfactory at the end of doupload() iirc

I installed hg on talos-r3-snow-001 through vnc from : http://mercurial.berkwood.com/binaries/Mercurial-1.5-py2.6-macosx10.6.zip so that I didn't have to install anything else (xcode, macports)

Comment on attachment 431932 [details] [diff] [review] sendchange-unitests being sent to talos-staging-master02 http://hg.mozilla.org/build/buildbot-configs/rev/2e782932a01c

In testing this on talos-staging, I've installed mercurial 1.3.1 on talos-r3-fed64-001 with: yum install mercurial

Tested in talos-staging. NOTE: @@ -187,17 +188,17 @@ class MozillaBuildFactory(BuildFactory): + command=['bash', '-l','-c', added this so that snowleopard could grab the correct PYTHONPATH from a local .bash_profile since both leopard and snowleopard use the same env.

Comment on attachment 435034 [details] [diff] [review] Buildbotcustom patch to add unittests to talos >-def generateTalosBranchObjects(branch, branch_config, PLATFORMS, SUITES, >- factory_class=TalosFactory): >+def generateTalosBranchObjects(branch, branch_config, PLATFORMS, SUITES, >+ ACTIVE_UNITTEST_PLATFORMS, factory_class=TalosFactory): > branchObjects = {'schedulers': [], 'builders': [], 'status': []} > branch_builders = [] >+ test_builders = [] > > branchName = branch_config['branch_name'] > buildBranch = branch_config['build_branch'] > talosCmd = branch_config['talos_command'] > > for platform, platform_config in PLATFORMS.items(): > for slave_platform in platform_config['slave_platforms']: > for suite, talosConfig in SUITES.items(): >@@ -1725,22 +1731,62 @@ def generateTalosBranchObjects(branch, b > treeStableTimer=0, > builderNames=[builder['name']], > numberOfBuildsToTrigger=tests, > ) > branchObjects['schedulers'].append(s) > branchObjects['builders'].append(builder) > branch_builders.append(builder['name']) > >+ if platform in ACTIVE_UNITTEST_PLATFORMS.keys(): >+ testTypes = [] >+ >+ if branch_config['platforms'][platform].get('enable_opt_unittests'): >+ testTypes.append('opt') >+ if branch_config['platforms'][platform].get('enable_debug_unittests'): >+ testTypes.append('debug') >+ >+ for test_type in testTypes: >+ test_builders = [] >+ triggeredUnittestBuilders = [] >+ >+ # create builder names for TinderboxMailNotifier >+ for suites_name, suites in branch_config['unittest_suites']: >+ test_builders.extend(generateTestBuilderNames( >+ '%s %s %s test' % (platform_name, branch, test_type), suites_name, suites)) >+ >+ triggeredUnittestBuilders.append(('%s-%s-%s-unittest' % (branch, slave_platform, test_type), test_builders, True)) >+ >+ for suites_name, suites in branch_config['unittest_suites']: >+ # Remove mochitest-a11y from test suites since the packaged builds are not >+ # built with a11y enabled >+ if platform.startswith("macosx") and 'mochitest-a11y' in suites: >+ # Create a new factory that doesn't have mochitest-a11y >+ suites = suites[:] >+ suites.remove('mochitest-a11y') So, I realize this a11y code is a mess, but won't this result in builder names getting generated for mochitest-a11y in the block above, and then no builders created below? >+ >+ # create the builders >+ branchObjects['builders'].extend(generateTestBuilder( >+ branch_config, branch, platform, "%s %s %s test" % (platform_name, branch, test_type), >+ "%s-%s-%s-unittest" % (branch, slave_platform, test_type), >+ suites_name, suites, branch_config.get('mochitest_leak_threshold', None), >+ branch_config.get('crashtest_leak_threshold', None), >+ platform_config[slave_platform]['slaves'])) >+ >+ for scheduler_name, test_builders, merge in triggeredUnittestBuilders: >+ scheduler_branch = ('%s-%s-%s-unittest' % (branch, platform, test_type)) >+ branchObjects['schedulers'].append(NoMergeScheduler(name=scheduler_name, >+ branch=scheduler_branch, builderNames=test_builders, treeStableTimer=0)) >@@ -5255,19 +5256,22 @@ class WinmoBuildFactory(MobileBuildFacto > > class UnittestPackagedBuildFactory(MozillaBuildFactory): > def __init__(self, platform, test_suites, env={}, productName='firefox', > mochitest_leak_threshold=None, crashtest_leak_threshold=None, > totalChunks=None, thisChunk=None, chunkByDir=None, downloadSymbols=True, > **kwargs): > self.platform = platform.split('-')[0] > >- self.env = MozillaEnvironments['%s-unittest' % self.platform].copy() >+ self.env = deepcopy(MozillaEnvironments['%s-unittest' % self.platform]) > self.env['MINIDUMP_STACKWALK'] = getPlatformMinidumpPath(self.platform) > self.env.update(env) >+ # copy in overwritten env vars >+ for key, value in env.items(): >+ self.env[key] = value > self.productName = productName > What problem did you run into that required a deepcopy of the environment? self.env should be a shallow dictionary, with only strings as values, so the original .copy() should have been sufficient to get an isolated copy of the environment. You've also left in 'self.env.update(env)', which does exactly what your new for loop does.

Comment on attachment 435037 [details] [diff] [review] production configs adding unittests to Talos Looks good here too. Are you planning on pushing macosx out as soon as it's ready?

(In reply to comment #37) > (From update of attachment 435037 [details] [diff] [review]) > Looks good here too. Are you planning on pushing macosx out as soon as it's > ready? Yes, we'll roll out Leopard and Snow Leopard right away. Fedora will follow when we've got all bugs known and tracked.

Alice I would like to know if these machines will stay in the long-term on staging for my site-staging.pp changes. > - talos-r3-fed-001 > - talos-r3-fed64-001 > - talos-r3-snow-001 > - talos-r3-leopard-001

so it turns out I didn't need the deepcopy, and I took the unittest_suites into the per-platform config so that the a11y block in generateTalosBranchObjects is no longer needed either.

just moved the unittest_suites into the branch[platforms] so we can take a11y out for macosx in the configs instead of in misc.py

same changes as in staging - also set reboot to 1 build

Comment on attachment 435290 [details] [diff] [review] Buildbotcustom patch to add unittests to talos Does it look like 'bash -l -c' has any effect on any other platform? It might be totally overkill, but we could also limit only running the -l switch on mac platforms.

Comment on attachment 435292 [details] [diff] [review] production configs adding unittests to Talos I find the deepcopy stuff a little weird, but it seems that you are avoiding having to re-write env and test names and such for unit test per-platform over and over - and that I do approve of.

carrying forward alice's r+ -- I just cleaned up an extra space and the import of deepcopy which is no longer needed here.

now with reboots set to 1 to match production

Comment on attachment 435304 [details] [diff] [review] staging configs adding unittests to Talos Ship it!

Comment on attachment 435303 [details] [diff] [review] Buildbotcustom patch to add unittests to talos We need to be able to have different suites of tests for opt/debug on the same platform. For example, right now we want to run a11y tests on macosx debug, but not on macosx opt. Can you adjust the config.py patch so that there's a default set of test suites that gets copied in for each platform, and override macosx opt does that it doesn't run a11y? Also, the TinderboxMailNotifier needs to be using errorparser="unittest" for the unittest builders.

Lukas you can run now talos-r3-leopard-001 if you want through staging to see that it works after the puppet changes. I will get you the snow leopard machine when the mounting /N bug gets resolved (bug 555790).

Tested in staging - adds ability to have different suites of tests for opt/debug on the same platform. Configs patch (to be attached next) has a default set of test suites that gets copied in for each platform, mac opt unittests are overridden to take out mochitest-a11y Also, added a separate TinderboxMailNotifier so as to use errorparser="unittest" for the unittest builders. Finally, in the configs I had set up ACTIVE_BRANCHES to make it easier to turn branches on and off through master.cfg but I realized that I had attached that to the ACTIVE_UNITTEST_PLATFORMS so I refactored that and in doing so added ACTIVE_UNITTEST_PLATFORMS to the generateTalosReleaseBranchObjects params since that gets called within generateTalosBranchObjects and then in turn calls generateTalosBranchObjects. This way it just cycles the platform's active Unittest platforms back around but won't create any additional builders.

flagging catlee for feedback on the changes specific to the new buildbotcustom handling of default unittests for both opt/debug

flagging catlee for feedback just on the changes specific to the new handling of default unittests (same as staging)

Current status: - pushing hard to get leopard/snow leopard in production by end of week, currently blocked on puppet working on snow leopard + finalization of buildbot-configs - winxp/win7 blocked on unitest packages as zips, blocked on installation of hg and getting successful staging runs - fed/fed64 blocked on people (needs investigation of failures so that they can be properly farmed out to appropriate teams) Current plan: 1. leopard/snow leopard working in production by end of week 2. after leopard/snow leopard up move on to the issues found during linux staging and run them down 3. get unit test packages as zips pushed to production 4. staging of win7/winxp machine

Comment on attachment 435733 [details] [diff] [review] Buildbotcustom patch to add unittests to talos >+ for scheduler_name, test_builders, merge in triggeredUnittestBuilders: >+ scheduler_branch = ('%s-%s-%s-unittest' % (branch, platform, test_type)) >+ branchObjects['schedulers'].append(NoMergeScheduler(name=scheduler_name, >+ branch=scheduler_branch, builderNames=test_builders, treeStableTimer=0)) >+ What if merge is True here? We'll want to use a regular scheduler in that case, right? >- command=['bash', '-c', >+ command=['bash', '-l','-c', Why do we need '-l' here?

Comment on attachment 435734 [details] [diff] [review] staging configs adding unittests to Talos jsreftest should be run on all platforms, not just macosx. and mochitest-a11y won't be running on macosx-debug if I'm reading this correctly.

Comment on attachment 435736 [details] [diff] [review] production configs adding unittests to Talos same as staging

> What if merge is True here? We'll want to use a regular scheduler in that > case, right? Funny enough merge is set to True by default in generateBranchObjects as well as in generateTalosBranchObjects so thanks for catching it. I put in a regular Scheduler option same as in the former. > > >- command=['bash', '-c', > >+ command=['bash', '-l','-c', > > Why do we need '-l' here? This is to source the local bash_profile which is needed on snow leopard. It has a different PYTHONPATH than the leopard slaves and yet the two share the same env since those are per platform (macosx) and not per slave_platform (leopard, snowleopard). Adding the -l doesn't affect slaves with no bash_profile override of PYTHONPATH so it seems like a safe thing to do.

Added jsreftest to default unittest set. In staging I have tested the configs and mac-debug builds do get a mochitest-a11y step in the mochitest-other builder with these configs.

same as for staging - added jsreftest to default unittest suites.

Catlee, disregard the comment about 'bash' '-l' because it appears that armenzg worked around this issue in his puppet installation of mercurial for leopard/snow leopard. I've tested in staging on puppet slaves without the -l and hg clone worked fine. Also, in this patch (which still handles the merge/nomerge) I shortened the builddir prefix to -u instead of -unittest because we were having installdmg.sh "failed to mount" on the longer builddir-named builder mozilla-central-leopard-debug-unittest-mochitest-other and with a shorter builddir name it no longer had an issue. Now all builddirs are along the lines of mozilla-central-leopard-debug-u-mochitest-other.

(In reply to comment #57) > > This is to source the local bash_profile which is needed on snow leopard. It > has a different PYTHONPATH than the leopard slaves and yet the two share the > same env since those are per platform (macosx) and not per slave_platform > (leopard, snowleopard). Adding the -l doesn't affect slaves with no > bash_profile override of PYTHONPATH so it seems like a safe thing to do. Lukas, the bash_profile is using PYTHONPATH=/Library/Python/2.5/site-packages instead of PYTHONPATH=/usr/local/lib/python2.5/site-packages (since it was non-existent). You might want to run your patch without -l and it should probably fine since both platforms now use the same PYTHONPATH.

Talos darwin slaves now have mercurial installed. We are ready puppet wise to turn unit tests on them.

taking a11y out of mac debug unittest suites until bug 524775 is fixed.

taking debug a11y out of mac unittest suites until bug 524775 is fixed.

Comment on attachment 436692 [details] [diff] [review] talos-pool amendments to accomodate packaged unittest changes in talos-r3 r+ from alice in IRC

Comment on attachment 436235 [details] [diff] [review] staging configs adding unittests to Talos http://hg.mozilla.org/build/buildbot-configs/rev/62ce31923cc0

Comment on attachment 436236 [details] [diff] [review] production configs adding unittests to Talos http://hg.mozilla.org/build/buildbot-configs/rev/02cfb6f3932d

Comment on attachment 436086 [details] [diff] [review] Buildbotcustom patch to add unittests to talos http://hg.mozilla.org/build/buildbotcustom/rev/d5ba95f1dcb8

the last, very important, piece of config update.

Comment on attachment 436697 [details] [diff] [review] Turns on sendchange notification for talos-master from production-master *stamp*

Comment on attachment 436697 [details] [diff] [review] Turns on sendchange notification for talos-master from production-master http://hg.mozilla.org/build/buildbot-configs/rev/9f1626be3147

I filed bug 558777 to track everything wrt to Leopard and Snow Leopard. I have transferred all related dependent bugs to it.

I have filed bug 558781 to track everything wrt to running unit tests for Windows XP and Windows 7. I have transferred all related dependent bugs to it.

I have filed bug 558782 to track everything wrt to running unit tests for Fedora and Fedora 64. I have transferred all related dependent bugs to it.

Bug 557294 will get us more Rev3 machines (post delivery imaging is also needed) which we need to enable unit tests on more branches than just mozilla-central. This will help us put our wait times back to normal on production-talos.

afaict, all the comments, patches and depbugs here are done. Which makes this bug officially FIXED! (let me know if I missed anything!)

The only thing left is to add XP debug unit tests in bug 614955 but I don't want to mess anymore with the dependencies.