Seen on talos-r3-xp-037 trying to talk to buildbot-master16: 2011-12-19 11:19:51-0800 [Broker,client] ReconnectingPBClientFactory.failedToGetPerspective 2011-12-19 11:19:51-0800 [Broker,client] While trying to connect: Traceback from remote host -- Traceback (most recent call last): File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/twisted/internet/defer.py", line 363, in unpause self._runCallbacks() ... File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/twisted/spread/pb.py", line 664, in registerReference raise Error("Maximum PB reference count exceeded. " twisted.spread.pb.Error: Maximum PB reference count exceeded. Goodbye. Connected fine after a reboot, so tried to hit the master during a reconfig ?

Also talos-r3-xp-039 at 2011-12-19 11:18:52-0800 [Broker,client] While trying to connect:

And talos-r3-xp-057 slightly older, full stack: 2011-12-16 18:22:01-0800 [Broker,client] ReconnectingPBClientFactory.failedToGetPerspective 2011-12-16 18:22:01-0800 [Broker,client] While trying to connect: Traceback from remote host -- Traceback (most recent call last): File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/twisted/internet/defer.py", line 363, in unpause self._runCallbacks() File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks self.result = callback(self.result, *args, **kw) File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/twisted/internet/defer.py", line 397, in _continue self.unpause() File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/twisted/internet/defer.py", line 363, in unpause self._runCallbacks() --- <exception caught here> --- File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks self.result = callback(self.result, *args, **kw) File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/twisted/spread/pb.py", line 763, in serialize return jelly(object, self.security, None, self) File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/twisted/spread/jelly.py", line 1122, in jelly return _Jellier(taster, persistentStore, invoker).jelly(object) File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/twisted/spread/jelly.py", line 475, in jelly return obj.jellyFor(self) File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/twisted/spread/flavors.py", line 127, in jellyFor return "remote", jellier.invoker.registerReference(self) File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/twisted/spread/pb.py", line 664, in registerReference raise Error("Maximum PB reference count exceeded. " twisted.spread.pb.Error: Maximum PB reference count exceeded. Goodbye.

jhford has a bug to add a couple more masters but it is blocked on IT to open firewalls for DB access (bug 708804).

jhford mention those are for tegras. I am asking dustin for more masters. Also CPU wio is happening on that master, which is not surprising.

found in triage: (In reply to Armen Zambrano G. [:armenzg] - (vactions from Dec. 24th & back on Jan. 9th) from comment #3) > jhford has a bug to add a couple more masters but it is blocked on IT to > open firewalls for DB access (bug 708804). bug#708804 now done. ...and so too is bug#712398.

Once buildbot-master21 is created, we should use our scripts to set it up as a Windows test master, enable it in slavealloc, and let it start grabbing Windows slaves from buildbot-master16.

I'll set it up in here.

lsblakk, catlee mentioned that you're setting up some masters. Do you have instructions on how to? I would like to try one myself. Thanks!

(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #8) > lsblakk, catlee mentioned that you're setting up some masters. > Do you have instructions on how to? I would like to try one myself. > Thanks! Armen, I just set up buildbot-master{22..27} following https://wiki.mozilla.org/ReleaseEngineering/Master_Setup and have added a few notes where I found it necessary. Am currently finished everything on my end and just trying to figure out the best practice for filing the IT bugs required. I will make notes in that wiki page with whatever I discover.

If buildbot-master16 is already windows only (win32+win64) and there really are too many builders, is there a plan to split win32 vs win64 ? Simply adding another master doesn't seem like it'll help any.

If we're hitting the PB limit, it's a problem with number-of-builders-per-slave, not number-of-slaves-per-master. So we need to reduce number of builders and/or bump the PB limit.

Should we split the win64 slaves into a separate master? or increase the PB limit?

I think that the limit to the reference count in Twisted is actually reasonable, so it may be time to look for ways to have fewer builders defined. But you can monkey-patch it. Details in the Twisted bug (#2045)

sorry, *buildbot* bug - http://trac.buildbot.net/ticket/2045

Does this happen when we do a reconfig that adds more builders? and then have to backout? I just don't know when this problem happens and how often.

I will leave this bug open for someone else to determine what is the way forward.

(In reply to Chris AtLee [:catlee] from comment #11) > If we're hitting the PB limit, it's a problem with > number-of-builders-per-slave, not number-of-slaves-per-master. So we need to > reduce number of builders and/or bump the PB limit. * Do we actually know how many builders we have per slave? * Does the new PB ref limit need to be a power-of-2? * Do we actually have a way to reduce the # of builders per slave? I assume this is a direct result of having (many project branches) x (many tests split into smaller parts)

It doesn't need to be a power of two.

(In reply to Chris Cooper [:coop] from comment #17) > (In reply to Chris AtLee [:catlee] from comment #11) > > If we're hitting the PB limit, it's a problem with > > number-of-builders-per-slave, not number-of-slaves-per-master. So we need to > > reduce number of builders and/or bump the PB limit. > > * Do we actually know how many builders we have per slave? You can count it locally by editing master.cfg for a test master and running checkconfig. > * Does the new PB ref limit need to be a power-of-2? > * Do we actually have a way to reduce the # of builders per slave? I assume > this is a direct result of having (many project branches) x (many tests > split into smaller parts) We'd have to consolidate builders e.g. doing all the different mochitest suites in the same builder, relying on runtime information to run the right suite. This also means tbpl would need updating since it looks at the builder name to determine what type of job something is.

Updating summary.

Not sure why this is marked as [buildduty]. This seems like more work than buildduty could hope to tackle in a given week in addition to everything else.

Unblocking bug 698843: with Thunderbird builders, the highest usage is talos-r3-fed-076 has 981 builders; limit is 1012; 96 percent of max. Still, we have very little wiggle room for new builders.

Dustin: I'm guessing here, but I think this will make slavealloc give all slavealloc-enabled slaves a MAX_BROKER_REFS of 2048. Does it look right to you?

This patch hacks the production-0.8 copy of hg.m.o/build/buildbot to have a [larger] hardcoded MAX_BROKER_REFS. Aiui, this doesn't affect any existing masters, but will affect any new masters.

Comment on attachment 623878 [details] [diff] [review] update slavealloc's buildbot.tac template looks right to me, untested

Did a quick'n'dirty test: 1. Started up buildmaster with import twisted.spread.pb twisted.spread.pb.MAX_BROKER_REFS = 2048 in the master's buildbot.tac. Connected a linux64 slave with no MAX_BROKER_REFS update. Forced a build; the slave picked it up and started building. Killed the job. 2. Stopped the master + slave. Commented out the MAX_BROKER_REFS lines on the master; added those lines to the slave. Restarted both, forced a build. The slave picked it up and started building.

Comment on attachment 623884 [details] [diff] [review] hack buildbot/master/buildbot/scripts/runner.py http://hg.mozilla.org/build/buildbot/rev/082cd6ddcb18 Landed on the production-0.8 branch. Any new masters created should have this fix. We still need to land+deploy to slavealloc, and manually update the existing masters. Catlee: is there anything else I need to do re: the buildbot repo?

I set up a new test master on dev-master01 via make -f Makefile.setup \ MASTER_NAME=preproduction-tests_master \ BASEDIR=/builds/buildbot/aki/test-master \ PYTHON=/usr/bin/python26 \ VIRTUALENV="/usr/bin/python26 /tools/misc-python/virtualenv.py" \ BUILDBOTCUSTOM_BRANCH=default \ BUILDBOTCONFIGS_BRANCH=default \ virtualenv deps install-buildbot master master-makefile The buildbot.tac had the MAX_BROKER_REFS fix.

Comment on attachment 623878 [details] [diff] [review] update slavealloc's buildbot.tac template http://hg.mozilla.org/build/tools/rev/2afbe1c9176d This needs to be deployed, per https://wiki.mozilla.org/ReleaseEngineering/Applications/Slavealloc#Deployment .

When pointing at http://staging-puppet.build.mozilla.org/staging/python-packages/ , it dies when trying to find sqlalchemy. Pointing it at repos/python/packages works.

Updated slavealloc with the above script; http://slavealloc.build.mozilla.org/gettac/linux-ix-slave03 gives me a tac with the MAX_BROKER_REFS lines.

Comment on attachment 624766 [details] [diff] [review] fix slavealloc script This will only work in scl1 and mtv1 right now, but since slavealloc is in scl1, this looks good.

Comment on attachment 624766 [details] [diff] [review] fix slavealloc script http://hg.mozilla.org/build/tools/rev/acbaea8e273b

Added the MAX_BROKER_REFS lines to existing enabled masters in production-masters.json.

This shouldn't land until we restart all our test masters, or fix via manhole.

Comment on attachment 624777 [details] [diff] [review] increase builder limit to 2048 in master.cfg http://hg.mozilla.org/build/buildbot-configs/rev/6da2ccee77a0

Comment on attachment 627374 [details] [diff] [review] up builder limit for build masters http://hg.mozilla.org/build/buildbot-configs/rev/45bff744e472

All the enabled masters (including bm34) have had this change deployed via the manhole.

Thanks Coop! -> RESO FIXED