To fix bug 1082031 we're upgrading Celery to 3.x and require a newer version of its "librrabbitmq" dependency. Right now that is installed in the global Python site-packages so I'm assuming it requires assistance from webops. The changes in used non-compiled packages can be found here: https://github.com/mozilla/kuma/pull/2949/files#diff-8 Those are *not* required to be installed globally by you as we're using Git submodules to vendor those with kuma. The package we *do* need to install in the root site-packages since it requires a compiler is here: https://github.com/mozilla/kuma/pull/2949/files#diff-7 librabbitmq 1.6.1, https://pypi.python.org/pypi/librabbitmq/1.6.1 It's maintained by the Celery author and can be trusted. Please let me know if you see any problems with using that version. Additionally we require another process to be set up on *one* of the celery nodes (similar to how we've set up just *one* celerbeat process for the whole cluster) which takes regular snapshots of the state of the Celery nodes. That allows us to use the integration with the Django admin and to monitor Celery tasks on the app level with lots of more information than just the queue level data. The command for this process is: python manage.py celerycam --freq=2.0 celerycam is the process to take snapshots (ha ha) of events. Please let me know if you have any questions, this needs to be a coordinated deploy right after merging https://github.com/mozilla/kuma/pull/2949. As I'm on holiday till the new year, I'm CCing :groovecoder and :ubernostrum in case this can get done in the meantime.

It looks like librabbitmq also requires the amqp library [1]. I have a preliminary set of RPMs that should work. As I haven't yet had a chance to destroy the MDN -dev environment, would it make sense for me to install them there for testing? N.B. When I created the RPMs for the two python packages, I tried to keep file names, etc. as similar as possible to what you'd normally get with a 'pip install'. On the current production servers, it looks like the librabbitmq files are installed as _pyrabbitmq.XX, e.g. /usr/lib64/python2.6/site-packages/_pyrabbitmq.so /usr/lib64/python2.6/site-packages/pylibrabbitmq-0.5.0-py2.6.egg-info/PKG-INFO /usr/lib64/python2.6/site-packages/pylibrabbitmq/__init__.py The RPMs I created have a slightly different naming scheme: /usr/lib64/python2.6/site-packages/_librabbitmq.so /usr/lib64/python2.6/site-packages/librabbitmq-1.6.1-py2.6.egg-info/PKG-INFO /usr/lib64/python2.6/site-packages/librabbitmq/__init__.py [1] https://github.com/celery/librabbitmq/blob/master/requirements/default.txt

I've installed the new RPMs on developer1.dev.webapp.scl3.mozilla.com 8:55:25 AM jezdez: the celery author forked pylibrabbitmq and called it just librabbitmq 8:55:39 AM jezdez: which is part of the changes in celery 3.x that we try to get to 8:55:53 AM jezdez: in theory they should not conflict with each other 8:55:57 AM jezdez: so installing the rpms would be ideal 8:56:13 AM cyliang: Ah. K. The name change (pylibrabbitmq -> librabbitmq) makes more sense. 8:56:56 AM jezdez: so assuming I get the fix the last broken test in the branch, do you think we could try out rpms today or tomorrow and assuming all goes well do the deploy on thurs? 8:57:14 AM cyliang: Try out the rpms on dev? or are you thinking of stage? 8:57:24 AM jezdez: I think dev makes sense, you’re right 8:57:29 AM jezdez: I just want to make sure all the files are there

Camera process deployed on stage via puppet.

For those keeping score at home: * new RPMS installed on developer1.stage.webapp.scl3.mozilla.com * new RPMS installed on developer admin host (needed for deploys) - test of a trivial push to prod (post-RPM install) went okay

* new RPMS installed in prod * BROKER settings change made in prod * celery camera process started up in prod (currently set to run on developer-celery1)