I pushed a code update to Balrog today, and the aus4-dev nodes picked it up around 5am PST. Later, Catlee noticed that he wasn't getting an update. When I started investigating, I found that I was getting inconsistent results from the same request. The URL in question is: https://aus4-dev.allizom.org/update/3/Firefox/11.0a1/20111218031140/Linux_x86_64-gcc3/en-US/nightly/Linux%203.0.0-12-generic%20%28GTK%202.24.6%29/default/default/update.xml?force=1 Sometimes I get a full XML response, as expected. Other times I get an internal server error. In the case of the latter, I get a traceback in the application log that doesn't match the current code on disk. _However_, the traceback *does* make sense against the previous version of the code. Catlee suggested that some of the workers might not be getting the new code right away, or something along those lines. He also suggested adding some extra logging that would log the PID its running under, and possibly the version of code, too. Would that be helpful in debugging this? Are there other things I could log that would help? I've noticed this sort of thing before and it's always seemed to work itself out over time, but this is the first time I've been able to confirm that the code being run doesn't match the latest code on disk.

Ben: If you can log the version of the code that would be incredibly helpful. Is there a file I can look at to see what version is on disk? I ran a manual code push and no changes were made, so it looks to be up to date. It could also be that the previous version of the code is cached. It's a best practice to kick Apache or touch the entry point after a code push which is currently not being done. If you can point me to a file which indicates the code version I'll setup a sync extras script to handle this on every push.

I spoke with Ben on IRC TL;DR I added a sync extras script to touch the .wsgi entrypoint when the git revision changed. Scripts committed to puppet, I tested and it appears to be working %100