The tl;dr is that we either need to make Python 2.7.x work on CentOS 6.x or just use CentOS 7 (which ships with 2.7.5). The argument for CentOS 6 is that we're already running it in production and have ironed out all the associated kinks (i.e. it's the devil we know). The argument for CentOS 7 is that it ships with Python 2.7 already, which may be a easier / better from a maintenance perspective. The goal of this bug is to determine which of these two potential choices is least painful (or, at least, has a level of pain we're willing to deal with).

As it turns out, the process of getting a usable Python 2.7 on CentOS 6 is not that bad - but it's not exactly production-quality either. In short, it's more janky than it is difficult, which is actually rather unfortunate, since the inverse would actually be better from a production perspective. There are any number of blog posts that walk through the process (so I won't repeat it here). The upshot is that on CentOS 6, both versions 2.6 and 2.7 must co-exist, since the former is a hard requirement at the system level. In order to avoid collisions, 2.7 is installed into alternate paths, and the many things having '2.7' appended or otherwise inserted into their names / locations. But, hey, it works! The issue is that there are now a bunch of loose files lying all over the place, which is literally the opposite of package management, and therefore highly questionable from a maintenance and administration perspective. Two possible ways of dealing with / looking at this situation: 1. Run a find before and after 'make altinstall', extract everything that was installed by Python (et al.), and then make a package out of it with fpm. This would necessitate editing the in-flight SPEC file during package generation to ensure that all perms are set correctly, but it's certainly do-able. This would give us a platform-specific deployable package for internal use. 2. Realise that we're running Packer-generated nodes anyways, and that as long as the box build process is known and repeatable, it doesn't really matter how anything is actually installed or maintained on the resulting nodes; if we need to modify, update, or otherwise manipulate a node, we'd simply re-build it anyway. The former is clearly the cleanest approach, but there is perhaps something to said for the latter. I'm not sure how I feel about it quite yet. In any case, there's still CentOS 7 to test, so we have some time to debate still. ;)

(In reply to Daniel Maher [:phrawzty] from comment #1) > 1. Run a find before and after 'make altinstall', extract everything that > was installed by Python (et al.), and then make a package out of it with > fpm. This would necessitate editing the in-flight SPEC file during package > generation to ensure that all perms are set correctly, but it's certainly > do-able. This would give us a platform-specific deployable package for > internal use. > > 2. Realise that we're running Packer-generated nodes anyways, and that as > long as the box build process is known and repeatable, it doesn't really > matter how anything is actually installed or maintained on the resulting > nodes; if we need to modify, update, or otherwise manipulate a node, we'd > simply re-build it anyway. > > The former is clearly the cleanest approach, but there is perhaps something > to said for the latter. I'm not sure how I feel about it quite yet. In any > case, there's still CentOS 7 to test, so we have some time to debate still. > ;) I totally agree that we should do the cleanest approach when we have the luxury of time (like we do now!), the second should be reserved for more grave circumstances (e.g. critical bug in production Right Now) where we know we're accruing technical debt and have a longer-term plan to pay it back down - the more fragile and complex some install process is, the more likely it'll break down and harm repeatability. To summarize our IRC convo - since we do have the time to do this right, let's test both CentOS 7 and also getting this working on 6, should be easy to spin up local vagrant or AWS VMs to test these.

Another possibility is to use the SCL[0] channel which makes Python 2.7.5 available on CentOS[1]. This gives us access to a stable, packaged, and upstream-supported instance of Python 2.7.x via the scl mechanism. The trade-off is that it's not the most recent in the 2.7 series. We'd also have to re-jigger our configs so that the scl instance is used instead of relying on that of the system. I put a up a quick gist[2] that provides an overview of the install details. [0] https://access.redhat.com/documentation/en-US/Red_Hat_Software_Collections/1/html-single/1.0_Release_Notes/index.html [1] http://wiki.centos.org/AdditionalResources/Repositories/SCL [2] https://gist.github.com/phrawzty/5db4212b04a74137d992

Ultimately it has been decided that both Python 2.7.5 and CentOS 6 are simply too antiquated to continue fighting with. We're forging ahead with CentOS 7 for now, with an eye towards Ubuntu down the line.