In order to properly deploy AUS3 to the dev machines, IT needs its database configuration in a file. cshields said we need to put '-dist' in the filename, too.

Started working on this yesterday. I ended up scope creeping this to be "make AUS deployable as a wsgi app", because that's really what needs to happen. This is looking like it involves a few things: 1) Make AUS installable, so that it can be easily put into a virtualenv 2) Recreate AUS-server.py as a Flask app, because it has much simpler WSGI support than the built-in Python stuff 3) Create a config file and wsgi runner.

I _think_ this patch has everything we need to let IT deploy to the development environment. (I'm still waiting on feedback from IT in bug 685169, so I won't be sure until they reply.) Specifically, this patch: - Moves the incoming client app into Flask, to make it easy to create a wsgi runner - Creates a wsgi running script - Changes 404s instead empty XML, to emulate AUS2's behaviour - Creates a simple config file, and a parser for it - Imports necessary vendor libraries (except Jinja2, because it's binary)

The first patch bitrotted a little bit, and I enhanced it with a couple of things: * Fix tests to run against vendor libraries * Change authorization to look at the passed-through REMOTE_USER rather than doing it in-app.

Post review I was playing out with patch and hit and error doing the PPC detection code, in short: $ venv/bin/python AUS-server.py 2011-10-27 23:11:26,285: * Running on http://127.0.0.1:8000/ 2011-10-27 23:11:26,285: * Restarting with reloader 2011-10-27 23:11:28,821: 127.0.0.1 - - [27/Oct/2011 23:11:28] "GET /update/3/Firefox/8.0/20111019081014/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/beta/Darwin%2010.8.0/default/default/update.xml HTTP/1.1" 500 - ... File "/Users/nthomas/github/mozilla/balrog.test2/venv/lib/python2.6/site-packages/flask/views.py", line 112, in dispatch_request return meth(*args, **kwargs) File "/Users/nthomas/github/mozilla/balrog.test2/auslib/client/views/client.py", line 45, in get query = self.getQueryFromURL(queryVersion, url) File "/Users/nthomas/github/mozilla/balrog.test2/auslib/client/views/client.py", line 33, in getQueryFromURL ua = self.headers.getfirstmatchingheader('User-Agent') AttributeError: 'ClientRequestView' object has no attribute 'headers'

(In reply to Nick Thomas [:nthomas] from comment #4) > Post review I was playing out with patch and hit and error doing the PPC > detection code, in short: > > $ venv/bin/python AUS-server.py > 2011-10-27 23:11:26,285: * Running on http://127.0.0.1:8000/ > 2011-10-27 23:11:26,285: * Restarting with reloader > 2011-10-27 23:11:28,821: 127.0.0.1 - - [27/Oct/2011 23:11:28] "GET > /update/3/Firefox/8.0/20111019081014/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/ > beta/Darwin%2010.8.0/default/default/update.xml HTTP/1.1" 500 - > ... > File > "/Users/nthomas/github/mozilla/balrog.test2/venv/lib/python2.6/site-packages/ > flask/views.py", line 112, in dispatch_request > return meth(*args, **kwargs) > File > "/Users/nthomas/github/mozilla/balrog.test2/auslib/client/views/client.py", > line 45, in get > query = self.getQueryFromURL(queryVersion, url) > File > "/Users/nthomas/github/mozilla/balrog.test2/auslib/client/views/client.py", > line 33, in getQueryFromURL > ua = self.headers.getfirstmatchingheader('User-Agent') > AttributeError: 'ClientRequestView' object has no attribute 'headers' Booo, I guess I didn't hit this because it's a Mac-only code path. Looks like I forgot to flask-ify this part.

Most of the new stuff in this patch is tests. I factored out the architecture detection to a helper method to make it easier to test (Flask wouldn't let me mock out flask.request.headers for some reason), and I also threw in a couple basic tests of get() for good measure. I was going to write more tests, but I realized that any end to end tests of get() here are pretty much duplicating the existing tests in aus-data-snapshots. I find them easier to understand/write, so I'm thinking we can stick with them going forward if everyone else agrees.

Comment on attachment 570053 [details] [diff] [review] fix issue with headers, add tests Sorry, just found a bug in requirepermission(), will repost.

Comment on attachment 570053 [details] [diff] [review] fix issue with headers, add tests Sorry again...got confused between two patches, this is actually OK.

Comment on attachment 570053 [details] [diff] [review] fix issue with headers, add tests >diff --git a/auslib/client/base.py b/auslib/client/base.py >+from auslib.AUS import AUS3 >+ >+app = Flask(__name__) >+AUS = AUS3() ... >+from auslib.client.views.client import * >diff --git a/auslib/client/views/client.py b/auslib/client/views/client.py >+from auslib.client.base import app, AUS Is this kind of circular importing going to cause issues later ?

(In reply to Nick Thomas [:nthomas] from comment #9) > Comment on attachment 570053 [details] [diff] [review] [diff] [details] [review] > fix issue with headers, add tests > > >diff --git a/auslib/client/base.py b/auslib/client/base.py > >+from auslib.AUS import AUS3 > >+ > >+app = Flask(__name__) > >+AUS = AUS3() > ... > >+from auslib.client.views.client import * > > >diff --git a/auslib/client/views/client.py b/auslib/client/views/client.py > >+from auslib.client.base import app, AUS > > Is this kind of circular importing going to cause issues later ? I _think_ it's okay since we're not doing reloads(). I haven't seen any issues in my local testing.

Comment on attachment 570053 [details] [diff] [review] fix issue with headers, add tests The interdiff looks fine, but I wondered about these binary files diff --git a/vendor/lib/python/simplejson/_speedups.so b/vendor/lib/python/simplejson/_speedups.so diff --git a/vendor/lib/python/sqlalchemy/cprocessors.so b/vendor/lib/python/sqlalchemy/cprocessors.so diff --git a/vendor/lib/python/sqlalchemy/cresultproxy.so b/vendor/lib/python/sqlalchemy/cresultproxy.so How does that sit with the binary-modules-by-rpm vs vendor for non-binary plan ?

(In reply to Nick Thomas [:nthomas] from comment #11) > Comment on attachment 570053 [details] [diff] [review] [diff] [details] [review] > fix issue with headers, add tests > > The interdiff looks fine, but I wondered about these binary files > > diff --git a/vendor/lib/python/simplejson/_speedups.so > b/vendor/lib/python/simplejson/_speedups.so > diff --git a/vendor/lib/python/sqlalchemy/cprocessors.so > b/vendor/lib/python/sqlalchemy/cprocessors.so > diff --git a/vendor/lib/python/sqlalchemy/cresultproxy.so > b/vendor/lib/python/sqlalchemy/cresultproxy.so > > How does that sit with the binary-modules-by-rpm vs vendor for non-binary > plan ? Darn it, I don't know how I missed that. Good catch...

Comment on attachment 570684 [details] [diff] [review] drop sqlalchemy/simplejson r+ assuming IT are happy with those binary packages.

Probably need to update the files in requirements/ too, just carry the r for that.

(In reply to Nick Thomas [:nthomas] from comment #11) > How does that sit with the binary-modules-by-rpm vs vendor for non-binary > plan ? We can install simplejson and sqlalchemy system-wide

(In reply to Corey Shields [:cshields] from comment #16) > (In reply to Nick Thomas [:nthomas] from comment #11) > > > How does that sit with the binary-modules-by-rpm vs vendor for non-binary > > plan ? > > We can install simplejson and sqlalchemy system-wide Are you concerned about different applications requiring different versions of these binary dependencies?

(In reply to Chris AtLee [:catlee] from comment #17) > Are you concerned about different applications requiring different versions > of these binary dependencies? AUS will be on dedicated nodes.. so no..

Comment on attachment 570684 [details] [diff] [review] drop sqlalchemy/simplejson (In reply to Nick Thomas [:nthomas] from comment #15) > Probably need to update the files in requirements/ too, just carry the r for > that. I did this, and pushed in: https://github.com/mozilla/balrog/commit/10761b5c5b3df90b9926c0bdf2b1ee11217e6e79 Jenkins picked up the commit and tested it, too! https://jenkins.mozilla.org/job/Balrog/25/

Actual environment set-up tracked in bugs 685169 and 684243

mass component change