We talked many times about wanting to be able to replay transactions that happen on the existing AUS server in Balrog, and make sure we get the same result.

my fault, but s/transactions/requests/

As a start, we could write something that uses patcher configs as a data source, and then compares requests made to aus3.m.o vs. balrog. For nightlies we might be able to do something really brain dead like: { 'versions': [1,2,3], 'channels': ['nightly', 'aurora'], 'platforms': ['mac-x86', 'mac64', 'win32', 'linux32'], 'distVersions': ['Windows NT 6.2', 'GTK something'], etc. } ....and then test all intersections.

Got started on this today: https://github.com/bhearsum/balrog/compare/mozilla:master...request-replay The code is rough, but gives output like: Found difference for /update/3/Firefox/10.0a1/20120222174716/WINNT_x86-msvc/af/nightly/Windows_NT 5.0/default/default/update.xml (aus3.mozilla.org vs. aus4-dev.allizom.org): --- +++ @@ -1,3 +1,6 @@ <?xml version="1.0"?> <updates> + <update type="minor" version="22.0a1" extensionVersion="22.0a1" buildID="20130318030947"> + <patch type="complete" URL="http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/03/2013-03-18-03-09-47-mozilla-central-l10n/firefox-22.0a1.af.win32.complete.mar" hashFunction="SHA512" hashValue="6c046061fd021acfef9d19ad887b98a5620eb1ccb0de0e1a96e325ff65fa886f5421cce47435953ab3ef6910b9ed4594bcc11c9da05196018887b83c6b49384c" size="27896110"/> + </update> </updates> Found difference for /update/3/Firefox/10.0a1/20120222174716/WINNT_x86-msvc/af/nightly/Windows_NT 6.2/default/default/update.xml (aus3.mozilla.org vs. aus4-dev.allizom.org): --- +++ @@ -1,6 +1,6 @@ <?xml version="1.0"?> <updates> - <update type="minor" displayVersion="22.0a1" appVersion="22.0a1" platformVersion="22.0a1" buildID="20130318030947"> - <patch type="complete" URL="http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/03/2013-03-18-03-09-47-mozilla-central-l10n/firefox-22.0a1.af.win32.complete.mar" hashFunction="sha512" hashValue="6c046061fd021acfef9d19ad887b98a5620eb1ccb0de0e1a96e325ff65fa886f5421cce47435953ab3ef6910b9ed4594bcc11c9da05196018887b83c6b49384c" size="27896110"/> + <update type="minor" version="22.0a1" extensionVersion="22.0a1" buildID="20130318030947"> + <patch type="complete" URL="http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/03/2013-03-18-03-09-47-mozilla-central-l10n/firefox-22.0a1.af.win32.complete.mar" hashFunction="SHA512" hashValue="6c046061fd021acfef9d19ad887b98a5620eb1ccb0de0e1a96e325ff65fa886f5421cce47435953ab3ef6910b9ed4594bcc11c9da05196018887b83c6b49384c" size="27896110"/> </update> </updates> ...which already reveals that we need to put in OS blocking rules. I'm not sure if Balrog is the right repo for this to live in, most of it might be better off in a tools library, since this isn't really part of the Balrog server infrastructure.

Nick and I chatted about this yesterday, here's a summary * I'm going to look into getting lists of unique requests for the nightly (and maybe ux, aurora, and other nightly-style branches) channel to aus3.mozilla.org from Metrics/IT. If we can get them with their response bodies we can avoid rerunning them against the production server. * Either way we can rerun them against aus4-dev and see what happens. * If we need to rerun them against aus3.m.o, we need to talk to Metrics and make sure that it won't distort any of their charts. * We want to aim to have some sort of continuous testing here. Whether that's something running in Buildbot, cron, jenkins - TBD. But we want these requests regularly run w/ a report generated. Ideally we'll be in a state where every query to aus4-dev returns exactly the same thing as aus3.m.o before we flip the switch on bug 832454. * I'll do some initial tests by hand to get an idea of the obvious inconsistencies we already have.

(In reply to Ben Hearsum [:bhearsum] from comment #4) > * I'll do some initial tests by hand to get an idea of the obvious > inconsistencies we already have. I did this today with: python compare-update-requests.py --version 10.0a1 --version 12.0a1 --version 22.0a1 --buildid 20130216031127 --buildid 20130220040207 --buildid 20110901030807 --locale de --locale en-US --locale ja --locale ja-JP-mac --channel nightly --channel nightly-ux --platform Linux_x86-gcc3 "GTK 2.17." "GTK 2.20." --platform Darwin_x86_64-gcc3-u-i386-x86_64 "Darwin 9" "Darwin 10" --platform Darwin_x86-gcc3-u-i386-x86_64 "Darwin 9" "Darwin 10" --platform WINNT_x86-msvc "Windows_NT 5.0" "Windows_NT 6.0" Results: * A bunch of inconsistencies due to not supporting OS deprecation (bug 748689) * A bunch of inconsistencies due to not supporting new attributes (bug 748698) * "sha512" is used as a hash function on aus3, "SHA512" is used on balrog (bug 853036) * UX channel served nothing from balrog because its rule wasn't there - I added it.

I set-up a Jenkins job to do regular runs against a static set of data with these scripts (http://10.134.48.37:8080/job/Balrog%20Snippet%20Comparison/). Once bug 853051 is fixed, hopefully we can plug in the data from Metrics pretty easily. I need to move this code into the Balrog repo so that we can get per-commit tests, so just feedback? for now.

Comment on attachment 729192 [details] [diff] [review] scripts to do snippet comparsion >diff --git a/release/generate-update-urls.py b/release/generate-update-urls.py >+from balrog.testing import compare_snippets, generate_paths compare_snippets isn't used here. >+DEFAULT_UPDATE_DATA = { >+ 'products': ['Firefox'], >+ 'versions': ['10.0a1', '12.0a1', '17.0a1'], >+ 'buildids': ['20120222174716', '20111201231412'], Are any nonsensical combinations a feature or just a bit of a test slowdown ? Or is the idea to call the script several times, with different sets of self-consistent args ? If so, you could just amend the default to one value for each. The alternative - tying version, buildID, and channel together somehow - seems nontrivial. >+ paths = generate_paths(update_data, force=args.force) I was surprised generate_paths() lived elsewhere, do you have other plans for it ? itertools is nice!

(In reply to Nick Thomas [:nthomas] from comment #7) > Comment on attachment 729192 [details] [diff] [review] > scripts to do snippet comparsion > > >diff --git a/release/generate-update-urls.py b/release/generate-update-urls.py > >+from balrog.testing import compare_snippets, generate_paths > > compare_snippets isn't used here. > > >+DEFAULT_UPDATE_DATA = { > >+ 'products': ['Firefox'], > >+ 'versions': ['10.0a1', '12.0a1', '17.0a1'], > >+ 'buildids': ['20120222174716', '20111201231412'], > > Are any nonsensical combinations a feature or just a bit of a test slowdown > ? The latter. This format was a pretty quick way to get a large number of mostly reasonable URLs. I think that we probably won't use this script in the end though...it'll be replaced w/ URLs we get from Metrics and URLs we can generate from patcher configs. > Or is the idea to call the script several times, with different sets of > self-consistent args ? If so, you could just amend the default to one value > for each. The alternative - tying version, buildID, and channel together > somehow - seems nontrivial. I hadn't thought of calling the script multiple times actually. I'm not sure what you mean by "amend the default to one value for each" though... > >+ paths = generate_paths(update_data, force=args.force) > > I was surprised generate_paths() lived elsewhere, do you have other plans > for it ? Nothing so far. I could move this into the script if that makes more sense. Upon second viewing it does seem pretty specific to generate-update-urls.py...

I spoke with Metrics about whether or not our replaying would skew them, it doesn't look it will: 10:51 < bhearsum> am i crazy in thinking that we ignore AUS data that comes from within our network? 10:51 < bhearsum> i seem to recall someone told me that once, but it's only a vague memory 10:56 < mreid> bhearsum, you're not crazy - we filter a bunch of IP addresses 10:57 < bhearsum> mreid: do you know if that list includes the RelEng network? i ask because i'm planning to do a bunch of queries on a regular basis, and i don't want to mess up any metrics 10:57 < mreid> bhearsum, I can give you the list of IPs being filtered 10:57 < bhearsum> that'd be great 10:59 < mreid> bhearsum, http://mreid.pastebin.mozilla.org/2253793 10:59 < deinspanjer> bhearsum: We basically filter just about everything coming from one of the IPs assigned to our datacenters. 11:00 < bhearsum> ah 11:00 < bhearsum> yeah, i think that list includes the outbound addresses for all of the RelEng stuff That pastebin included the following: Filter string Filter position Stop on filter Positive match 63.245.208 0 N N 63.245.209 0 N N 63.245.21 0 N N 63.245.220 0 N N 63.245.221 0 N N 63.245.222 0 N N 63.245.223 0 N N 180.92.184 0 N N 59.151.50 0 N N 72.26.221.66 0 N N 72.26.221.67 0 N N 209.10.217.226 0 N N And according to https://bugzilla.mozilla.org/show_bug.cgi?id=646076#c14, that includes the hide nats for at least mtv1/scl1, so I assume it also includes SCL3.

Not a whole lot of changes here: * Move into the Balrog repo * Put generate_paths in generate-update-paths.py * Fix some bugs in AUS-server.py that I noticed (unrelated to this bug) I also generated what I think is a decent set of queries for now. I'm hosting it at https://people.mozilla.com/~bhearsum/update-paths.txt for lack of a better place. It covers all platform on the Firefox nightly channel across 13 locales and 2 OS versions for each platform. Do you think this is enough until bug 853051 gets fixed? I'm still not 100% sure I understand your comment about the default update data either, let me know if there's changes that you still want me to make around that - I'm not trying to skirt that one ;).

Comment on attachment 730816 [details] [diff] [review] v2 We can let my comment about single valued defaults lapse, it was based on mis-grokking your intentions. The update_paths.txt looks fine for testing nightlies, except for wondering about the duplicates in there. Looks like 260 unique urls there but 390 lines in total.

(In reply to Nick Thomas [:nthomas] from comment #11) > Comment on attachment 730816 [details] [diff] [review] > v2 > > We can let my comment about single valued defaults lapse, it was based on > mis-grokking your intentions. > > The update_paths.txt looks fine for testing nightlies, except for wondering > about the duplicates in there. Looks like 260 unique urls there but 390 > lines in total. Heh, I passed "--buildid 20130314030914 --buildid 20130314030914" in some of my runs, which explains all those dupes. I'll remove them.

Commit pushed to master at https://github.com/mozilla/balrog https://github.com/mozilla/balrog/commit/a8ca8f426538b91bc297932f289da9d6a66c1315 bug 832465: develop script/automation to replay real update requests against balrog. r=nthomas

Leaving this bug open to switch over to Metrics report based URLs after bug 853051 is fixed. Nothing to do until that time.

Turns out that this broke the tests: ====================================================================== ERROR: Failure: ImportError (No module named requests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/tmp/tmp.gm6PeezuBG/lib/python2.6/site-packages/nose/loader.py", line 390, in loadTestsFromName addr.filename, addr.module) File "/tmp/tmp.gm6PeezuBG/lib/python2.6/site-packages/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/tmp/tmp.gm6PeezuBG/lib/python2.6/site-packages/nose/importer.py", line 86, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/var/lib/jenkins/jobs/Balrog/workspace/auslib/util/testing.py", line 3, in <module> import requests ImportError: No module named requests Probably need to make nosetests ignore this file...

Commit pushed to master at https://github.com/mozilla/balrog https://github.com/mozilla/balrog/commit/10bfe12fd0797599104dc8dfa23364093b7c2f41 bug 832465: develop script/automation to replay real update requests against balrog - be explicit about where the tests are. r=catlee

Comment on attachment 732794 [details] [diff] [review] be explicit about where the tests are This greened up the test.

I ran some tests at various ||izations, and it looks like -j8 is about what we want. It ran this test in 8min30sec. -j10, 16, 32, 48, and 64 all took just over 7min -- so we're clearly reaching our peak around -j10. Best to keep us a bit under that peak to lower the chance of getting bad responses from zeus/AUS. I had to use our own retry here because requests' doesn't retry on ISE 500, which gets thrown from time to time.

Comment on attachment 738614 [details] [diff] [review] ||ize tests; use update paths from metrics Review of attachment 738614 [details] [diff] [review]: ----------------------------------------------------------------- lgtm

Commit pushed to master at https://github.com/mozilla/balrog https://github.com/mozilla/balrog/commit/489f4cc7c1e54a78463b67273537faea65adcfbe bug 832465: develop script/automation to replay real update requests against balrog. r=nthomas

Most recent run has: Tested 5828 paths. Pass count: 909 Fail count: 4919