We serve more than 1.5 TB/day for .mar files on ftp.m.o, which averages to 200 Mbps. The lions share of that is likely to be updates for nightly and aurora. Since we are currently having trouble with Zeus traffic levels I'm proposing we move that to the CDN, and bypassing bouncer. This means we just point to download.cdn.mozilla.net instead of ftp.mozilla.org when submitting into Balrog, and acknowledge that we won't have any logs as a result. I don't think setting up bouncer products for each nightly just to get hits on download.m.o is useful, given the data is not processed in any accessible way. We'll have to add download.cdn.mozilla.net to the domain_whitelist in balrog.ini and admin.ini.

Getting this done for the current updates has become high priority, doing that right now. Hopefully it will be trivial to enable it for future updates afterwards.

Okay, Nightly and Aurora traffic should be on the CDN. Here's what we did: * Downloaded the Firefox-mozilla-{central,aurora}-nightly-latest release blobs from Balrog. * Copied them to -cdn versions. * s/ftp.mozilla.org/download.cdn.mozilla.net/g in them * Uploaded to Balrog with the new names * Repointed nightly and aurora rules at them With this in place, all nightly and aurora Firefox updates should be on the CDN instead of FTP. Because we've used a new release name, the next set of builds will NOT be automatically published. We need to do one of the following to get the next set published: 1) Adjust our CI to publish nightlies that point at download.cdn.mozilla.net automatically (instead of ftp.mozilla.org). Repoint nightly/aurora rules at original release blobs after the next set completes (making sure to wait for at least some l10n jobs). 2) Repeat the steps above after tomorrow's nightlies finish.

Automation uses URLs printed by post_upload.py. We can: 1) change http://hg.mozilla.org/build/tools/file/b5405137c3dc/stage/production.ini#l15 and point ALL bits to CDN. This means that the tests will be hitting CDN too 2) add hacks to post_upload.py to treat MAR files differently (have different domain for them) 3) add hacks to automation to replace ftp.mozilla.org with cdn domain The first option is the cleanest one, not sure how comfortable we with serving ALL nightly files form CDNs.

If we are comfortable with option 1, we can use this patch. The file needs to be copied to the productdelivery svn module by someone with access (I lost mine at some point).

Comment on attachment 8580135 [details] [diff] [review] cdn.diff Review of attachment 8580135 [details] [diff] [review]: ----------------------------------------------------------------- Do we know if our build and test machines can route to the CDNs? If so, how do we make sure it doesn't accidentally change in the future? I'm still not sure it's great to move this traffic to the CDNs, especially on a whim. I'd be more comfortable if we could _only_ change what's in the blob....

Comment on attachment 8580135 [details] [diff] [review] cdn.diff Just chatted with catlee. We came to the conclusion that option 3 would be clearer here.

Something like this maybe?

Comment on attachment 8580181 [details] [diff] [review] replace_canonical_urls-tools.diff Review of attachment 8580181 [details] [diff] [review]: ----------------------------------------------------------------- Hiding this all the way down here kindof sucks. It would be nicer to do it at the script level if possible... We should make sure that all of the files that we point at are on the CDNs. It looks like that's the case (eg, download.cdn.mozilla.net/pub/mozilla.org/b2g/nightly/2015-03-19-00-12-27-mozilla-b2g34_v2_1-flame-kk/b2g-flame-gecko-update.mar works), but there might be an edge case that I'm missing...

Thanks for picking this work up. FWIW, I started on a patch pretty similar to this. Memory asserts that the origin server has access to all files, but there's a slight chance we might break things with future changes. No doubt it will change with s!ftp!S3! anyway.

I updated the blobs like in comment#2 today.

This is another, more indirect approach.

Tracking for 38 and 39.

I've pointed Nightly and Aurora updates back to the latest builds, via ftp.m.o because we have more traffic headroom there now. We have a month to decide how to switch to the CDN.

Moar test. Try in production? :)

Comment on attachment 8581580 [details] [diff] [review] replace_canonical_urls-tools-1.diff Review of attachment 8581580 [details] [diff] [review]: ----------------------------------------------------------------- ::: lib/python/balrog/submitter/cli.py @@ +306,5 @@ > class MultipleUpdatesNightlyMixin(object): > + > + def _maybe_replace_canocical_url(self, url): > + # self._replace_canocical_url() and self.url_replacements come from > + # NightlySubmitterBase. Make sure they exist in concrete classes This is pretty nasty....are you sure it's necessary? I don't see url_replacements ever getting passed to Release* classes.

Comment on attachment 8580722 [details] [diff] [review] balrog_replace_urls-mozharness.diff Review of attachment 8580722 [details] [diff] [review]: ----------------------------------------------------------------- ::: configs/balrog/production.py @@ +4,5 @@ > 'balrog_api_root': 'https://aus4-admin.mozilla.org/api', > 'ignore_failures': False, > + 'url_replacements': [ > + ('http://ftp.mozilla.org/pub/mozilla.org', > + 'http://download.cdn.mozilla.net/pub/mozilla.org'), Minor nit: this would read better with both URLs on the same line IMO. If we ever add a second one it will be harder to see the before and after URLs at a glance.

Fix a typo. I ran this against localhost (nothing running) with the following patch applied https://gist.github.com/rail/5330ebb3d49da9deefb7 (just to dump data): python scripts/updates/balrog-submitter.py --build-properties ~/tmp/b.json --api-root https://localhost/api --username ffxbld -t nightly --credentials-file ~/tmp/oauth.txt --verbose --url-replacement ftp.mozilla.org,download.cdn.mozilla.net Replacing http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/03/2015-03-23-03-02-03-mozilla-central/firefox-39.0a1.en-US.linux-i686.complete.mar with http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/2015/03/2015-03-23-03-02-03-mozilla-central/firefox-39.0a1.en-US.linux-i686.complete.mar Replacing http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/03/2015-03-23-03-02-03-mozilla-central/firefox-39.0a1.en-US.linux-i686.partial.20150322030216-20150323030203.mar with http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/2015/03/2015-03-23-03-02-03-mozilla-central/firefox-39.0a1.en-US.linux-i686.partial.20150322030216-20150323030203.mar {'product': u'Firefox', 'hashFunction': u'sha512', 'schema_version': 4, 'alias': 'null', 'copyTo': '["Firefox-mozilla-central-nightly-latest"]', 'version': u'39.0a1', 'data': '{"platformVersion": "39.0a1", "partials": [{"fileUrl": "http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/2015/03/2015-03-23-03-02-03-mozilla-central/firefox-39.0a1.en-US.linux-i686.partial.20150322030216-20150323030203.mar", "hashValue": "a8ec9c0ba52cb153511c0842f8a2109d056978e93e0aeecb852ae2b93fbe3ff905de948814c34b195998f20eb7c013e7425a688eee1b4213e208b8aeb91be9ff", "from": "Firefox-mozilla-central-nightly-20150322030216", "filesize": 8244610}], "completes": [{"fileUrl": "http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/2015/03/2015-03-23-03-02-03-mozilla-central/firefox-39.0a1.en-US.linux-i686.complete.mar", "hashValue": "c4ea1e8548582691c49b36c9bf0d0ba5b5b22e9b2313486cac9926e9ab1d95cb297f32f755127f5043d92c0c9dcc7aa13166bafcd761b62a27fed97b64cd6b20", "from": "*", "filesize": 54133063}], "buildID": "20150323030203", "displayVersion": "39.0a1", "appVersion": "39.0a1"}'}

(In reply to Ben Hearsum [:bhearsum] from comment #17) > Comment on attachment 8581580 [details] [diff] [review] > replace_canonical_urls-tools-1.diff > > Review of attachment 8581580 [details] [diff] [review]: > ----------------------------------------------------------------- > > ::: lib/python/balrog/submitter/cli.py > @@ +306,5 @@ > > class MultipleUpdatesNightlyMixin(object): > > + > > + def _maybe_replace_canocical_url(self, url): > > + # self._replace_canocical_url() and self.url_replacements come from > > + # NightlySubmitterBase. Make sure they exist in concrete classes > > This is pretty nasty....are you sure it's necessary? I don't see > url_replacements ever getting passed to Release* classes. You mean I should remove the guards?

(In reply to Rail Aliiev [:rail] from comment #20) > (In reply to Ben Hearsum [:bhearsum] from comment #17) > > Comment on attachment 8581580 [details] [diff] [review] > > replace_canonical_urls-tools-1.diff > > > > Review of attachment 8581580 [details] [diff] [review]: > > ----------------------------------------------------------------- > > > > ::: lib/python/balrog/submitter/cli.py > > @@ +306,5 @@ > > > class MultipleUpdatesNightlyMixin(object): > > > + > > > + def _maybe_replace_canocical_url(self, url): > > > + # self._replace_canocical_url() and self.url_replacements come from > > > + # NightlySubmitterBase. Make sure they exist in concrete classes > > > > This is pretty nasty....are you sure it's necessary? I don't see > > url_replacements ever getting passed to Release* classes. > > You mean I should remove the guards? I don't see any reason for them to exist, so I would hesistantly say yes. They don't actually affect any behaviour right now, right?

Interdiff https://gist.github.com/rail/b17f8011ba632ca6d4f8

Comment on attachment 8581713 [details] [diff] [review] replace_canonical_urls-tools.diff https://hg.mozilla.org/build/tools/rev/789a3138bd4d

Comment on attachment 8580722 [details] [diff] [review] balrog_replace_urls-mozharness.diff https://hg.mozilla.org/build/mozharness/rev/a814625fcd39

Make python 2.6 happier.

Comment on attachment 8581765 [details] [diff] [review] replace_canonical_urls-tools-1.diff r=bhearsum in IRC https://hg.mozilla.org/build/tools/rev/0d0c1165741c

Of course, skipIf if 2.7+ only... green: https://travis-ci.org/rail/build-tools/builds/55529377

Comment on attachment 8581836 [details] [diff] [review] replace_canonical_urls-tools-2.diff https://hg.mozilla.org/build/tools/rev/bccc7e199158

mozharness production tag moved to: https://hg.mozilla.org/build/mozharness/rev/production

For nighlty and l10n which are not in mozharness yet

I updated the blobs like in comment#2 today.

In-tree bump

Comment on attachment 8582388 [details] [diff] [review] mozilla-inbound_bump_mozharness_rev.diff Review of attachment 8582388 [details] [diff] [review]: ----------------------------------------------------------------- I don't think you need review for mozharness version bumps, but r+ anyways.

Comment on attachment 8582044 [details] [diff] [review] configs-nighlty.diff https://hg.mozilla.org/build/buildbot-configs/rev/b804f613e118

Comment on attachment 8582046 [details] [diff] [review] custom-nightly.diff https://hg.mozilla.org/build/buildbotcustom/rev/8a55bdc3828a

Comment on attachment 8582388 [details] [diff] [review] mozilla-inbound_bump_mozharness_rev.diff mshal jinxed me in https://hg.mozilla.org/integration/mozilla-inbound/rev/50d681158e9f :)

In production: https://hg.mozilla.org/build/buildbot-configs/rev/b804f613e118

In production: https://hg.mozilla.org/build/buildbot-configs/rev/22f9faca403c

In production: https://hg.mozilla.org/build/buildbotcustom/rev/8a55bdc3828a

L10N repacks worked fine (from yesterday), Android worked fine, waiting for m-c en-US build.

Just checked mac en-US nightly, the partial in Firefox-mozilla-central-nightly-20150325030206 looks good. The complete is pointing to https://queue.taskcluster.net/v1/task/dUi7oY11RxaXt7hjH_wE8A/artifacts/public/build/firefox-39.0a1.en-US.mac.complete.mar. Is this expected?

http://buildbot-master86.bb.releng.scl3.mozilla.com:8001/builders/OS%20X%2010.7%20mozilla-central%20nightly/builds/4 properties look interesting. None of the properties contain "https://queue.taskcluster.net/", but if you look to the script output, some URLs use ftp.m.o, some queue.t.n. balrog-submitter.py itself is a bit confused which one to use. :) CCing Mike, he may have some info.

It looks like I might just need to add a 'write_to_file=True' argument to set_buildbot_property() when mozharness is uploading the arguments to TC. I'm going to test that out on maple...

This is untested at the moment, but it should avoid overriding completeMarUrl and partialMarUrl. We still need to keep the property conditions in there though to avoid stomping on packageUrl, so these are just renamed to completeMarUrlTC and partialMarUrlTC, which shouldn't be used by anything else.

Comment on attachment 8583847 [details] [diff] [review] disable-mar-TC https://hg.mozilla.org/build/mozharness/rev/db4fdf68f146

Live!

I guess it is fixed for 38 & 37.

(In reply to Sylvestre Ledru [:sylvestre] from comment #48) > I guess it is fixed for 38 & 37. These flags really don't apply here...this isn't in-tree code that's being tweaked, and aurora and nightly are already at 39 & 40...

Looks like this was fixed while 39 was in nightly so we can call it fixed for 39+. Untracking.