Closed
Bug 1144985
Opened 10 years ago
Closed 10 years ago
Serve nightly & aurora updates from the CDN
Categories
(Release Engineering :: General, defect)
Tracking
(firefox37 fixed, firefox38 fixed, firefox39 fixed)
RESOLVED
FIXED
People
(Reporter: nthomas, Assigned: rail)
References
Details
Attachments
(8 files, 6 obsolete files)
(deleted),
patch
|
bhearsum
:
review+
rail
:
checked-in+
|
Details | Diff | Splinter Review |
(deleted),
patch
|
bhearsum
:
review+
rail
:
checked-in+
|
Details | Diff | Splinter Review |
(deleted),
patch
|
rail
:
review+
rail
:
checked-in+
|
Details | Diff | Splinter Review |
(deleted),
patch
|
bhearsum
:
review+
rail
:
checked-in+
|
Details | Diff | Splinter Review |
(deleted),
patch
|
bhearsum
:
review+
rail
:
checked-in+
|
Details | Diff | Splinter Review |
(deleted),
patch
|
bhearsum
:
review+
rail
:
checked-in+
|
Details | Diff | Splinter Review |
(deleted),
patch
|
bhearsum
:
review+
rail
:
checked-in+
|
Details | Diff | Splinter Review |
(deleted),
patch
|
rail
:
review+
mshal
:
checked-in+
|
Details | Diff | Splinter Review |
We serve more than 1.5 TB/day for .mar files on ftp.m.o, which averages to 200 Mbps. The lions share of that is likely to be updates for nightly and aurora. Since we are currently having trouble with Zeus traffic levels I'm proposing we move that to the CDN, and bypassing bouncer.
This means we just point to download.cdn.mozilla.net instead of ftp.mozilla.org when submitting into Balrog, and acknowledge that we won't have any logs as a result. I don't think setting up bouncer products for each nightly just to get hits on download.m.o is useful, given the data is not processed in any accessible way.
We'll have to add download.cdn.mozilla.net to the domain_whitelist in balrog.ini and admin.ini.
Comment 1•10 years ago
|
||
Getting this done for the current updates has become high priority, doing that right now. Hopefully it will be trivial to enable it for future updates afterwards.
Assignee: nobody → bhearsum
Severity: normal → critical
Comment 2•10 years ago
|
||
Okay, Nightly and Aurora traffic should be on the CDN. Here's what we did:
* Downloaded the Firefox-mozilla-{central,aurora}-nightly-latest release blobs from Balrog.
* Copied them to -cdn versions.
* s/ftp.mozilla.org/download.cdn.mozilla.net/g in them
* Uploaded to Balrog with the new names
* Repointed nightly and aurora rules at them
With this in place, all nightly and aurora Firefox updates should be on the CDN instead of FTP. Because we've used a new release name, the next set of builds will NOT be automatically published. We need to do one of the following to get the next set published:
1) Adjust our CI to publish nightlies that point at download.cdn.mozilla.net automatically (instead of ftp.mozilla.org). Repoint nightly/aurora rules at original release blobs after the next set completes (making sure to wait for at least some l10n jobs).
2) Repeat the steps above after tomorrow's nightlies finish.
Assignee | ||
Comment 3•10 years ago
|
||
Automation uses URLs printed by post_upload.py. We can:
1) change http://hg.mozilla.org/build/tools/file/b5405137c3dc/stage/production.ini#l15 and point ALL bits to CDN. This means that the tests will be hitting CDN too
2) add hacks to post_upload.py to treat MAR files differently (have different domain for them)
3) add hacks to automation to replace ftp.mozilla.org with cdn domain
The first option is the cleanest one, not sure how comfortable we with serving ALL nightly files form CDNs.
Assignee | ||
Comment 4•10 years ago
|
||
If we are comfortable with option 1, we can use this patch.
The file needs to be copied to the productdelivery svn module by someone with access (I lost mine at some point).
Attachment #8580135 -
Flags: review?(bhearsum)
Comment 5•10 years ago
|
||
Comment on attachment 8580135 [details] [diff] [review]
cdn.diff
Review of attachment 8580135 [details] [diff] [review]:
-----------------------------------------------------------------
Do we know if our build and test machines can route to the CDNs? If so, how do we make sure it doesn't accidentally change in the future?
I'm still not sure it's great to move this traffic to the CDNs, especially on a whim. I'd be more comfortable if we could _only_ change what's in the blob....
Assignee | ||
Comment 6•10 years ago
|
||
Comment on attachment 8580135 [details] [diff] [review]
cdn.diff
Just chatted with catlee. We came to the conclusion that option 3 would be clearer here.
Attachment #8580135 -
Flags: review?(bhearsum)
Assignee | ||
Comment 7•10 years ago
|
||
Something like this maybe?
Attachment #8580181 -
Flags: review?(bhearsum)
Comment 8•10 years ago
|
||
Comment on attachment 8580181 [details] [diff] [review]
replace_canonical_urls-tools.diff
Review of attachment 8580181 [details] [diff] [review]:
-----------------------------------------------------------------
Hiding this all the way down here kindof sucks. It would be nicer to do it at the script level if possible...
We should make sure that all of the files that we point at are on the CDNs. It looks like that's the case (eg, download.cdn.mozilla.net/pub/mozilla.org/b2g/nightly/2015-03-19-00-12-27-mozilla-b2g34_v2_1-flame-kk/b2g-flame-gecko-update.mar works), but there might be an edge case that I'm missing...
Reporter | ||
Comment 9•10 years ago
|
||
Thanks for picking this work up. FWIW, I started on a patch pretty similar to this. Memory asserts that the origin server has access to all files, but there's a slight chance we might break things with future changes. No doubt it will change with s!ftp!S3! anyway.
Assignee | ||
Comment 10•10 years ago
|
||
I updated the blobs like in comment#2 today.
Assignee | ||
Comment 11•10 years ago
|
||
This is another, more indirect approach.
Attachment #8580181 -
Attachment is obsolete: true
Attachment #8580181 -
Flags: review?(bhearsum)
Attachment #8580721 -
Flags: feedback?(bhearsum)
Assignee | ||
Comment 12•10 years ago
|
||
Attachment #8580722 -
Flags: feedback?(bhearsum)
Tracking for 38 and 39.
status-firefox37:
--- → affected
status-firefox38:
--- → affected
tracking-firefox38:
--- → +
tracking-firefox39:
--- → +
Reporter | ||
Comment 14•10 years ago
|
||
I've pointed Nightly and Aurora updates back to the latest builds, via ftp.m.o because we have more traffic headroom there now. We have a month to decide how to switch to the CDN.
Assignee | ||
Comment 15•10 years ago
|
||
Attachment #8580721 -
Attachment is obsolete: true
Attachment #8580721 -
Flags: feedback?(bhearsum)
Attachment #8581574 -
Flags: feedback?(bhearsum)
Assignee | ||
Comment 16•10 years ago
|
||
Moar test. Try in production? :)
Attachment #8581574 -
Attachment is obsolete: true
Attachment #8581574 -
Flags: feedback?(bhearsum)
Attachment #8581580 -
Flags: review?(bhearsum)
Assignee | ||
Updated•10 years ago
|
Attachment #8580722 -
Flags: feedback?(bhearsum) → review?(bhearsum)
Comment 17•10 years ago
|
||
Comment on attachment 8581580 [details] [diff] [review]
replace_canonical_urls-tools-1.diff
Review of attachment 8581580 [details] [diff] [review]:
-----------------------------------------------------------------
::: lib/python/balrog/submitter/cli.py
@@ +306,5 @@
> class MultipleUpdatesNightlyMixin(object):
> +
> + def _maybe_replace_canocical_url(self, url):
> + # self._replace_canocical_url() and self.url_replacements come from
> + # NightlySubmitterBase. Make sure they exist in concrete classes
This is pretty nasty....are you sure it's necessary? I don't see url_replacements ever getting passed to Release* classes.
Comment 18•10 years ago
|
||
Comment on attachment 8580722 [details] [diff] [review]
balrog_replace_urls-mozharness.diff
Review of attachment 8580722 [details] [diff] [review]:
-----------------------------------------------------------------
::: configs/balrog/production.py
@@ +4,5 @@
> 'balrog_api_root': 'https://aus4-admin.mozilla.org/api',
> 'ignore_failures': False,
> + 'url_replacements': [
> + ('http://ftp.mozilla.org/pub/mozilla.org',
> + 'http://download.cdn.mozilla.net/pub/mozilla.org'),
Minor nit: this would read better with both URLs on the same line IMO. If we ever add a second one it will be harder to see the before and after URLs at a glance.
Attachment #8580722 -
Flags: review?(bhearsum) → review+
Assignee | ||
Comment 19•10 years ago
|
||
Fix a typo.
I ran this against localhost (nothing running) with the following patch applied https://gist.github.com/rail/5330ebb3d49da9deefb7 (just to dump data):
python scripts/updates/balrog-submitter.py --build-properties ~/tmp/b.json --api-root https://localhost/api --username ffxbld -t nightly --credentials-file ~/tmp/oauth.txt --verbose --url-replacement ftp.mozilla.org,download.cdn.mozilla.net
Replacing http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/03/2015-03-23-03-02-03-mozilla-central/firefox-39.0a1.en-US.linux-i686.complete.mar with http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/2015/03/2015-03-23-03-02-03-mozilla-central/firefox-39.0a1.en-US.linux-i686.complete.mar
Replacing http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/03/2015-03-23-03-02-03-mozilla-central/firefox-39.0a1.en-US.linux-i686.partial.20150322030216-20150323030203.mar with http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/2015/03/2015-03-23-03-02-03-mozilla-central/firefox-39.0a1.en-US.linux-i686.partial.20150322030216-20150323030203.mar
{'product': u'Firefox', 'hashFunction': u'sha512', 'schema_version': 4, 'alias': 'null', 'copyTo': '["Firefox-mozilla-central-nightly-latest"]', 'version': u'39.0a1', 'data': '{"platformVersion": "39.0a1", "partials": [{"fileUrl": "http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/2015/03/2015-03-23-03-02-03-mozilla-central/firefox-39.0a1.en-US.linux-i686.partial.20150322030216-20150323030203.mar", "hashValue": "a8ec9c0ba52cb153511c0842f8a2109d056978e93e0aeecb852ae2b93fbe3ff905de948814c34b195998f20eb7c013e7425a688eee1b4213e208b8aeb91be9ff", "from": "Firefox-mozilla-central-nightly-20150322030216", "filesize": 8244610}], "completes": [{"fileUrl": "http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/2015/03/2015-03-23-03-02-03-mozilla-central/firefox-39.0a1.en-US.linux-i686.complete.mar", "hashValue": "c4ea1e8548582691c49b36c9bf0d0ba5b5b22e9b2313486cac9926e9ab1d95cb297f32f755127f5043d92c0c9dcc7aa13166bafcd761b62a27fed97b64cd6b20", "from": "*", "filesize": 54133063}], "buildID": "20150323030203", "displayVersion": "39.0a1", "appVersion": "39.0a1"}'}
Attachment #8581580 -
Attachment is obsolete: true
Attachment #8581580 -
Flags: review?(bhearsum)
Attachment #8581592 -
Flags: review?(bhearsum)
Assignee | ||
Comment 20•10 years ago
|
||
(In reply to Ben Hearsum [:bhearsum] from comment #17)
> Comment on attachment 8581580 [details] [diff] [review]
> replace_canonical_urls-tools-1.diff
>
> Review of attachment 8581580 [details] [diff] [review]:
> -----------------------------------------------------------------
>
> ::: lib/python/balrog/submitter/cli.py
> @@ +306,5 @@
> > class MultipleUpdatesNightlyMixin(object):
> > +
> > + def _maybe_replace_canocical_url(self, url):
> > + # self._replace_canocical_url() and self.url_replacements come from
> > + # NightlySubmitterBase. Make sure they exist in concrete classes
>
> This is pretty nasty....are you sure it's necessary? I don't see
> url_replacements ever getting passed to Release* classes.
You mean I should remove the guards?
Assignee | ||
Updated•10 years ago
|
Attachment #8580135 -
Attachment is obsolete: true
Comment 21•10 years ago
|
||
(In reply to Rail Aliiev [:rail] from comment #20)
> (In reply to Ben Hearsum [:bhearsum] from comment #17)
> > Comment on attachment 8581580 [details] [diff] [review]
> > replace_canonical_urls-tools-1.diff
> >
> > Review of attachment 8581580 [details] [diff] [review]:
> > -----------------------------------------------------------------
> >
> > ::: lib/python/balrog/submitter/cli.py
> > @@ +306,5 @@
> > > class MultipleUpdatesNightlyMixin(object):
> > > +
> > > + def _maybe_replace_canocical_url(self, url):
> > > + # self._replace_canocical_url() and self.url_replacements come from
> > > + # NightlySubmitterBase. Make sure they exist in concrete classes
> >
> > This is pretty nasty....are you sure it's necessary? I don't see
> > url_replacements ever getting passed to Release* classes.
>
> You mean I should remove the guards?
I don't see any reason for them to exist, so I would hesistantly say yes. They don't actually affect any behaviour right now, right?
Assignee | ||
Comment 22•10 years ago
|
||
Attachment #8581592 -
Attachment is obsolete: true
Attachment #8581592 -
Flags: review?(bhearsum)
Attachment #8581713 -
Flags: review?(bhearsum)
Updated•10 years ago
|
Attachment #8581713 -
Flags: review?(bhearsum) → review+
Assignee | ||
Comment 23•10 years ago
|
||
Comment on attachment 8581713 [details] [diff] [review]
replace_canonical_urls-tools.diff
https://hg.mozilla.org/build/tools/rev/789a3138bd4d
Attachment #8581713 -
Flags: checked-in+
Assignee | ||
Comment 24•10 years ago
|
||
Comment on attachment 8580722 [details] [diff] [review]
balrog_replace_urls-mozharness.diff
https://hg.mozilla.org/build/mozharness/rev/a814625fcd39
Attachment #8580722 -
Flags: checked-in+
Updated•10 years ago
|
Assignee: bhearsum → rail
Assignee | ||
Comment 25•10 years ago
|
||
Make python 2.6 happier.
Attachment #8581765 -
Flags: review?(bhearsum)
Assignee | ||
Comment 26•10 years ago
|
||
Comment on attachment 8581765 [details] [diff] [review]
replace_canonical_urls-tools-1.diff
r=bhearsum in IRC
https://hg.mozilla.org/build/tools/rev/0d0c1165741c
Attachment #8581765 -
Flags: review?(bhearsum)
Attachment #8581765 -
Flags: review+
Attachment #8581765 -
Flags: checked-in+
Assignee | ||
Comment 27•10 years ago
|
||
Of course, skipIf if 2.7+ only...
green: https://travis-ci.org/rail/build-tools/builds/55529377
Attachment #8581836 -
Flags: review?(bhearsum)
Updated•10 years ago
|
Attachment #8581836 -
Flags: review?(bhearsum) → review+
Assignee | ||
Comment 28•10 years ago
|
||
Comment on attachment 8581836 [details] [diff] [review]
replace_canonical_urls-tools-2.diff
https://hg.mozilla.org/build/tools/rev/bccc7e199158
Attachment #8581836 -
Flags: checked-in+
Comment 29•10 years ago
|
||
mozharness production tag moved to: https://hg.mozilla.org/build/mozharness/rev/production
Assignee | ||
Comment 30•10 years ago
|
||
For nighlty and l10n which are not in mozharness yet
Attachment #8582044 -
Flags: review?(bhearsum)
Assignee | ||
Comment 31•10 years ago
|
||
Attachment #8582046 -
Flags: review?(bhearsum)
Assignee | ||
Comment 32•10 years ago
|
||
I updated the blobs like in comment#2 today.
Comment 34•10 years ago
|
||
Comment on attachment 8582388 [details] [diff] [review]
mozilla-inbound_bump_mozharness_rev.diff
Review of attachment 8582388 [details] [diff] [review]:
-----------------------------------------------------------------
I don't think you need review for mozharness version bumps, but r+ anyways.
Attachment #8582388 -
Flags: review?(bhearsum) → review+
Updated•10 years ago
|
Attachment #8582046 -
Flags: review?(bhearsum) → review+
Updated•10 years ago
|
Attachment #8582044 -
Flags: review?(bhearsum) → review+
Assignee | ||
Comment 35•10 years ago
|
||
Comment on attachment 8582044 [details] [diff] [review]
configs-nighlty.diff
https://hg.mozilla.org/build/buildbot-configs/rev/b804f613e118
Attachment #8582044 -
Flags: checked-in+
Assignee | ||
Comment 36•10 years ago
|
||
Comment on attachment 8582046 [details] [diff] [review]
custom-nightly.diff
https://hg.mozilla.org/build/buildbotcustom/rev/8a55bdc3828a
Attachment #8582046 -
Flags: checked-in+
Assignee | ||
Comment 37•10 years ago
|
||
Comment on attachment 8582388 [details] [diff] [review]
mozilla-inbound_bump_mozharness_rev.diff
mshal jinxed me in https://hg.mozilla.org/integration/mozilla-inbound/rev/50d681158e9f :)
Attachment #8582388 -
Flags: checked-in+
Comment 38•10 years ago
|
||
Comment 39•10 years ago
|
||
Comment 40•10 years ago
|
||
In production: https://hg.mozilla.org/build/buildbotcustom/rev/8a55bdc3828a
Assignee | ||
Comment 41•10 years ago
|
||
L10N repacks worked fine (from yesterday), Android worked fine, waiting for m-c en-US build.
Assignee | ||
Comment 42•10 years ago
|
||
Just checked mac en-US nightly, the partial in Firefox-mozilla-central-nightly-20150325030206 looks good. The complete is pointing to https://queue.taskcluster.net/v1/task/dUi7oY11RxaXt7hjH_wE8A/artifacts/public/build/firefox-39.0a1.en-US.mac.complete.mar. Is this expected?
Assignee | ||
Comment 43•10 years ago
|
||
http://buildbot-master86.bb.releng.scl3.mozilla.com:8001/builders/OS%20X%2010.7%20mozilla-central%20nightly/builds/4 properties look interesting. None of the properties contain "https://queue.taskcluster.net/", but if you look to the script output, some URLs use ftp.m.o, some queue.t.n. balrog-submitter.py itself is a bit confused which one to use. :)
CCing Mike, he may have some info.
Comment 44•10 years ago
|
||
It looks like I might just need to add a 'write_to_file=True' argument to set_buildbot_property() when mozharness is uploading the arguments to TC. I'm going to test that out on maple...
Comment 45•10 years ago
|
||
This is untested at the moment, but it should avoid overriding completeMarUrl and partialMarUrl. We still need to keep the property conditions in there though to avoid stomping on packageUrl, so these are just renamed to completeMarUrlTC and partialMarUrlTC, which shouldn't be used by anything else.
Attachment #8583847 -
Flags: review?(rail)
Assignee | ||
Updated•10 years ago
|
Attachment #8583847 -
Flags: review?(rail) → review+
Comment 46•10 years ago
|
||
Comment on attachment 8583847 [details] [diff] [review]
disable-mar-TC
https://hg.mozilla.org/build/mozharness/rev/db4fdf68f146
Attachment #8583847 -
Flags: checked-in+
Assignee | ||
Comment 47•10 years ago
|
||
Live!
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Comment 48•10 years ago
|
||
I guess it is fixed for 38 & 37.
Comment 49•10 years ago
|
||
(In reply to Sylvestre Ledru [:sylvestre] from comment #48)
> I guess it is fixed for 38 & 37.
These flags really don't apply here...this isn't in-tree code that's being tweaked, and aurora and nightly are already at 39 & 40...
Looks like this was fixed while 39 was in nightly so we can call it fixed for 39+. Untracking.
Updated•6 years ago
|
Component: General Automation → General
You need to log in
before you can comment on or make changes to this bug.
Description
•