Closed
Bug 1300060
Opened 8 years ago
Closed 8 years ago
SHA1 repacks break uptake monitoring by timing out in release promotion
Categories
(Release Engineering :: Release Automation: Other, defect)
Release Engineering
Release Automation: Other
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: mtabara, Assigned: rail)
References
Details
Attachments
(1 file)
(deleted),
text/x-github-pull-request
|
mtabara
:
review+
rail
:
checked-in+
|
Details |
In https://bugzilla.mozilla.org/show_bug.cgi?id=1290179#c12 we enabled SHA1 repacks and added them to the stuff we check as part of the uptake monitoring step in release promotion.
Because SHA1 repacks take a bit longer to finish up and currently the builder is not chained as a dependency to uptake monitoring builder, the latter one started and timed-out for 5 retries in a row, eventually giving up as failed.
I'm thinking of three possible solutions here:
1. Chain up just the sha1 repacks builder as a dependency to uptake monitoring builder.
2. Chain the entire release-{}-{}_partner_repacks_copy_to_releases as a dependency to uptake monitoring builder to cope with existing uptake monitoring dependencies (push-to-releases)
3. Bump the uptake monitoring sleep/pause times to cover a larger time window to allow sha1 repacks to finish.
From https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-beta-noarch/release-mozilla-beta-firefox_uptake_monitoring-bm70-build1-build2.txt.gz
00:39:32 INFO - Current uptake for Firefox-49.0b9-sha1 is 0
00:39:32 INFO - Requesting https://bounceradmin.mozilla.com/api/uptake/?product=Firefox-49.0b9-stub&os=win from tuxedo
00:39:32 INFO - Starting new HTTPS connection (1): bounceradmin.mozilla.com
/builds/slave/rel-m-beta-fx_uptk_mntr-000000/build/venv/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
Assignee | ||
Comment 1•8 years ago
|
||
(In reply to Mihai Tabara [:mtabara] from comment #0)
> 1. Chain up just the sha1 repacks builder as a dependency to uptake
> monitoring builder.
My main concern here is that we don't want to be hard blocked by partner repacks. They may fail for some reason, but this should not be a blocker, esp for betas.
> 2. Chain the entire release-{}-{}_partner_repacks_copy_to_releases as a
> dependency to uptake monitoring builder to cope with existing uptake
> monitoring dependencies (push-to-releases)
1) without 2) doesn't make too much sense, because we have to wait for 2) to make uptake monitoring work. The same concerns apply here.
> 3. Bump the uptake monitoring sleep/pause times to cover a larger time
> window to allow sha1 repacks to finish.
I think 2) is better, because you still have to wait for it to make this work.
I have 2 more options:
4) Separate the sha1 repacks graph branch from partner repacks, create 3rd push to mirrors for win32-sha1, create separate uptake monitoring for this, create a separate postrelease task to update bouncer aliases and make them depend on each other (and postrelease on the main publish release task). Adds a lot complexity though.
5) Almost the same as 4), but without separate uptake monitoring - the main uptake monitoring should depend on the main push to mirrors and sha1 push to mirrors (or we can merge pushes together)
2 or 5, pick one! :)
Reporter | ||
Comment 2•8 years ago
|
||
Ah, right.
1) is non-sense since the push-partner-repacks-to-releases is needed anyway before talking about any uptake monitoring
2) is indeed the easiest option to be altered but as you said, adds some heavyburden-blocking on the general work-flow of release that is not needed
3) yeah - that was my main thinking as well. Uptake monitoring doesn't even have to start and keep timing-out in a loop until we have that partner_to_push_to_releases ready anyway.
At second thoughts, I don't like any of these.
I'll go with 5 I guess, seems the cleanest way. Unless there is way to tweak 2 to avoid the "blocking" part?
Reporter | ||
Comment 3•8 years ago
|
||
Will resume to my WIP this week as we're yet again at the beginning of a beta-cycle and we can test the things out.
Reporter | ||
Comment 4•8 years ago
|
||
Bleah, I need to address this soon, 50.0b2 brings this up yet again https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-beta-noarch/release-mozilla-beta-firefox_uptake_monitoring-bm94-build1-build5.txt.gz
Most likely I'll run uptake monitoring manually or rerun once sha1 is done.
Assignee | ||
Comment 6•8 years ago
|
||
Attachment #8799925 -
Flags: review?(mtabara)
Reporter | ||
Comment 7•8 years ago
|
||
Comment on attachment 8799925 [details]
PR
Somehow I imagined this would be more complicated, given https://bugzilla.mozilla.org/show_bug.cgi?id=1300060#c2 but other than that, looks awesome, r+.
Apologies for procrastinating and thanks for taking care of it.
Attachment #8799925 -
Flags: review?(mtabara) → review+
Assignee | ||
Comment 8•8 years ago
|
||
Comment on attachment 8799925 [details]
PR
deployed
Attachment #8799925 -
Flags: checked-in+
Assignee | ||
Updated•8 years ago
|
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Reporter | ||
Comment 9•8 years ago
|
||
Uptake monitoring worked like a charm this time in 50.0b7 => https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-beta-noarch/release-mozilla-beta-firefox_uptake_monitoring-bm70-build1-build9.txt.gz
...
09:26:55 <release-notifications>w8_4uEGvQbe4y85S81Z0KA: [beetmover] firefox mozilla-beta partner repacks push to releases has completed successfully! Yay!
09:26:55 <release-notifications>https://mozilla-release-logs.s3.amazonaws.com/mozilla-beta/firefox-50.0b7/build1/[beetmover]_firefox_mozilla-beta_partner_repacks_push_to_releases-all-w8_4uEGvQbe4y85S81Z0KA-0
09:27:57 <release-notifications> bQTOqnaNQdufcyO6z5bptA: firefox mozilla-beta uptake monitoring has completed successfully! Yay!
release-notifications> https://mozilla-release-logs.s3.amazonaws.com/mozilla-beta/firefox-50.0b7/build1/firefox_mozilla-beta_uptake_monitoring-all-bQTOqnaNQdufcyO6z5bptA-0
...
You need to log in
before you can comment on or make changes to this bug.
Description
•