Closed Bug 1300060 Opened 8 years ago Closed 8 years ago

SHA1 repacks break uptake monitoring by timing out in release promotion

Categories

(Release Engineering :: Release Automation: Other, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: mtabara, Assigned: rail)

References

Details

Attachments

(1 file)

(deleted), text/x-github-pull-request
mtabara
: review+
Details
In https://bugzilla.mozilla.org/show_bug.cgi?id=1290179#c12 we enabled SHA1 repacks and added them to the stuff we check as part of the uptake monitoring step in release promotion. Because SHA1 repacks take a bit longer to finish up and currently the builder is not chained as a dependency to uptake monitoring builder, the latter one started and timed-out for 5 retries in a row, eventually giving up as failed. I'm thinking of three possible solutions here: 1. Chain up just the sha1 repacks builder as a dependency to uptake monitoring builder. 2. Chain the entire release-{}-{}_partner_repacks_copy_to_releases as a dependency to uptake monitoring builder to cope with existing uptake monitoring dependencies (push-to-releases) 3. Bump the uptake monitoring sleep/pause times to cover a larger time window to allow sha1 repacks to finish. From https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-beta-noarch/release-mozilla-beta-firefox_uptake_monitoring-bm70-build1-build2.txt.gz 00:39:32 INFO - Current uptake for Firefox-49.0b9-sha1 is 0 00:39:32 INFO - Requesting https://bounceradmin.mozilla.com/api/uptake/?product=Firefox-49.0b9-stub&os=win from tuxedo 00:39:32 INFO - Starting new HTTPS connection (1): bounceradmin.mozilla.com /builds/slave/rel-m-beta-fx_uptk_mntr-000000/build/venv/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
(In reply to Mihai Tabara [:mtabara] from comment #0) > 1. Chain up just the sha1 repacks builder as a dependency to uptake > monitoring builder. My main concern here is that we don't want to be hard blocked by partner repacks. They may fail for some reason, but this should not be a blocker, esp for betas. > 2. Chain the entire release-{}-{}_partner_repacks_copy_to_releases as a > dependency to uptake monitoring builder to cope with existing uptake > monitoring dependencies (push-to-releases) 1) without 2) doesn't make too much sense, because we have to wait for 2) to make uptake monitoring work. The same concerns apply here. > 3. Bump the uptake monitoring sleep/pause times to cover a larger time > window to allow sha1 repacks to finish. I think 2) is better, because you still have to wait for it to make this work. I have 2 more options: 4) Separate the sha1 repacks graph branch from partner repacks, create 3rd push to mirrors for win32-sha1, create separate uptake monitoring for this, create a separate postrelease task to update bouncer aliases and make them depend on each other (and postrelease on the main publish release task). Adds a lot complexity though. 5) Almost the same as 4), but without separate uptake monitoring - the main uptake monitoring should depend on the main push to mirrors and sha1 push to mirrors (or we can merge pushes together) 2 or 5, pick one! :)
Ah, right. 1) is non-sense since the push-partner-repacks-to-releases is needed anyway before talking about any uptake monitoring 2) is indeed the easiest option to be altered but as you said, adds some heavyburden-blocking on the general work-flow of release that is not needed 3) yeah - that was my main thinking as well. Uptake monitoring doesn't even have to start and keep timing-out in a loop until we have that partner_to_push_to_releases ready anyway. At second thoughts, I don't like any of these. I'll go with 5 I guess, seems the cleanest way. Unless there is way to tweak 2 to avoid the "blocking" part?
Will resume to my WIP this week as we're yet again at the beginning of a beta-cycle and we can test the things out.
Bleah, I need to address this soon, 50.0b2 brings this up yet again https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-beta-noarch/release-mozilla-beta-firefox_uptake_monitoring-bm94-build1-build5.txt.gz Most likely I'll run uptake monitoring manually or rerun once sha1 is done.
Sorry, hijacking! :)
Assignee: mtabara → rail
Attached file PR (deleted) —
Attachment #8799925 - Flags: review?(mtabara)
Comment on attachment 8799925 [details] PR Somehow I imagined this would be more complicated, given https://bugzilla.mozilla.org/show_bug.cgi?id=1300060#c2 but other than that, looks awesome, r+. Apologies for procrastinating and thanks for taking care of it.
Attachment #8799925 - Flags: review?(mtabara) → review+
Comment on attachment 8799925 [details] PR deployed
Attachment #8799925 - Flags: checked-in+
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Uptake monitoring worked like a charm this time in 50.0b7 => https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-beta-noarch/release-mozilla-beta-firefox_uptake_monitoring-bm70-build1-build9.txt.gz ... 09:26:55 <release-notifications>w8_4uEGvQbe4y85S81Z0KA: [beetmover] firefox mozilla-beta partner repacks push to releases has completed successfully! Yay! 09:26:55 <release-notifications>https://mozilla-release-logs.s3.amazonaws.com/mozilla-beta/firefox-50.0b7/build1/[beetmover]_firefox_mozilla-beta_partner_repacks_push_to_releases-all-w8_4uEGvQbe4y85S81Z0KA-0 09:27:57 <release-notifications> bQTOqnaNQdufcyO6z5bptA: firefox mozilla-beta uptake monitoring has completed successfully! Yay! release-notifications> https://mozilla-release-logs.s3.amazonaws.com/mozilla-beta/firefox-50.0b7/build1/firefox_mozilla-beta_uptake_monitoring-all-bQTOqnaNQdufcyO6z5bptA-0 ...
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: