first for testing purposes, then for real once QE sign off.

context: POA: catlee> can we create SSL-only bouncer aliases for 58.0bX devedition to check if we can restore updates for users on 58.0b1? ... nthomas> can probably update fileUrls for special cases, and use a different bouncer product for those entries 22:09:08 <•catlee> do we even need to use bouncer I wonder? 22:09:15 if we're going to be updating balrog 22:10:06 <•nthomas> hmm, interesting idea testing so far: 15:05:14 bogdan_maris: could you check if 58.0b1 still fails to update when you get a chance? <bogdan_maris> catlee: Sure thing 15:40:45 catlee: still fails if I test on aurora-cdntest 15:41:41 catlee: aurora channel as well nick is mocking something up. Testing a patch similar to: https://diff.pastebin.mozilla.org/9073277

Should we use SSL by default? It's just toggling "ssl-only" to True everywhere in bouncer configs, https://dxr.mozilla.org/mozilla-central/rev/b056526be38e96b3e381b7e90cd8254ad1d96d9d/testing/mozharness/configs/releases/bouncer_firefox_release.py#9 for example.

(In reply to Rail Aliiev [:rail] ⌚️ET from comment #2) > Should we use SSL by default? It's just toggling "ssl-only" to True > everywhere in bouncer configs, > https://dxr.mozilla.org/mozilla-central/rev/ > b056526be38e96b3e381b7e90cd8254ad1d96d9d/testing/mozharness/configs/releases/ > bouncer_firefox_release.py#9 for example. I think we should...but not just yet :)

AIUI cert pinning is enabled on SSL downloads and we intentionally chose not to use cert pinning on app update per bug 1063111 (please read the bug for the rationale). catlee / jlund, will these SSL downloads be cert pinned?

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #4) > AIUI cert pinning is enabled on SSL downloads ... Sounds like you are referring to cdn.mozilla.net appearing in https://hg.mozilla.org/releases/mozilla-beta/file/default/security/manager/ssl/StaticHPKPins.h#l674, which presumably includes our SSL CDN download-installer.cdn.mozilla.net. Should we separate the one-off rescue of 58.0b1 users from the broader question of serving updates over SSL ?

I've added a Devedition-58.0b5-build1-SSL release to Balrog, which has a modified fileUrls block to download mar files from https://archive.mozilla.org. We can't use download-installer.cdn.mozilla.net without a change to the whitelist used by Balrog. Also added rules 687 and 688 to use that release for requests coming from DevEd 58.0b1 (buildID 20171103003834). Was able to update 58.0b1 mac en-US on aurora-cdntest OK. Bogdan, could you please test aurora-cdntest to see if it fixes up the issues you've been seeing with 58.0b1.

Rerunning the automated update verify as well, although it's using wget for the downloads.

(In reply to Nick Thomas [:nthomas] (UTC+13) from comment #6) > I've added a Devedition-58.0b5-build1-SSL release to Balrog, which has a > modified fileUrls block to download mar files from > https://archive.mozilla.org. We can't use download-installer.cdn.mozilla.net > without a change to the whitelist used by Balrog. > > Also added rules 687 and 688 to use that release for requests coming from > DevEd 58.0b1 (buildID 20171103003834). Was able to update 58.0b1 mac en-US > on aurora-cdntest OK. > > Bogdan, could you please test aurora-cdntest to see if it fixes up the > issues you've been seeing with 58.0b1. Retested across platforms (Windows 10 64bit, macOS 10.13 and Ubuntu 16.04 32bit) and DevEd 58.0b1 now updates to 58.0b5 without issues.

Great! I've added a Balrog scheduled change to do this on the aurora channel too, to go with the main change for 58.0b5.

Updates are live.

We're hitting some update verify failures for 58.0b6 because of the special rule.

The changes to the u.v. configs are to fix for 58.0b6 testing, while the patcher config is for subsequent releases. As we discussed, we'll have a static 58.0b1 -> 58.0b5 SSL path for the 58 cycle, then likely remove that in 59 when users have had a chance to update. We're kinda flying blind since we don't know how many users are affected by the download issue QE saw.

Comment on attachment 8931511 [details] [diff] [review] [tools] Fix update verify https://hg.mozilla.org/build/tools/rev/017ead33eb4260f2de190c36391e60f12d54e425 https://hg.mozilla.org/build/tools/rev/c43e705ae16463f39e9216de73749fee9b453775 Will rerun the failing u.v. jobs.

Green!