See here for more details https://github.com/mozilla/releasewarrior/blob/master/how-tos/setup-staging-release.md#misc

It would be best if we could do update verify tests from old real releases to a new staging release. One big problem with this is the updater (which comes from the real release) expects the mar file to be signed with the release cert, and we've obviously been reluctant to use that in staging. One suggestion is to compile the updater twice in each release build - once with the release cert, and once with the dep cert. The dep updater would be stashed somewhere for use in staging releases. Another option might be to unpack the real release and modify the updater on the fly, swapping out the release cert from https://hg.mozilla.org/mozilla-central/file/default/toolkit/mozapps/update/updater/release_primary.der with dep1.der. It's easy enough to read the release cert into python and locate it in the updater binary, but some padding or other care might be needed because the dep cert is 10 bytes shorter. There's a related issue with the mar-channel-id stored in the mar file, but update verify already has support to modify update-settings.ini before calling the updater.

I talked to Nick yesterday and he said he would look into this

I looked at the current state of jamun: * the most recent staging release (58.0b1 build17) had trouble submitting data to a custom staging Balrog, so we didn't try to run update verify. There's work going on in bug 1398236 to use the genuine stage Balrog * the last time we ran update verify (https://mzl.la/2gdL6on) there were tag issues, which seem to be resolved now * in the tools repo the configs use ftp_server_to="https://ftp.stage.mozaws.net/pub", but the ssl cert has expired. Bug 1408797 to fix that up * bug 1407321 is changing which tools repo we bump in the updates task But most importantly for this bug * we're using dep signing on all desktop platforms (via in-tree scheduling and tc jobs); the buildbot config in mozilla/project_branches.py setting dep_signing_servers is a red herring * a real release updater (from a previous release) isn't going to like the a dep-signed mar file * confirmed that by doing a 56.0b4 (real) -> 58.0b1 build17 (staging) update test, and getting "ERROR: Error verifying signature." * which leads us back to comment #1, I will talk with rstrong about those options and his ideas

Johan - Aki spotted that you originally landed this on jamun in early August as part of https://hg.mozilla.org/projects/jamun/rev/ec6531e7818a, but it was reverted by Rail in mid-September in https://hg.mozilla.org/projects/jamun/rev/ee42b4e2a575. Rail didn't remember why that was done like that, my guess is that it was resetting jamun to known beta code. We still have release signing in the release config at https://hg.mozilla.org/build/buildbot-configs/file/default/mozilla/project_branches.py#l352, so all tasks in the releasetasks graph like l10n and partial updates have release signing, just not the en-US builds. If we take this patch, and change the partials for the next staging release to be after all the watersheds (eg 56.0b9 or later) then we're a bit closer to having staging working. Just need to make sure we don't submit to prod beetmover buckets or balrog, which I think is already in place.

Comment on attachment 8919629 [details] [diff] [review] [gecko] Restore release signing for en-US builds Review of attachment 8919629 [details] [diff] [review]: ----------------------------------------------------------------- That patch rings a bell :) As jamun isn't listed in [1], we should be safe from uploading to the prod buckets. [1] https://hg.mozilla.org/projects/jamun/file/189b15d8e3f2/taskcluster/taskgraph/util/scriptworker.py#l72

Comment on attachment 8919629 [details] [diff] [review] Landed on jamun at https://hg.mozilla.org/projects/jamun/rev/c5d33b6948c6cea3b5091b3ea0221141fdadec70. Staging release "Firefox-58.0b1-build20" triggered. It will start once CI builds are finished.

Thanks Johan. I've started and email thread with rstrong & mhowell for the longer term strategy of using dep signing for staging releases.

I've started 58.0b1 build21 in staging with the signing fix, and all the other balrog goodness that everyone else has been landing. Using https://public.etherpad-mozilla.org/p/staging-releases-jamun to keep track of it.

The status of build21 was almost working cleanly. 1, There was problem starting the release. To use recent partials of '57.0b9build1,57.0b8build3' I needed to import those releases from the prod ship-it to dev ship-it, because release sanity wanted to look up the l10n changesets. Probably https://dxr.mozilla.org/build-central/source/tools/lib/python/kickoff/api.py#116 being used at https://dxr.mozilla.org/build-central/source/tools/lib/python/kickoff/__init__.py#138 Need a solution here that avoid manual db changes. 2, All proceeded without issue until update testing. a) Failures on mac and windows because no partials were offered by Balrog. Turns out the Firefox-57.0b8-build3 and Firefox-57.0b9-build1 releases were no in Balrog stage, very likely because of bug 1408800. Ben is going to fix this up. b) Failures on Linux because it's querying on the beta-cdntest channel while the other platforms are using beta-localtest. As we have disabled pushing to releases, localtest is the correct choice. Easy patch for that coming up. I did a test run on beta-localtest for one linux u.v. and it was green, see https://tools.taskcluster.net/groups/W-jEIrIMRPWwjjWpREDLqg.

This parameter is passed all the way through releaserunner to releasetasks and used at https://github.com/mozilla-releng/releasetasks/blob/master/releasetasks/templates/desktop/tc_update_verify.yml.tmpl#L52. The buildbot jobs use the channel specified at https://hg.mozilla.org/projects/jamun/file/c2a953e7113a/testing/mozharness/configs/releases/dev_updates_firefox_beta.py#l31

Comment on attachment 8920517 [details] [diff] [review] [buildbot-configs] Use localtest channels for linux32/linux64 Makes sense to me!

Comment on attachment 8920517 [details] [diff] [review] [buildbot-configs] Use localtest channels for linux32/linux64 https://hg.mozilla.org/build/buildbot-configs/rev/110b0f82c2c9d6ad52962227ea1ce1a1f7a520aa

Comment on attachment 8920517 [details] [diff] [review] [buildbot-configs] Use localtest channels for linux32/linux64 Turns out this modified maple instead of jamun, so I landed https://hg.mozilla.org/build/buildbot-configs/rev/a652901b318eac6c4f5b0924a16802efdc4a8861 to actually change jamun. Since they both set push_to_releases_automatic to False I left this original patch landed.