This bug is to track effort to obtain a staging release of Devedition, based on jamun.

So far: * sanity checks blocks me from moving forward. First the long revision, then absence of partials and finally the l10n changests. I think there's a miscommunication somewhere in the configs for this. The l10n that ship-it pulls automatically are mozilla-beta based (since jamun is mozilla-beta based) while the build itself is based on `projects/jamun`. For this reason, some (~20) locales are missing from https://hg.mozilla.org/releases/l10n/mozilla-aurora/<locale>/raw-file/<revision>. Need to either update this l10n changeset to remove them or find the place to tweak the configs to avoid hitting this from the first place. * I also needed to push a fix[3] in tools under bug 1358607 for a syntax errors. I'm surprised the automatc tests didn't catch that from the first place Notes to self: * file bugs to cleanup old filtering system[1] as its being default now. All filtering logic moved to release-runner.yml[2] under puppet. * once we have this bug fixed, we need to tweak [2] to encompass Devedition too [1]: https://hg.mozilla.org/build/tools/file/tip/lib/python/kickoff/__init__.py#l19 [2]: https://hg.mozilla.org/build/puppet/file/tip/modules/releaserunner/templates/release-runner.yml.erb#l50 [3]: https://hg.mozilla.org/build/tools/rev/314a14daf2f1

:rail - I think I need a bit of help. Not sure where to look next to fix the l10n changeset issue.

The issue is that you added multiple partial versions, but they don't exist in ship-it. release runner tries to fetch the l10n changesets used in the previous versions and fails to find them. I created Devedition-54.0b6-build1 without any partials and used it in b7 created by you. This should help

Taking it for today

pushed https://hg.mozilla.org/projects/jamun/rev/4c2d260af74e6dbd0b0f9e406083868d335d914e to address the l10n 404s

Comment on attachment 8867241 [details] Bug 1364225 - Devedition staging release based on jamun https://reviewboard.mozilla.org/r/138798/#review142110

Comment on attachment 8867241 [details] Bug 1364225 - Devedition staging release based on jamun https://hg.mozilla.org/build/buildbot-configs/rev/b648e9ff9bf04e5f7ddfec3a30f76b2c3fec7ece

Comment on attachment 8867270 [details] Bug 1364225 - Devedition staging release based on jamun https://reviewboard.mozilla.org/r/138806/#review142154

configs incoming

Comment on attachment 8867270 [details] Bug 1364225 - Devedition staging release based on jamun https://hg.mozilla.org/build/tools/rev/14fad99b7a3524ea944b88805c69df159a910738

Comment on attachment 8867301 [details] [diff] [review] devedtion-jamun-configs.diff Review of attachment 8867301 [details] [diff] [review]: ----------------------------------------------------------------- ::: mozilla/config.py @@ +2456,4 @@ > BRANCHES['mozilla-beta']['beetmover_buckets'] = { > "firefox": "net-mozaws-prod-delivery-firefox", > "fennec": "net-mozaws-prod-delivery-archive", > + "devedition": "net-mozaws-prod-delivery-firefox", Is this the right bucket? I guess it is, I just don't fully understand how products map to buckets... ::: mozilla/project_branches.py @@ +177,4 @@ > 'android-x86': {}, > }, > 'pgo_strategy': 'per-checkin', > 'enable_release_promotion': True, I think you need to update beetmover_buckets in this file too, it currently only has "firefox". @@ +188,5 @@ > + }, > + 'uptake_monitoring_platforms': { > + "firefox": ("linux", "linux64", "win32", "win64", "macosx64"), > + "fennec": ("android-api-15", "android-x86"), > + "devedition": ("linux", "linux64", "win32", "win64", "macosx64"), Shouldn't these be -devedition?

(In reply to Ben Hearsum (:bhearsum) from comment #14) > Comment on attachment 8867301 [details] [diff] [review] > devedtion-jamun-configs.diff > > Review of attachment 8867301 [details] [diff] [review]: > ----------------------------------------------------------------- > > ::: mozilla/config.py > @@ +2456,4 @@ > > BRANCHES['mozilla-beta']['beetmover_buckets'] = { > > "firefox": "net-mozaws-prod-delivery-firefox", > > "fennec": "net-mozaws-prod-delivery-archive", > > + "devedition": "net-mozaws-prod-delivery-firefox", > > Is this the right bucket? I guess it is, I just don't fully understand how > products map to buckets... I'm not 100% sure, but it should be. :) > > ::: mozilla/project_branches.py > @@ +177,4 @@ > > 'android-x86': {}, > > }, > > 'pgo_strategy': 'per-checkin', > > 'enable_release_promotion': True, > > I think you need to update beetmover_buckets in this file too, it currently > only has "firefox". Good point! > > @@ +188,5 @@ > > + }, > > + 'uptake_monitoring_platforms': { > > + "firefox": ("linux", "linux64", "win32", "win64", "macosx64"), > > + "fennec": ("android-api-15", "android-x86"), > > + "devedition": ("linux", "linux64", "win32", "win64", "macosx64"), > > Shouldn't these be -devedition? This reminded me more things to change. I'll rework the patch.

(In reply to Rail Aliiev [:rail] ⌚️ET from comment #15) > (In reply to Ben Hearsum (:bhearsum) from comment #14) > > Comment on attachment 8867301 [details] [diff] [review] > > devedtion-jamun-configs.diff > > > > Review of attachment 8867301 [details] [diff] [review]: > > ----------------------------------------------------------------- > > > > ::: mozilla/config.py > > @@ +2456,4 @@ > > > BRANCHES['mozilla-beta']['beetmover_buckets'] = { > > > "firefox": "net-mozaws-prod-delivery-firefox", > > > "fennec": "net-mozaws-prod-delivery-archive", > > > + "devedition": "net-mozaws-prod-delivery-firefox", > > > > Is this the right bucket? I guess it is, I just don't fully understand how > > products map to buckets... > > I'm not 100% sure, but it should be. :) Few notes here as I think this might be trickier than we think: So my initial thoughts were ... * `net-mozaws-prod-delivery-firefox` is the S3 bucket that backfills archive.mozilla.org/pub/firefox with all its directories (e.g. candidates, releases, nightly, etc). We have different credentials to push to ~releases vs ~nightlies. The counterpart `net-mozaws-prod-delivery-archive` applies the same for Fennec. * Devedition assumingly should go under /pub/firefox/{candidates,releases}/$version but since it shares the same versioning system with betas, destinations will overlap * old beetmover (the Mozharness one) can't push artifacts if they already exist and will result in error should that happen However ... I just noticed the existence of http://archive.mozilla.org/pub/devedition/ (`tinderbox-builds` being the only subdirectory) which means some credentials do have access to that bucket subdirectory. My money is on the `net-mozaws-prod-delivery-firefox` credentials just because I don't see another set of credentials under puppet[1] but we'll need to double check in the automation anyway. To conclude, I'd say since we're here, we could/should: * double check which creds push to ~/pub/devedition * double check with ClousOps the subdirs permissions these creds have * attempt to either: ** separate to a potentially third devedition bucket named `net-mozaws-prod-delivery-devedition` (probably the cleanest way) ** have CloudOps provide us a separate set of credentials that can only push to ~/pub/devedion but keep the same bucket @rail: what do you think? Also, worth mentioning: * we also need to update beetmover script to grab that `product` config via arguments[2] rather than hardcoded class dict[3] * beetmover paths were nicely done in a generic way per product (rail++) so we should be covered from this perspective. But this[4] aso reinforces that we need the aforementioned bucket separation between devedition and firefox [1]: https://dxr.mozilla.org/build-central/source/puppet/manifests/moco-config.pp [2]: https://hg.mozilla.org/releases/mozilla-beta/file/tip/testing/mozharness/scripts/release/beet_mover.py#l36 [3]: https://hg.mozilla.org/releases/mozilla-beta/file/tip/testing/mozharness/scripts/release/beet_mover.py#l154 [4]: https://hg.mozilla.org/releases/mozilla-beta/file/tip/testing/mozharness/scripts/release/beet_mover.py#l203

The smallest one :)

Nothing impressive

comments incoming

Comment on attachment 8867986 [details] [diff] [review] bb-configs.diff Review of attachment 8867986 [details] [diff] [review]: ----------------------------------------------------------------- The patch is mostly about moving some variables from BRANCH[branch][var] to BRANCH[branch][var][product]. This should let us start a staging release and concentrate on the run-time changes (in-tree configs and patcher configs). ::: mozilla/config.py @@ +2411,5 @@ > # used by process/release.py > BRANCHES['mozilla-beta']['tuxedoServerUrl'] = "https://bounceradmin.mozilla.com/api" > BRANCHES['mozilla-beta']['bouncer_submitter_config'] = { > "firefox": "releases/bouncer_firefox_beta.py", > + "devedition": "releases/bouncer_firefox_devedition.py", These are in-tree config files. I'll attach them after we prove them working on jamun. @@ +2478,4 @@ > BRANCHES['mozilla-beta']['beetmover_buckets'] = { > "firefox": "net-mozaws-prod-delivery-firefox", > "fennec": "net-mozaws-prod-delivery-archive", > + "devedition": "net-mozaws-prod-delivery-firefox", Still to be verified. @@ +2482,5 @@ > } > BRANCHES['mozilla-beta']['uptake_monitoring_platforms'] = { > "firefox": ("linux", "linux64", "win32", "win64", "macosx64"), > "fennec": ("android-api-15", "android-x86"), > + "devedition": ("linux", "linux64", "win32", "win64", "macosx64"), The platforms listed here are (almost) nothing to do with the platform names we used in buildbot. We can reuse the same platform names here to avoid extra conversion them to bouncer platforms. @@ +2557,5 @@ > + "devedition": { > + "linux": { > + "signed": "gecko.v2.mozilla-beta.signed-nightly.revision.{rev}.devedition-l10n.linux-opt.en-US", > + "unsigned": "gecko.v2.mozilla-beta.revision.{rev}.devedition-l10n.linux-opt.en-US", > + }, Similar here. I didn't use linux-devedition buildbot platform, because we shouldn't use anything from that config.py dictionary in release promotion. Let's stick with the regular platform names to avoid various functions to map these platforms to some other platforms (ftp, bouncer, etc).

Comment on attachment 8867984 [details] [diff] [review] bbcustom.diff Review of attachment 8867984 [details] [diff] [review]: ----------------------------------------------------------------- ::: common.py @@ +109,5 @@ > 'graphics': 'gfx', > 'firefox_tag_source': 'fx_tag_src', > 'firefox_tag_l10n': 'fx_tag_l10n', > 'firefox': 'fx', > + 'devedition': 'dev', Out of my experience, names including the product are already almost 30chars long and we'll need to use only two letters here (see `firefox` and `fennec`). I would also note that "dev" might be a bit confusing as it somehow misleads to thinking about staging environment or something alike. However, not sure though which are better alternatives "dv", "ed", "de" ?

Comment on attachment 8867986 [details] [diff] [review] bb-configs.diff Review of attachment 8867986 [details] [diff] [review]: ----------------------------------------------------------------- ::: mozilla/builder_master.cfg @@ +183,5 @@ > + branch_config=BRANCHES[branch], branch_name=branch, > + product="firefox", > + secrets=getattr(passwords, 'secrets', None) > + )) > + if BRANCHES[branch].get('enable_release_promotion', {}).get("devedition"): Nitpicking: /s/if/elif to avoid useless checks? @@ +189,5 @@ > + branch_config=BRANCHES[branch], branch_name=branch, > + product="devedition", > + secrets=getattr(passwords, 'secrets', None) > + )) > + if BRANCHES[branch].get('enable_release_promotion', {}).get("fennec"): Nitpicking: /s/if/elif as above ::: mozilla/config.py @@ +2232,5 @@ > BRANCHES['mozilla-release']['enable_nightly'] = False > BRANCHES['mozilla-release']['enable_blocklist_update'] = True > BRANCHES['mozilla-release']['enabled_products'] = ['firefox', 'mobile'] > ### Release Promotion > +BRANCHES['mozilla-release']['enable_release_promotion'] = { sweet! @@ +2277,5 @@ > BRANCHES['mozilla-release']['platforms']['win32']['dep_signing_servers'] = 'release-signing' > BRANCHES['mozilla-release']['platforms']['win64']['dep_signing_servers'] = 'release-signing' > # used by releasetasks > BRANCHES['mozilla-release']['bouncer_enabled'] = True > +BRANCHES['mozilla-release']['postrelease_version_bump_enabled'] = { Hm, question: why only this and not the others too? (bouncer, shipit, etc) @@ +2279,5 @@ > # used by releasetasks > BRANCHES['mozilla-release']['bouncer_enabled'] = True > +BRANCHES['mozilla-release']['postrelease_version_bump_enabled'] = { > + "firefox": True, > + "fennec": True, This doesn't hurt, for now. But it's supposed to be set to False. As part of bug 1347635 all these will be turned to True. @@ +2423,5 @@ > BRANCHES['mozilla-beta']['postrelease_version_bump_config'] = { > "firefox": 'releases/postrelease_firefox_beta.py', > # configs are generic so can be reused > "fennec": 'releases/postrelease_firefox_beta.py', > + "devedition": "disabled", Reminder we need to alter this before going live. The version bump script or the BB custom don't know to handle this value and will most likely result in IOError for not finding that path. @@ +2463,5 @@ > BRANCHES['mozilla-beta']['update_verify_enabled'] = True > +BRANCHES['mozilla-beta']['postrelease_version_bump_enabled'] = { > + "firefox": True, > + "devedition": False, > + "fennec": True, This doesn't hurt, for now. But it's supposed to be set to False. As part of bug 1347635 all these will be turned to True. @@ +2478,4 @@ > BRANCHES['mozilla-beta']['beetmover_buckets'] = { > "firefox": "net-mozaws-prod-delivery-firefox", > "fennec": "net-mozaws-prod-delivery-archive", > + "devedition": "net-mozaws-prod-delivery-firefox", Where in release automation I should look to understand who's pushing the artifacts to http://archive.mozilla.org/pub/devedition/tinderbox-builds/ ? ::: mozilla/universal_master_sqlite.cfg @@ +171,5 @@ > + branch_config=BRANCHES[branch], branch_name=branch, > + product="firefox", > + secrets=getattr(passwords, 'secrets', None) > + )) > + if BRANCHES[branch].get('enable_release_promotion', {}).get("devedition"): Nitpicking: /s/if/elif as in builder_master.cfg @@ +177,5 @@ > + branch_config=BRANCHES[branch], branch_name=branch, > + product="devedition", > + secrets=getattr(passwords, 'secrets', None) > + )) > + if BRANCHES[branch].get('enable_release_promotion', {}).get("fennec"): Nitpicking: /s/if/elif as in builder_master.cfg

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #21) > > + 'devedition': 'dev', > > Out of my experience, names including the product are already almost 30chars > long and we'll need to use only two letters here (see `firefox` and > `fennec`). I would also note that "dev" might be a bit confusing as it > somehow misleads to thinking about staging environment or something alike. > However, not sure though which are better alternatives "dv", "ed", "de" ? Sure, "de" it is! :)

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #22) > Nitpicking: /s/if/elif to avoid useless checks? I think we want to check all of the product, so we are able to enable multiple products per branch. > > > +BRANCHES['mozilla-release']['postrelease_version_bump_enabled'] = { > > Hm, question: why only this and not the others too? (bouncer, shipit, etc) Probably I should comment it. The idea is that devedtion tracks firefox bit to bit (fennec doesn't). So I thought that there is no need to waste compute time for noops. > @@ +2279,5 @@ > > # used by releasetasks > > BRANCHES['mozilla-release']['bouncer_enabled'] = True > > +BRANCHES['mozilla-release']['postrelease_version_bump_enabled'] = { > > + "firefox": True, > > + "fennec": True, > > This doesn't hurt, for now. But it's supposed to be set to False. As part of > bug 1347635 all these will be turned to True. AFAIK we bump the in-tree version in fennec, even thought it doesn't go through release runner. I just wanted to reflect it the reality, so we don't forget to flip the flag when we integrate them. > > @@ +2423,5 @@ > > BRANCHES['mozilla-beta']['postrelease_version_bump_config'] = { > > "firefox": 'releases/postrelease_firefox_beta.py', > > # configs are generic so can be reused > > "fennec": 'releases/postrelease_firefox_beta.py', > > + "devedition": "disabled", > > Reminder we need to alter this before going live. The version bump script or > the BB custom don't know to handle this value and will most likely result in > IOError for not finding that path. This is just to prevent KeyErrors down the path. This value shouldn't be used anywhere, because the step is disabled for devedition. > @@ +2478,4 @@ > > BRANCHES['mozilla-beta']['beetmover_buckets'] = { > > "firefox": "net-mozaws-prod-delivery-firefox", > > "fennec": "net-mozaws-prod-delivery-archive", > > + "devedition": "net-mozaws-prod-delivery-firefox", > > Where in release automation I should look to understand who's pushing the > artifacts to http://archive.mozilla.org/pub/devedition/tinderbox-builds/ ? Those are our regular CI builds. I think it's https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla/config.py#24

Comment on attachment 8867984 [details] [diff] [review] bbcustom.diff https://hg.mozilla.org/build/buildbotcustom/rev/32ecfc7dc1122304f133b43d19a84e481bd59434

Comment on attachment 8867986 [details] [diff] [review] bb-configs.diff https://hg.mozilla.org/build/buildbot-configs/rev/91ab94840f5d2218dfaba3b0c0456eaf18e099be

(In reply to Rail Aliiev [:rail] ⌚️ET from comment #24) > (In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #22) > > Nitpicking: /s/if/elif to avoid useless checks? > > I think we want to check all of the product, so we are able to enable > multiple products per branch. > Doh, silly me, that's right! > > > > > > +BRANCHES['mozilla-release']['postrelease_version_bump_enabled'] = { > > > > Hm, question: why only this and not the others too? (bouncer, shipit, etc) > > Probably I should comment it. The idea is that devedtion tracks firefox bit > to bit (fennec doesn't). So I thought that there is no need to waste compute > time for noops. Makes sense. But in the end, we would still need those builders enabled? I mean, some devedition builder will have to push the bouncer entries for example, or mark the release as shipped, ain't they? > > > @@ +2279,5 @@ > > > # used by releasetasks > > > BRANCHES['mozilla-release']['bouncer_enabled'] = True > > > +BRANCHES['mozilla-release']['postrelease_version_bump_enabled'] = { > > > + "firefox": True, > > > + "fennec": True, > > > > This doesn't hurt, for now. But it's supposed to be set to False. As part of > > bug 1347635 all these will be turned to True. > > AFAIK we bump the in-tree version in fennec, even thought it doesn't go > through release runner. I just wanted to reflect it the reality, so we don't > forget to flip the flag when we integrate them. Makes sense. However, no variables from http://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla/config.py are used for Fennec at the moment as we run that "release-runner" part via https://hg.mozilla.org/build/tools/file/tip/buildfarm/release/releasetasks_graph_gen.py + https://github.com/mozilla/releasetasks/blob/master/releasetasks/release_configs/prod_mozilla-beta_fennec_full_graph.yml#L32 . So having that variable is definitely harmless in config.py, just wanted to make sure it's not adding confusion. > > > > @@ +2423,5 @@ > > > BRANCHES['mozilla-beta']['postrelease_version_bump_config'] = { > > > "firefox": 'releases/postrelease_firefox_beta.py', > > > # configs are generic so can be reused > > > "fennec": 'releases/postrelease_firefox_beta.py', > > > + "devedition": "disabled", > > > > Reminder we need to alter this before going live. The version bump script or > > the BB custom don't know to handle this value and will most likely result in > > IOError for not finding that path. > > This is just to prevent KeyErrors down the path. This value shouldn't be > used anywhere, because the step is disabled for devedition. I agree. Was just a reminder for later ^_^ > > @@ +2478,4 @@ > > > BRANCHES['mozilla-beta']['beetmover_buckets'] = { > > > "firefox": "net-mozaws-prod-delivery-firefox", > > > "fennec": "net-mozaws-prod-delivery-archive", > > > + "devedition": "net-mozaws-prod-delivery-firefox", > > > > Where in release automation I should look to understand who's pushing the > > artifacts to http://archive.mozilla.org/pub/devedition/tinderbox-builds/ ? > > Those are our regular CI builds. I think it's > https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla/config. > py#24 Ah, sorry, I'll rephrase my question: who transfers the artifacts from CI builds under the S3 bucket? I only know how release-side system work and nightlies. I don't have too much context of how this step is tackled for CI builds, especially since our CI builds are actually "nightly" graphs with some of steps disabled.

In production: https://hg.mozilla.org/build/buildbot-configs/rev/91ab94840f5d

In production: https://hg.mozilla.org/build/buildbotcustom/rev/32ecfc7dc112

I'll open a PR in releasetasks as well to add the missing `--product` argument there.

Looking at the last staging build we have on jamun, turns out we need to land these patches in releasetasks/in-tree to prevent devedition pollute the Firefox bucket as it does right now under {stage}/pub/firefox/candidates/54.0b7-candidates/.

Comment on attachment 8868291 [details] Bug 1364225 - Releasetasks - pass product via configs for all beetmover script occurrences. https://github.com/mozilla/releasetasks/commit/a35f7082eb36027d8c06355e9ff178ce1326579e

Comment on attachment 8868286 [details] [diff] [review] In-tree beetmover to take product in a generic way via configs. https://hg.mozilla.org/projects/jamun/rev/d150c65de671

Comment on attachment 8868537 [details] Bug 1364225 - Enable Devedition in ship-it https://reviewboard.mozilla.org/r/140152/#review143468

Comment on attachment 8868537 [details] Bug 1364225 - Enable Devedition in ship-it remote: https://hg.mozilla.org/build/puppet/rev/a2b962989da32fe11dc7e8a3ab54c3856e29be28 remote: https://hg.mozilla.org/build/puppet/rev/24ad01c5137e5e7578b2e99312410303973c537b

Comment on attachment 8867985 [details] [diff] [review] tools.diff :rail checked-in this yesterday. https://hg.mozilla.org/build/tools/rev/8a9951b2888c

Pushed by mtabara@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/9bba670dbdc3 in-tree beetmover to take product in a generic way via configs. r=rail a=release DONTBUILD

https://hg.mozilla.org/releases/mozilla-beta/rev/93a43397cf6c https://hg.mozilla.org/releases/mozilla-release/rev/4fba047d14bc https://hg.mozilla.org/releases/mozilla-esr52/rev/bb14f704d641

https://hg.mozilla.org/releases/mozilla-esr52/rev/0dc7c28f2064 (FIREFOX_ESR_52_1_X_RELBRANCH)

In production: https://hg.mozilla.org/build/buildbot-configs/rev/91ab94840f5d

In production: https://hg.mozilla.org/build/buildbotcustom/rev/32ecfc7dc112

Comment on attachment 8869355 [details] Bug 1364225 - adjust devedition beetmover buckets to archive, rather than firefox. https://reviewboard.mozilla.org/r/141002/#review144480 The change is in line with bug 1365991 comment 7 and dxr doesn't show any other places where it could be needed. Lgtm!

https://hg.mozilla.org/build/buildbot-configs/rev/c93ccd0f33ac Bug 1364225 - adjust devedition beetmover buckets to archive, rather than firefox. r=jlorenzo

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #47) > https://hg.mozilla.org/build/buildbot-configs/rev/c93ccd0f33ac > Bug 1364225 - adjust devedition beetmover buckets to archive, rather than > firefox. r=jlorenzo We're unblocked on beetmover now - http://ftp.stage.mozaws.net/pub/devedition/candidates/54.0b7-candidates/build9/

Pretty much done with this.

Removing leave-open keyword from resolved bugs, per :sylvestre.