Generating a taskgraph with esr60 configs[1] leads to some discrepancies. The things I've noticed are: * balrog-ach-linux-nightly/opt => Bad scope "project:releng:balrog:server:dep". Worker type is good, though. * release-balrog-submit-toplevel-firefox => * archive_domain points to staging * "dep" scope is there * worker type is "invalid" * `channel-names` and `rules_to_update` are empty. By the way, we need to create a new specific balrog rule aliased `esr60`. This allows to not break users who want to remain on esr52, for now. * release-balrog-scheduling-firefox => * worker type is "invalid" * "dep" scope is there * `publish_rules` is empty There are probably other issues I didn't see. [1] https://bugzilla.mozilla.org/show_bug.cgi?id=1434889#c4

Hello Simon! 60.0esr goes to build on April 30th. Would you feel comfortable looking into this bug by then?

Should be fine - what're the commands (or where are the docs) used to generate this taskgraph?

(In reply to Simon Fraser [:sfraser] ⌚️GMT from comment #2) > Should be fine - what're the commands (or where are the docs) used to > generate this taskgraph? You should be able to use something like this from a mozilla-beta repo: ../braindump/taskcluster/taskgraph-diff/taskgraph-gen.py --params-dir ../braindump/taskcluster/taskgraph-diff/params-pre-partner-repacks --overwrite -j8 --full esr60 You could limit to specific params within params-pre-partner-repacks to avoid unnecessary work. Be sure to pull the latest revision of braindump, too.

I see `target_tasks_method: ship_firefox` - do we still want it to be called that, or do we need a separate ship_esr?

For the balrog rules here, I'd like to add rules to the esr & esr-*test channels that are basically clones of rule 521. That is, with: Alias: esr60 Comment: esr60 rule Initial mapping: esr52 (as we don't have a 60 build yet) A lower priority and 0 background rate, to ensure it's not yet used. This will let us get some rule IDs to add to the config. Does this seem reasonable?

Not yet finished, there are a few more things to do. Will add a comment with the ones I know about

Not sure of the status of android esr. If it's happening, these two need changing: taskcluster/ci//push-apk/kind.yml taskcluster/ci//google-play-strings/kind.yml Are we doing snaps? taskcluster/ci//release-snap-push/kind.yml lines 45-64 taskcluster/ci//release-secondary-update-verify-config/kind.yml Config files needed: taskcluster/ci//release-generate-checksums/kind.yml lines 62- taskcluster/ci//release-bouncer-check/kind.yml lines 47+ These depend on the balrog rule ID which we don't know yet, and the channel the RCs will be in (which I don't know) staskcluster/ci//release-secondary-balrog-submit-toplevel/kind.yml taskcluster/ci//release-balrog-scheduling/kind.yml taskcluster/ci//release-secondary-balrog-scheduling/kind.yml These three mention esr45, but not esr52. Do we need esr60 here? taskcluster/ci//build/windows.yml taskcluster/ci//build/linux.yml taskcluster/ci//build/macosx.yml

As discussed on irc, I'll split the diff up to the relevant bugs

Comment on attachment 8969275 [details] [diff] [review] esr60_draft.diff Review of attachment 8969275 [details] [diff] [review]: ----------------------------------------------------------------- (In reply to Simon Fraser [:sfraser] ⌚️GMT from comment #5) > For the balrog rules here, I'd like to add rules to the esr & esr-*test > channels that are basically clones of rule 521. That is, with: > > Alias: esr60 > Comment: esr60 rule > Initial mapping: esr52 (as we don't have a 60 build yet) > A lower priority and 0 background rate, to ensure it's not yet used. > > This will let us get some rule IDs to add to the config. Does this seem > reasonable? This should be mostly fine, except you'll need a version filter on it as well to avoid versions <60.0 getting offered updates to the latest version (we don't do that until esr52 is officially desupported - which could be a long time). I'd suggest using ">=60.0" for this. ::: taskcluster/ci/release-balrog-submit-toplevel/kind.yml @@ +71,4 @@ > birch: ["firefox-release-cdntest", "firefox-release-localtest"] > mozilla-beta: ["firefox-beta-cdntest", "firefox-beta-localtest"] > mozilla-release: ["firefox-release-cdntest", "firefox-release-localtest"] > + mozilla-esr60: ["firefox-esr-cdntest", "firefox-esr-localtest"] I think you'll need "esr60" in both of these alias names. ::: taskcluster/ci/release-update-verify-config/kind.yml @@ +58,4 @@ > maple: beta > mozilla-beta: beta > mozilla-release: nonbeta > + mozilla-esr60: nonbeta I think this is OK. It will likely cause esr52 -> esr60 update tests to run. Even though we won't immediately ship these updates, it doesn't hurt to run the tests early.

Where should this land, for your testing? Jamun?

Comment on attachment 8969327 [details] [diff] [review] balrog_draft.diff Review of attachment 8969327 [details] [diff] [review]: ----------------------------------------------------------------- LGTM! Thank you! ::: taskcluster/ci/release-balrog-scheduling/kind.yml @@ +44,3 @@ > mozilla-beta: [32] > mozilla-release: [145] > + mozilla-esr60: [] TODO: Put rule ID whenever known.

It seems we need to whitelist the staging archive.mozilla.org at [1]. It made the partial tasks fail[2] [1] https://searchfox.org/mozilla-central/rev/59a9a86553e9bfd9277202748ff791fd9bc0713b/taskcluster/docker/funsize-update-generator/scripts/funsize.py#46 [2] https://tools.taskcluster.net/groups/H0mR0zVhT66ohtOLFl-OZw/tasks/CtaWS2vwSYW6oOiBsuAfUQ/runs/0/logs/public%2Flogs%2Flive_backing.log#L75

Pushed the temp hack to unblock staging releases until I add the proper fixes: https://hg.mozilla.org/projects/jamun/rev/8082db5de26e

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #13) > Pushed the temp hack to unblock staging releases until I add the proper > fixes: > https://hg.mozilla.org/projects/jamun/rev/8082db5de26e Non-hack correct fix for this is under review here https://bugzilla.mozilla.org/show_bug.cgi?id=1453273#c12

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #14) > (In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #13) > > Pushed the temp hack to unblock staging releases until I add the proper > > fixes: > > https://hg.mozilla.org/projects/jamun/rev/8082db5de26e > > Non-hack correct fix for this is under review here > https://bugzilla.mozilla.org/show_bug.cgi?id=1453273#c12 https://hg.mozilla.org/projects/jamun/rev/185083f7f079

Simon, Mihai, Can you help drop some state for me, so we can get relevant patches landed?

Status update: * for balrog, fixes are part of the following patches that landed on jamun, in this order: https://hg.mozilla.org/projects/jamun/rev/cf2e01720435 https://hg.mozilla.org/projects/jamun/rev/c4b233b286a4 https://hg.mozilla.org/projects/jamun/rev/8082db5de26e https://hg.mozilla.org/projects/jamun/rev/185083f7f079 Might be also landed by sfraser here. I already checked the esr60 graphs with these patches produce the right bits. Leftover: test a staging esr60 release on jamun to make sure all the balrog jobs are using the dep-workers. After that, I'll graft these to central and beta. @Callek: if you can help me with the staging release test today (if not already), I'll take care of the grafting whatever we have here for balrog jobs as well, by EOD.

The balrog esr60 rules still need adding on the live servers, so we can fill in those options.

Comment on attachment 8969275 [details] [diff] [review] esr60_draft.diff Review of attachment 8969275 [details] [diff] [review]: ----------------------------------------------------------------- Clearing review. We talked on IRC last week. The decision was to split the patch into smaller ones, if I remember correctly. Please r? me again, if I missed something.

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #17) > Status update: > > * for balrog, fixes are part of the following patches that landed on jamun, > in this order: > https://hg.mozilla.org/projects/jamun/rev/cf2e01720435 > https://hg.mozilla.org/projects/jamun/rev/c4b233b286a4 > https://hg.mozilla.org/projects/jamun/rev/8082db5de26e > https://hg.mozilla.org/projects/jamun/rev/185083f7f079 > > Might be also landed by sfraser here. > > I already checked the esr60 graphs with these patches produce the right bits. > Leftover: test a staging esr60 release on jamun to make sure all the balrog > jobs are using the dep-workers. > > After that, I'll graft these to central and beta. > > @Callek: if you can help me with the staging release test today (if not > already), I'll take care of the grafting whatever we have here for balrog > jobs as well, by EOD. I landed this on inbound. Some of the stuff was already there from other grafts I suppose. https://hg.mozilla.org/integration/mozilla-inbound/rev/1527cfbcd067b594afc71fc2a7673f5bffed1a94 Letting this ride the sheriffingh to central + merge tomorrow to get to beta and eventually to esr60. @sfraser: do we still need to land anything else besides my patch?

Tried again after backout https://hg.mozilla.org/integration/mozilla-inbound/rev/db04d03e84185756c423bd40141fa45e7004d05aa

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #20) > I landed this on inbound. Some of the stuff was already there from other > grafts I suppose. > https://hg.mozilla.org/integration/mozilla-inbound/rev/ > 1527cfbcd067b594afc71fc2a7673f5bffed1a94 > > Letting this ride the sheriffingh to central + merge tomorrow to get to beta > and eventually to esr60. > > @sfraser: do we still need to land anything else besides my patch? There's patches in https://bugzilla.mozilla.org/show_bug.cgi?id=1453253#c6 that don't seem to be there, and I think they're still needed. I'll schedule the creation of balrog rules, then we can fill out the rule IDs in the diffs

Patches from this bug are tracked to be landed to inbound and some to be cherry-picked to release & esr60. Closing this bug but work continues in bug 1457090.

I massaged the balrog_draft.diff: - populated the 806 rule - removed the secondary balrog diffs - secondary balrog is for RCs only, and we don't do RCs on ESR. Thinking we should a) land this on esr60 post-merge, and b) graft it around once the trees open.

Comment on attachment 8971324 [details] [diff] [review] balrog.diff https://hg.mozilla.org/releases/mozilla-esr60/rev/4f344a27631e

https://hg.mozilla.org/releases/mozilla-esr60/rev/58ec70321aa3 https://hg.mozilla.org/releases/mozilla-esr60/rev/cd1552c9487e

Pushed by mozilla@hocat.ca: https://hg.mozilla.org/integration/mozilla-inbound/rev/c5cd2d794698 Taskgraph support for balrog esr60/jamun tasks p=sfraser r=aki https://hg.mozilla.org/integration/mozilla-inbound/rev/ce028e2b39b8 fix esr60 balrog config. p=sfraser r=jlorenzo

https://hg.mozilla.org/mozilla-central/rev/c5cd2d794698 https://hg.mozilla.org/mozilla-central/rev/ce028e2b39b8