Generating a taskgraph with esr60 configs[1] leads to some discrepancies. The things I've noticed are: * beetmover-checksums-ach-linux-nightly/opt, beetmover-repackage-ach-linux-nightly, beetmover-source-linux64-source, firefox-push-to-cdns => * Bad scope "project:releng:beetmover:bucket:dep". * Worker type is good, though. There are probably other issues I didn't see. [1] https://bugzilla.mozilla.org/show_bug.cgi?id=1434889#c4

* release-generate-checksums-firefox-beetmover => * bad scope and bad worker.

Hello Mihai! 60.0esr goes to build on April 30th. Would you feel comfortable looking into this bug by then?

(In reply to Johan Lorenzo [:jlorenzo] from comment #2) > Hello Mihai! 60.0esr goes to build on April 30th. Would you feel comfortable > looking into this bug by then? Absolutely!

I got carried away in Bug 1453253 and drafted this. Not sure if it's complete yet

(In reply to Simon Fraser [:sfraser] ⌚️GMT from comment #4) Worker type looks good. We also need to fix the bucket scope. Would it also be possible to make jamun use the dev instances? Tasks like https://tools.taskcluster.net/groups/H0mR0zVhT66ohtOLFl-OZw/tasks/QUWNgEDQTuqlVSxierukiw/details failed at CoT in our first 60.0esr staging release.

Double-checked for all beetmovers, over all promote/push/ship: * (not used) beetmover * (not used) beetmover-l10n * beetmover-cdns * beetmover-checksums * beetmover-repackage * beetmover-source * post-beetmover-checksums-dummy * post-beetmover-dummy * release-beetmover-signed-langpacks * release-generate-checksums-beetmover Part of the hunks already landed on jamun here[1] and other part were already mentioned by Simon in the previous patchset[2]. For future use of this (next esr bump or w\e), I'm making the other patch obsolete and keep this one to have a good record of what changed overall for beetmover. [1]: https://hg.mozilla.org/projects/jamun/diff/f200dea1d257/taskcluster/taskgraph/util/scriptworker.py [2]: https://bugzilla.mozilla.org/attachment.cgi?id=8969285&action=edit

The above patch is what we need to make sure esr60 works, the actual repo. As to jamun, which is behaving like jamun, normally we'd have to change nothing. But ... while we wrote the kinds and transforms, we left some of the tasks without proper way of handling the scopes, hence they are hardcoded. Out of the beetmover types above, they are: * beetmover-checksums[1] * beetmover-repackage[2] * beetmover-source[3](which actually uses the beetmover transforms) Easy fix is to: * hardcode the below lines with the dev worker and make sure we don't uplift that to beta/esr when we're done Correct fix is to: * rewrite all that logic to better handle the scopes via by-project in TC [1]: https://hg.mozilla.org/releases/mozilla-beta/file/tip/taskcluster/taskgraph/transforms/beetmover_checksums.py#l103 [2]: https://hg.mozilla.org/releases/mozilla-beta/file/tip/taskcluster/taskgraph/transforms/beetmover_repackage.py#l249 [3]: https://hg.mozilla.org/releases/mozilla-beta/file/tip/taskcluster/taskgraph/transforms/beetmover.py#l392

Landed rest of the hunks in https://hg.mozilla.org/projects/jamun/rev/cf2e01720435b19d7fb1320c48a91728b39bddaa

Pushed the temp hack to unblock staging releases until I add the proper fixes: https://hg.mozilla.org/projects/jamun/rev/c4b233b286a443549cbf88820b3dd9c93de70ef6

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #9) > Pushed the temp hack to unblock staging releases until I add the proper > fixes: > https://hg.mozilla.org/projects/jamun/rev/ > c4b233b286a443549cbf88820b3dd9c93de70ef6 Non-hack correct fix for this is under review here https://bugzilla.mozilla.org/show_bug.cgi?id=1453273#c12

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #10) > (In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #9) > > Pushed the temp hack to unblock staging releases until I add the proper > > fixes: > > https://hg.mozilla.org/projects/jamun/rev/ > > c4b233b286a443549cbf88820b3dd9c93de70ef6 > > Non-hack correct fix for this is under review here > https://bugzilla.mozilla.org/show_bug.cgi?id=1453273#c12 https://hg.mozilla.org/projects/jamun/rev/185083f7f079

Hey Mihai, Any insight into what still needs landing, and where, specifically? What pieces still need testing, and/or what testing has been done. (Skimming this bug I found myself confused, though it could be PEBKAC figured a n-i may help me gather the state)

Status update: * for beetmover, fixes are part of the following patches that landed on jamun, in this order: https://hg.mozilla.org/projects/jamun/rev/cf2e01720435 https://hg.mozilla.org/projects/jamun/rev/c4b233b286a4 https://hg.mozilla.org/projects/jamun/rev/8082db5de26e https://hg.mozilla.org/projects/jamun/rev/185083f7f079 I already checked the esr60 graphs with these patches produce the right bits. Leftover: test a staging esr60 release on jamun to make sure all the beetmover jobs are using the dep-workers. After that, I'll graft these to central and beta. @Callek: if you can help me with the staging release test today (if not already), I'll take care of the grafting by EOD.

We're getting better here, but one thing I notice is that esr52 doesn't have a stub installer, and we have code in beetmover to exclude the stub installer. However I don't think nightly and release machinery has ever been attempted without the stub installer, so I'm wondering is it *ok* if we build and ship the stub installer with esr60, Robert?

Matt Howell owns the installers now so let's ask him.

I don't think a stub installer built on the ESR channel would even work; I haven't actually tried it, but we never built any specific support in there for ESR, and I think that means it would try to install a release build. So, no, I'm afraid it's probably *not* okay.

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #13) > Status update: > > * for beetmover, fixes are part of the following patches that landed on > jamun, in this order: > https://hg.mozilla.org/projects/jamun/rev/cf2e01720435 > https://hg.mozilla.org/projects/jamun/rev/c4b233b286a4 > https://hg.mozilla.org/projects/jamun/rev/8082db5de26e > https://hg.mozilla.org/projects/jamun/rev/185083f7f079 > > I already checked the esr60 graphs with these patches produce the right bits. > Leftover: test a staging esr60 release on jamun to make sure all the > beetmover jobs are using the dep-workers. > > After that, I'll graft these to central and beta. > > @Callek: if you can help me with the staging release test today (if not > already), I'll take care of the grafting by EOD. I landed this on inbound. Some of the stuff was already there from other grafts I suppose. https://hg.mozilla.org/integration/mozilla-inbound/rev/1527cfbcd067b594afc71fc2a7673f5bffed1a94 Letting this ride the sheriffingh to central + merge tomorrow to get to beta and eventually to esr60.

Tried again after backout https://hg.mozilla.org/integration/mozilla-inbound/rev/db04d03e84185756c423bd40141fa45e7004d05aa

We failed to graft this patch in time on beta for esr60 mergeduty. We'll wait until the merge is completed and then we'll land this patch to release & esr60. Work for this is tracked in bug 1457090.