Generating a taskgraph with esr60 configs[1] leads to some discrepancies. The things I've noticed are: * checksums-signing-ach-linux-nightly/opt, release-generate-checksums-firefox-signing, release-source-signing-linux54-source, repackage-signing-ach-linux-nightly/opt => * Bad scope "project:releng:signing:cert:dep-signing". * Bad worker-type "depsigning" There are probably other issues I didn't see. [1] https://bugzilla.mozilla.org/show_bug.cgi?id=1434889#c4

Hello Aki! 60.0esr goes to build on April 30th. Would you feel comfortable looking into this bug by then?

On my radar. If partner repacks go long, this may be fodder for someone else to tackle.

Working on similar stuff in bug 1453255 so I can take this along.

Double-checked for all signing jobs, over all promote/push/ship: * build-signing * checksums-signing * nightly-l10n-signing * partials-signing * release-beetmover-signed-langpacks * release-generate-checksums-signing * release-sign-and-push-langpacks (are mozilla-esr60 langpacks listed publicly) * release-source-signing * repackage-signing A subset of the hunks already landed on jamun here[1]. I'm going to land the rest tomorrow before we start a staging release. For future use of this (next esr bump or w\e), I'm stating the patch here just so we have a good record of what changed overall for signing. [1]: https://hg.mozilla.org/projects/jamun/diff/f200dea1d257/taskcluster/taskgraph/util/scriptworker.py

00:58:19 <mtabara> are mozilla-esr60 langpacks supposed to be listed publicly? 00:58:39 <aki> c.allek knows best, but i'd guess probably not until esr60 is live 01:01:27 <mtabara> sounds reasonable to me, thanks @Callek: can you please confirm we don't need to add (yet) esr60 here[1] [1]: https://hg.mozilla.org/releases/mozilla-beta/file/tip/taskcluster/ci/release-sign-and-push-langpacks/kind.yml#l34

The above patch is what we need to make sure esr60 works as expected (the actual repo, not jamun). As for jamun, out of the list of *signing* jobs above, there's one that doesn't have the scopes/get-worker_type_by_scope, hence they are hardcoded. Out of the signing types above, that's: * `partials-signing `[1] Easy fix is to: * hardcode the below lines with the dev worker and make sure we don't uplift that to beta/esr when we're done Correct fix is to: * rewrite all that logic to better handle the scopes via scopes/get-worker_type_by_scope [1]: https://hg.mozilla.org/releases/mozilla-beta/file/tip/taskcluster/taskgraph/transforms/partials_signing.py#l86

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #5) > 00:58:19 <mtabara> are mozilla-esr60 langpacks supposed to be listed > publicly? > 00:58:39 <aki> c.allek knows best, but i'd guess probably not until esr60 is > live > 01:01:27 <mtabara> sounds reasonable to me, thanks > > @Callek: can you please confirm we don't need to add (yet) esr60 here[1] > > [1]: > https://hg.mozilla.org/releases/mozilla-beta/file/tip/taskcluster/ci/release- > sign-and-push-langpacks/kind.yml#l34 Langpacks for esr60 *must* be submitted as "unlisted"

(In reply to Justin Wood (:Callek) from comment #7) > (In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #5) > > 00:58:19 <mtabara> are mozilla-esr60 langpacks supposed to be listed > > publicly? > > 00:58:39 <aki> c.allek knows best, but i'd guess probably not until esr60 is > > live > > 01:01:27 <mtabara> sounds reasonable to me, thanks > > > > @Callek: can you please confirm we don't need to add (yet) esr60 here[1] > > > > [1]: > > https://hg.mozilla.org/releases/mozilla-beta/file/tip/taskcluster/ci/release- > > sign-and-push-langpacks/kind.yml#l34 > > Langpacks for esr60 *must* be submitted as "unlisted" Cool, good to know. Nothing to do there then. Thanks!

Dropping more lines here for posterity: 01:41:47 <Callek> aki: mtabara|afk: I answered in bug, but even once esr60 is live, we can't list langpacks in amo as "listed" and must stay "unlisted" 01:41:58 <aki> makes sense 01:42:27 <Callek> that is no difference on AMO side vs today, (with esr52) and is primarily because "most recent langpack is what users are presented with while browsing" which if min/max gecko versions don't match then those users will get a "this addon is not compatible with your version" type of error message 01:42:49 <Callek> though they could find a compat version in "Other Versions" its just not worth it for ESR with todays AMO

Landed rest of the hunks in https://hg.mozilla.org/projects/jamun/rev/cf2e01720435b19d7fb1320c48a91728b39bddaa

Pushed the temp hack to unblock staging releases until I add the proper fixes: https://hg.mozilla.org/projects/jamun/rev/c4b233b286a443549cbf88820b3dd9c93de70ef6

We have a bunch of tasks across {signing,beetmover,balrog} that have had the worker-type hardcoded. This is to address this in one single patch. Signing (current bug) * partials-signing Beetmover (bug 1453255) * beetmover-checksums * beetmover-repackage * beetmover-source (which actually uses beetmover kind to perform this) Balrog (bug 1453253) * balrog

Comment on attachment 8969709 [details] [diff] [review] in-tree {signing,beetmover,balrog} worker-type tweaks for esr60 Thanks! I think we landed at least some of this patch on mozilla-beta yesterday.

mihai, do you have further state here, what is required to land to central/beta?

Status update: * for signing, fixes are part of the following patches that landed on jamun, in this order: https://hg.mozilla.org/projects/jamun/rev/cf2e01720435 https://hg.mozilla.org/projects/jamun/rev/c4b233b286a4 https://hg.mozilla.org/projects/jamun/rev/8082db5de26e https://hg.mozilla.org/projects/jamun/rev/185083f7f079 I already checked the esr60 graphs with these patches produce the right bits. Leftover: test a staging esr60 release on jamun to make sure all the signing jobs are using the dep-workers. After that, I'll graft these to central and beta. @Callek: if you can help me with the staging release test today (if not already), I'll take care of the grafting by EOD.

Pushed by mtabara@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/1527cfbcd067 1453255,1453253 - add esr60 fixes in automation. r=aki

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #16) > Status update: > > * for signing, fixes are part of the following patches that landed on jamun, > in this order: > https://hg.mozilla.org/projects/jamun/rev/cf2e01720435 > https://hg.mozilla.org/projects/jamun/rev/c4b233b286a4 > https://hg.mozilla.org/projects/jamun/rev/8082db5de26e > https://hg.mozilla.org/projects/jamun/rev/185083f7f079 > > I already checked the esr60 graphs with these patches produce the right bits. > Leftover: test a staging esr60 release on jamun to make sure all the signing > jobs are using the dep-workers. > > After that, I'll graft these to central and beta. > > @Callek: if you can help me with the staging release test today (if not > already), I'll take care of the grafting by EOD. I landed this on inbound. Some of the stuff was already there from other grafts I suppose. https://hg.mozilla.org/integration/mozilla-inbound/rev/1527cfbcd067b594afc71fc2a7673f5bffed1a94 Letting this ride the sheriffingh to central + merge tomorrow to get to beta and eventually to esr60.

Backed out changeset 1527cfbcd067 (bug 1453273) for failing Geko decision task on a CLOSED TREE Backout: https://hg.mozilla.org/integration/mozilla-inbound/rev/2db2fd52dd668ee4f168a716f21755c61b8966c6 Failure push: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=1527cfbcd067b594afc71fc2a7673f5bffed1a94 Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=175569942&repo=mozilla-inbound&lineNumber=1699 [task 2018-04-25T18:14:56.909Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/release_notifications.py", line 39, in add_notifications [task 2018-04-25T18:14:56.909Z] for job in jobs: [task 2018-04-25T18:14:56.910Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/beetmover_cdns.py", line 100, in make_beetmover_cdns_worker [task 2018-04-25T18:14:56.910Z] for job in jobs: [task 2018-04-25T18:14:56.910Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/beetmover_cdns.py", line 62, in make_beetmover_cdns_description [task 2018-04-25T18:14:56.910Z] for job in jobs: [task 2018-04-25T18:14:56.910Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/beetmover_cdns.py", line 56, in validate [task 2018-04-25T18:14:56.910Z] "In cdns-signing ({!r} kind) task for {!r}:".format(config.kind, label)) [task 2018-04-25T18:14:56.910Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/schema.py", line 31, in validate_schema [task 2018-04-25T18:14:56.911Z] raise Exception('\n'.join(msg) + '\n' + pprint.pformat(obj)) [task 2018-04-25T18:14:56.911Z] Exception: In cdns-signing (u'beetmover-cdns' kind) task for 'firefox-push-to-cdns': [task 2018-04-25T18:14:56.911Z] extra keys not allowed @ data['worker-type'] [task 2018-04-25T18:14:56.911Z] {u'dependencies': {u'release-generate-checksums-firefox-beetmover': u'release-generate-checksums-firefox-beetmover'}, [task 2018-04-25T18:14:56.911Z] u'job-from': u'kind.yml', [task 2018-04-25T18:14:56.911Z] 'name': 'firefox-push-to-cdns', [task 2018-04-25T18:14:56.911Z] 'product': 'firefox', [task 2018-04-25T18:14:56.911Z] 'run-on-projects': [], [task 2018-04-25T18:14:56.912Z] 'shipping-phase': 'push', [task 2018-04-25T18:14:56.912Z] 'shipping-product': 'firefox', [task 2018-04-25T18:14:56.912Z] 'treeherder-platform': 'Linux64/opt', [task 2018-04-25T18:14:56.912Z] 'worker-type': {'by-project': {'default': 'scriptworker-prov-v1/beetmoverworker-dev', [task 2018-04-25T18:14:56.912Z] 'mozilla-beta': 'scriptworker-prov-v1/beetmoverworker-v1', [task 2018-04-25T18:14:56.912Z] 'mozilla-esr60': 'scriptworker-prov-v1/beetmoverworker-v1', [task 2018-04-25T18:14:56.912Z] 'mozilla-release': 'scriptworker-prov-v1/beetmoverworker-v1'}}} [taskcluster 2018-04-25 18:14:57.418Z] === Task Finished ===

Pushed by mtabara@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/db04d03e8418 1453255,1453253 - add esr60 fixes in automation. r=aki CLOSED TREE

Tried again after backout https://hg.mozilla.org/integration/mozilla-inbound/rev/db04d03e84185756c423bd40141fa45e7004d05aa

We failed to graft this patch in time on beta for esr60 mergeduty. We'll wait until the merge is completed and then we'll land this patch to release & esr60. Work for this is tracked in bug 1457090.

Comment 22 was grafted to mozilla-esr60 as https://hg.mozilla.org/releases/mozilla-esr60/rev/28107b91f58a