This will generate a task which looks plausible. It doesn't set build_props in extra and it looks like desktop_partner_repacks.py does use that, so I'm mostly looking for a sanity check on kind.yml.

(In reply to Nick Thomas [:nthomas] (UTC+13) from comment #1) > It doesn't set build_props in extra and it looks like desktop_partner_repacks.py does use that, so I'm > mostly looking for a sanity check on kind.yml. I don't think it uses them. That would require querying TC. I bet you mean it uses regular buildbot properties, which mozharness dumps into buildprops.json. See https://dxr.mozilla.org/mozilla-central/rev/72ee4800d4156931c89b58bd807af4a3083702bb/testing/mozharness/scripts/desktop_partner_repacks.py#110

Comment on attachment 8930857 [details] [diff] [review] [gecko] WIP v1 Review of attachment 8930857 [details] [diff] [review]: ----------------------------------------------------------------- ::: taskcluster/ci/release-partner-repack/kind.yml @@ +22,5 @@ > + release-promotion: true > + worker: > + properties: > + repack_manifests_url: git@github.com:mozilla-partners/repack-manifests.git > + # pass something extra to maple to avoid uploads into prod bucket ?? maybe create a dummy repack manifest (at least) and make it `by-project`? It will still upload somewhere. We can probably use different `partner_repack_config` in buildbot-configs for maple and pass different --s3cfg (more puppet!). Or just remove --s3cfg for maple, so it doesn't copy to s3. @@ +25,5 @@ > + repack_manifests_url: git@github.com:mozilla-partners/repack-manifests.git > + # pass something extra to maple to avoid uploads into prod bucket ?? > + > +jobs: > + firefox-linux: I wonder if you can use `only-for-build-platforms` here. Not sure how to deal with the routes though in this case.

The s3cfg is tricky, because the more public repacks (like funnelcake) specify s3cfg in the their repack.cfg, while the private ones do not and get a value from the command line. Somewhat unusually, the repack.cfg trumps the command line. One option is to create a dummy repack-manifests repo, which includes a simple repack for staging, and provision a staging S3 bucket and some credentials via puppet (I think this is what you meant). Or possibly our own bucket and do cross-account access using the existing credentials. Or just aim low, and make a bogus repack manifests url and expect the job to fail, but still ensure it's has the right deps.

With TC notifications, routes, and a full set of platforms (but not EME-free yet). Fails mach taskgraph target-graph with an error about the by-product, which works just fine in notifications, and in the very similar taskcluster/ci/release-bouncer-aliases/kind.yml. Grrr.

For now I'd like to enable the builders, and will pass a non-existent repack_manifest_url in the task definition for initial TC tests. Later I'll add an empty manifest in a new github repo in mozilla-releng. The "--s3cfg", "NO_UPLOADS_IN_STAGING", guards against pointing to a real manifest repo, at least for the case of our more private repacks.

Comment on attachment 8932610 [details] [diff] [review] [buildbot-configs] Enable partner repacks on maple https://hg.mozilla.org/build/buildbot-configs/rev/9dd25c1371175aa69ca6a114a74ec944b4a63500 https://hg.mozilla.org/build/buildbot-configs/rev/700bd517902ee900fe8d7b5eedf072933a375e87 FTR the change in eme_free_repacks_platforms is due bug 1404048 and co. The builds are no-ops right now.

Adds a transform to handle by-project definition of repack_manifests_url, deps should work now.

* adds a new partner repack kind to run jobs on the bbb * creates partner repack jobs for 5 platforms, and EME-free jobs for mac/win32/win64. They start doing work after all l10n jobs are done and beetmoved (post-beetmover-dummy) * the repack_manifest_url differs between these two types (the git repo contains an xml for the repo util, describing which partner repos to clone) * there's a transform to resolve the by-project of repack_manifest_url. This seems like it should be a more general transform in TC * the values for repack_manifest_url are slightly hacky. mozilla-releng/staging-repack-manifests.git just has tools and the repack scripts in it, no actual repacks to avoid uploads into production buckets. Then set prod values for prod branches

On that last point, this is way of having no-op jobs on other branches, like ESR. If there's a better way to enable/disable jobs by branches then I'm happy to switch to it.

Comment on attachment 8933409 [details] [diff] [review] [gecko] Add partner repacks >diff --git a/taskcluster/ci/release-partner-repack/kind.yml b/taskcluster/ci/release-partner-repack/kind.yml >new file mode 100644 >--- /dev/null >+++ b/taskcluster/ci/release-partner-repack/kind.yml >@@ -0,0 +1,174 @@ >+# This Source Code Form is subject to the terms of the Mozilla Public >+# License, v. 2.0. If a copy of the MPL was not distributed with this >+# file, You can obtain one at http://mozilla.org/MPL/2.0/. >+ >+loader: taskgraph.loader.transform:loader >+ >+transforms: >+ - taskgraph.transforms.release_deps:transforms >+ - taskgraph.transforms.partner_repack:transforms >+ - taskgraph.transforms.job:transforms >+ - taskgraph.transforms.task:transforms >+ >+kind-dependencies: >+ - post-beetmover-dummy >+ >+job-defaults: >+ description: Release Promotion partner repacks >+ worker-type: buildbot-bridge/buildbot-bridge >+ run-on-projects: [] >+ shipping-product: firefox >+ shipping-phase: promote >+ run: >+ using: buildbot >+ release-promotion: true >+ product: firefox >+ notifications: >+ completed: >+ by-project: >+ maple: >+ - "release-drivers-staging" >+ try: >+ #- "{task[tags][createdForUser]}" >+ default: >+ - "release-drivers" >+ failed: >+ by-project: >+ maple: >+ - "release-drivers-staging" >+ try: >+ #- "{task[tags][createdForUser]}" >+ default: >+ - "release-drivers" >+ exception: >+ by-project: >+ maple: >+ - "release-drivers-staging" >+ try: >+ #- "{task[tags][createdForUser]}" >+ default: >+ - "release-drivers" >+ index: >+ type: release >+ product: firefox >+ >+jobs: >+ firefox-linux: >+ label: firefox linux partner repacks >+ worker: >+ properties: >+ repack_manifests_url: >+ by-project: >+ mozilla-beta: "git@github.com:mozilla-partners/repack-manifests.git" >+ mozilla-release: "git@github.com:mozilla-partners/repack-manifests.git" >+ default: "git@github.com:mozilla-releng/staging-repack-manifests.git" >+ run: >+ buildername: release-{branch}-firefox-linux_partner_repacks >+ routes: >+ - index.releases.v1.{branch}.{revision}.{product}.{underscore_version}.build{build_number}.partner_repacks.linux >+ - index.releases.v1.{branch}.latest.{product}.latest.partner_repacks.linux >+ >+ firefox-linux64: >+ label: firefox linux64 partner repacks >+ worker: >+ properties: >+ repack_manifests_url: >+ by-project: >+ mozilla-beta: "git@github.com:mozilla-partners/repack-manifests.git" >+ mozilla-release: "git@github.com:mozilla-partners/repack-manifests.git" >+ default: "git@github.com:mozilla-releng/staging-repack-manifests.git" >+ run: >+ buildername: release-{branch}-firefox-linux64_partner_repacks >+ routes: >+ - index.releases.v1.{branch}.{revision}.{product}.{underscore_version}.build{build_number}.partner_repacks.linux64 >+ - index.releases.v1.{branch}.latest.{product}.latest.partner_repacks.linux64 The general case (linux, linux64, macosx, etc.) seem to be the same except for the platform. We could potentially have a `firefox` job type with a list of platforms, and run a for-loop per. However, a) this isn't a blocker, and b) this becomes less of a win since we have to also define the eme-free varieties. >diff --git a/taskcluster/taskgraph/target_tasks.py b/taskcluster/taskgraph/target_tasks.py >--- a/taskcluster/taskgraph/target_tasks.py >+++ b/taskcluster/taskgraph/target_tasks.py >@@ -319,17 +319,17 @@ def target_tasks_promote_firefox(full_ta > 'build', 'build-signing', 'repackage', 'repackage-signing', > 'nightly-l10n', 'nightly-l10n-signing', 'repackage-l10n', > 'release-update-verify', 'release-buildbot-update-verify', > # Disabled for now because of missing scopes. > #'release-binary-transparency', > 'partials', 'partials-signing', > 'beetmover-repackage', 'release-source', > 'release-source-signing', 'beetmover-source', >- 'balrog', >+ 'balrog', 'release-partner-repack', > ] > > def filter(task): > platform = task.attributes.get('build_platform') > > # Android has its own promotion. > if platform and 'android' in platform: > return False >@@ -347,25 +347,24 @@ def target_tasks_promote_firefox(full_ta > if task.label in beta_tasks: > return True > > # TODO add shipping_product / shipping_phase > if task.kind in ('release-binary-transparency', > 'partials', 'partials-signing', 'beetmover-repackage', > 'release-source', 'nightly-l10n', 'nightly-l10n-signing', > 'release-source-signing', 'beetmover-source', >- 'repackage-l10n', 'balrog', >+ 'repackage-l10n', 'balrog', 'release-partner-repack', > ): > return True Ideally the below block will add the tasks, and we don't have to add to the above deprecated block. Do you know if this works without adding the above change? > if task.attributes.get('shipping_product') == 'firefox' and \ > task.attributes.get('shipping_phase') == 'promote': > return True

(In reply to Aki Sasaki [:aki] from comment #11) > The general case (linux, linux64, macosx, etc.) seem to be the same except > for the platform. We could potentially have a `firefox` job type with a list > of platforms, and run a for-loop per. However, a) this isn't a blocker, and > b) this becomes less of a win since we have to also define the eme-free > varieties. The label, buildername, and routes vary, but could probably be parametrized. I'll have another look to see if I can squash that into blocks for eme-free and not. > Ideally the below block will add the tasks, and we don't have to add to the > above deprecated block. > Do you know if this works without adding the above change? Yes, it does. I'll update the patch for this, at least. Thanks for the review.

I wasn't able to find a nice way to parametrize kinds.yml, there just isn't first class support there yet. https://hg.mozilla.org/projects/maple/rev/3109a1d2f7bcd342d7ca7ec0bcaa31893314a211 lands the reviewed patch, merged the changes which remove allow_kinds and maintaining flake8 cleanness.