We need to tweak/add in-tree beetmover kinds/tasks to support Firefox and migrate from old beetmover[1]. [1]: https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/scripts/release/beet_mover.py

@Rail: Quick question as I dived deep into in-tree graph generation and things are getting clearer now. Why do we have the following kinds defined as possibilities to reuse? 1) under relpro action config for promote_fennec[1] * repackage, repackage signing - they are not being used in Fennec at all. * beetmover, beetmover-signing, checksums, release bouncer sub - they don't exist in the CI graph. Is this because we're not actually reusing the tasks but their definitions in the promote_fennec graph? 2) under relpro action config for publish fennec[2] * repackage, repackage-signing, beetmover-repackage, repackage-signing - the same, not used in any-related Fennec graph * beetmover-l10n - we don't have Fennec single locales * checksums-signing - same question as for promote_fennec Overall question: * when we say we're reusing kinds, we mean to have their definitions available in that graph generation or does it mean to actually reuse those tasks if possible? [1]: https://hg.mozilla.org/projects/maple/file/tip/taskcluster/taskgraph/actions/release_promotion.py#l25 [2]: https://hg.mozilla.org/projects/maple/file/tip/taskcluster/taskgraph/actions/release_promotion.py#l35

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #1) > @Rail: > > Quick question as I dived deep into in-tree graph generation and things are > getting clearer now. > > Why do we have the following kinds defined as possibilities to reuse? > > 1) under relpro action config for promote_fennec[1] > * repackage, repackage signing - they are not being used in Fennec at all. AFAIK they are used for single locale Android repack, and those are disabled ATM. > * beetmover, beetmover-signing, checksums, release bouncer sub - they don't > exist in the CI graph. Is this because we're not actually reusing the tasks > but their definitions in the promote_fennec graph? The same here. > 2) under relpro action config for publish fennec[2] > * repackage, repackage-signing, beetmover-repackage, repackage-signing - > the same, not used in any-related Fennec graph > * beetmover-l10n - we don't have Fennec single locales > * checksums-signing - same question as for promote_fennec Yep, the same. :) > Overall question: > * when we say we're reusing kinds, we mean to have their definitions > available in that graph generation or does it mean to actually reuse those > tasks if possible? It means that we actually reuse the tasks. In other words, the new graph doesn't schedule those tasks if they can be found in the previous_graph_ids. I assume we automagically add deps and use the artifacts from those tasks.

Got it, thanks. I finally understoooooooood how the graphs work. Beetmover jobs incoming!

Not sure what's going on. Without adding maple_desktop_promotion in scriptworker/util.py, I get the tasks I expect but the scopes are wrong (have action: push-to-staging). To fix that I added the target task method under proper alias/scope but then I end up with lots of devedition build tasks, changes routes from firefox.latest to *nightly*devedition latest and such. Am I missing something here?

Same questions as above, I've updated the patch with including the new kind in the action `publish_firefox` as well, before I forget.

Comment on attachment 8926137 [details] [diff] [review] Add beetmover push to candidates for Firefox. >diff --git a/taskcluster/taskgraph/target_tasks.py b/taskcluster/taskgraph/target_tasks.py >--- a/taskcluster/taskgraph/target_tasks.py >+++ b/taskcluster/taskgraph/target_tasks.py >@@ -314,44 +314,47 @@ def target_tasks_mozilla_beta_desktop_pr > ignore_kinds=[], > allow_l10n=True)] > allow_kinds = [ > 'build', 'build-signing', 'repackage', 'repackage-signing', > 'nightly-l10n', 'nightly-l10n-signing', 'repackage-l10n', > 'release-update-verify', 'release-buildbot-update-verify', > 'upload-generated-sources', > 'partials', 'partials-signing', >+ 'beetmover-repackage', > ] We don't need this. > > def filter(task): > platform = task.attributes.get('build_platform') > > # Android has its own promotion. > if platform and 'android' in platform: > return False > > if task.kind not in allow_kinds: > return False > >+ if task.kind in ('beetmover-repackage'): >+ return True >+ We also have to filter by product. If beetmover-repackage doesn't have a product flag, we need to set one. We need this before this patch lands on maple. >diff --git a/taskcluster/taskgraph/util/scriptworker.py b/taskcluster/taskgraph/util/scriptworker.py >--- a/taskcluster/taskgraph/util/scriptworker.py >+++ b/taskcluster/taskgraph/util/scriptworker.py >@@ -84,16 +84,17 @@ BEETMOVER_SCOPE_ALIAS_TO_PROJECT = [[ > ]] > > """The set of all beetmover release target tasks. > > Used for both `BEETMOVER_SCOPE_ALIAS_TO_TARGET_TASK` and `get_release_build_number` > """ > BEETMOVER_CANDIDATES_TARGET_TASKS = set([ > 'candidates_fennec', >+ 'maple_desktop_promotion', We should also include mozilla-beta_desktop_promotion.

(In reply to Aki Sasaki [:aki] from comment #6) > > 'partials', 'partials-signing', > >+ 'beetmover-repackage', > We don't need this. I agree. Why don't we just kill this allow_kinds var along with the check then? Function will default to returning False anyway, unless we return True. > >+ if task.kind in ('beetmover-repackage'): > >+ return True > >+ > > We also have to filter by product. If beetmover-repackage doesn't have a > product flag, we need to set one. We need this before this patch lands on > maple. 01:00:06 <mtabara> hm, I think I'll take that back. not sure where you mean by "here" as temp fix - here? https://hg.mozilla.org/projects/maple/file/tip/taskcluster/taskgraph/transforms/beetmover_repackage.py#l344 01:00:59 <mtabara> also, this doesn't fix the problem that we have the devedition builders imported. we need to add the extra:product:firefox to more tasks, not just beetmover, wouldn't we? > > BEETMOVER_CANDIDATES_TARGET_TASKS = set([ > > 'candidates_fennec', > >+ 'maple_desktop_promotion', > > We should also include mozilla-beta_desktop_promotion. Good idea.

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #7) > (In reply to Aki Sasaki [:aki] from comment #6) > > > 'partials', 'partials-signing', > > >+ 'beetmover-repackage', > > We don't need this. > > I agree. Why don't we just kill this allow_kinds var along with the check > then? Function will default to returning False anyway, unless we return True. Hm, I misspoke. We probably want to keep allow_kinds, and keep this line. > > > >+ if task.kind in ('beetmover-repackage'): > > >+ return True > > >+ > > > > We also have to filter by product. If beetmover-repackage doesn't have a > > product flag, we need to set one. We need this before this patch lands on > > maple. > > 01:00:06 <mtabara> hm, I think I'll take that back. not sure where you mean > by "here" as temp fix - here? > https://hg.mozilla.org/projects/maple/file/tip/taskcluster/taskgraph/ > transforms/beetmover_repackage.py#l344 e.g. https://hg.mozilla.org/projects/maple/file/tip/taskcluster/ci/beetmover-cdns/kind.yml#l26 > 01:00:59 <mtabara> also, this doesn't fix the problem that we have the > devedition builders imported. we need to add the extra:product:firefox to > more tasks, not just beetmover, wouldn't we? No, we explicitly have to add product to beetmover-repackage, and change this > >+ if task.kind in ('beetmover-repackage'): > >+ return True to if task.task.get('extra', {}).get('product') == 'firefox': if task.kind in ('beetmover-repackage', ): like we do in candidates_fennec.

Actually, you’re right again. We might have to filter out ‘devedition’ from build_platform like we do for fennec.

Comment on attachment 8926961 [details] [diff] [review] Add beetmover push to candidates for Firefox. Thanks! and awesome. This can land on maple. >diff --git a/taskcluster/ci/beetmover-repackage/kind.yml b/taskcluster/ci/beetmover-repackage/kind.yml >--- a/taskcluster/ci/beetmover-repackage/kind.yml >+++ b/taskcluster/ci/beetmover-repackage/kind.yml >@@ -15,8 +15,11 @@ kind-dependencies: > - partials-signing > > only-for-build-platforms: > - linux-nightly/opt > - linux64-nightly/opt > - macosx64-nightly/opt > - win32-nightly/opt > - win64-nightly/opt >+ >+# FIXME: temporary hack until we find a better home for this >+product: firefox We'll have to address this when we add devedition beetmover-repackage. We'll possibly set product by build platform at that point? >diff --git a/taskcluster/taskgraph/target_tasks.py b/taskcluster/taskgraph/target_tasks.py >--- a/taskcluster/taskgraph/target_tasks.py >+++ b/taskcluster/taskgraph/target_tasks.py <snip> >+ if platform and 'devedition' in platform: >+ return False >+ > if task.kind not in allow_kinds: > return False > > # Allow for beta_tasks; these will get optimized out to point to > # the previous graph using ``previous_graph_ids`` and > # ``previous_graph_kinds``. > if task.label in beta_tasks: > return True > > # TODO: do we need to filter on product, too? >+ > # is this target tasks method only for firefox? if so, we should rename it > if task.kind in ('release-update-verify', 'release-buildbot-update-verify', >- 'partials', 'partial-signing', >- ): >+ 'partials', 'partials-signing'): > return True >+ >+ if task.task.get('extra', {}).get('product') == 'firefox': >+ if task.kind in ('beetmover-repackage', >+ ): >+ return True >+ I think this will be nicer if/when I get my shipping-product patch landed.

Comment on attachment 8926961 [details] [diff] [review] Add beetmover push to candidates for Firefox. https://hg.mozilla.org/projects/maple/rev/8fedffdb162905be7670a1833e4464513330a62f

Currently fixing some hiccups here but getting closer. Rail landed the funsize changes which means I need to deal with the partials handling in-tree/beetmoverscript and templates before I can see artifacts in the bucket. I'm very close, hope to finish up this part tomorrow.

https://hg.mozilla.org/projects/maple/rev/631d4ee73a5ea752c0338361018b4db381cbf186 to fix beetmover-repackage l10n stuff.

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #14) > https://hg.mozilla.org/projects/maple/rev/ > 631d4ee73a5ea752c0338361018b4db381cbf186 to fix beetmover-repackage l10n > stuff. Follow-up staging release[1] shows beetmover-l10n is now working too. Off to the next items in the beetmover bucket that need fixing. [1]: https://tools.taskcluster.net/groups/QhNYiURETkqk6kEanP5K8A

Need some work to fix the partials names. Following through in a bit.

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #16) > Need some work to fix the partials names. Following through in a bit. Attempting to fix this - https://hg.mozilla.org/projects/maple/rev/fe62c0ca34443cfe834f7af3e618fb33fe1b0dff

Follow-up fix: https://hg.mozilla.org/projects/maple/rev/072f1b5cee28

partial mar correctly renamed within beetmover - http://ftp.stage.mozaws.net/pub/firefox/candidates/58.0b6-candidates/build14/update/linux-i686/af/

To sum-up, patches to land in-tree are: https://hg.mozilla.org/projects/maple/rev/8fedffdb162905be7670a1833e4464513330a62f https://hg.mozilla.org/projects/maple/rev/631d4ee73a5ea752c0338361018b4db381cbf186 https://hg.mozilla.org/projects/maple/rev/fe62c0ca34443cfe834f7af3e618fb33fe1b0dff https://hg.mozilla.org/projects/maple/rev/072f1b5cee28826875c893b226a0d3a61df17909 To check before landing on central: * make sure the changes don't affect the beetmover-repackage for nightlies

Marking this as fixed, patches are part of bug 1423081.