Staging creds for release bucket for now

Comment on attachment 8842220 [details] [diff] [review] beetmover.diff Review of attachment 8842220 [details] [diff] [review]: ----------------------------------------------------------------- Bleah, I meant to split them nicely before but

I've pinned beetmoverworker-{1,2} to my puppet env for now.

https://hg.mozilla.org/projects/jamun/rev/374872d263535faf0e0024f622d112f578756175 bug 1343393 - enable staging fennec via candidates_fennec target_tasks_method

This patch: * adds a candidates_fennec target_task_method, which I am currently calling from https://tools.taskcluster.net/hooks/#project-releng/candidates-fennec-dev * this function disables balrog, since a) I think we might be pushing to the wrong channel atm, and b) we don't have release update channel support yet * adds a `build_number` to the task.payload in beetmover if a) this is a candidates_fennec graph, and b) BUILD_NUMBER is defined in the environment. * a candidates_fennec graph without BUILD_NUMBER in the env will throw a ValueError. * Removes jamun from the valid branches. We can patch jamun for testing, but let's not have it ride the trains * Changes the beetmover bucket logic! Since we have both 'push-to-nightly' and 'push-to-candidates' live on jamun, I needed something other than the project to define the bucket scopes. I'm using the new `get_scope_from_target_method_and_project` function, which looks at both target_task_method and project, and only gives you privileged scopes if both the project and target_task_method align. What do you think? (Go ahead and land your other util.scriptworker changes when you're ready; I can deal with bitrot.)

Comment on attachment 8842289 [details] [diff] [review] candidates_fennec.diff Review of attachment 8842289 [details] [diff] [review]: ----------------------------------------------------------------- Looks good, none of the comments are blocking! Thanks for verbossed comments! ::: taskcluster/taskgraph/target_tasks.py @@ +228,5 @@ > + """Select the set of tasks required for a candidates build of fennec. The > + nightly build process involves a pipeline of builds, signing, > + and, eventually, uploading the tasks to balrog.""" > + filtered_for_project = target_tasks_nightly(full_task_graph, parameters) > + def filter(task): I think the nightly check doesn't hurt but is useless here as it will always return True, because filtered_for_project is already and instance of `target_tasks_nightly` which is at its turn a filtering in the full task graph retaining *just* the android-based for which the task.attributes returns *nightly*. So I think it's safe to simplify this to: `def filter(task): return task.kind not in ['balrog'] ` ::: taskcluster/taskgraph/transforms/task.py @@ +15,5 @@ > > from taskgraph.util.treeherder import split_symbol > from taskgraph.transforms.base import TransformSequence > from taskgraph.util.schema import validate_schema > +from taskgraph.util.scriptworker import get_release_build_number In bug 1343349 I'm adding a new ~/taskcluster/taskgraph/util/release.py in order to add some stuff that's not related with the scopes. Initially I was gonna suggest we move stuff like this beetmover-buildnumber thing + my routes from bug 1343349 there to avoid polluting the scriptworker.py scopes script but I changed my mind. I think your approach is better as we might reuse some of those scopes dicts in the util/helper functions as you did with buildnumber ++ ::: taskcluster/taskgraph/util/scriptworker.py @@ -52,5 @@ > BEETMOVER_SCOPE_ALIAS_TO_PROJECT = [[ > 'all-nightly-branches', set([ > 'mozilla-central', > 'mozilla-aurora', > - 'jamun', Good call here! @@ +57,1 @@ > 'mozilla-beta', Before Monday, aren't we supposed to delete the "mozilla-beta" and "mozilla-release" from here to make sure the release scopes are pointed at? @@ +79,5 @@ > BEETMOVER_SCOPE_ALIAS_TO_TARGET_TASK = [[ > 'all-nightly-tasks', set([ > 'nightly_fennec', > 'nightly_linux', > 'mozilla_beta_tasks', Normally, I'd say we need to delete the "mozilla-beta" and "mozilla-release" from here and add them in the "all-release-tasks" section but I guess we'd be calling the "fennec_candidates" target task method anyway for the real beta 1 on Monday, right? @@ +176,5 @@ > return alias_to_scope_map[alias] > return alias_to_scope_map['default'] > > > +def get_scope_from_target_method_and_project(alias_to_tasks_map, alias_to_project_map, Good job on refactoring this to take both into account!

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #6) > @@ +57,1 @@ > > 'mozilla-beta', > > Before Monday, aren't we supposed to delete the "mozilla-beta" and > "mozilla-release" from here to make sure the release scopes are pointed at? > > @@ +79,5 @@ > > BEETMOVER_SCOPE_ALIAS_TO_TARGET_TASK = [[ > > 'all-nightly-tasks', set([ > > 'nightly_fennec', > > 'nightly_linux', > > 'mozilla_beta_tasks', > > Normally, I'd say we need to delete the "mozilla-beta" and "mozilla-release" > from here and add them in the "all-release-tasks" section but I guess we'd > be calling the "fennec_candidates" target task method anyway for the real > beta 1 on Monday, right? > Bah, nevermind these two questions from above comment. I just reread https://bugzilla.mozilla.org/show_bug.cgi?id=1343144#c3 :) Forgot about the push-to-nightly for CI on beta and release.

Pushed by mtabara@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/232a19e40a52 add graph logic to handle a fennec beta build. r=mtabara DONTBUILD

@aki: hope you don't mind, I took the liberty to land the patch for you on inbound, to save some time.

Comparing http://archive.mozilla.org/pub/mobile/releases/52.0b8/ with the staging fennec build "53.0" out of jamun, my observations are these: 1. At the root folder level we're missing the following Dir logs/ Dir source/ File KEY 4K 20-Feb-2017 22:15 File SHA512SUMS 14K 20-Feb-2017 22:15 File SHA512SUMS.asc 836 20-Feb-2017 22:15 The Key and SHA512SUMS are coming from BBB jobs if I'm not wrong. Not sure what we're gonna do with the logs since we don't have a centralized BB way to do that. But I'm sure there are workarounds (to add them in beetmover or so). 2. At the single locale level, we seem to be having an extra "ur" locale but I guess that's fine. Most likelt that's been added recently in aurora and has been riding the trains. 3. At the en-US locale, for both Fennecx86 and Fennec: a) we seem to be missing: File fennec-52.0b8.en-US.android-arm.crashreporter-symbols-full.zip 453M 20-Feb-2017 21:16 File fennec-52.0b8.en-US.android-arm.crashreporter-symbols.zip 97M 20-Feb-2017 21:17 b) we're missing the /host/bin/{mar,mbsdiff}. I see they're present in beetmover templates but they're not being specified in the upstreamArtifacts, see last fennec staging task[2] c) checksums look good, except that we're short on the {mar,mbsdiff} artifacts 4. At the "multi" locale we're 100% accurate for both Fennecx86 and Fennec. [1]: https://github.com/mozilla-releng/beetmoverscript/blob/master/beetmoverscript/templates/fennec_candidates.yml#L102 [2]: https://tools.taskcluster.net/task-group-inspector/#/QQjEh3q1TOu2liv57iHawg/OzMj6pdsQJSM6H9oKIu7yg?_k=mlp0pv

https://hg.mozilla.org/integration/mozilla-inbound/rev/ec8ac10801d6c054a3e74f89fd1857c0bdac638b bug 1343393 - fix flake8 linting. r=bustage

Comment on attachment 8842625 [details] Bug 1343393 - add missing script_config in beetmoverworker. https://reviewboard.mozilla.org/r/116396/#review118026

Comment on attachment 8842625 [details] Bug 1343393 - add missing script_config in beetmoverworker. https://hg.mozilla.org/build/puppet/rev/acca7d430371

https://hg.mozilla.org/mozilla-central/rev/232a19e40a52 https://hg.mozilla.org/mozilla-central/rev/ec8ac10801d6

Rail, it looks like we might have a good candidates build for Fennec now. I could use a hand running the candidates->releases portion Thursday if you have cycles? or I can pester someone else :)

(In reply to Wes Kocher (:KWierso) from comment #15) > https://hg.mozilla.org/mozilla-central/rev/232a19e40a52 > https://hg.mozilla.org/mozilla-central/rev/ec8ac10801d6 :aki melded these two and uplifted them to aurora: https://hg.mozilla.org/releases/mozilla-aurora/rev/15eece89ef86

(In reply to Aki Sasaki [:aki] from comment #16) > Rail, it looks like we might have a good candidates build for Fennec now. > I could use a hand running the candidates->releases portion Thursday if you > have cycles? or I can pester someone else :) Sure!

Thanks! To be clear, I or someone else can do it, but will probably need pointers or help.

(In reply to Aki Sasaki [:aki] from comment #19) > Thanks! To be clear, I or someone else can do it, but will probably need > pointers or help. I can help / volunteer to help here with some pointers. I'm looking forward for a break from {beetmover,balrog} work.

I think we're done with the staging candidates build portion. :kmoir is working on the releasetasks graph1 triggering the candidates_fennec graph in bug 1343649, and :mtabara is working on the candidates->release portion. We can keep using this as a tracking bug, though.

The releasetasks graph work are both in progress and tracked in the phase2 relpro trello board. I'm going to resolve this bug.

Removing leave-open keyword from resolved bugs, per :sylvestre.