+++ This bug was initially created as a clone of Bug #1282186 +++ where balrogscript is the likely result of 'the generic method' used by scriptworker in: bug 1244181

like 1282188 (beetmover) and 1282186 (signing), we will want to support publishing on a developer channel. e.g. 'nightly-test' depending on scopes in the graph and, eventually, whether graph is part of 'chain of trust'

...I'm going to patch (in this bug) desktop and fennec

Comment on attachment 8806972 [details] Bug 1282187 - create task that publishes nightlies on update server via scriptworker -> balrogscript. This is currently a near-done WIP. I expect to need to change some minor things with the payload format here, but in the meantime this is indeed the bulk of it

Comment on attachment 8806972 [details] Bug 1282187 - create task that publishes nightlies on update server via scriptworker -> balrogscript. https://reviewboard.mozilla.org/r/90212/#review90102 Looks good. I ran a try run and the resulting task graph looks good. The only thing is that I would advise naming beetmover_task as balrog_task in taskcluster/taskgraph/task/balrog.py for clarity.

Comment on attachment 8806972 [details] Bug 1282187 - create task that publishes nightlies on update server via scriptworker -> balrogscript. https://reviewboard.mozilla.org/r/90212/#review91194 ::: taskcluster/taskgraph/transforms/task.py:493 (Diff revision 1) > > +@payload_builder('balrog') > +def build_balrog_payload(config, task, task_def): > + worker = task['worker'] > + > + task_def['payload'] = { patch looks great. I have some more updated info with regards to the payload. beetmoverscript: https://github.com/mozilla-releng/beetmoverscript/pull/9 and balrogworker: https://github.com/mozilla-releng/funsize-balrogworker/pull/2 are both nearing the stage where they both can handle creating a manifest and submitting to balrog. the payload body from the graph should be as follows: ``` task_def['payload'] = { "parent_task_artifacts_url": "https://queue.taskcluster.net/v1/task/<taskID>/artifacts/public", "signing_cert": "dep" } ``` where <taskID> should be interpolated to the taskid of the beetmoverscript task that beet moves the signingscript task's artifacts. iow - the second beetmoverscript task's taskid that handles the signed artfacts signing_cert's value doesn't really matter. It probably should be 'dep' for projects/date based nightlies, and "nightly" for central/aurora. However, in our world it's ignored but needed satisfy balrogworker's current expected payload. If you are curious, it is used if balrogworker is also uploading to s3 prior to balrog submission. in our world, we use beetmover for that so we skip s3 upload: https://github.com/mozilla-releng/funsize-balrogworker/blob/master/bin/balrogworker.py#L70 once that is on date and beetmoverscript+balrogworker PRs get deployed, we should be looking in good shape for balrog submission

I can look at this tomorrow if you are working on something else. otherwise, see comment 7 for latest state

(In reply to Jordan Lund (:jlund) from comment #8) > I can look at this tomorrow if you are working on something else. otherwise, > see comment 7 for latest state I'll continue on this, but we have some small pieces to discuss today as well (probably at, or just after, the tc-nightly mtg)

(In reply to Justin Wood (:Callek) from comment #9) > (In reply to Jordan Lund (:jlund) from comment #8) > > I can look at this tomorrow if you are working on something else. otherwise, > > see comment 7 for latest state > > I'll continue on this, but we have some small pieces to discuss today as > well (probably at, or just after, the tc-nightly mtg) via irc: <Callek> sooo, we create the balrog manifest from "signed" -- I recognize we need balrog to depend on the signing of the apk for fennec, but for desktop we'll be signing some needed stuff in two passes, .mar and .tar.bz2 -- which does balrogworker need to depend on (or does it need both)? <jlund>: looks like for desktop we only care about the .mar in terms of balrog submission 11:09:25 yeah. we don't care about hte tar.bz2

Comment on attachment 8806972 [details] Bug 1282187 - create task that publishes nightlies on update server via scriptworker -> balrogscript. https://reviewboard.mozilla.org/r/90212/#review92136 r+ with two comments that are not date blocking but should be addressed or commented on. ::: taskcluster/taskgraph/task/balrog.py:19 (Diff revision 2) > + """ > + > + @classmethod > + def get_inputs(cls, kind, path, config, params, loaded_tasks): > + if config.get('kind-dependencies', []) != ["beetmover"]: > + raise Exception("Balrog kinds must depend on builds or signing builds") copy/paste from beetmoverscript taskgraph? shouldn't this be we depend on beetmover tasks? ::: taskcluster/taskgraph/transforms/task.py:318 (Diff revision 2) > Required('update_manifest'): bool, > + }, { > + Required('implementation'): 'balrog', > + > + # taskid of the signed beetmoved task > + Required('taskid', default="invalid"): taskref_or_string, should we include the payload requirements in this schema like beetmover tasks?

(In reply to Jordan Lund (:jlund) from comment #12) > Comment on attachment 8806972 [details] > Bug 1282187 - create task that publishes nightlies on update server via > scriptworker -> balrogscript. > > https://reviewboard.mozilla.org/r/90212/#review92136 > > r+ with two comments that are not date blocking but should be addressed or > commented on. > > ::: taskcluster/taskgraph/transforms/task.py:318 > (Diff revision 2) > > Required('update_manifest'): bool, > > + }, { > > + Required('implementation'): 'balrog', > > + > > + # taskid of the signed beetmoved task > > + Required('taskid', default="invalid"): taskref_or_string, > > should we include the payload requirements in this schema like beetmover > tasks? *Maybe*. I wasn't really a fan of potential duplication, nor am I a big fan of a direct payload passthrough outside of `implementation` of the worker (below). Either way, this isn't hard to change.

Comment on attachment 8806972 [details] Bug 1282187 - create task that publishes nightlies on update server via scriptworker -> balrogscript. https://reviewboard.mozilla.org/r/90212/#review92298 ::: taskcluster/taskgraph/transforms/balrog.py:84 (Diff revision 3) > + 'description': "{} Balrog".format( > + dep_job.task["metadata"]["description"]), > + # do we have to define worker type somewhere? > + 'worker-type': 'scriptworker-prov-v1/balrogworker-v1', > + 'worker': {'implementation': 'balrog', > + 'taskid': {"task-reference": There's something wrong here - it doesn't properly subtitute later on. "payload": { "signing_cert": "dep", "parent_task_artifacts_url": "https://queue.taskcluster.net/v1/task/{u'task-reference': u'<beetmover>'}/artifacts/public" },

Landed a bustage fix... https://hg.mozilla.org/projects/date/rev/b5ec815f13d099e6a35dab57ba6eb49f51b8ec8a

Do you mind landing another to change `_` to `-` in that description attribute? The `_` in the payload are fine, but everything *before* the task definition should be `-`-separated.

Comment on attachment 8806972 [details] Bug 1282187 - create task that publishes nightlies on update server via scriptworker -> balrogscript. https://reviewboard.mozilla.org/r/90212/#review92298 > There's something wrong here - it doesn't properly subtitute later on. > > "payload": { > "signing_cert": "dep", > "parent_task_artifacts_url": "https://queue.taskcluster.net/v1/task/{u'task-reference': u'<beetmover>'}/artifacts/public" > }, Thanks Callek for pushing fix for this - https://hg.mozilla.org/projects/date/rev/b5ec815f13d099e6a35dab57ba6eb49f51b8ec8a