The template is at [1], and we can use that, BUT: a) we have to make sure the build_number is correct every time, and b) we trigger the release off tip, rather than a specified revision. Or we have to fix the revision info in the hook before triggering. This is onerous and mistake-prone enough that we should just fix the releasetasks->fennec build graph step. [1] https://tools.taskcluster.net/hooks/#project-releng/candidates-fennec-dev

also, the current hook points at jamun. But the non-dev one would point at m-b, if we had one.

I'm looking at this now, but still trying to understand how to implement

I'm not sure how yet either. I'm going to take a look at it just to make sure this is a reasonable request; let's add our findings here?

Looks like we have {{ revision }} and {{ buildNumber }} already for some tasks. We can create a new role for releases. This looks doable, though I definitely don't know details.

Per Callek: "There is api to get and set hook data. We can probably set the hook via a relpromo sanity check for rev and build number, until a better thing exists, if we want." This is possible, since cron tasks use it. If we're able to submit new hook data, then triggering the hook, a) that's one solution, and allows us to have a hook afterwards if we want, which may be nice, but b) it also seems like more steps than submitting it directly?

Reading the api documentation, it seems we might be able to use triggerHook with a specific payload, might need getTriggerToken as well https://docs.taskcluster.net/reference/core/taskcluster-hooks/references/api#triggerHook Going to investigate more

So I have some initial code to trigger the hook, (payload data not complete). Not sure what client this should run with with adequate scopes to trigger the hook within release promotion sanity check, as suggested above.

I might be understanding this wrong, but: * we need a task with enough scopes to trigger the build graph, via hook or other means * we have a initial graph, triggered by releasetasks, which will cause the build graph to trigger If we have a superset of scopes in the releasetasks decision task, then the task that triggers the build graph can have the appropriate subset of scopes to either create the build decision task itself, or trigger a hook. Does that work, or am I missing something?

(Also, having the candidates_fennec decision task as part of the releasetasks initial graph still seems simpler to me than updating a hook and then triggering, but I'm going to leave implementation details to the implementer.)

kmoir> Kim Moir aki: so I understand what you are saying re the including the candidates_fennec decision task, it indeed does seem easier 2:34 PM <aki> kmoir: yeah, we were just discussing it. it's another task in the graph, which looks like the hook's task definition 2:34 PM <kmoir> Kim Moir so I would translate the information here https://queue.taskcluster.net/v1/task/DIqtGj1oSJWaB1gWRGxRNQ (task definition for hook) 2:34 PM <aki> so i think that means another template 2:35 PM <kmoir> Kim Moir to a template in the release graphs for fennec? 2:35 PM <aki> yes. we'd have to templatize the BUILD_NUMBER and i think the GECKO_HEAD_REV 2:35 PM to {{ buildNumber }} and {{ revision }} i think 2:35 PM templatizing the requester email is a bonus 2:36 PM GECKO_HEAD_REF too maybe 2:36 PM <kmoir> Kim Moir okay 2:36 PM — aki looks at the on-push decision task def 2:36 PM <aki> https://queue.taskcluster.net/v1/task/DMayDarvTByhzQIceGinmA sets both GECKO_HEAD_REF and GECKO_HEAD_REV 2:38 PM <kmoir> Kim Moir okay will go start on that 2:38 PM <aki> let me know if you need a hand with anything 2:38 PM <kmoir> Kim Moir okay sure 2:38 PM thanks

Here are the changes I have so far, would like to know if I'm on the right track https://github.com/kmoir/releasetasks/pull/1/files

Commented in the PR; I think you're on the right track!

From the comments, :kmoir is working on this.

Tested kmoir's PR [1] today, similarly to how I tested the second graph in the dev-staging environment. I'll follow-up with some minor tweaks for the graph itself in the same PR. Some iterations are still needed to make it work, but good news is, we've gotten our selves a scheduled subgraph[2] via the [3]. We need to iterate from here to see what's going wrong with the tasks. There seem to be some quotes issues for starters and maybe something else. We'll see. Note: we needed to alter the buildpromotion client from the release-runner-dev-instance[4] to also encompass the hook-candidates-dev scopes. That is adding "assume:hook-id:project-releng/candidates-fennec-dev" there and under the superset of scopes from ~/releasetasks/templates/mobile/release_graph.yml. [1]: https://github.com/mozilla/releasetasks/pull/218/files#diff-cf87c293dd164e9050bc14f472753a26R17 [2]: https://tools.taskcluster.net/task-group-inspector/#/xZGmvfEmSVqgM1wTZYF6cA/qL9gpxrKQXG7q7HL1j-JaA?_k=wwf8uk [3]: https://github.com/mozilla/releasewarrior/blob/master/how-tos/relpro.md#how-1 [4]: https://tools.taskcluster.net/auth/clients/#project%252freleng%252freleasepromotion%252fproduction

@kmoir: sorry I hijacked your PR but I felt it was easier rather than melding and merging and then deal with bitrot. I updated your PR with a few minor tweaks - https://github.com/mozilla/releasetasks/pull/218/commits/d71e6a916a57c8cc045a35dd29fcf1445b172bec If we want that fennec_candidates toggled by a variable, we need to tweak tools repo as well. For that reason, I enforced it for now within release_graph.yml

Following up with some more information and status update here: a) The last[1] graph triggered on Friday resulted in green tasks for source generation and signing the source. At least those two tasks are working which is great \o/ b) I played a bit more today with the decision task which needed just few minor yml syntax fixes. Triggered a bunch of graphs, the last one working is here[2]. Eventually the decision task turned green and generated this staging-fennec beta build graph beauty from [3] \o/ I updated the PR. c) we still need to debug the "beetmover_generation" which blocks 'source builder's successful artifacts to be transfered to S3. The logs can be found at [4]. Something is fishy as I can't reproduce it locally. The template for the task is 99% identical to the firefox one which works/worked greatly for all past week releases. I'll debug more tomorrow morning. Note to self: * to be honest, I was expecting the fennec-candidate staging beta tasks to be scheduled within the graph itself (so under the source builder, checksum and decision task). Not sure why they're not there but I may have understood wrong how these things work within TC. I need to read more in the docs. * I think I broke some tests within km.oir's PR with my patches. sorry for that, I'll fix those tomorrow * if we're fast enough, we might get this working so that we can trigger the official beta 1 via this method. Otherwise, we can default to what :aki said here [5]. \o/ [1]: https://tools.taskcluster.net/task-group-inspector/#/xZGmvfEmSVqgM1wTZYF6cA/x481LGKbQV2tebaFV6zBJg?_k=whcyvv [2]: https://tools.taskcluster.net/task-group-inspector/#/oj8z6k-AQruZ1VbwJLILBQ/_EQQGvwGQWC5FQq8fSwc4A?_k=hkku0d [3]: https://tools.taskcluster.net/task-group-inspector/#/_EQQGvwGQWC5FQq8fSwc4A?_k=vt2njp [4]: https://public-artifacts.taskcluster.net/0M9vFpjQSY2Mg7J4l1O1yg/0/public/logs/live_backing.log [5]: https://gist.github.com/escapewindow/5819efb7bab47be9868c7375317efaae#2017-03-03

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #16) > Following up with some more information and status update here: Awesome work! > Note to self: > * to be honest, I was expecting the fennec-candidate staging beta tasks to > be scheduled within the graph itself (so under the source builder, checksum > and decision task). Not sure why they're not there but I may have understood > wrong how these things work within TC. I need to read more in the docs. This is as expected. The decision task is part of the releasetasks graph 1, but it generates its own graph and submits it separately. Each task in the candidates_fennec graph has a taskGroupId that corresponds to the decision task's taskId (this is convention in mach taskgraph). Which is why I was waffling between 2 graphs and 3 graphs: the first releasetasks graph triggers another candidates_fennec graph, and the 2nd releasetasks graph triggers push-to-releases. > * I think I broke some tests within km.oir's PR with my patches. sorry for > that, I'll fix those tomorrow I think it needed some changes to be landable, so no worries. > * if we're fast enough, we might get this working so that we can trigger the > official beta 1 via this method. Otherwise, we can default to what :aki said > here [5]. a) awesome, and b) if we do get ready to launch beta 1 build 1 with this method, we probably need to change the task's scopes. Instead of using the assume:hook-id:project-releng/candidates-fennec-dev scope (which is jamun-specific), we should use the assume:project:releng:release:mozilla-beta scope (which is m-b specific). https://tools.taskcluster.net/auth/roles/#project:releng:release:mozilla-beta

(In reply to Aki Sasaki [:aki] from comment #17) > > Note to self: > > * to be honest, I was expecting the fennec-candidate staging beta tasks to > > be scheduled within the graph itself (so under the source builder, checksum > > and decision task). Not sure why they're not there but I may have understood > > wrong how these things work within TC. I need to read more in the docs. > > This is as expected. The decision task is part of the releasetasks graph 1, > but it generates its own graph and submits it separately. Each task in the > candidates_fennec graph has a taskGroupId that corresponds to the decision > task's taskId (this is convention in mach taskgraph). Which is why I was > waffling between 2 graphs and 3 graphs: the first releasetasks graph > triggers another candidates_fennec graph, and the 2nd releasetasks graph > triggers push-to-releases. Yep, it now makes sense, thanks for clarifying this Aki! > > * if we're fast enough, we might get this working so that we can trigger the > > official beta 1 via this method. Otherwise, we can default to what :aki said > > here [5]. > > a) awesome, and b) if we do get ready to launch beta 1 build 1 with this > method, we probably need to change the task's scopes. Instead of using the > assume:hook-id:project-releng/candidates-fennec-dev scope (which is > jamun-specific), we should use the > assume:project:releng:release:mozilla-beta scope (which is m-b specific). > > https://tools.taskcluster.net/auth/roles/#project:releng:release:mozilla-beta I agree. Actually, we'll need to tweak a bit more than that: * changing the scope within the template as you mentioned to point to m-b + adding the actual role scope within the releng/buildpromotion TC client that lies beneath the release runner production instance, if not already * get r? from :rail on kmoir's PR and land that to master in releasetasks * (done) land this under tools - https://bugzilla.mozilla.org/show_bug.cgi?id=1343990#c8 I've changed the branches and added some debuggin code on the dev-release runner instance. But we won't be able to do the same for the production RR instance, so all the releasetasks / tools changes need to land in their corresponding master branches beforehand.

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #16) > c) we still need to debug the "beetmover_generation" which blocks 'source > builder's successful artifacts to be transfered to S3. The logs can be found > at [4]. Something is fishy as I can't reproduce it locally. The template for > the task is 99% identical to the firefox one which works/worked greatly for > all past week releases. I'll debug more tomorrow morning. Yet another nut cracked and strikedthrough! Apparently I was using a week-old revision while triggering the release graph 1. Johan pointed out that I'm missing these[1] changes which have been uplifted in the meantime. This is why there were differences between firefox and fennec graphs. I retriggered the graph and we're past that failing point with high chances of seeing the beetmover-relates tasks go green! \o/ Last graph triggered lies here[2] while the decision task generated graph lies here[3]. If this works, we're done with tasks from releasetasks graph 1 and need to move on to debugging the signing-tasks which fail in the decision-task graph for CoT issues. I suspec it's a matter of scopes or something, I'll double check with :jlorenzo who has better understading than me here. jlorenzo++ for the help! [1]: https://hg.mozilla.org/projects/jamun/rev/566f16455446 [2]: https://tools.taskcluster.net/task-group-inspector/#/4n4rC7aJRByyTF6G0TRGHg/Kco3kErqSs2XjZQxTxhYmQ?_k=sduock [3]: https://tools.taskcluster.net/task-group-inspector/#/JFeTDIFzSM6AQxlZRC55hQ?_k=82wqe3

Sum-up of the current terminology so that we all talk in the same language. Graph 1 Contains: * jamun source generationun * jamun source signing * Generate beetmover docker image * [beetmover] mobile jamun move source bundle checksums * [beetmover] mobile jamun move source bundles * mobile jamun checksums builder * [D] jamun candidates_fennec - this is a decision task! The graph above is generated either via: i) hook https://tools.taskcluster.net/hooks/#project-releng/candidates-fennec-dev ii) release runner script https://github.com/mozilla/releasewarrior/blob/master/how-tos/relpro.md#how-1 Example: https://tools.taskcluster.net/task-group-inspector/#/oj8z6k-AQruZ1VbwJLILBQ/ --- Graph 2: Contains * the nightly graph with building, signing, checksums and beetmover stuff The graph above is generated via: i) is generated and submitted by the decision task from Graph 1. That is because the decision task is part of the releasetasks graph 1, but it generates its own graph (this one, graph2) and submits it separately. Each task from this graph have the taskGroupId set to the decision task's taskId (this is convention in mach taskgraph). Example: https://tools.taskcluster.net/task-group-inspector/#/_EQQGvwGQWC5FQq8fSwc4A/ (generated and chained to the decision task from graph 1 above) --- Graph 3: Contains: * push to releases * publish to balrog * post-release tasks (mark as shipped, version bump, bouncer aliases, etc) The graph above is generated via: i) release runner script https://github.com/mozilla/releasewarrior/blob/master/how-tos/relpro.md#how-1 Example: lost my example from Friday but it's basically sa simple graph containing a single task, the "push-to-releases" task. The other post-release builders need some tweaks before they can be enabled.

Status update here: 1. beetmover generate image task is now green 2. the beetmover checksums and beetmover source transfer tasks are failing because of 13:00:20 INFO - Running main action method: upload_bits 13:00:20 INFO - uploading artifacts to s3... 13:00:21 INFO - uploading 'source_checksum' deliverable for 'all' locale 13:00:21 INFO - Writing to file /mozharness/build/firefox-53.0b1.source.checksums.beet 13:00:21 INFO - Contents: 13:00:21 INFO - f16d9c56295237655ae91653f9a0435e478cc46786dc474be5a19650e7a0d995f5799ff731b8ddcf462d325a5bf7f74cd11d23616ecd1255c5b433ebde4500c8 sha512 654 source/firefox-53.0b1.source.checksums 13:00:21 INFO - uploading to s3 with key: pub/firefox/candidates/53.0b1-candidates/build3/source/firefox-53.0b1.source.checksums 13:00:21 INFO - Checking if `pub/firefox/candidates/53.0b1-candidates/build3/source/firefox-53.0b1.source.checksums` already exists 13:00:21 INFO - Uploading to `pub/firefox/candidates/53.0b1-candidates/build3/source/firefox-53.0b1.source.checksums` 13:00:21 INFO - retry: Calling <lambda> with args: (), kwargs: {}, attempt #1 13:00:21 INFO - retry: attempt #1 caught exception: S3ResponseError: 403 Forbidden 13:00:21 INFO - <?xml version="1.0" encoding="UTF-8"?> 13:00:21 INFO - <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>38F8B88A87203691</RequestId><HostId>eTUEUI/qiQ4nPtJz48/Kf34DG02XAKchJTqYLfLiR3eafiTgQ2N0mvNjRrRhrIAO1DXKwmZZJwk=</HostId></Error> More details: * https://tools.taskcluster.net/task-group-inspector/#/4n4rC7aJRByyTF6G0TRGHg/zQnTHtHORue0mZQhfyKJpA?_k=dm9bh2 * https://tools.taskcluster.net/task-group-inspector/#/4n4rC7aJRByyTF6G0TRGHg/lPP2BrVTTLu5n9icy-3Xmg?_k=j66upj We've slipped somewhere some platform/product wrong strings. Will follow-up on the beetmover script in the tree to fix this. 3. The source generation task is extremely tricky. It shows as completed but there's certain things that look very suspicious: * there are no logs * there are no CoT logs * there is nothing in the task that confirms CoT worked properly. For this particular reason, :jlorenzo thiks somehow the Cot was bypassed. Maybe because the TaskGroupId of the signing source task is not a taskId but instead a taskGroupId. We need to investigate more here. 4. The signing tasks for all the tasks within the second graph (for more stuff on which graph is which, see more in my previous comment[0]- they failing because of CoT verification errors. See more here[1]. All tasks in graph 2 have TaskGroupId set to the decision task from graph 1. I think that step works smoothly in CoT verification but fails when the chain in build against the TaskGroupId of the decision task which is the taskGroupId of graph 1 (so basically Cot verifying the signing task againt their "grand-parent", the graph 1 taskGroupId). Johan suggested this might happen because the graph 1 we generate via releasetasks is a TaskGraphId, not a TaskId, hence the scriptworker CoT fails. There are workarounds to this but we decided to wait for :aki's opinion here as well. I couldn't find a way to alter releasestasks to send a TaskId instead of a TaskGraphId via [2]. From chat, for reference: aki-away> i thought i fixed the decision.decision issue 13:04:52 <aki-away> i suppose it'll look for it even if it doesn't require it? i can patch scriptworker for that probably 13:14:32 <jlorenzo> aki-away: I think we found the issue, releasetasks fill a taskgroupId for us, which shouldn't be the case for the first decision task 13:15:07 <jlorenzo> fix should be: change releasetasks to not generate a taskgroupId (aka task_graph in releasetasks) for Fennec beta 13:16:03 <aki-away> ok. the first decision task may not pass cot unless it runs mach taskgraph, but this is a good first step I teamed-up with :jlorenzo this morning to solve various things in the nightly graph and it was super useful. GG team work, jlorenzo++ [0]: https://bugzilla.mozilla.org/show_bug.cgi?id=1343649#c20 [1]: https://public-artifacts.taskcluster.net/WuA-l_GyQWqfmHlKQTSuTg/0/public/logs/chain_of_trust.log [2]: https://docs.taskcluster.net/reference/platform/scheduler/api-docs

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #21) > 3. The source generation task is extremely tricky. It shows as completed but > there's certain things that look very suspicious: > * there are no logs > * there are no CoT logs > * there is nothing in the task that confirms CoT worked properly. > > For this particular reason, :jlorenzo thiks somehow the Cot was bypassed. > Maybe because the TaskGroupId of the signing source task is not a taskId but > instead a taskGroupId. We need to investigate more here. a) releasetasks graphs don't have CoT enabled, so you can ignore that completely. b) if you wanted to add CoT generation to the source task, I believe you need to set task.payload.features.chainOfTrust: true c) verifying chain of trust on the releasetasks graphs will be broken unless we make the decision task use `mach taskgraph`. > 4. The signing tasks for all the tasks within the second graph (for more > stuff on which graph is which, see more in my previous comment[0]- they > failing because of CoT verification errors. See more here[1]. All tasks in > graph 2 have TaskGroupId set to the decision task from graph 1. I think that > step works smoothly in CoT verification but fails when the chain in build > against the TaskGroupId of the decision task which is the taskGroupId of > graph 1 (so basically Cot verifying the signing task againt their > "grand-parent", the graph 1 taskGroupId). Johan suggested this might happen > because the graph 1 we generate via releasetasks is a TaskGraphId, not a > TaskId, hence the scriptworker CoT fails. There are workarounds to this but > we decided to wait for :aki's opinion here as well. > > I couldn't find a way to alter releasestasks to send a TaskId instead of a > TaskGraphId via [2]. > > From chat, for reference: > aki-away> i thought i fixed the decision.decision issue > 13:04:52 <aki-away> i suppose it'll look for it even if it doesn't require > it? i can patch scriptworker for that probably > 13:14:32 <jlorenzo> aki-away: I think we found the issue, releasetasks fill > a taskgroupId for us, which shouldn't be the case for the first decision > task > 13:15:07 <jlorenzo> fix should be: change releasetasks to not generate a > taskgroupId (aka task_graph in releasetasks) for Fennec beta > 13:16:03 <aki-away> ok. the first decision task may not pass cot unless it > runs mach taskgraph, but this is a good first step Yeah, I think I need to make some scriptworker-side changes. > I teamed-up with :jlorenzo this morning to solve various things in the > nightly graph and it was super useful. GG team work, jlorenzo++ Thanks both of you!

(In reply to Aki Sasaki [:aki] from comment #22) > a) releasetasks graphs don't have CoT enabled, so you can ignore that > completely. > b) if you wanted to add CoT generation to the source task, I believe you > need to set task.payload.features.chainOfTrust: true > c) verifying chain of trust on the releasetasks graphs will be broken unless > we make the decision task use `mach taskgraph`. Makes sense now. We use releasetasks for graph templates + scheduler to submit the graph. As opposed to the "nightly" way, with CoT enabled via the `mach taskgraph` way. Okay, got it, thanks!

https://hg.mozilla.org/build/puppet/rev/d32ff53ee739bb9c58f1ed1f7d434371382e3dee bug 1343649 - update balrog,beetmover,signing scriptworkers to scriptworker 2.5.0. r=callek

https://hg.mozilla.org/build/puppet/rev/745141d6f938211c6b96f6943f0bd0f2f7791f98 bug 1343649 - update balrog,beetmover,signing scriptworkers to scriptworker 2.6.0. r=callek

Status update: * release graph 1 - source builder needs more attention than I initially thought. I'll file a separate bug to track it as it's not blocking the release for now. It will require to make some new changes and configs in mozharnes. Additionally, mozharness relpro beetmover needs some masssaging too to deal with fennec as the S3 paths are 'firefox' hardcoded. * decision task nighyl graph: I've triggered another staging release based on jamun and we're past the CoT failing point in signing task. Some beetmoverjos are failing due to build_number missing. I'm investigating now to come up with a fix, might be my fault. Graph 1 - https://tools.taskcluster.net/task-group-inspector/#/DvL2un6XQ2afI8WJ5Bef5g/ Decision task generated nightly graph: https://tools.taskcluster.net/task-group-inspector/#/MM-8iSmKQK2M3spbiZ7bWg/

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #26) > Status update: > * release graph 1 - source builder needs more attention than I initially > thought. I'll file a separate bug to track it as it's not blocking the > release for now. It will require to make some new changes and configs in > mozharnes. Additionally, mozharness relpro beetmover needs some masssaging > too to deal with fennec as the S3 paths are 'firefox' hardcoded. > * decision task nighyl graph: I've triggered another staging release based > on jamun and we're past the CoT failing point in signing task. Some > beetmoverjos are failing due to build_number missing. I'm investigating now > to come up with a fix, might be my fault. > > Graph 1 - > https://tools.taskcluster.net/task-group-inspector/#/DvL2un6XQ2afI8WJ5Bef5g/ > Decision task generated nightly graph: > https://tools.taskcluster.net/task-group-inspector/#/MM-8iSmKQK2M3spbiZ7bWg/ Fixed beetmoverscript here Bug 1343649 - fix build_number and version within various action types. r=jlorenzo https://github.com/mozilla-releng/beetmoverscript/commit/549e60a5fdc8f2a967624198126a5c87168aeb4b Bumped beetmoverscript to 0.4.1 https://hg.mozilla.org/build/puppet/rev/a3d2ceca1514 Wrote a quick script to get all the failing beetmover jobs from last triggered graph and rerun those. All beetmover jobs resulted in green tasks and the artifacts are correctly uploaded to http://bucketlister-delivery.stage.mozaws.net/pub/mobile/candidates/53.0b1-candidates/build3/ Again last decision task generated graph lies under - https://tools.taskcluster.net/task-group-inspector/#/MM-8iSmKQK2M3spbiZ7bWg/

So in order to trigger the first graph via hook, we added the necessary hook[1] on Friday but apparently we forgot to also add the corresponding role. While attempting to trigger the hook today for Fennec 53.0b1 it failed with the following message: Missing at least one of the following scopes: queue:create-task:aws-provisioner-v1/gecko-decision queue:define-task:aws-provisioner-v1/gecko-decision queue:task-group-id:gecko-level-3/H2yiqYp_RXqI1ZYcAfbQrA queue:schedule-task:gecko-level-3/H2yiqYp_RXqI1ZYcAfbQrA/H2yiqYp_RXqI1ZYcAfbQrA Only scopes present are: have assume:hook-id:project-releng/candidates-fennec-beta Not sure I got this right but in my understanding each hook has to have a corresponding role in order to be able to trigger tasks. Since we didn't, it default to a "fictional" role which it's only scope is itself "assume:role ...". I created the role[2] associated with the hook and granted it only one scope "assume:project:releng:release:mozilla-beta" which pretty much expands to our needs. I vimdiffed the differences against the dev one [3] and things look the same way (except the obvious jamun vs "mozilla-beta" differences). Triggering the graph worked - https://tools.taskcluster.net/task-group-inspector/#/SejnPXCNRMSw1UdcP4VazA?_k=84xa9s [1]: https://tools.taskcluster.net/hooks/#project-releng/candidates-fennec-beta [2]: https://tools.taskcluster.net/auth/roles/#hook-id:project-releng%252fcandidates-fennec-beta [3]: https://tools.taskcluster.net/auth/roles/#hook-id:project-releng%252fcandidates-fennec-dev

* for betas, we no longer need two graphs, we can just have one single releasetasks graph, just like we have for Firefox. (the additional nightly graph generated via the decision task can be bound via the task). * we still have 5-6 weeks to improve the beta cycle before we hit the merge again so I think it's safe to focus on betas now.

Comment on attachment 8846710 [details] Bug 1343649 - Add candidates_fennec boolean config for release runner. https://reviewboard.mozilla.org/r/119712/#review121554

Comment on attachment 8846710 [details] Bug 1343649 - Add candidates_fennec boolean config for release runner. https://hg.mozilla.org/build/tools/rev/a16e32a3a6ac + bustage fix https://hg.mozilla.org/build/tools/rev/6b539a22fea2

Attempt no. 1 to schedula full beta graph via releasetasks: https://tools.taskcluster.net/task-group-inspector/#/iOGln3OKRIyMzWORVs06dg?_k=3njzgq

Comment on attachment 8846692 [details] Bug 1343649 - meld the graphs into a single one to rule them all. r=rail https://github.com/mozilla/releasetasks/commit/4be3542d0bfd8661b42ff28dc798d2f072138bec

We have variations of (formerly knwown "graph1" from comment 20) that are currently working via the releasetasks so I think it's safe to close this bug now. The remaining related builders (source builder, checkeums + related issues) are tracked in bug 1346892 as part of phase2 effort.