This might live as a poc or expand into something more but... There is some value in generating a list of tasks similar to all_builders.txt [1] that could be used when selecting what tasks to submit. Initially this could be used for mozci to build the try syntax to pass to `mach taskcluster-graph`. [1] http://people.mozilla.org/~armenzg/permanent/all_builders.txt

Bug 1174313 - Mach target for building job list for all/specific projects

Comment on attachment 8622647 [details] MozReview Request: Bug 1174313 - Mach target for building job list for all/specific projects Here is a quick hacky-please-dont-judge-me mach target for generating a list of jobs based on our in tree job flags. Either you can pass in '--project' and limit the output, or don't pass it in and get everything. By default our mach taskcluster-graph target will take any project, and if we don't have a branch specific config, it will default to a base set of jobs. I think this format matches the example you showed me. Here is the output of not passing in --project: https://gist.github.com/gregarndt/9dadb0ad43e10508aabf

https://reviewboard.mozilla.org/r/11423/#review9787 ::: testing/taskcluster/mach_commands.py:470 (Diff revision 1) > + def create_job_list(self, **params): Add project=None to the signature. Could you also please add a doc string under the function?

The output looks good. However, I'm still trying to figure out how to use to make it more useful for the scheduling work we're looking into. For instance, this job will not have any jobs triggered in taskcluster: https://treeherder.allizom.org/#/jobs?repo=try&revision=1b891687a19f If I want to schedule "linux64_gecko try opt test gaia-build-unit", how could I map it to something that will create the correct task graph?

Comment on attachment 8622647 [details] MozReview Request: Bug 1174313 - Mach target for building job list for all/specific projects https://reviewboard.mozilla.org/r/11423/#review9897 Ship It!

Comment on attachment 8622647 [details] MozReview Request: Bug 1174313 - Mach target for building job list for all/specific projects I realized that the **params is the way mach passes parameters into a function so ignore what I said.

> If I want to schedule "linux64_gecko try opt test gaia-build-unit", how > could I map it to something that will create the correct task graph? so that output is <build flag> <project> <build type> <build or test> <test flag> this would translate in a try command: try -b o -p linux64_gecko -u gaia-build-unit I tried it out, and here are my steps. This is somewhat painful so I hope that we can work on making it better: 1. Pushed to try with flags that didn't include building b2g-desktop and running gaia-build-unit... 2. Remember that oh wait, I want to build b2gdesktop, so I took the hg commit hash (3d4efab3bc24) that was pushed to try and looked it up here: https://treeherder.mozilla.org/api/project/try/revision-lookup/?revision=3d4efab3bc24 3. Used the revision_hash returned by that to construct the taskcluster-graph command I need: mach taskcluster-graph --project try --message "try: -b o -p linux64_gecko -u gaia-build-unit" --base-repository http://hg.mozilla.org/mozilla-central --head-repository http://hg.mozilla.org/try --head-rev 3d4efab3bc24 --owner garndt@mozilla.com --revision-hash 4504e49c9b5d2d5f4ec2d2945e35652a03cc66cc | taskcluster run-graph --verbose Here you can see my try push that didn't trigger the b2g build, and then I added the jobs by using `mach taskcluster-graph` https://treeherder.allizom.org/#/jobs?repo=try&revision=3d4efab3bc24

garndt, this is exactly what I had in mind! I will make it easier and propose an API to help building tools for this purpose. We should review this together to see what makes sense for such API.

Why does --base-repository point to mozilla-central instead of try? What was the --revision-hash again? This is the error that I get. MacAir mozilla-inbound hg:[default!] $ ./mach taskcluster-graph --project try --message "try: -b o -p linux64_gecko -u gaia-build-unit" --base-repository http://hg.mozilla.org/mozilla-central --head-repository http://hg.mozilla.org/try --head-rev 1b891687a19f --owner armenzg@mozilla.com --revision-hash 17e1e266875bf3eda8d80008e3ccb0a2d7796d49 | taskcluster run-graph --verbose Taskgraph creation error Error: Authorization Failed { "message": "Authorization Failed", "error": { "info": "None of the scope-sets was satisfied", "scopesets": [ [ "queue:route:tc-treeherder-stage.try.17e1e266875bf3eda8d80008e3ccb0a2d7796d49", "docker-worker:capability:device:loopbackVideo", "queue:create-task:aws-provisioner-v1/test-c4-2xlarge", "queue:create-task:aws-provisioner-v1/build-c4-2xlarge", "docker-worker:image:taskcluster/tester:0.3.5", "queue:route:index.buildbot.revisions.1b891687a19f.try.linux64_gecko", "queue:define-task:aws-provisioner-v1/test-c4-2xlarge", "queue:define-task:aws-provisioner-v1/build-c4-2xlarge", "queue:route:index.buildbot.branches.try.linux64_gecko", "queue:route:index.gecko.v1.try.revision.linux.1b891687a19f.b2g-desktop.opt", "docker-worker:cache:workspace-b2g-desktop-objects-opt", "docker-worker:image:taskcluster/builder:0.5.5", "docker-worker:cache:tc-vcs", "queue:route:index.gecko.v1.try.latest.linux.b2g-desktop.opt", "queue:define-task:aws-provisioner-v1/b2gtest", "queue:route:tc-treeherder.try.17e1e266875bf3eda8d80008e3ccb0a2d7796d49", "docker-worker:capability:device:loopbackAudio", "docker-worker:cache:linux-cache", "queue:define-task:aws-provisioner-v1/b2g-desktop-opt" ] ] } }

regarding base-repository, this was taken from our decision task logic [1]. My understanding is that try pushes are pushes applied on top of m-c, but perhaps I'm mistaken about that. re: authorization failure, I think this is because of the scopes for device capabilities. Since they are just loopback devices, and part of every task, I added the scopes for temporary credentials. These changes can take up to an hour to propagate and you will need to log out and back in again I think. Let me know if there is anything else I can help with. [1] https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/decision/try.yml#48

I'm thinking of closing this bug for now and re-open it once we have a clear plan (to be discussed on Whistler). Thanks a lot for your help and for that patch! It's pretty useful and I think it will very useful once I know how to use the generated data. What do you think?

and: > What was the --revision-hash again? Is that an internal value to treeherder?

Yup, I agree that we could close this now. --revision-hash is a hash that treeherder generates for a set of jobs (at least that's what I remember). mozilla-taskcluster has some code for calculating it, which I think is a copy from treeherder: https://github.com/taskcluster/mozilla-taskcluster/blob/master/src/treeherder/resultset.js#L15-40

We decided not to land this. Nevertheless, it was very informative and helped us understand the issue better! Thanks Greg!

Resetting Version and Target Milestone that accidentally got changed...

This is actually useful for me to generate the data necessary to determine what new jobs we're adding after some code changes.

Not working on this at the moment.

I think this is 100% bitrotted, and probably irrelevant, after the in-tree taskgraph stuff.