We're currently doing a full checkout for toolchain tasks even though they need very few files. They are a prime candidate for sparse checkouts. And, I've written this patch a few times because I was annoyed with how long Try pushes were taking. So I'm just going to submit it :)

Comment on attachment 8910476 [details] Bug 1401725 - Use sparse checkouts for toolchain tasks; https://reviewboard.mozilla.org/r/181906/#review187430 ::: build/sparse-profiles/toolchain-build:4 (Diff revision 4) > +%include build/sparse-profiles/mach > + > +[include] > +path:browser/config/tooltool-manifests/ > + > +path:build/ > +path:taskcluster/ > + I wonder if it wouldn't be better to just %include build/sparse-profiles/taskgraph, which should be a superset (and if not, there's a problem in the taskgraph)

Comment on attachment 8910476 [details] Bug 1401725 - Use sparse checkouts for toolchain tasks; https://reviewboard.mozilla.org/r/181906/#review187430 > I wonder if it wouldn't be better to just %include build/sparse-profiles/taskgraph, which should be a superset (and if not, there's a problem in the taskgraph) I agree. I've made the change. The toolchain profile still adds tools/rewriting/ because of that one toolchain task. That's kinda wonky to me and is possibly deserving of its own bug. I'll file something.

Pushed by gszorc@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/af6c2544903e Use sparse checkouts for toolchain tasks; r=glandium

I had to back this out for the following decision task failure: https://public-artifacts.taskcluster.net/PcpguSSvR2qaFMuhPLwQBA/0/public/logs/live_backing.log [task 2017-09-21T14:39:23.168306Z] "GET /integration/autoland/json-automationrelevance/af6c2544903e3281baf169b682daeef515f77ee3 HTTP/1.1" 200 None [task 2017-09-21T14:39:23.174459Z] 1 commits influencing task scheduling: [task 2017-09-21T14:39:23.174537Z] af6c2544903e Bug 1401725 - Use sparse checkouts for toolchain tasks; r=glandium [task 2017-09-21T14:39:23.546879Z] "GET /index/v1/task/gecko.cache.level-3.toolchains.v1.linux64-binutils.2856d9293652b9192c77948dfb65c9a55fc465d4d13e99b509a43d665c39e28a HTTP/1.1" 404 452 [task 2017-09-21T14:39:23.548141Z] Traceback (most recent call last): [task 2017-09-21T14:39:23.548250Z] File "/builds/worker/checkouts/gecko/taskcluster/mach_commands.py", line 154, in taskgraph_decision [task 2017-09-21T14:39:23.548347Z] return taskgraph.decision.taskgraph_decision(options) [task 2017-09-21T14:39:23.548522Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/decision.py", line 138, in taskgraph_decision [task 2017-09-21T14:39:23.548657Z] write_artifact('task-graph.json', tgg.morphed_task_graph.to_json()) [task 2017-09-21T14:39:23.548813Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 184, in morphed_task_graph [task 2017-09-21T14:39:23.549009Z] return self._run_until('morphed_task_graph') [task 2017-09-21T14:39:23.549142Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 289, in _run_until [task 2017-09-21T14:39:23.549275Z] k, v = self._run.next() [task 2017-09-21T14:39:23.549387Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 276, in _run [task 2017-09-21T14:39:23.549495Z] do_not_optimize) [task 2017-09-21T14:39:23.549667Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 42, in optimize_task_graph [task 2017-09-21T14:39:23.549785Z] existing_tasks=existing_tasks) [task 2017-09-21T14:39:23.549934Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 96, in annotate_task_graph [task 2017-09-21T14:39:23.550046Z] opt_result = optimize_task(task, params) [task 2017-09-21T14:39:23.550174Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 54, in optimize_task [task 2017-09-21T14:39:23.550274Z] opt_result = opt_fn(task, params, *args) [task 2017-09-21T14:39:23.550354Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 175, in opt_index_search [task 2017-09-21T14:39:23.550472Z] use_proxy=bool(os.environ.get('TASK_ID'))) [task 2017-09-21T14:39:23.550542Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/taskcluster.py", line 90, in find_task_id [task 2017-09-21T14:39:23.550637Z] raise KeyError("index path {} not found".format(index_path)) [task 2017-09-21T14:39:23.550732Z] KeyError: u'index path gecko.cache.level-3.toolchains.v1.linux64-binutils.2856d9293652b9192c77948dfb65c9a55fc465d4d13e99b509a43d665c39e28a not found'

Backout by gszorc@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/e66f516aa655 Backed out changeset af6c2544903e for uncovering a taskgraph bug

Comment on attachment 8910765 [details] Bug 1401725 - Catch HTTP 404 errors during taskgraph optimization; https://reviewboard.mozilla.org/r/182240/#review187562 See the last cset in bug 1383880.

Pushed by gszorc@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/bffb6b9d7212 Use sparse checkouts for toolchain tasks; r=glandium