"when: file_changed:" is great, but probably not quite sufficient for bug 1357513. I want a similar mechanism that triggers a job when a test file is changed. "when: file_changed: @TEST@", or "when: test_changed"...something like that. What is a test file? A file listed in an active test manifest, ideally including support files. It looks like the existing TestResolver might do the job. https://dxr.mozilla.org/mozilla-central/rev/a374c35469935a874fefe64d3e07003fc5bc8884/python/mozbuild/mozbuild/testing.py#179 A complication is that a test might be skipped on some platforms, but we don't want that to stop test verification on other platforms. Ideally, test_changed would trigger test verification on platforms where the modified test is enabled; I think a reasonable alternative would be to trigger test verification on all platforms, allowing the harness to recognize the platform skip and no-op where appropriate.

I've had a look at implementing "when: tests-changed" paralleling the approach for "when: files-changed" and gotten that working down to the level of determining whether a path corresponds to a test file. I had hoped to use TestResolver for this, but attempts to instantiate TestResolver in the decision task fail at: https://dxr.mozilla.org/mozilla-central/rev/bdb2387396b4a74dfefb7c983733eed3625e906a/python/mozbuild/mozbuild/testing.py#188 [task 2017-06-02T17:07:02.622846Z] File "/home/worker/checkouts/gecko/taskcluster/mach_commands.py", line 164, in taskgraph_decision [task 2017-06-02T17:07:02.622900Z] return taskgraph.decision.taskgraph_decision(options) [task 2017-06-02T17:07:02.622967Z] File "/home/worker/checkouts/gecko/taskcluster/taskgraph/decision.py", line 136, in taskgraph_decision [task 2017-06-02T17:07:02.623023Z] write_artifact('task-graph.json', tgg.morphed_task_graph.to_json()) [task 2017-06-02T17:07:02.623087Z] File "/home/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 180, in morphed_task_graph [task 2017-06-02T17:07:02.623130Z] return self._run_until('morphed_task_graph') [task 2017-06-02T17:07:02.623187Z] File "/home/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 286, in _run_until [task 2017-06-02T17:07:02.623229Z] k, v = self._run.next() [task 2017-06-02T17:07:02.623288Z] File "/home/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 274, in _run [task 2017-06-02T17:07:02.623329Z] do_not_optimize) [task 2017-06-02T17:07:02.623388Z] File "/home/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 43, in optimize_task_graph [task 2017-06-02T17:07:02.623423Z] existing_tasks=existing_tasks) [task 2017-06-02T17:07:02.623508Z] File "/home/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 128, in annotate_task_graph [task 2017-06-02T17:07:02.623557Z] opt_result = optimize_task(task, params) [task 2017-06-02T17:07:02.623614Z] File "/home/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 83, in optimize_task [task 2017-06-02T17:07:02.623656Z] opt_result = opt_fn(task, params, *args) [task 2017-06-02T17:07:02.623710Z] File "/home/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 262, in opt_tests_changed [task 2017-06-02T17:07:02.623758Z] changed = files_changed.check_tests(params, flavor) [task 2017-06-02T17:07:02.623819Z] File "/home/worker/checkouts/gecko/taskcluster/taskgraph/files_changed.py", line 94, in check_tests [task 2017-06-02T17:07:02.623853Z] if test_match(path, flavor): [task 2017-06-02T17:07:02.623917Z] File "/home/worker/checkouts/gecko/taskcluster/taskgraph/files_changed.py", line 75, in test_match [task 2017-06-02T17:07:02.623959Z] resolver = build_obj._spawn(TestResolver) [task 2017-06-02T17:07:02.624008Z] File "/home/worker/checkouts/gecko/python/mozbuild/mozbuild/base.py", line 623, in _spawn [task 2017-06-02T17:07:02.624045Z] topobjdir=self.topobjdir) [task 2017-06-02T17:07:02.624100Z] File "/home/worker/checkouts/gecko/python/mozbuild/mozbuild/testing.py", line 195, in __init__ [task 2017-06-02T17:07:02.624148Z] self.topsrcdir, 'build', 'gen_test_backend.py'), [task 2017-06-02T17:07:02.624208Z] File "/home/worker/checkouts/gecko/python/mozbuild/mozbuild/base.py", line 469, in _run_make [task 2017-06-02T17:07:02.624249Z] args = self._make_path() [task 2017-06-02T17:07:02.624300Z] File "/home/worker/checkouts/gecko/python/mozbuild/mozbuild/base.py", line 600, in _make_path [task 2017-06-02T17:07:02.624341Z] raise Exception('Could not find a suitable make implementation.') TestResolver runs make("backend.TestManifestBackend") to generate pickle representations of the tests dictionary, which it then reads in and searches to satisfy requests. https://treeherder.mozilla.org/#/jobs?repo=try&revision=fcdfcc2d60de3d44850a9f95c43e6ce5994fe750 https://treeherder.mozilla.org/#/jobs?repo=try&revision=b2ea767b238014f983a92c548a163aba1f62f5cd

The experience in comment 1 reminds me that "proper" examination of test manifests is linked to the build system, and as such cannot be performed pre-build: We can't determine if a file is a test file in the decision task. If instead, the manifest lookup is performed post-build, perhaps in a normal test task, it may be able to use all-tests.pkl, etc generated at build time. Also, post-build, the manifest lookup can be performed in the context of a specific platform, properly accounting for os, debug, etc annotations. Let's try a different approach...