Test verification (TV/TVg/TVw tasks) is initiated in CI when a push modifies files with names that appear to be tests: a quick inspection of the modified file names is performed, comparing to known test name patterns, like "test_*.js". In the TV/TVg/TVw task, mozharness then does a more comprehensive lookup: Load all supported test manifests, lookup modified file names in the manifests to determine if test verification can proceed. This two-stage comparison is inherently inefficient: Sometimes TV tasks are spun up only to find there are no supported tests to verify; sometimes TV tasks are not started even for valid tests.

Increased efficiency is possible if the decision task loads and inspects test manifests.

A complication to keep in mind: Ideally test manifests need to be consulted with a variety of platform-specific mozinfo data, to allow for tests that are skipped on specific platforms.

This idea was part of the original TV design, but was rejected for concerns over performance of the decision task. :ahal points out that a lot has changed since then and the incremental cost may now be negligible.

I imagine there is a similar issue for per-test test coverage, but I am not familiar with that feature and unsure of how much it is used today.

This is slightly tangential to bug 1639324. I think we could solve that other one before this one is finished. Though from a project management perspective, I agree it makes sense to do this first.

Note the reason we're considering this now is that we are loading test manifests anyway for taskgraph chunking:
https://searchfox.org/mozilla-central/rev/5e4d4827aa005d031580d2d17a01bae1af138b2e/taskcluster/taskgraph/util/chunking.py#211

So the additional perf impact of re-using that data for TV should be minimal. Note there's even a guess_mozinfo_from_task function that is used expressly for filtering skipped tests. So that problem is also solved :).

The reason I made this block smart-scheduling is that the TV tasks are often causing unnecessary builds to run with |mach try auto|. So solving this will improve people's perception of it (even if $$ savings is minimal).

Another title for this bug could be "Run TV under |mach try auto|", as fixing this will allow us to run it on try auto pushes again. See also:
https://mozilla-hub.atlassian.net/browse/RELENG-447