Test-Verify should run tests affected by modified support files. This might require chunking if there are many tests in the .ini file which lists the support files and the tests. https://hg.mozilla.org/integration/autoland/rev/5db4200c90a80fd7e966daf0b30e1742ab6e56da modified the support file browser/components/resistfingerprinting/test/mochitest/file_animation_api.html but not the test, the failure of browser/components/resistfingerprinting/test/mochitest/test_animation_api.htm went unnoticed until later: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=67203041d4c9a4dd8f8579df928baeeae084c4ba&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable

Verification based on support files is a fairly hard problem. It seems to me that some test types do not list their support files, or do not fully list dependencies, in manifests. Also, most support files seem to be associated with manifests (default section) rather than individual tests. We need to be careful not to over-test / run out of test time trying to test everything when a common support file is modified. I don't have any specific plans for solving this.

How bad would it be to verify all tests that are in the same directory as the support file (when it isn't listed in the INI)? I'm asking because we have a similar problem for code coverage (if we want to build a tryselector based on coverage data, we need a way to handle support file changes).

(In reply to Marco Castelluccio [:marco] (PTO until March 29) from comment #2) > How bad would it be to verify all tests that are in the same directory as > the support file (when it isn't listed in the INI)? I'm asking because we > have a similar problem for code coverage (if we want to build a tryselector > based on coverage data, we need a way to handle support file changes). It depends on the directory: On the "bad" side, consider dom/base/test, with about 500 tests. On Android, it takes about an hour to run those tests once, so I would expect it would take at least 50 hours (test repetitions + browser start/stops, etc) to run test-verify on the whole directory. On the "good" side, some directories (manifests) have only a few tests.