Many reftests compare a "test" file to a "reference" file, as captured in a manifest with something like: == test.html test-ref.html 'mach reftest test.html' runs the test. 'mach reftest test-ref.html' does not run the test and complains REFTEST ERROR | EXCEPTION: No tests to run The issue is the same when the reference file is requested directly on the runreftest.py command line. Currently, when a reftest reference file is modified in a changeset, test-verify tries to run reftests for that reference file, resulting in this error. If the corresponding test file is also modified, that test is run for verification, and that succeeds. I think the desired behavior is: - when the test and reference file are both modified, the test is verified - when either the test or the reference are modified, the test is verified

An example: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=be79515b9f2afc703b5bcfc14546d9dad7692437&filter-searchStr=test-verify&filter-tier=1&filter-tier=2&filter-tier=3

The python reftest manifest does not provide enough information to differentiate the test from the reference...and even if it did, it would be awkward to use that in test verification. Instead, I make use of the convention that reference files are usually named "<test>-ref.html" to detect and differentiate reference files and handle them correctly. Works great when reference file is modified alone or when both test and reference are modified: https://treeherder.mozilla.org/#/jobs?repo=try&revision=8976b22e219fe0d7b5ea8806a6e8b3cfa2a2c284&filter-tier=1&filter-tier=2&filter-tier=3 https://treeherder.mozilla.org/#/jobs?repo=try&revision=3760d172819c0795113c420c25ab7894758bf2e7&filter-tier=1&filter-tier=2&filter-tier=3

Comment on attachment 8912693 [details] [diff] [review] improve reftest reference file verification Review of attachment 8912693 [details] [diff] [review]: ----------------------------------------------------------------- ::: testing/mozharness/mozharness/mozilla/testing/verify_tools.py @@ +147,5 @@ > for file in files: > if suite in ['reftest', 'crashtest']: > file = os.path.join(self.reftest_test_dir, file) > + if suite == 'reftest': > + nonref = file.replace('-ref.', '.') about about the reference of about:blank or other oddities? we should find a way to write unittests for these edge cases.

(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #3) > about about the reference of about:blank or other oddities? we should find > a way to write unittests for these edge cases. I think about:blank is okay: Usually the test is '== <test>.html about:blank'. If <test>.html changes, we request that test be run from the harness, and all's well. (If about:blank changes, we have bigger problems!!)

Pushed by gbrown@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/45f0f5d939ee Improve test-verify behavior when reftest reference file is modified; r=jmaher