If a bad path is passed to TV-bf, it runs green and does nothing. I think it should fail and report the bad path.

Some examples of failures and passes at https://treeherder.mozilla.org/#/jobs?repo=try&revision=8ba248923bfe1c04406d76653122e224c04a2caf (I have fixed the lint failure.) Do you have ideas for related logging improvements?

Comment on attachment 8999199 [details] [diff] [review] log requested path, fail if path invalid/not found Review of attachment 8999199 [details] [diff] [review]: ----------------------------------------------------------------- overall this is a positive thing. In terms of what else we can do- I would like to consider turning the job orange or maybe red if we cannot find the path specified. There are 2 cases here: 1) the file is not found 2) we tried to run the file (i.e. launched the browser) and no tests were run If we were to turn the job orange, that would need to account for tests that have duplicate names which we skip, likewise some other random exceptions with names ( like test-oop-extensions: https://searchfox.org/mozilla-central/source/browser/components/extensions/test/browser/browser-remote.ini ) for tv-bf we should be more specific- we are running this with the intention of a specified test and the job should fail. Overall I think you caught the main case (#1 above) and we will have to sort out some details. What other logging can we do- maybe details about total files found, total files not able to run (ex: skip-if), total chunks, paths for finding files. Getting this in sooner will help out.

Pushed by gbrown@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/71af60e17c8b Improve mozharness diagnostics in per-test runs, especially for TV-bf; r=jmaher

Pushed by gbrown@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/a71e4a20a0a6 More mozharness logging updates for test-verify; r=me,a=test-only

A few more log statements, inspired by Joel's feedback. There's no reasonably convenient way to identify skipped reftests, or wpt. I also looked at logging in the taskcluster code, but I think it is better not to clutter the decision task.

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&selectedJob=201290342&revision=615776a8b2ef9e06b39b498cc51bf888657d84c2 [task 2018-09-24T22:55:35.085Z] 22:55:35 INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/mochitest/tests/mochitest.ini (9801 active, 472 skipped) [task 2018-09-24T22:55:35.149Z] 22:55:35 INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/mochitest/chrome/chrome.ini (980 active, 104 skipped) [task 2018-09-24T22:55:35.354Z] 22:55:35 INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/mochitest/browser/browser-chrome.ini (4122 active, 432 skipped) [task 2018-09-24T22:55:35.367Z] 22:55:35 INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/mochitest/a11y/a11y.ini (298 active, 7 skipped) [task 2018-09-24T22:55:35.505Z] 22:55:35 INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/xpcshell/tests/xpcshell.ini (2988 active, 123 skipped) [task 2018-09-24T22:55:36.958Z] 22:55:36 INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/reftest/tests/layout/reftests/reftest.list (23848 tests) [task 2018-09-24T22:55:37.157Z] 22:55:37 INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/reftest/tests/testing/crashtest/crashtests.list (3603 tests) [task 2018-09-24T22:55:38.570Z] 22:55:38 INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/jsreftest/tests/jstests.list (33989 tests) [task 2018-09-24T22:55:38.571Z] 22:55:38 INFO - Per-test run found test testing/mochitest/tests/Harness_sanity/test_sanity.html (plain/None) [task 2018-09-24T22:55:38.572Z] 22:55:38 INFO - 'testing/mochitest/tests/Harness_sanity/test_importInMainProcess.html' has been skipped on this platform.

Pushed by gbrown@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/d66dcdb73140 Improve diagnostics for test-verify and per-test coverage; r=me,a=test-only