I'm trying to run this command: ./mach try fuzzy toolkit/components/microformats/test/marionette/test_standards.py And getting the response: UNKNOWN TEST: toolkit/components/microformats/test/marionette/test_standards.py Is running marionette tests like this supported? If not, what are the steps to support this?

This is not a Marionette harness question. So moving into build system. Please adjust if the component is not the correct one.

The fact that you're getting UNKNOWN_TEST suggests that the build system doesn't know about it, which is strange. Basically this function isn't returning any results: https://searchfox.org/mozilla-central/source/testing/mozbase/moztest/moztest/resolve.py#323 It's possible you need to have a build for it to get recognized. However, even if we solved this, it looks like marionette isn't hooked up to |mach try fuzzy <path>| yet. This second part is easy to fix though and involves adding a task regex here: https://searchfox.org/mozilla-central/source/testing/mozbase/moztest/moztest/resolve.py#68

The tests under testing/marionette/harness/marionette_harness/ don't raise UNKNOWN TEST, just the mentioned above. I'll add the task regex for this bug, so it integrates nicely with bug 1429463. When I'll do comprehensive testing for bug 1429463 I'll open bug reports for outliers like the one above.

After adding the task reged, selecting tests and directories with `mach try fuzzy` works now. Failing try push: https://treeherder.mozilla.org/#/jobs?repo=try&revision=1cf3738e84bc9727fa446c38dd5d714dbe2e283c On linux,, macos and windows: [task 2018-08-21T13:55:00.904Z] 13:55:00 INFO - usage: runtests.py [options] test_file_or_dir <test_file_or_dir> ... [task 2018-08-21T13:55:00.906Z] 13:55:00 INFO - runtests.py: error: Test file(s) not found: browser/components/migration/tests/marionette The path doesn't exist on the test machine probably because there's no complete source checkout. This could be fixed by rewriting the paths in MOZHARNESS_TEST_PATH to point to abs_marionette_tests_dir, here: https://dxr.mozilla.org/mozilla-central/rev/4e56a2f51ad739ca52046723448f3129a58f1666/testing/mozharness/scripts/marionette.py#310. I'd do an interactive task to see how the paths should be mapped, but I don't have the scopes to do that. For Android, it's a different story: [task 2018-08-21T13:58:56.684Z] 13:58:56 INFO - usage: runtests.py [options] test_file_or_dir <test_file_or_dir> ... [task 2018-08-21T13:58:56.685Z] 13:58:56 INFO - runtests.py: error: unrecognized arguments: browser/components/migration/tests/marionette It seems that the Android command parser does not support taking a directory with the tests. I don't get it why this is happening, since it's using the same argument parser (which supports the 'tests' argument). Maybe it's because of the argument ordering?

(In reply to Tudor-Gabriel Vijiala [:tvijiala] from comment #5) > I don't have the scopes to do that. Actually I do, I was just logged out. Trying it out now.

I don't think the first error is happening due to lack of source checkout (none of the "test" tasks have a source checkout). Rather it's likely a bug in testing/mozharness/scripts/marionette.py around how the MOZHARNESS_TEST_PATHS env is handled. If you download a "common.tests.zip" build artifact, you'll see all the marionette test files, e.g "marionette/tests/browser/components/migration/tests/marionette" So possibly we're not mapping that relative path to the proper location in the extracted tests.zip. I'm not totally surprised this isn't working as this code path has likely never been hit until now. I haven't looked into the Android issue yet, but let's see if we can get it working on desktop first.

The paths that we were passing to the Marionette test runner were wrong, as it expects paths relative to abs_marionette_tests_dir.

(In reply to Marco Castelluccio [:marco] from comment #9) > Created attachment 9026011 [details] [diff] [review] > Pass test paths relative to abs_marionette_tests_dir to the Marionette test > runner > > The paths that we were passing to the Marionette test runner were wrong, as > it expects paths relative to abs_marionette_tests_dir. Actually, relative to dirs['abs_test_install_dir']/marionette/tests, I need to update the commit message.

I've landed the first patch in bug 1480141.

Comment on attachment 9026011 [details] [diff] [review] Pass test paths relative to abs_marionette_tests_dir to the Marionette test runner >+ paths = [os.path.join(dirs['abs_test_install_dir'], 'marionette', 'tests', p) It would be great to get this into the `query_abs_dirs()` method, but that can be done at a later time if still necessary. Thanks for the fix!

Pushed by mcastelluccio@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/eed5fefba089 Pass test paths relative to the Marionette test install dir to the Marionette test runner. r=whimboo