The initial implementation of "test verification" only runs on Linux. Running on Android should find intermittent issues specific to those platforms, providing benefit to overall test reliability. The test verification support in desktop_unittest.py needs to be ported to android_emulator_unittest.py. Tests often run slower on Android than on desktop; consider modifying max run time and/or iteration counts on Android.

There are a few other pieces that need to land before I can proceed here, but this bit is straight-forward.

Comment on attachment 8918855 [details] [diff] [review] tc config changes to run test-verify on Android, tier 3 Review of attachment 8918855 [details] [diff] [review]: ----------------------------------------------------------------- this is a great step forward

This updates the mozharness script for android with similar code to the test-verification support in desktop_unittest.py. The suite mapping is a little confusing, but I'll be re-visiting that in bug 1391085. https://treeherder.mozilla.org/#/jobs?repo=try&revision=5f4ab2205596376da645ab2bae8d2bea7948cad9&filter-tier=1&filter-tier=2&filter-tier=3

Comment on attachment 8919415 [details] [diff] [review] update mozharness for test-verify on Android Review of attachment 8919415 [details] [diff] [review]: ----------------------------------------------------------------- rubber stamping this- nothing stands out as wrong. ::: testing/mozharness/scripts/android_emulator_unittest.py @@ +156,5 @@ > dirs['abs_test_install_dir'], 'mochitest') > + dirs['abs_reftest_dir'] = os.path.join( > + dirs['abs_test_install_dir'], 'reftest') > + dirs['abs_xpcshell_dir'] = os.path.join( > + dirs['abs_test_install_dir'], 'xpcshell') do you need one for robocop at all?

(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #4) > ::: testing/mozharness/scripts/android_emulator_unittest.py > @@ +156,5 @@ > > dirs['abs_test_install_dir'], 'mochitest') > > + dirs['abs_reftest_dir'] = os.path.join( > > + dirs['abs_test_install_dir'], 'reftest') > > + dirs['abs_xpcshell_dir'] = os.path.join( > > + dirs['abs_test_install_dir'], 'xpcshell') > > do you need one for robocop at all? Robocop uses the mochitest directory, so I don't need to make allowance for it here. Currently robocop tests are not verified -- the harness does not support --verify and mozharness does not recognize robocop test files. I've filed bug 1409511 to follow-up, eventually.

Pushed by gbrown@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/55f3eaaf482d Schedule test-verify on Android, tier 3; r=jmaher https://hg.mozilla.org/integration/mozilla-inbound/rev/015ee9f1c358 Add test-verify support to Android mozharness script; r=jmaher

https://hg.mozilla.org/mozilla-central/rev/55f3eaaf482d https://hg.mozilla.org/mozilla-central/rev/015ee9f1c358