At the moment, we run one special instrumentation suite in automation: Robocop. We have two others building but not being run: Browser JUnit 3 [1] and Background Services JUnit 3. This meta bug tracks getting those running in automation. There are a lot of pieces to this ticket, but I think broadly we need to: 1) get a new mozharness (ScriptFactory) job running on cedar. 2) write enough mozharness glue to prepare the test environment and execute an in-tree harness. 3) write said in-tree harness.

(In reply to Nick Alexander :nalexander from comment #0) > At the moment, we run one special instrumentation suite in automation: > Robocop. We have two others building but not being run: Browser JUnit 3 [1] > and Background Services JUnit 3. This meta bug tracks getting those running > in automation. > > There are a lot of pieces to this ticket, but I think broadly we need to: > > 1) get a new mozharness (ScriptFactory) job running on cedar. > 2) write enough mozharness glue to prepare the test environment and execute > an in-tree harness. > 3) write said in-tree harness. Bug 903537 tracks writing an in-tree harness for running instrumentation tests. This is the part that I can personally move on most easily.

> 1) get a new mozharness (ScriptFactory) job running on cedar. I've filed Bug 1064010 to track getting something running on cedar. > 2) write enough mozharness glue to prepare the test environment and execute > an in-tree harness. I've filed Bug 1064012 to track getting a mozharness script up and running.

This is like Bug 1238678 but for Android instrumentation tests instead of Android unit tests. It used to be that the former were JUnit 3 and the latter JUnit 4, although now instrumentation tests can use JUnit 4 if the choose too.

Review commit: https://reviewboard.mozilla.org/r/39039/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/39039/

Review commit: https://reviewboard.mozilla.org/r/39041/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/39041/

Review commit: https://reviewboard.mozilla.org/r/39043/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/39043/

Comment on attachment 8728697 [details] MozReview Request: Bug 1064004 - Add Espresso instrumentation test support to Gradle. r?mcomella mcomella: this isn't really ready for review, but the other bits are, so I wanted to pre-flight the groundwork. You can see this working in https://treeherder.mozilla.org/#/jobs?repo=try&revision=0f51765c4ab2&selectedJob=17847321, which just runs one (non-Espresso) instrumentation test. This is performing a --disable-compile-environment build *without Gecko libraries*, so it's not sensible yet; but it leads the way to running DB-only instrumentation tests, at least.

Comment on attachment 8728696 [details] MozReview Request: Bug 1064004 - Add TaskCluster job to run Android instrumentation tests in automation. r?dustin https://reviewboard.mozilla.org/r/39041/#review35877 ::: testing/taskcluster/tasks/builds/android_api_15_androidTest.yml:18 (Diff revision 1) > + - 'docker-worker:cache:level-{{level}}-{{project}}-build-android-api-15-frontend-workspace' This will share the workspace with `testing/taskcluster/tasks/builds/android_api_15_frontend.yml`. Is that intentional?

Comment on attachment 8728695 [details] MozReview Request: Bug 1064004 - Pre: Allow running Android emulators in headless mode. r?gbrown https://reviewboard.mozilla.org/r/39039/#review35891 Can I get some context? Why run with no audio, window, or gpu? Obviously you don't need to for this case, but wouldn't it be better to have audio and gpu available for browser tests? And window for debugging? On another note, I came to the conclusion that different emulator startup logic is needed for interactive (mach) and automated (mozharness) environments. In our existing tier 1 emulator tests, mozharness (android_emulator_unittest.py) starts the emulator, tries to verify, automatically kills and retries and logs diagnostics, etc. A lot of that seemed an inappropriate bother for the mach command (where if it fails 1 time in a 100, you can just run it again) -- something to think about.

Comment on attachment 8728697 [details] MozReview Request: Bug 1064004 - Add Espresso instrumentation test support to Gradle. r?mcomella I still don't have much context on how gradle works but these changes seem reasonable.

We have completed our launch of our new Firefox on Android. The development of the new versions use GitHub for issue tracking. If the bug report still reproduces in a current version of [Firefox on Android nightly](https://play.google.com/store/apps/details?id=org.mozilla.fenix) an issue can be reported at the [Fenix GitHub project](https://github.com/mozilla-mobile/fenix/). If you want to discuss your report please use [Mozilla's chat](https://wiki.mozilla.org/Matrix#Connect_to_Matrix) server https://chat.mozilla.org and join the [#fenix](https://chat.mozilla.org/#/room/#fenix:mozilla.org) channel.