There's a good deal of preparation needed to run a Robocop test on device. Off the top of my head, we: * push the Robocop APK; * start the mochi.test server; * prepare a PAC file; * push a profile to the device; * push a robotium.config to the device; * launch instrumentation with specific configuration options; * and after the test is complete, arrange to kill the process. It's pretty hard to do that from within an IDE. However, IDEs are really good at pushing APKs only when needed and launching instrumentation tests. This ticket tracks adding a flag to |mach robocop| that does everything above up to "launch instrumentation". Instead of launching any tests, |mach robocop --serve| gets everything started, launches Fennec with a test profile, and then just waits. The user can then launch instrumentation tests into this testing environment from the IDE. It's not smooth yet but it's helping me locally.

For context, this is a rehashed version of Part 8 of Bug 929654. That got r- from jmaher, largely of concerns with --no-autorun: see https://bugzilla.mozilla.org/show_bug.cgi?id=929654#c21. That flag got added by another ticket, I wager because doing the right thing is just too painful :( Also from that ticket, I wrote: """ I'm sorry, I didn't give this good context. The purpose of this is to let you "run Fennec in testing mode" and keep the mochi.test web server up, while you rebuild and redeploy the Robocop APK from an IDE. That is, in a terminal, I run |mach robocop --serve| and then I can develop my tests in Eclipse. Killing the mochi.test server after 10 minutes doesn't serve this goal. """ I hope the ticket description elaborates on this.

Bug 1169476 -- Implement |mach robocop --serve|. r?gbrown This adds a flag to |mach robocop| that does everything to run a Robocop test except launch the actual test. Instead of launching the test, it starts the mochi.test server and launches Fennec with a test profile; then it sits and waits forever. This allows regular Java IDEs (IntelliJ, but previously Eclipse) to run Robocop tests like regular instrumentation tests, "injecting" them into the prepared testing environment. It's quite nice!

Comment on attachment 8612636 [details] MozReview Request: Bug 1169476 -- Implement |mach robocop --serve|. r?gbrown https://reviewboard.mozilla.org/r/9603/#review8485 ::: mobile/android/tests/browser/robocop/UITest.java (Diff revision 1) > - // Don't show the first run experience. > - intent.putExtra(BrowserApp.EXTRA_SKIP_STARTPANE, true); > - intent.putExtra("args", "-no-remote -profile " + mProfile); It looks like the EXTRA_SKIP_STARTPANE bit is lost in the move. I'm not sure of the history here, but worth checking. ::: mobile/android/tests/browser/robocop/BaseRobocopTest.java:212 (Diff revision 1) > + Log.i(LOGTAG, "Trying to keep started activity alive."); > + setActivity(null); Seems fragile, but I don't have a better suggestion. ::: build/mobile/robocop/LaunchFennecWithConfigurationActivity.java:31 (Diff revision 1) > + final String configFile = FennecNativeDriver.getFile(BaseRobocopTest.DEFAULT_ROOT_PATH + "/robotium.config"); DEFAULT_ROOT_PATH might not work. Is there no way to get the proper root? ::: testing/mochitest/runtestsremote.py:584 (Diff revision 1) > + "-e", "quit", "1", > + "-e", "finish", "1", I would consolidate "quit" and "finish" into one "quit": If --serve, there is no quit-or-finish; if no --serve, quit-and-finish. ::: testing/mochitest/runtestsremote.py:601 (Diff revision 1) > + dm.default_timeout = 7 * 24 * 60 * 60 # One week, in seconds. Something like sys.maxint might be more informative. ::: testing/mochitest/runtestsremote.py:598 (Diff revision 1) > + options.browserArgs = ["start", > + "-n", "org.mozilla.roboexample.test/org.mozilla.gecko.LaunchFennecWithConfigurationActivity", > + "&&", "cat"] You might not need "&& cat" if you set options.timeout to something huge.

https://reviewboard.mozilla.org/r/9603/#review8511 > DEFAULT_ROOT_PATH might not work. Is there no way to get the proper root? Sadly, no. We could make it more robust by also trying /data/local, I suppose. This is strictly for local testers, and even more it's just for testers running from IntelliJ. 99% of these testers will be fine with /mnt/sdcard. > It looks like the EXTRA_SKIP_STARTPANE bit is lost in the move. I'm not sure of the history here, but worth checking. Great catch! > I would consolidate "quit" and "finish" into one "quit": If --serve, there is no quit-or-finish; if no --serve, quit-and-finish. Sure, I'll do this. > You might not need "&& cat" if you set options.timeout to something huge. The slave process will exit, though, and the test harness will exit after it. I arrived at && cat after trying to be more forgiving about the process ending and found this easier. We'll see how it holds up in the wild. > Something like sys.maxint might be more informative. Sounds reasonable, I'll do this.

gbrown: I tried to push the updated commit back to RB, so you could see the changes with the nice interdiff slider, but all my rebasing and amending confused the system :(

sorry Nick, had to back this out for test failures/crashes like https://treeherder.mozilla.org/logviewer.html#?job_id=3275457&repo=fx-team

Backout: https://hg.mozilla.org/integration/fx-team/rev/55c42e800046 https://hg.mozilla.org/integration/fx-team/rev/049b2fdb0711

Fixed and re-landed. Looked healthy on try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=6c3fae69b336 -- the rc5 looks like a legit issue.