We need to run the GeckoView JUnit tests in automation. These are in the org.mozilla.geckoview.test module. They are more-or-less standard Android tests and can be run with instructions similar to how it's described here: https://developer.android.com/studio/test/command-line.html#RunTestsGradle These tests don't require any special environment, so anything should be suitable (emulator, hardware, whatever).

P1 because we want to run tests before we ship Klar+GeckoView.

Some initial results: https://treeherder.mozilla.org/#/jobs?repo=try&revision=d70b35759dbe101ca76e989f674a512f40aeddc0 https://taskcluster-artifacts.net/dAHbU_H4QpGuavXgyFXqjw/0/public/logs/live_backing.log task 2018-04-05T22:08:09.290Z] 22:08:09 CRITICAL - ADBTimeoutError: args: adb -s emulator-5554 wait-for-device shell am instrument -w org.mozilla.geckoview.test/android.support.test.runner.AndroidJUnitRunner; echo rc=$?, exitcode: None, stdout: [task 2018-04-05T22:08:09.291Z] 22:08:09 INFO - org.mozilla.geckoview.test.ContentDelegateTest: [task 2018-04-05T22:08:09.292Z] 22:08:09 INFO - Error in titleChange(org.mozilla.geckoview.test.ContentDelegateTest): [task 2018-04-05T22:08:09.292Z] 22:08:09 INFO - java.lang.AssertionError: Timed out after 10000ms [task 2018-04-05T22:08:09.293Z] 22:08:09 INFO - at org.junit.Assert.fail(Assert.java:88) [task 2018-04-05T22:08:09.294Z] 22:08:09 INFO - at org.mozilla.geckoview.test.rule.GeckoSessionTestRule$5.run(GeckoSessionTestRule.java:891) [task 2018-04-05T22:08:09.294Z] 22:08:09 INFO - at android.os.Handler.handleCallback(Handler.java:730) [task 2018-04-05T22:08:09.295Z] 22:08:09 INFO - at android.os.Handler.dispatchMessage(Handler.java:92) [task 2018-04-05T22:08:09.296Z] 22:08:09 INFO - at org.mozilla.geckoview.test.rule.GeckoSessionTestRule.loopUntilIdle(GeckoSessionTestRule.java:915) [task 2018-04-05T22:08:09.297Z] 22:08:09 INFO - at org.mozilla.geckoview.test.rule.GeckoSessionTestRule.openSession(GeckoSessionTestRule.java:788) [task 2018-04-05T22:08:09.298Z] 22:08:09 INFO - at org.mozilla.geckoview.test.rule.GeckoSessionTestRule.prepareStatement(GeckoSessionTestRule.java:729) [task 2018-04-05T22:08:09.299Z] 22:08:09 INFO - at org.mozilla.geckoview.test.rule.GeckoSessionTestRule$3.evaluate(GeckoSessionTestRule.java:842) [task 2018-04-05T22:08:09.300Z] 22:08:09 INFO - at android.support.test.internal.statement.UiThreadStatement$1.run(UiThreadStatement.java:44) [task 2018-04-05T22:08:09.301Z] 22:08:09 INFO - at android.app.Instrumentation$SyncRunnable.run(Instrumentation.java:1719) [task 2018-04-05T22:08:09.301Z] 22:08:09 INFO - at android.os.Handler.handleCallback(Handler.java:730) [task 2018-04-05T22:08:09.302Z] 22:08:09 INFO - at android.os.Handler.dispatchMessage(Handler.java:92) [task 2018-04-05T22:08:09.303Z] 22:08:09 INFO - at android.os.Looper.loop(Looper.java:137) [task 2018-04-05T22:08:09.303Z] 22:08:09 INFO - at android.app.ActivityThread.main(ActivityThread.java:5103) [task 2018-04-05T22:08:09.303Z] 22:08:09 INFO - at java.lang.reflect.Method.invokeNative(Native Method) [task 2018-04-05T22:08:09.304Z] 22:08:09 INFO - at java.lang.reflect.Method.invoke(Method.java:525) [task 2018-04-05T22:08:09.304Z] 22:08:09 INFO - at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:737) [task 2018-04-05T22:08:09.304Z] 22:08:09 INFO - at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:553) [task 2018-04-05T22:08:09.304Z] 22:08:09 INFO - at dalvik.system.NativeStart.main(Native Method) [task 2018-04-05T22:08:09.304Z] 22:08:09 INFO - Error in download(org.mozilla.geckoview.test.ContentDelegateTest): [task 2018-04-05T22:08:09.304Z] 22:08:09 INFO - java.lang.AssertionError: Timed out after 10000ms [task 2018-04-05T22:08:09.304Z] 22:08:09 INFO - at org.junit.Assert.fail(Assert.java:88) [task 2018-04-05T22:08:09.304Z] 22:08:09 INFO - at org.mozilla.geckoview.test.rule.GeckoSessionTestRule$5.run(GeckoSessionTestRule.java:891) [task 2018-04-05T22:08:09.305Z] 22:08:09 INFO - at android.os.Handler.handleCallback(Handler.java:730) [task 2018-04-05T22:08:09.305Z] 22:08:09 INFO - at android.os.Handler.dispatchMessage(Handler.java:92) [task 2018-04-05T22:08:09.305Z] 22:08:09 INFO - at org.mozilla.geckoview.test.rule.GeckoSessionTestRule.loopUntilIdle(GeckoSessionTestRule.java:915) [task 2018-04-05T22:08:09.305Z] 22:08:09 INFO - at org.mozilla.geckoview.test.rule.GeckoSessionTestRule.openSession(GeckoSessionTestRule.java:788) [task 2018-04-05T22:08:09.305Z] 22:08:09 INFO - at org.mozilla.geckoview.test.rule.GeckoSessionTestRule.prepareStatement(GeckoSessionTestRule.java:729) [task 2018-04-05T22:08:09.305Z] 22:08:09 INFO - at org.mozilla.geckoview.test.rule.GeckoSessionTestRule$3.evaluate(GeckoSessionTestRule.java:842) [task 2018-04-05T22:08:09.305Z] 22:08:09 INFO - at android.support.test.internal.statement.UiThreadStatement$1.run(UiThreadStatement.java:44) [task 2018-04-05T22:08:09.306Z] 22:08:09 INFO - at android.app.Instrumentation$SyncRunnable.run(Instrumentation.java:1719) [task 2018-04-05T22:08:09.306Z] 22:08:09 INFO - at android.os.Handler.handleCallback(Handler.java:730) [task 2018-04-05T22:08:09.306Z] 22:08:09 INFO - at android.os.Handler.dispatchMessage(Handler.java:92) [task 2018-04-05T22:08:09.306Z] 22:08:09 INFO - at android.os.Looper.loop(Looper.java:137) [task 2018-04-05T22:08:09.306Z] 22:08:09 INFO - at android.app.ActivityThread.main(ActivityThread.java:5103) [task 2018-04-05T22:08:09.306Z] 22:08:09 INFO - at java.lang.reflect.Method.invokeNative(Native Method) [task 2018-04-05T22:08:09.307Z] 22:08:09 INFO - at java.lang.reflect.Method.invoke(Method.java:525) [task 2018-04-05T22:08:09.307Z] 22:08:09 INFO - at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:737) [task 2018-04-05T22:08:09.307Z] 22:08:09 INFO - at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:553) [task 2018-04-05T22:08:09.307Z] 22:08:09 INFO - at dalvik.system.NativeStart.main(Native Method) [task 2018-04-05T22:08:09.308Z] 22:08:09 INFO - org.mozilla.geckoview.test.GeckoSessionTestRuleTest:.................... [task 2018-04-05T22:08:09.308Z] 22:08:09 INFO - Error in createOpenSession_withSettings(org.mozilla.geckoview.test.GeckoSessionTestRuleTest): ... :jchen -- Are these "java.lang.AssertionError: Timed out after 10000ms" errors expected?

I tried increasing the timeout to 60000ms, but that didn't help much: https://treeherder.mozilla.org/#/jobs?repo=try&revision=840428be63075924b3ee9a99f088d34300e2105e

Some of the failures are expected. Locally I get 4 failures under e10s and 3 under non-e10s. You should try with non-e10s by adding, > mDefaultSettings.setBoolean(GeckoSessionSettings.USE_MULTIPROCESS, false); at the end of `GeckoSessionTestRule()` near line 536 of GeckoSessionTestRule.java. Also, FWIW, we will eventually need a mochitest server started up for some of the tests that require connection to a server (e.g. SSL tests).

(In reply to Jim Chen [:jchen] [:darchons] from comment #4) > Also, FWIW, we will eventually need a mochitest server started up for some > of the tests that require connection to a server (e.g. SSL tests). That should be easy, but...Recall that the mochitest harness creates a test profile with network proxies like "mochi.test", pushes that profile to the device and then runs Firefox for Android with -profile. Will we need similar facilities for these tests?

That should be pretty easy. "adb instrument" takes its arguments through "-e key value", so we can use that to pass in e10s flag, env vars, and Gecko command line arguments like -profile.

I should say our tests don't support these arguments right now, but if you come up with a list of arguments the test harness passes in, we can go ahead and add support for them.

OK, I'll create and push a profile for the network proxy support and we'll plan on using -profile to communicate the profile location. And run a mochitest-like server.

Another issue is the ability to skip troublesome tests. I notice https://developer.mozilla.org/en-US/docs/Mozilla/Android-specific_test_suites#android-test talks about using @Ignore, but I could not get that to work (tried applying to the failing test in ContentDelegateTest.kt - build fails).

Hm that worked for me locally ("import org.junit.Ignore" and then add "@Ignore" in front of "@Test fun download()"). BTW related to bug 1452239, I think we want to eventually use the -r option for "adb instrument", which will continuously output raw results from individual test cases, and we can translate that into the treeherder format. I think that will provide a better treeherder experience (and less risk for timeout due to no output). This work can go in a follow-up bug though.

(In reply to Jim Chen [:jchen] [:darchons] from comment #10) > Hm that worked for me locally ("import org.junit.Ignore" and then add > "@Ignore" in front of "@Test fun download()"). Yes, indeed, that works. Must have had a typo before? Thanks!

With -r and some minor logging improvements: https://treeherder.mozilla.org/#/jobs?repo=try&revision=9a6b523d852c6d291451584de5e51b9e54f58c6e

I still need to finish off the harness, so cannot check-in yet, but let's look at the task definition now anyway.

Comment on attachment 8967825 [details] [diff] [review] add geckoview junit test task Review of attachment 8967825 [details] [diff] [review]: ----------------------------------------------------------------- r=me with a few nits ::: taskcluster/ci/test/misc.yml @@ +25,5 @@ > + suite: junit > + treeherder-symbol: gv-junit > + instance-size: xlarge > + loopback-video: true > + e10s: false isn't geckoview e10s? ::: testing/mozharness/mozharness/mozilla/testing/testbase.py @@ +413,4 @@ > 'mochitest-plain-gpu': 'mochitest', > 'mochitest-gl': 'mochitest', > 'geckoview': 'mochitest', > + 'junit': 'mochitest', why does junit need the mochitest harness bits to download?

(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #15) > > + e10s: false > > isn't geckoview e10s? Good catch - thanks! > ::: testing/mozharness/mozharness/mozilla/testing/testbase.py > @@ +413,4 @@ > > 'mochitest-plain-gpu': 'mochitest', > > 'mochitest-gl': 'mochitest', > > 'geckoview': 'mochitest', > > + 'junit': 'mochitest', > > why does junit need the mochitest harness bits to download? The harness does not reflect it yet, but will need to start a web server. I'm planning to leverage some mochitest harness code.

I've started a wiki page describing geckoview-junit tests at https://developer.mozilla.org/en-US/docs/Mozilla/Geckoview-Junit_Tests Contributions welcome!

Now with e10s, profile, crash reporting and symbols: https://treeherder.mozilla.org/#/jobs?repo=try&revision=80f86d7124fe506c6932a2164734de3b65b6961f

This simple mach command drives the runjunit.py test harness. All the normal Android test checks (device connected? app installed? host-utils found?) are in place. Implementation generally parallels existing code for other mochitest-related tests. gbrown@mozpad:~/src$ ./mach geckoview-junit No Android devices connected. Start an emulator? (Y/n) Starting emulator running Android 4.3... Running the arm emulator; be sure to install an arm APK! 0:00.31 adb INFO adbd running as root 0:00.62 adb INFO su -c supported 0:00.92 adb INFO su 0 supported 0:01.43 adb INFO /system/bin/ls -a supported 0:01.63 adb INFO Native cp support: True 0:01.83 adb INFO Native chmod -R support: True pk12util: PKCS12 IMPORT SUCCESSFUL ...

gbrown@mozpad:~/src$ ./mach help geckoview-junit usage: mach [global arguments] geckoview-junit [command arguments] Run remote geckoview junit tests. Global Arguments: -v, --verbose Print verbose output. -l FILENAME, --log-file FILENAME Filename to write log data to. --log-interval Prefix log line with interval from last message rather than relative time. Note that this is NOT execution time if there are parallel operations. --log-no-times Do not prefix log lines with times. By default, mach will prefix each output line with the time since command start. -h, --help Show this help message. --debug-command Start a Python debugger when command is dispatched. --settings FILENAME Path to settings file. Command Parameters: test_filters Test filter(s): class and/or method names of test(s) to run. Command Arguments: -h, --help show this help message and exit --appname APP Test package name. --adbpath ADBPATH Path to adb executable. --deviceSerial DEVICESERIAL adb serial number of remote device. --remoteTestRoot REMOTETESTROOT Remote directory to use as test root (eg. /mnt/sdcard/tests or /data/local/tests). --disable-e10s Disable multiprocess mode in test app. --max-time MAX_TIME Max time in seconds to wait for tests (default 2400s). --runner RUNNER Test runner name. --symbols-path SYMBOLSPATH Path to directory containing breakpad symbols, or the URL of a zip file containing symbols. --utility-path UTILITYPATH Path to directory containing host utility programs. --certificate-path CERTPATH Path to directory containing certificate store. --http-port HTTPPORT Port of the web server for http traffic. --remote-webserver REMOTEWEBSERVER IP address of the webserver. --ssl-port SSLPORT Port of the web server for https traffic. Output Logging: Each option represents a possible logging format and takes a filename to write that format to, or '-' to write to stdout. Some options are provided by the mozlog utility; see https://firefox-source- docs.mozilla.org/mozbase/mozlog.html for extended documentation. --log-unittest LOG_UNITTEST Unittest style output (provided by mozlog) ...

https://treeherder.mozilla.org/#/jobs?repo=try&revision=f414ca0a30f0bc726baa181912b01f26f8b2ea68 demonstrates lots of good things: passing tests, treeherder-friendly logging, profile setup, crash reporting and symbols, and server setup. I think we have everything we need to run these as a tier 1 job...except that there are several test failures and at least one crash due to a non-local network connection attempt. :jchen -- Can you have a look at the test failures? Let me know if there is anything lacking in the test harness or environment.

JUnitTestRunner inherits from MochitestDesktop simply to take advantage of mochitest's startServer/stopServer. Many options and class variables need to be set in order to use the mochitest servers: hence, server_init(). run_tests() starts the tests with something like 'adb shell am instrument -w -r ...' and uses an output callback to monitor the raw output from that command, line-by-line. Raw output is logged as structured logging "process output", and is also translated into structured logging test_start/test_end calls. Otherwise, this is a pretty simple test harness.

Comment on attachment 8968714 [details] [diff] [review] add 'geckoview-junit' mach command Review of attachment 8968714 [details] [diff] [review]: ----------------------------------------------------------------- I have one question, will r+ if it is clarified. ::: testing/mochitest/mach_commands.py @@ +275,5 @@ > + with open(path, 'r') as fh: > + imp.load_module('mochitest', fh, path, > + ('.py', 'r', imp.PY_SOURCE)) > + > + from runjunit import JunitArgumentParser here you load_module 'mochitest', and above in run_geckoview_junit_test you load_module 'runjunit'. My confusion is this line, where is runjunit defined as?

Comment on attachment 8968745 [details] [diff] [review] add runjunit.py test harness Review of attachment 8968745 [details] [diff] [review]: ----------------------------------------------------------------- a few small nits ::: testing/mochitest/runjunit.py @@ +165,5 @@ > + env["MOZ_CRASHREPORTER_NO_REPORT"] = "1" > + env["MOZ_CRASHREPORTER_SHUTDOWN"] = "1" > + env["XPCOM_DEBUG_BREAK"] = "stack" > + env["DISABLE_UNSAFE_CPOW_WARNINGS"] = "1" > + env["MOZ_DISABLE_NONLOCAL_CONNECTIONS"] = "1" on android don't we want to connect to a remote server? @@ +217,5 @@ > + if full_name == self.current_full_name: > + if status == '0': > + message = '' > + status = 'PASS' > + self.pass_count = self.pass_count + 1 why not use += 1? @@ +221,5 @@ > + self.pass_count = self.pass_count + 1 > + else: > + message = 'status %s' % status > + status = 'FAIL' > + self.fail_count = self.fail_count + 1 += 1 ? @@ +227,5 @@ > + self.test_started = False > + else: > + if self.test_started: > + # next test started without reporting previous status > + self.fail_count = self.fail_count + 1 += 1? @@ +366,5 @@ > + help="Port of the web server for https traffic.") > + # Remaining arguments are test filters. > + self.add_argument("test_filters", > + nargs="*", > + help="Test filter(s): class and/or method names of test(s) to run.") it seems like many of these cli options are already defined for remote testing or desktop testing- is there a way to share these command line parsers? Maybe using mixins like we do in mozharness?

(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #25) > ::: testing/mochitest/runjunit.py > @@ +165,5 @@ > > + env["MOZ_CRASHREPORTER_NO_REPORT"] = "1" > > + env["MOZ_CRASHREPORTER_SHUTDOWN"] = "1" > > + env["XPCOM_DEBUG_BREAK"] = "stack" > > + env["DISABLE_UNSAFE_CPOW_WARNINGS"] = "1" > > + env["MOZ_DISABLE_NONLOCAL_CONNECTIONS"] = "1" > > on android don't we want to connect to a remote server? I think we want the same as mochitests: Only allow white-listed proxied connections to our webserver. Otherwise tests may be dependent on external resources, leading to non-determinism and keeping us off of tier-1.

(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #25) > it seems like many of these cli options are already defined for remote > testing or desktop testing- is there a way to share these command line > parsers? Maybe using mixins like we do in mozharness? There are definitely lots of common options across test harnesses. One obstacle to sharing between say mochitest and reftest is the code location: Put it in mozbase somewhere? A mozoptions module?? At any rate, I think cleaning up the options is a separate bug.

In setup_junit_argument_parser (some of the earlier junit-related code involved in the mach command), if we load_module("runjunit.py"), it fails because the base class cannot be imported: ImportError: cannot import name MochitestDesktop File "/home/gbrown/src/python/mach/mach/main.py", line 359, in run return self._run(argv) File "/home/gbrown/src/python/mach/mach/main.py", line 414, in _run args = parser.parse_args(argv) File "/usr/lib/python2.7/argparse.py", line 1701, in parse_args args, argv = self.parse_known_args(args, namespace) File "/usr/lib/python2.7/argparse.py", line 1733, in parse_known_args namespace, args = self._parse_known_args(args, namespace) File "/usr/lib/python2.7/argparse.py", line 1942, in _parse_known_args stop_index = consume_positionals(start_index) File "/usr/lib/python2.7/argparse.py", line 1898, in consume_positionals take_action(action, args) File "/usr/lib/python2.7/argparse.py", line 1807, in take_action action(self, namespace, argument_values, option_string) File "/home/gbrown/src/python/mach/mach/dispatcher.py", line 172, in __call__ if handler.parser: File "/home/gbrown/src/python/mach/mach/decorators.py", line 76, in parser self._parser = self._parser() File "/home/gbrown/src/testing/mochitest/mach_commands.py", line 275, in setup_junit_argument_parser ('.py', 'r', imp.PY_SOURCE)) File "/home/gbrown/objdirs/droid/_tests/testing/mochitest/runjunit.py", line 21, in <module> from runtests import MochitestDesktop, update_mozinfo so we need to load_module("runtests.py") first. After that, load_module("runjunit.py") will succeed, but, afaict, that's not necessary: a simple import of runjunit works fine. I've added a comment and simplified the imports accordingly.

Comment on attachment 8969054 [details] [diff] [review] add 'geckoview-junit' mach command Review of attachment 8969054 [details] [diff] [review]: ----------------------------------------------------------------- this looks much better, thanks.

I'll check in most of this, to make 'mach geckoview-junit' available, but won't enable junit in automation until we resolve the failing tests (comment 22).

Or maybe land as tier 2 until they're consistently green? I think most of the failures are due to a startup race that I'm looking at now.

Pushed by gbrown@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/5247cde8ef0c Add runjunit.py, a test harness for junit tests on Android devices; r=jmaher https://hg.mozilla.org/integration/mozilla-inbound/rev/05412fcc00a7 Add Android mach command 'geckoview-junit'; r=jmaher

(In reply to Jim Chen [:jchen] [:darchons] from comment #31) > Or maybe land as tier 2 until they're consistently green? I think most of > the failures are due to a startup race that I'm looking at now. We could do that, but I hate to start running tests that consistently fail: The sheriffs will still file bugs and star the failures, and that can cause confusion (triagers trying to find ownership, without realizing its tier 2, etc). It sounds like we are close; let's revisit in a few days.

https://hg.mozilla.org/mozilla-central/rev/5247cde8ef0c https://hg.mozilla.org/mozilla-central/rev/05412fcc00a7

Geoff, I filed bug 1456209 to address problems with the test framework. There are also some issues with the test harness that I found, * In my try runs, the debug test run has a run time of ~90 minutes, so we need to increase the taskcluster timeout or split the debug run into two. * We will have tests that make use of the mochitest host aliases (e.g. https://example.com), and those don't seem to work right now. I worked around it by adding a `locations` parameter at [1]. * The test proxy server and/or the SSL tunnel don't seem to be working. Loading https://example.com in a test always gave me a connection refused error, after fixing the `locations` problem above. * The test runner uses status code -3 to mean "ignored" and -4 to mean "known fail", so we need to handle those. The full list of status codes is at [2]. * There are a lot of intermittent "missing test completion status" errors. I think there's something going on with the code that pipes adb output. With these and bug 1456209 fixed, we should have some green runs! [1] https://searchfox.org/mozilla-central/rev/fcd5cb3515d0e06592bba42e378f1b9518497e3d/testing/mochitest/runjunit.py#125 [2] https://android.googlesource.com/platform/frameworks/testing/+/master/support/src/android/support/test/internal/runner/listener/InstrumentationResultPrinter.java#62

Thanks Jim. I'm on it -- hope to have some solutions here soon.

I think it will be hard to chunk geckoview-junit because the harness does not have a manifest: tests are baked in to the apk and we run everything at once. Hopefully this will do.

I don't know much about this stuff, but I've tried to make this parallel the code in runtests.py and it seems to work. https://treeherder.mozilla.org/#/jobs?repo=try&revision=a1b3d300585f132f71382f57c5cb696b35df7085

Comment on attachment 8970690 [details] [diff] [review] support skipped and expected-fail tests Review of attachment 8970690 [details] [diff] [review]: ----------------------------------------------------------------- nice and simple

Comment on attachment 8970691 [details] [diff] [review] increase geckoview-junit max-run-time on debug Review of attachment 8970691 [details] [diff] [review]: ----------------------------------------------------------------- will these run faster on hardware? If so then there is less concern. I think we should find a way to make a manifest if possible- how can we skip these tests and query what tests we have, etc... ? Maybe we build different apk files for each chunk? Either way, for now and the near future a single chunk sounds reasonable.

Comment on attachment 8970692 [details] [diff] [review] support locations in geckoview-junit profile Review of attachment 8970692 [details] [diff] [review]: ----------------------------------------------------------------- ::: testing/mochitest/runjunit.py @@ +182,5 @@ > + @property > + def locations(self): > + if self._locations is not None: > + return self._locations > + locations_file = os.path.join(here, 'server-locations.txt') do we copy server-locations.txt to the same directory as runjunit.py? If so, then no concerns. Ensure we handle both automation and mach.

Awesome work! The multipleLoads test is perma-fail right now (bug 1441059). The onNewSession_calledForNewWindow and onNewSession_doesNotAllowOpened tests are flaky (we're trying to fix that through bug 1456190). So ignoring those three, it's looking really good! (In reply to Geoff Brown [:gbrown] from comment #38) > Created attachment 8970691 [details] [diff] [review] > increase geckoview-junit max-run-time on debug > > I think it will be hard to chunk geckoview-junit because the harness does > not have a manifest: tests are baked in to the apk and we run everything at > once. Hopefully this will do. The junit harness actually supports that natively [1], so might be worth looking into in the future. [1] https://developer.android.com/training/testing/junit-runner.html#sharding-tests

(In reply to Joel Maher ( :jmaher - limited bugzilla access until May 1st) (UTC+2) from comment #41) > will these run faster on hardware? If so then there is less concern. I > think we should find a way to make a manifest if possible- how can we skip > these tests and query what tests we have, etc... ? Maybe we build different > apk files for each chunk? They likely will run faster on hardware, or an x86 emulator. There is native support for skipping tests (@Ignore in the source code) and selecting specific tests to run, so chunking was my only concern related to a manifest -- and with :jchen's tip (comment 43), I'm less concerned about that now. > Either way, for now and the near future a single chunk sounds reasonable. I'll go ahead with this, and consider sharding another day.

(In reply to Joel Maher ( :jmaher - limited bugzilla access until May 1st) (UTC+2) from comment #42) > do we copy server-locations.txt to the same directory as runjunit.py? If > so, then no concerns. Ensure we handle both automation and mach. It's just like mochitest/runtests.py and works fine locally.

Pushed by gbrown@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/1b29fa5bb802 Process more geckoview-junit status codes; r=jmaher https://hg.mozilla.org/integration/mozilla-inbound/rev/dbcb5f8ea056 Add new Android test task geckoview-junit, but do not run it yet; r=jmaher https://hg.mozilla.org/integration/mozilla-inbound/rev/ec40107c1c13 Increase timeout for geckoview-junit on android debug; r=jmaher https://hg.mozilla.org/integration/mozilla-inbound/rev/fe6ddc27db72 Support profile locations in geckoview-junit; r=jmaher

I included the task definition in this check-in, but disabled it. To test on try, we'll still need to remove the comment at: https://hg.mozilla.org/integration/mozilla-inbound/rev/dbcb5f8ea056f2307b867e9ceafa4772b91c2a00#l3.12 Still to do: - last few failing tests (comment 43) - possible lost output / missing status code problem

Until those tests get fixed in their bugs, feel free to @Ignore them for now.

https://hg.mozilla.org/mozilla-central/rev/1b29fa5bb802 https://hg.mozilla.org/mozilla-central/rev/dbcb5f8ea056 https://hg.mozilla.org/mozilla-central/rev/ec40107c1c13 https://hg.mozilla.org/mozilla-central/rev/fe6ddc27db72

[geckoview:klar:p1] because running these tests is a Klar+GeckoView blocker.

As suggested, @Ignore failing tests to get a green run. We can start running geckoview-junit on trunk with this change and bug 1457662. https://treeherder.mozilla.org/#/jobs?repo=try&revision=30d3cf2e1cbf8cc0ea25c628b271022dde716591

Comment on attachment 8972008 [details] [diff] [review] skip failing tests There are still some debug-only failures...I'll revise a bit...

Looking good now: https://treeherder.mozilla.org/#/jobs?repo=try&revision=b1c1d14b5362caf408d5fdaadae126790222a2c3

Pushed by gbrown@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/e131aaef5d89 Run new Android test task, geckoview-junit; r=me,a=test-only https://hg.mozilla.org/integration/mozilla-inbound/rev/a340a6ab8489 Skip remaining failures in geckoview-junit tests; r=jchen

https://hg.mozilla.org/mozilla-central/rev/e131aaef5d89 https://hg.mozilla.org/mozilla-central/rev/a340a6ab8489