https://treeherder.mozilla.org/logviewer.html#?job_id=8817899&repo=fx-team 04:21:52 INFO - :app:testAutomationDebugUnitTest 04:21:52 INFO - org.mozilla.gecko.db.BrowserProviderVisitsTest > testDefaultSortingOrder FAILED 04:21:52 INFO - java.lang.RuntimeException at BrowserProviderVisitsTest.java:130 04:21:52 INFO - Caused by: java.lang.IllegalStateException 04:22:10 INFO - 423 tests completed, 1 failed, 2 skipped 04:22:10 INFO - :app:testAutomationDebugUnitTest FAILED 04:22:10 INFO - FAILURE: Build failed with an exception. 04:22:10 INFO - * What went wrong: 04:22:10 INFO - Execution failed for task ':app:testAutomationDebugUnitTest'. 04:22:10 INFO - > There were failing tests. See the report at: file:///home/worker/workspace/build/src/obj-firefox/gradle/build/mobile/android/app/reports/tests/automationDebug/index.html i guess caused by bug 1046709

Here's the fail specifically: https://public-artifacts.taskcluster.net/SZGzUVCvTdiw2JpQy1gQyw/0/public/android/unittest/automationDebug/classes/org.mozilla.gecko.db.BrowserProviderVisitsTest.html

Trying to get this to show up as a starrable intermittent in Treeherder.

This is an unacceptable intermittent failure rate. Really, bug 1046709 should have been backed out rather than being allowed to stay in the tree with this level of test failures, but it's been a day at this point, and the patch seems to have other things on top of it now. Do you have a plan for fixing this bug this week?

(In reply to David Baron [:dbaron] ⌚️UTC-7 from comment #4) > This is an unacceptable intermittent failure rate. Really, bug 1046709 > should have been backed out rather than being allowed to stay in the tree > with this level of test failures, was because tier 2 Tier 2: Jobs are shown by default on Treeherder, but are not sheriff-managed. Results will be shown on Treeherder "for information only". New test failures/bustage will not result in a backout, but a tracking bug will be filed when observed. https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy

Will have a fix shortly.

Nick, I've been going through these tests [0], and the problem seems to be that somehow, db connection isn't cleared up correctly in-between individual tests. See [1]. Something is stuck with an invalid connection pointer. Provided that in an @After method I'm calling BrowserProvider's `shutdown` (which ends up disposing of underlying database connections), I'm not sure how that might be. As an experiment (and somewhat out of frustration), I've looked closely at any differences in how TabsProvider tests work (Bug 1260478), and modified my BrowserProvider tests accordingly, but of course that didn't help at all. The main difference is that I'm re-creating content provider clients in @Before, and releasing them in @After. To add to confusion, I would expect this failure to occur for every test run, but it only happens for one of the tests (not any in particular, I think the 2nd test that gets run? I'm not sure actually which). Perhaps that's a hint? Removing that call to shutdown() results in this exact failure in every test, which is what one would expect. In Android Studio, everything works peachy. How is `./gradlew test` different from running these tests in AS, in a way that's relevant to this? Until this is figured out, I can back-out these tests in the morning, to at least get build failures to stop. Unless you can, perhaps, point to an obviously stupid thing that I'm doing :-) Or some another form of divine intervention occurs. :/ [0] https://dxr.mozilla.org/mozilla-central/source/mobile/android/tests/background/junit4/src/org/mozilla/gecko/db/BrowserProviderVisitsTest.java [1] https://public-artifacts.taskcluster.net/SZGzUVCvTdiw2JpQy1gQyw/0/public/android/unittest/automationDebug/classes/org.mozilla.gecko.db.BrowserProviderVisitsTest.html

Review commit: https://reviewboard.mozilla.org/r/48563/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/48563/

Comment on attachment 8744476 [details] MozReview Request: Bug 1265979 - missing call to provider.shutdown() causing test failures r=sebastian https://reviewboard.mozilla.org/r/48563/#review45277 ::: mobile/android/tests/background/junit4/src/org/mozilla/gecko/sync/repositories/android/VisitsHelperTest.java:89 (Diff revision 1) > - Assert.assertEquals(10, recentVisits.size()); > + Assert.assertEquals(10, recentVisits.size()); > - for (int i = 0; i < recentVisits.size(); i++) { > + for (int i = 0; i < recentVisits.size(); i++) { > - JSONObject v = (JSONObject) recentVisits.get(i); > + JSONObject v = (JSONObject) recentVisits.get(i); > - Long date = (Long) v.get("date"); > + Long date = (Long) v.get("date"); > - Integer type = (Integer) v.get("type"); > + Integer type = (Integer) v.get("type"); > - Assert.assertEquals(Long.valueOf(baseDate - i * 100), date); > + Assert.assertEquals(Long.valueOf(baseDate - i * 101), date); Without this r+ :)

Comment on attachment 8744476 [details] MozReview Request: Bug 1265979 - missing call to provider.shutdown() causing test failures r=sebastian Review request updated; see interdiff: https://reviewboard.mozilla.org/r/48563/diff/1-2/

https://hg.mozilla.org/integration/fx-team/rev/b50652d71510d898b7e1f94d8f8d4330b7d1a486 Bug 1265979 - missing call to provider.shutdown() causing test failures r=sebastian

Let's summarize what we have found in case this happens again and someone stumbles upon this bug: One of the tests did not call shutdown() on the content provider. Robolectric will create a new database instance for every test. Keeping and reusing references between tests will lead to the IllegalStateException ("Illegal connection pointer 1. Current pointers for thread [..]"). In addition to that the test order seems to be randomized (or at least different depending on the machine they are running on): Intermittent instead of perma fail.