When I enable UITest exception handling and fix bug 1105795, testAboutHomeVisibility still fails with: SimpleTest START TEST-START | testAboutHomeVisibility TEST-PASS | testAboutHomeVisibility | Robocop tests need the test device screen to be powered on. - TEST-PASS | testAboutHomeVisibility | Given message occurred for registered event: {"type":"Gecko:Ready"} - Gecko:Ready should equal Gecko:Ready EventExpecter: no longer listening for Gecko:Ready TEST-PASS | testAboutHomeVisibility | The title argument is not null - should not equal null TEST-PASS | testAboutHomeVisibility | The url argument is not null - about:home should not equal null TEST-PASS | testAboutHomeVisibility | The toolbar is not in the editing state - TEST-PASS | testAboutHomeVisibility | The Toolbar title is - should equal TEST-PASS | testAboutHomeVisibility | The HomePager is visible - TEST-PASS | testAboutHomeVisibility | The HomePager is visible - TEST-PASS | testAboutHomeVisibility | The current HomePager panel is top_sites - 3 should equal 3 TEST-PASS | testAboutHomeVisibility | url is not null - /robocop/robocop_blank_01.html should not equal null TEST-PASS | testAboutHomeVisibility | url is not null - /robocop/robocop_blank_01.html should not equal null TEST-PASS | testAboutHomeVisibility | The toolbar is not in the editing state - TEST-PASS | testAboutHomeVisibility | Waiting for Toolbar to enter editing mode. - TEST-PASS | testAboutHomeVisibility | Waiting for UrlEditText to be input method target. - TEST-PASS | testAboutHomeVisibility | url is not null - http://mochi.test:8888/tests/robocop/robocop_blank_01.html should not equal null TEST-PASS | testAboutHomeVisibility | The toolbar is in the editing state - TEST-PASS | testAboutHomeVisibility | The UrlEditText is the input method target - TEST-PASS | testAboutHomeVisibility | The toolbar is in the editing state - TEST-PASS | testAboutHomeVisibility | initiatingAction is not null - org.mozilla.gecko.tests.components.ToolbarComponent$2@419a6f18 should not equal null ToolbarTitleTextChangeVerifier: stored title, "". TEST-PASS | testAboutHomeVisibility | Given message occurred for registered event: {"type":"DOMTitleChanged","title":"Browser Blank Page 01","tabID":0} - DOMTitleChanged should equal DOMTitleChanged TEST-PASS | testAboutHomeVisibility | Given message occurred for registered event: {"bgColor":"transparent","errorType":"","type":"DOMContentLoaded","metadata":null,"tabID":0} - DOMContentLoaded should equal DOMContentLoaded EventExpecter: no longer listening for DOMContentLoaded EventExpecter: no longer listening for DOMTitleChanged ToolbarTitleTextChangeVerifier: state changed to title, "Browser Blank Page 01". ToolbarTitleTextChangeVerifier: was satisfied. TEST-PASS | testAboutHomeVisibility | Waiting for Toolbar to exit editing mode. - TEST-PASS | testAboutHomeVisibility | The title argument is not null - Browser Blank Page 01 should not equal null TEST-PASS | testAboutHomeVisibility | The url argument is not null - /robocop/robocop_blank_01.html should not equal null TEST-PASS | testAboutHomeVisibility | The toolbar is not in the editing state - TEST-PASS | testAboutHomeVisibility | The Toolbar title is Browser Blank Page 01 - Browser Blank Page 01 should equal Browser Blank Page 01 TEST-PASS | testAboutHomeVisibility | The HomePager is not visible - TEST-PASS | testAboutHomeVisibility | url is not null - /robocop/robocop_blank_02.html should not equal null TEST-PASS | testAboutHomeVisibility | url is not null - /robocop/robocop_blank_02.html should not equal null TEST-PASS | testAboutHomeVisibility | The toolbar is not in the editing state - TEST-PASS | testAboutHomeVisibility | Waiting for Toolbar to enter editing mode. - TEST-PASS | testAboutHomeVisibility | Waiting for UrlEditText to be input method target. - TEST-PASS | testAboutHomeVisibility | url is not null - http://mochi.test:8888/tests/robocop/robocop_blank_02.html should not equal null TEST-PASS | testAboutHomeVisibility | The toolbar is in the editing state - TEST-PASS | testAboutHomeVisibility | The UrlEditText is the input method target - TEST-PASS | testAboutHomeVisibility | The toolbar is in the editing state - TEST-PASS | testAboutHomeVisibility | initiatingAction is not null - org.mozilla.gecko.tests.components.ToolbarComponent$2@419b6038 should not equal null ToolbarTitleTextChangeVerifier: stored title, "Browser Blank Page 01". TEST-PASS | testAboutHomeVisibility | Given message occurred for registered event: {"type":"DOMTitleChanged","title":"Browser Blank Page 02","tabID":0} - DOMTitleChanged should equal DOMTitleChanged TEST-PASS | testAboutHomeVisibility | Given message occurred for registered event: {"bgColor":"transparent","errorType":"","type":"DOMContentLoaded","metadata":null,"tabID":0} - DOMContentLoaded should equal DOMContentLoaded EventExpecter: no longer listening for DOMContentLoaded EventExpecter: no longer listening for DOMTitleChanged ToolbarTitleTextChangeVerifier: state changed to title, "Browser Blank Page 02". ToolbarTitleTextChangeVerifier: was satisfied. TEST-PASS | testAboutHomeVisibility | Waiting for Toolbar to exit editing mode. - TEST-PASS | testAboutHomeVisibility | The title argument is not null - Browser Blank Page 02 should not equal null TEST-PASS | testAboutHomeVisibility | The url argument is not null - /robocop/robocop_blank_02.html should not equal null TEST-PASS | testAboutHomeVisibility | The toolbar is not in the editing state - TEST-PASS | testAboutHomeVisibility | The Toolbar title is Browser Blank Page 02 - Browser Blank Page 02 should equal Browser Blank Page 02 TEST-PASS | testAboutHomeVisibility | The HomePager is not visible - TEST-PASS | testAboutHomeVisibility | The toolbar is not in the editing state - TEST-PASS | testAboutHomeVisibility | Waiting for Toolbar to enter editing mode. - TEST-PASS | testAboutHomeVisibility | Waiting for UrlEditText to be input method target. - TEST-PASS | testAboutHomeVisibility | The HomePager is visible - TEST-PASS | testAboutHomeVisibility | The HomePager is visible - TEST-PASS | testAboutHomeVisibility | The current HomePager panel is top_sites - 3 should equal 3 TEST-PASS | testAboutHomeVisibility | The toolbar is in the editing state - TEST-PASS | testAboutHomeVisibility | Waiting for Toolbar to exit editing mode. - TEST-PASS | testAboutHomeVisibility | The HomePager is not visible - TEST-PASS | testAboutHomeVisibility | url is not null - about:home should not equal null TEST-PASS | testAboutHomeVisibility | url is not null - about:home should not equal null TEST-PASS | testAboutHomeVisibility | The toolbar is not in the editing state - TEST-PASS | testAboutHomeVisibility | Waiting for Toolbar to enter editing mode. - TEST-PASS | testAboutHomeVisibility | Waiting for UrlEditText to be input method target. - TEST-PASS | testAboutHomeVisibility | url is not null - about:home should not equal null TEST-PASS | testAboutHomeVisibility | The toolbar is in the editing state - TEST-PASS | testAboutHomeVisibility | The UrlEditText is the input method target - TEST-PASS | testAboutHomeVisibility | The toolbar is in the editing state - TEST-PASS | testAboutHomeVisibility | initiatingAction is not null - org.mozilla.gecko.tests.components.ToolbarComponent$2@41bcaea0 should not equal null ToolbarTitleTextChangeVerifier: stored title, "Browser Blank Page 02". TEST-PASS | testAboutHomeVisibility | Given message occurred for registered event: {"type":"DOMTitleChanged","title":"Fennec Home","tabID":0} - DOMTitleChanged should equal DOMTitleChanged TEST-PASS | testAboutHomeVisibility | Given message occurred for registered event: {"bgColor":"transparent","errorType":"","type":"DOMContentLoaded","metadata":null,"tabID":0} - DOMContentLoaded should equal DOMContentLoaded EventExpecter: no longer listening for DOMContentLoaded EventExpecter: no longer listening for DOMTitleChanged ToolbarTitleTextChangeVerifier: state changed to title, "". ToolbarTitleTextChangeVerifier: was satisfied. TEST-PASS | testAboutHomeVisibility | Waiting for Toolbar to exit editing mode. - TEST-PASS | testAboutHomeVisibility | The title argument is not null - should not equal null TEST-PASS | testAboutHomeVisibility | The url argument is not null - about:home should not equal null TEST-PASS | testAboutHomeVisibility | The toolbar is not in the editing state - TEST-PASS | testAboutHomeVisibility | The Toolbar title is - should equal TEST-PASS | testAboutHomeVisibility | The HomePager is visible - TEST-PASS | testAboutHomeVisibility | The HomePager is visible - TEST-PASS | testAboutHomeVisibility | The current HomePager panel is top_sites - 3 should equal 3 TEST-PASS | testAboutHomeVisibility | The HomePager is visible - TEST-PASS | testAboutHomeVisibility | The HomePager is visible - TEST-UNEXPECTED-FAIL | testAboutHomeVisibility | The current HomePager panel is bookmarks - got 3, expected 4 1 ERROR Exception caught during test! - junit.framework.AssertionFailedError: TEST-UNEXPECTED-FAIL | testAboutHomeVisibility | The current HomePager panel is bookmarks - got 3, expected 4 at junit.framework.Assert.fail(Assert.java:47) at org.mozilla.gecko.FennecMochitestAssert._logMochitestResult(FennecMochitestAssert.java:128) at org.mozilla.gecko.FennecMochitestAssert.ok(FennecMochitestAssert.java:150) at org.mozilla.gecko.FennecMochitestAssert.is(FennecMochitestAssert.java:156) at org.mozilla.gecko.tests.helpers.AssertionHelper.fAssertEquals(AssertionHelper.java:58) at org.mozilla.gecko.tests.components.AboutHomeComponent.assertCurrentPanel(AboutHomeComponent.java:78) at org.mozilla.gecko.tests.testAboutHomeVisibility.testAboutHomeVisibility(testAboutHomeVisibility.java:48) at java.lang.reflect.Method.invokeNative(Native Method) at java.lang.reflect.Method.invoke(Method.java:511) at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214) at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199) at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:186) at org.mozilla.gecko.tests.UITest.runTest(UITest.java:102) at junit.framework.TestCase.runBare(TestCase.java:127) at junit.framework.TestResult$1.protect(TestResult.java:106) at junit.framework.TestResult.runProtected(TestResult.java:124) at junit.framework.TestResult.run(TestResult.java:109) at junit.framework.TestCase.run(TestCase.java:118) at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:169) at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:154) at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:537) at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1551) TEST-UNEXPECTED-FAIL | testAboutHomeVisibility | Exception caught - junit.framework.AssertionFailedError: TEST-UNEXPECTED-FAIL | testAboutHomeVisibility | The current HomePager panel is bookmarks - got 3, expected 4 TEST-OK | testAboutHomeVisibility | TEST-START | Shutdown Passed: 76 Failed: 2 Todo: 0 SimpleTest FINISHED (And sometimes the Bookmarks check passes but the History one fails.)

Instead of navigating to the panel and immediately asserting, this waits for the panel change to occur before asserting -- works great, in local tests. I need to verify on try still.

Comment on attachment 8529970 [details] [diff] [review] wait for panel before asserting Review of attachment 8529970 [details] [diff] [review]: ----------------------------------------------------------------- The waits should be in the transition-starting methods, not in the assertions, e.g. ToolbarComponent.enterEditingMode/commitEditingMode. The reason for this is there are a lot of state transitions that can occur when an assertion occurs and it's much easier for the transitioning-starting methods to wait for their associated state to change than to catch each and every one of the state changes in the assertions. In this case, I believe the problem line is: AboutHome.navigateToBuiltinPanelType(PanelType.BOOKMARKS) So navigateToBuiltinPanelType should be doing the waiting. lgtm w/ that change.

OK. This seems to work just as well.

Comment on attachment 8531492 [details] [diff] [review] wait for panel before asserting Review of attachment 8531492 [details] [diff] [review]: ----------------------------------------------------------------- Nice, nice. lgtm w/ green try.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=9dd49c68402d https://hg.mozilla.org/integration/mozilla-inbound/rev/31d8c6265814