I have noticed that when running the entire test suite we are getting some flakiness. E.g. Window Management test counts the amount of open windows when asserting and had 1 more than expected which appears to be from the test that ran before We should probably restart the browser between test modules

Is there actually 1 extra window open, or are we holding a stale reference?

Yes, I see the same problem; basically Marionette gets confused when switching windows sometimes, and switches to the wrong one. It may be a timing issue. Definitely needs to be fixed.

On my machine, this problem is easily reproduced if Firefox is in the foreground while the tests run. If it's in the background, obscured by a Terminal window for example, it does not reproduce. I'm guessing this is a timing issue.

So it looks like this is what's happening: - some tests use window.open in an execute_script to open a new window to run a test, e.g., http://mxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/tests/unit/test_findelement.py#131 - the test then continues immediately, without waiting for the window to load - sometimes, this means that the test will be executed before the window is really open - then, when the test teardown happens (e.g., http://mxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/tests/unit/test_findelement.py#134), it will close the wrong window

I tracked it down to a test file that opens a window and doesnt clean it up. I have added a tearDown to close all windows but the main one.

Comment on attachment 646576 [details] [diff] [review] add test cleanup Review of attachment 646576 [details] [diff] [review]: ----------------------------------------------------------------- Thanks!

landed in http://hg.mozilla.org/integration/mozilla-inbound/rev/52f5b60aa1d6