This has been happening for a while, but they've just been dropping into the catch-all bug 614146. For example: https://tbpl.mozilla.org/php/getParsedLog.php?id=7357931&tree=Mozilla-Inbound Rev4 MacOSX Snow Leopard 10.6 mozilla-inbound debug test mochitests-3/5 on 2011-11-11 14:11:17 PST for push 259bae11858b The screenshot shows no visible browser windows; the "Utilities" folder is open in the Finder. --DOMWINDOW == 12 (0x146fd14b8) [serial = 2553] [outer = 0x0] [url = about:blank] --DOMWINDOW == 11 (0x145f11688) [serial = 2548] [outer = 0x0] [url = about:blank] --DOMWINDOW == 10 (0x1469b35b8) [serial = 2543] [outer = 0x0] [url = about:blank] --DOMWINDOW == 9 (0x14793d568) [serial = 2593] [outer = 0x0] [url = about:blank] --DOMWINDOW == 8 (0x146da6858) [serial = 2568] [outer = 0x0] [url = about:blank] --DOMWINDOW == 7 (0x11b68ca98) [serial = 12] [outer = 0x0] [url = http://mochi.test:8888/tests/?autorun=1&closeWhenDone=1&consoleLevel=INFO&totalChunks=5&thisChunk=3&chunkByDir=4&hideResultsTable=1] --DOMWINDOW == 6 (0x118f663e8) [serial = 2] [outer = 0x0] [url = about:blank] --DOMWINDOW == 5 (0x152d25f48) [serial = 5274] [outer = 0x0] [url = about:blank] WARNING: NS_ENSURE_TRUE(asyncCloseWasCalled) failed: file /builds/slave/m-in-osx64-dbg/build/storage/src/mozStorageConnection.cpp, line 835 WARNING: not an nsIRDFRemoteDataSource: 'remote != nsnull', file /builds/slave/m-in-osx64-dbg/build/rdf/datasource/src/nsLocalStore.cpp, line 312 WARNING: not an nsIRDFRemoteDataSource: 'remote != nsnull', file /builds/slave/m-in-osx64-dbg/build/rdf/datasource/src/nsLocalStore.cpp, line 312 TEST-UNEXPECTED-FAIL | Shutdown | application timed out after 330 seconds with no output Can't trigger Breakpad, just killing process INFO | automation.py | Application ran for: 0:12:25.677817 INFO | automation.py | Reading PID log: /var/folders/Hs/HsDn6a9SG8idoIya6p9mtE+++TI/-Tmp-/tmpjaUZu_pidlog TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | missing output line for total leaks! INFO | runtests.py | Running tests: end. program finished with exit code 247 elapsedTime=751.297119 TinderboxPrint: mochitest-plain-3<br/>14253/0/2845&nbsp;<em class="testfail">L-FAIL</em> Unknown Error: command finished with exit code: 247

Approximately 100% of recent cases of bug 614146 fall under this bug. Some examples from the past couple days: https://tbpl.mozilla.org/php/getParsedLog.php?id=7358681&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=7338548&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=7338133&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=7334635&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=7330654&tree=Firefox https://tbpl.mozilla.org/php/getParsedLog.php?id=7318174&tree=Mozilla-Inbound Moving this to Server Ops because it seems to be specific to the new rev4 slaves. Feel free to reassign it elsewhere if appropriate.

This isn't actionable for us - releng should investigate, and if it's a hardware issue, we'll take care of it.

Joel, have you seen this or done any triage?

I don't see anything in the screenshots in these logs that I have checked (comment 14,13) that points to machine configuration. Interesting is that the NightlyDebug application is showing in the menubar but there are no windows of Firefox being drawn on screen. The state of the finder window in the screenshots suggest that NightlyDebug also has focus. Comparing the last common bits of the failing log: WARNING: not an nsIRDFRemoteDataSource: 'remote != nsnull', file /builds/slave/srv-cen-osx64-dbg/build/rdf/datasource/src/nsLocalStore.cpp, line 312 WARNING: NS_ENSURE_TRUE(asyncCloseWasCalled) failed: file /builds/slave/srv-cen-osx64-dbg/build/storage/src/mozStorageConnection.cpp, line 835 WARNING: not an nsIRDFRemoteDataSource: 'remote != nsnull', file /builds/slave/srv-cen-osx64-dbg/build/rdf/datasource/src/nsLocalStore.cpp, line 312 TEST-UNEXPECTED-FAIL | Shutdown | application timed out after 330 seconds with no output With a passing log [1]: WARNING: not an nsIRDFRemoteDataSource: 'remote != nsnull', file /builds/slave/srv-cen-osx64-dbg/build/rdf/datasource/src/nsLocalStore.cpp, line 312 WARNING: not an nsIRDFRemoteDataSource: 'remote != nsnull', file /builds/slave/srv-cen-osx64-dbg/build/rdf/datasource/src/nsLocalStore.cpp, line 312 WARNING: not an nsIRDFRemoteDataSource: 'remote != nsnull', file /builds/slave/srv-cen-osx64-dbg/build/rdf/datasource/src/nsLocalStore.cpp, line 312 WARNING: NS_ENSURE_TRUE(mTextInputHandler) failed: file /builds/slave/srv-cen-osx64-dbg/build/widget/src/cocoa/nsChildView.mm, line 4068 WARNING: OOPDeinit() without successful OOPInit(): file /builds/slave/srv-cen-osx64-dbg/build/toolkit/crashreporter/nsExceptionHandler.cpp, line 1938 nsStringStats => mAllocCount: 2459159 => mReallocCount: 205499 => mFreeCount: 2459159 => mShareCount: 3770692 => mAdoptCount: 230551 => mAdoptFreeCount: 230551 INFO | automation.py | Application ran for: 0:06:50.225722 INFO | automation.py | Reading PID log: /var/folders/Hs/HsDn6a9SG8idoIya6p9mtE+++TI/-Tmp-/tmpTUsQWwpidlog It looks like the difference here is that asyncCloseWasCalled fails when the browser hangs. I wonder if this is an async timing issue that is exacerbated, not caused, by the hardware change. [1] https://tbpl.mozilla.org/php/getParsedLog.php?id=7397619&tree=Services-Central&full=1

This issue is happening on Fedora 64 (comment 23, 20) and Windows (comment 22,21). Updating bug to reflect that this is not a rev4 osx specific issue. I am also going to mark this as a dup of bug 614146 since there is no reason to track OS X failures separately from Windows and Fedora ones, especially as they are now being reported in either bug.