Hit this on shutdown, sdwilsh says it's probably the favicon service messing something up: WARNING: SQL statement 'SELECT f.data, f.mime_type FROM moz_favicons f WHERE url = ?1' was not finalized: file c:/Home/src/mozilla/electrolysis/storage/src/mozS torageConnection.cpp, line 509 ###!!! ASSERTION: sqlite3_close failed. There are probably outstanding statement s that are listed above!: 'srv == SQLITE_OK', file c:/Home/src/mozilla/electroly sis/storage/src/mozStorageConnection.cpp, line 522

stack: ntdll.dll!_DbgBreakPoint@0() xul.dll!Break(const char * aMsg=0x0012ef1c) Line 489 C++ xul.dll!NS_DebugBreak_P(unsigned int aSeverity=0x00000001, const char * aStr=0x11aea8c0, const char * aExpr=0x11aea8a8, const char * aFile=0x11aea860, int aLine=0x0000020a) Line 354 + 0xc bytes C++ xul.dll!mozilla::storage::Connection::Close() Line 522 + 0x21 bytes C++ xul.dll!mozilla::storage::Connection::~Connection() Line 264 C++ xul.dll!mozilla::storage::Connection::`scalar deleting destructor'() + 0xf bytes C++ xul.dll!mozilla::storage::Connection::Release() Line 273 + 0x8e bytes C++ xul.dll!nsCOMPtr<mozIStorageConnection>::~nsCOMPtr<mozIStorageConnection>() Line 511 C++ xul.dll!nsNavBookmarks::~nsNavBookmarks() Line 112 + 0x136 bytes C++ xul.dll!nsNavBookmarks::`scalar deleting destructor'() + 0xf bytes C++ xul.dll!nsNavBookmarks::Release() Line 117 + 0xcb bytes C++ xul.dll!DoDeferredRelease<nsISupports *>(nsTArray<nsISupports *> & array={...}) Line 493 + 0xe bytes C++ xul.dll!XPCJSRuntime::GCCallback(JSContext * cx=0x0354f9e0, JSGCStatus status=JSGC_END) Line 766 + 0xf bytes C++ xul.dll!DOMGCCallback(JSContext * cx=0x0354f9e0, JSGCStatus status=JSGC_END) Line 3707 + 0x17 bytes C++ js3250.dll!js_GC(JSContext * cx=0x0354f9e0, JSGCInvocationKind gckind=GC_NORMAL) Line 3840 + 0x9 bytes C++ js3250.dll!js_DestroyContext(JSContext * cx=0x0354f9e0, JSDestroyContextMode mode=JSDCM_FORCE_GC) Line 714 + 0xb bytes C++ js3250.dll!JS_DestroyContext(JSContext * cx=0x0354f9e0) Line 1067 + 0xb bytes C++ xul.dll!XPCJSContextStack::~XPCJSContextStack() Line 64 + 0x10 bytes C++ xul.dll!XPCJSContextStack::`scalar deleting destructor'() + 0xf bytes C++ xul.dll!XPCPerThreadData::CleanupAllThreads() Line 542 + 0x27 bytes C++ xul.dll!nsXPConnect::~nsXPConnect() Line 144 C++ xul.dll!nsXPConnect::`scalar deleting destructor'() + 0xf bytes C++ xul.dll!nsXPConnect::Release() Line 62 + 0x93 bytes C++ xul.dll!nsScriptSecurityManager::Shutdown() Line 3379 + 0x1d bytes C++ xul.dll!CapsModuleDtor(nsIModule * thisModules=0x018b9a88) Line 504 C++ xul.dll!nsGenericModule::Shutdown() Line 340 + 0xc bytes C++ xul.dll!nsGenericModule::~nsGenericModule() Line 243 C++ xul.dll!nsGenericModule::`scalar deleting destructor'() + 0xf bytes C++ xul.dll!nsGenericModule::Release() Line 245 + 0x8e bytes C++ xul.dll!nsCOMPtr<nsIModule>::assign_assuming_AddRef(nsIModule * newPtr=0x00000000) Line 496 C++ xul.dll!nsCOMPtr<nsIModule>::assign_with_AddRef(nsISupports * rawPtr=0x00000000) Line 1182 C++ xul.dll!nsCOMPtr<nsIModule>::operator=(nsIModule * rhs=0x00000000) Line 641 C++ xul.dll!info_ClearEntry(PLDHashTable * table=0x01811e08, PLDHashEntryHdr * entry=0x01820454) Line 71 C++ xul.dll!PL_DHashTableFinish(PLDHashTable * table=0x01811e08) Line 399 + 0x12 bytes C xul.dll!nsStaticModuleLoader::ReleaseModules() Line 70 + 0x9 bytes C++ xul.dll!nsComponentManagerImpl::Shutdown() Line 747 C++ xul.dll!mozilla::ShutdownXPCOM(nsIServiceManager * servMgr=0x00000000) Line 908 + 0xb bytes C++ xul.dll!NS_ShutdownXPCOM_P(nsIServiceManager * servMgr=0x01811d7c) Line 784 + 0x9 bytes C++ xul.dll!ScopedXPCOMStartup::~ScopedXPCOMStartup() Line 1013 + 0xb bytes C++ xul.dll!XRE_main(int argc=0x00000003, char * * argv=0x00c1d5f8, const nsXREAppData * aAppData=0x00c1dd48) Line 3470 C++ firefox.exe!NS_internal_main(int argc=0x00000003, char * * argv=0x00c1d5f8) Line 156 + 0x12 bytes C++ firefox.exe!wmain(int argc=0x00000003, wchar_t * * argv=0x00c1b318) Line 110 + 0xd bytes C++ firefox.exe!__tmainCRTStartup() Line 583 + 0x19 bytes C firefox.exe!wmainCRTStartup() Line 403 C kernel32.dll!_BaseProcessStart@4() + 0x23 bytes

while working on bug 478718 i hit this. i can reproduce this constantly now, so i'll take a look if i can find what's up.

This looks interesting: 0[d334a0]: Finalizing statement 'SELECT f.data, f.mime_type FROM moz_favicons f WHERE url = ?1' 0[d334a0]: WARNING: An event was posted to a thread that will never run it (rejected): file c:/mozilla-central/xpcom/threads/nsThread.cpp, line 362 0[d334a0]: WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x8000FFFF: file c:/mozilla-central/storage/src/mozStorageStatement.cpp, line 401

we use the statement in GetDataAsync, that is used by the anno protocol handler. sounds like the thread this was created on is now died.

mozStorageStatement.cpp :: finalize() 394 if (mCachedAsyncStatement) { 395 nsCOMPtr<nsIRunnable> event = 396 new AsyncStatementFinalizer(mCachedAsyncStatement); 397 NS_ENSURE_TRUE(event, NS_ERROR_OUT_OF_MEMORY); 398 399 nsCOMPtr<nsIEventTarget> target = mDBConnection->getAsyncExecutionTarget(); 400 nsresult rv = target->Dispatch(event, NS_DISPATCH_NORMAL); 401 NS_ENSURE_SUCCESS(rv, rv); 402 mCachedAsyncStatement = NULL; 403 } So, target here is not anymore a valid thread to dispatch an event, Shawn, could we check NS_FAILED and in case spawn a new thread and dispatch the finalize event to the new thread?

I don't think you want to be spawning any threads after we've started shutting down XPCOM.

(In reply to comment #6) > So, target here is not anymore a valid thread to dispatch an event, Shawn, > could we check NS_FAILED and in case spawn a new thread and dispatch the > finalize event to the new thread? The only way it's going to not be accepting any more events is if it's been told to shutdown. This happens when the threadmanager tells it to (xpcom-shutdown-threads) and when you call close on a connection. Nobody else will tell the thread to shutdown, so one of these two things is happening.

but xpcom-shutdown-threads comes after xpcom-shutdown, where this runs.

right, so you'll have to get a backtrace on whoever is calling Shutdown on the thread in question.

unassigning and tentatively depends on bug 507199, could be the same issue.

this should now be fixed.

Still happens for me (mozilla-central 34525ff5b1f1+)

ops, right, in this case target thread exists, but it is unable to process further events. Since we are already doing that in the case of target does not exists, Shawn could not we just finalize on the main thread, so nsresult rv = target->Dispatch(event, NS_DISPATCH_NORMAL); if (NS_FAILED(rv)) (void)::sqlite3_finalize(mCachedAsyncStatement); we are already doing the same if the thread does not exists anymore.

If the thread is not accepting events because it's shutting down, we won't hand it out, so that shouldn't be the issue: http://mxr.mozilla.org/mozilla-central/source/storage/src/mozStorageConnection.cpp#283

Did this fell off the radar?

> strgcmps.dll!mozilla::storage::Statement::Finalize() Line 395 C++ strgcmps.dll!mozilla::storage::Statement::~Statement() Line 345 C++ strgcmps.dll!mozilla::storage::Statement::`scalar deleting destructor'() + 0xf bytes C++ strgcmps.dll!mozilla::storage::Statement::Release() Line 348 + 0x90 bytes C++ places.dll!nsCOMPtr<mozIStorageStatement>::~nsCOMPtr<mozIStorageStatement>() Line 511 C++ places.dll!nsFaviconService::~nsFaviconService() Line 161 + 0x4d bytes C++ places.dll!nsFaviconService::`scalar deleting destructor'() + 0xf bytes C++ places.dll!nsFaviconService::Release() Line 141 + 0xd4 bytes C++ xpcom_core.dll!nsCOMPtr_base::assign_assuming_AddRef(nsISupports * newPtr=0x00000000) Line 457 C++ xpcom_core.dll!nsCOMPtr_base::assign_with_AddRef(nsISupports * rawPtr=0x00000000) Line 90 C++ xpcom_core.dll!nsCOMPtr<nsISupports>::operator=(nsISupports * rhs=0x00000000) Line 976 C++ xpcom_core.dll!FreeServiceContractIDEntryEnumerate(PLDHashTable * aTable=0x00e10b70, PLDHashEntryHdr * aHdr=0x00e13b80, unsigned int aNumber=83, void * aData=0x00000000) Line 1736 C++ xpcom_core.dll!PL_DHashTableEnumerate(PLDHashTable * table=0x00e10b70, PLDHashOperator (PLDHashTable *, PLDHashEntryHdr *, unsigned int, void *)* etor=0x0048f280, void * arg=0x00000000) Line 754 + 0x19 bytes C xpcom_core.dll!nsComponentManagerImpl::FreeServices() Line 1748 + 0x13 bytes C++ xpcom_core.dll!mozilla::ShutdownXPCOM(nsIServiceManager * servMgr=0x00000000) Line 831 C++ xpcom_core.dll!NS_ShutdownXPCOM_P(nsIServiceManager * servMgr=0x00000000) Line 744 + 0x9 bytes C++ xpcom.dll!NS_ShutdownXPCOM(nsIServiceManager * svcMgr=0x00000000) Line 179 + 0xa bytes C++ xpcshell.exe!main(int argc=15, char * * argv=0x00d2eb74, char * * envp=0x00d23e30) Line 1795 + 0x8 bytes C++ xpcshell.exe!__tmainCRTStartup() Line 586 + 0x19 bytes C xpcshell.exe!mainCRTStartup() Line 403 C

sounds like we are trying to finalize the statement when the service destructor runs, but the connection here has been already closed, and the statement should have been already finalized before this point.

in my case we were not initing nsPlacesDBFlush, for this reason the statements finalization was not done by it, instead it was running later at the destructor level (could even be after xpcom-shutdown-threads). since now nsPlacesDBFlush is a category observer, if there is nothing that notifies it, it is not initialized and we fail to finalize statements. In bug 478718 i'm trying to move shutdown work to history service, and just leave connection close to the sync service. Btw, i don't know if previous cases related to this error could be the same issue.

a lot of changes to shutdown have been done in 4.0, so that it's no more "spawning" events around to do crazy stuff. I suspect this bug is wfm now, any new stack would be quite different.