It's #47 top crasher in 9.0b3 and first appeared in 9.0a1/20110916. The regression range is: http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=f2a2adaaacba&tochange=39b192706927 The likely culprit is bug 683441. Signature je_free | plds4.dll@0x168f UUID 604d7732-0676-46ff-8441-afcca2111129 Date Processed 2011-11-29 02:34:43.608093 Uptime 903 Last Crash 3.2 weeks before submission Install Age 3.1 days since version was first installed. Install Time 2011-11-26 06:58:51 Product Firefox Version 9.0 Build ID 20111122192043 Release Channel beta OS Windows NT OS Version 5.1.2600 Service Pack 3 Build Architecture x86 Build Architecture Info GenuineIntel family 15 model 1 stepping 2 Crash Reason EXCEPTION_ACCESS_VIOLATION_READ Crash Address 0x1000 App Notes AdapterVendorID: 8086, AdapterDeviceID: 2562, AdapterSubsysID: 71481462, AdapterDriverVersion: 6.14.10.4020 D3D10 Layers? D3D10 Layers- D3D9 Layers? D3D9 Layers- EMCheckCompatibility True Frame Module Signature [Expand] Source 0 mozutils.dll je_free memory/jemalloc/jemalloc.c:6260 1 plds4.dll plds4.dll@0x168f 2 plds4.dll DefaultFreeEntry nsprpub/lib/ds/plhash.c:87 3 plds4.dll PL_HashTableDestroy nsprpub/lib/ds/plhash.c:147 4 nssutil3.dll SECOID_Shutdown security/nss/lib/util/secoid.c:2161 5 nss3.dll nss_Shutdown security/nss/lib/nss/nssinit.c:1101 6 nss3.dll NSS_Shutdown security/nss/lib/nss/nssinit.c:1156 7 xul.dll nsNSSComponent::ShutdownNSS security/manager/ssl/src/nsNSSComponent.cpp:1918 8 xul.dll nsNSSComponent::DoProfileBeforeChange security/manager/ssl/src/nsNSSComponent.cpp:2597 9 xul.dll nsNSSComponent::Observe security/manager/ssl/src/nsNSSComponent.cpp:2231 10 xul.dll nsObserverList::NotifyObservers xpcom/ds/nsObserverList.cpp:130 11 xul.dll nsObserverService::NotifyObservers xpcom/ds/nsObserverService.cpp:182 12 xul.dll xul.dll@0xbc1e43 13 xul.dll ScopedXPCOMStartup::~ScopedXPCOMStartup toolkit/xre/nsAppRunner.cpp:1080 14 xul.dll XRE_main toolkit/xre/nsAppRunner.cpp:3587 15 firefox.exe wmain toolkit/xre/nsWindowsWMain.cpp:107 16 firefox.exe firefox.exe@0x4033 17 firefox.exe __tmainCRTStartup crtexe.c:594 18 xul.dll nsRefreshDriver::~nsRefreshDriver layout/base/nsRefreshDriver.cpp:122 19 firefox.exe _SEH_epilog4 More reports at: https://crash-stats.mozilla.com/report/list?signature=je_free%20|%20plds4.dll%400x168f

The crash address is an interestingly round number, and the list of crashes shows that pretty consistently (0x1000, 0x2000, 0xb000, etc). Page boundaries? Don't know if accessing memory that low could be exploited or not. Also appears to be a shutdown crash which would be tricky to convince users to do as part of an attack. Guessing at sg:moderate as an appropriate rating.

(In reply to Scoobidiver from comment #0) > It's #47 top crasher in 9.0b3 and first appeared in 9.0a1/20110916. > The regression range is: > http://hg.mozilla.org/mozilla-central/ > pushloghtml?fromchange=f2a2adaaacba&tochange=39b192706927 > The likely culprit is bug 683441. The likely culprit is bug 677501 or bug 680440.

I'm not able to reproduce these crashes, but having glanced at the code the only way I can think of that could explain the problem is if somehow the dynOidHash values are allocated from the crt and freed with jemalloc. I see that nspr still imports the _malloc_crt symbol, though I haven't looked where it is being used from. If that is the real problem, the corresponding culprit would actually be new style jemalloc (bug 515492) and bug 678195 which switches to it everywhere. Ted, Kyle, what do you think?

Kyle Huey@KYLEHUEY-PC /c/dev/mozilla-beta/obj-i686-pc-mingw32/nsprpub $ grep 'malloc_crt' -r ./ Binary file ./lib/ds/plds4.dll matches Binary file ./lib/ds/plds4.ilk matches ./lib/ds/plds4.map: 0001:000019d6 __malloc_crt 100029d6 f mozcrt:MSVCR100.dll ./lib/ds/plds4.map: 0004:00000278 __imp___malloc_crt 10008278 mozcrt:MSVCR100.dll Binary file ./lib/ds/plds4.pdb matches Binary file ./lib/libc/src/plc4.dll matches Binary file ./lib/libc/src/plc4.ilk matches ./lib/libc/src/plc4.map: 0001:00002888 __malloc_crt 10003888 f mozcrt:MSVCR100.dll ./lib/libc/src/plc4.map: 0004:00000298 __imp___malloc_crt 10008298 mozcrt:MSVCR100.dll Binary file ./lib/libc/src/plc4.pdb matches Binary file ./pr/src/nspr4.dll matches Binary file ./pr/src/nspr4.ilk matches ./pr/src/nspr4.map: 0001:00032e50 __malloc_crt 10033e50 f mozcrt:MSVCR100.dll ./pr/src/nspr4.map: 0004:00000774 __imp___malloc_crt 10041774 mozcrt:MSVCR100.dll Binary file ./pr/src/nspr4.pdb matches That's consistent with malloc_crt only being used in the CRT glue and not in our actual code. This looks like an allocator mismatch at first glance (as do some other bugs that I've seen lately) but I haven't been able to actually find any mismatch.

Kyle Huey@KYLEHUEY-PC /c/dev/mozilla-beta/obj-i686-pc-mingw32/nsprpub $ dumpbin.exe -IMPORTS ./lib/ds/plds4.dll Microsoft (R) COFF/PE Dumper Version 10.00.40219.01 Copyright (C) Microsoft Corporation. All rights reserved. Dump of file ./lib/ds/plds4.dll File Type: DLL Section contains the following imports: nspr4.dll 100082F4 Import Address Table 10008184 Import Name Table 0 time date stamp 0 Index of first forwarder reference 52 PR_DestroyLock 85 PR_Free 166 PR_Unlock F5 PR_NewLock 27 PR_CallOnce E7 PR_Lock ED PR_Malloc 2D PR_CeilingLog2 MSVCR100.dll 1000824C Import Address Table 100080DC Import Name Table 0 time date stamp 0 Index of first forwarder reference 2B1 _initterm_e 131 __CppXcptFilter 153 __clean_type_info_names_internal 48D _unlock 15B __dllonexit 323 _lock 3C9 _onexit 221 _except_handler4_common 2B0 _initterm 1FB _crt_debugger_hook 219 _encoded_null 334 _malloc_crt 5F8 strcmp 5D3 memset 5CF memcpy 1C5 _amsg_exit mozutils.dll 100082C4 Import Address Table 10008154 Import Name Table 0 time date stamp 0 Index of first forwarder reference 5 frex KERNEL32.dll 100081D4 Import Address Table 10008064 Import Name Table 0 time date stamp 0 Index of first forwarder reference 300 IsDebuggerPresent 4A5 SetUnhandledExceptionFilter 4D3 UnhandledExceptionFilter 1C0 GetCurrentProcess 4C0 TerminateProcess 279 GetSystemTimeAsFileTime 1C1 GetCurrentProcessId 1C5 GetCurrentThreadId 2E9 InterlockedCompareExchange 4B2 Sleep 2EC InterlockedExchange CA DecodePointer 293 GetTickCount EA EncodePointer 3A7 QueryPerformanceCounter DE DisableThreadLibraryCalls Kyle Huey@KYLEHUEY-PC /c/dev/mozilla-beta/obj-i686-pc-mingw32/nsprpub $ dumpbin.exe -IMPORTS ./pr/src/nspr4.dll Microsoft (R) COFF/PE Dumper Version 10.00.40219.01 Copyright (C) Microsoft Corporation. All rights reserved. Dump of file ./pr/src/nspr4.dll File Type: DLL Section contains the following imports: ADVAPI32.dll 10041534 Import Address Table 1004108C Import Name Table 0 time date stamp 0 Index of first forwarder reference 20 AllocateAndInitializeSid 76 CopySid 136 GetLengthSid 15A GetTokenInformation 1F7 OpenProcessToken 120 FreeSid 2B6 SetSecurityDescriptorDacl 10 AddAccessAllowedAce 176 InitializeAcl 2B7 SetSecurityDescriptorGroup 2B8 SetSecurityDescriptorOwner 177 InitializeSecurityDescriptor WSOCK32.dll 100418D8 Import Address Table 10041430 Import Name Table 0 time date stamp 0 Index of first forwarder reference Ordinal 52 Ordinal 54 Ordinal 9 Ordinal 8 Ordinal 53 Ordinal 51 Ordinal 18 Ordinal 151 Ordinal 5 Ordinal 6 Ordinal 22 Ordinal 17 Ordinal 20 Ordinal 19 Ordinal 16 Ordinal 13 Ordinal 2 Ordinal 4 Ordinal 1 Ordinal 23 Ordinal 12 Ordinal 3 Ordinal 21 Ordinal 115 Ordinal 116 Ordinal 57 Ordinal 7 Ordinal 111 Ordinal 14 Ordinal 15 WINMM.dll 100418A8 Import Address Table 10041400 Import Name Table 0 time date stamp 0 Index of first forwarder reference 94 timeGetTime MSVCR100.dll 1004175C Import Address Table 100412B4 Import Name Table 0 time date stamp 0 Index of first forwarder reference 219 _encoded_null 2B0 _initterm 2B1 _initterm_e 1C5 _amsg_exit 131 __CppXcptFilter 1FB _crt_debugger_hook 334 _malloc_crt 153 __clean_type_info_names_internal 48D _unlock 15B __dllonexit 323 _lock 3C9 _onexit 221 _except_handler4_common 471 _time64 1BA _access 39E _mbspbrk 35E _mbsdec 437 _stat64i32 364 _mbsinc 22A _exit 1D4 _beginthreadex 441 _stricmp 607 strpbrk 21C _environ 5DE qsort 5FF strftime 3C6 _mktime64 5A3 isalpha 321 _localtime64 603 strncmp 5FD strerror 5F5 strcat 608 strrchr 60A strstr 59B getenv 61B tolower 60E strtol 60F strtoul 5F7 strchr 5CD memchr 5AB isspace 5A5 isdigit 5EF sprintf 583 fprintf 55C abort 596 fwrite 579 fflush 5F8 strcmp 581 fopen 5EB setvbuf 576 fclose 5F3 sscanf 165 __iob_func 600 strlen 5FA strcpy 5CF memcpy 566 atoi 5D3 memset 21E _errno 3DB _putenv mozutils.dll 10041994 Import Address Table 100414EC Import Name Table 0 time date stamp 0 Index of first forwarder reference C strdup 7 malloc 3 calloc B realloc 5 frex 4 free KERNEL32.dll 10041598 Import Address Table 100410F0 Import Name Table 0 time date stamp 0 Index of first forwarder reference 279 GetSystemTimeAsFileTime 1C1 GetCurrentProcessId 1C5 GetCurrentThreadId 304 IsProcessorFeaturePresent 300 IsDebuggerPresent 4A5 SetUnhandledExceptionFilter 4D3 UnhandledExceptionFilter 2E9 InterlockedCompareExchange CA DecodePointer EA EncodePointer 378 OpenFileMappingA 3A7 QueryPerformanceCounter 293 GetTickCount 383 OpenSemaphoreA 33F LoadLibraryW 339 LeaveCriticalSection 4D4 UnlockFile 352 LockFile 400 RemoveDirectoryA 7C CreateDirectoryA 35E MoveFileA 1FF GetHandleInformation 470 SetHandleInformation 1EC GetFileInformationByHandle 1D2 GetDriveTypeA 473 SetLastError 1F8 GetFullPathNameA D3 DeleteFileA 143 FindNextFileA 132 FindFirstFileA 12E FindClose 157 FlushFileBuffers 413 ResumeThread 1C4 GetCurrentThread E8 DuplicateHandle 4C6 TlsFree 4C5 TlsAlloc 286 GetThreadContext 3FE ReleaseSemaphore 466 SetFilePointer AB CreateSemaphoreA 525 WriteFile 1C0 GetCurrentProcess 4D6 UnmapViewOfFile 357 MapViewOfFile 15D FormatMessageA 89 CreateFileMappingA 2A3 GetVersionExA 4C0 TerminateProcess 4F9 WaitForSingleObject 1DF GetExitCodeProcess 1D8 GetEnvironmentStrings 160 FreeEnvironmentStringsA A4 CreateProcessA 277 GetSystemTime 4BD SystemTimeToFileTime 4C8 TlsSetValue D1 DeleteCriticalSection 2E2 InitializeCriticalSection 4C7 TlsGetValue 3C0 ReadFile 88 CreateFileA 4BA SuspendThread 4B2 Sleep 264 GetStdHandle 499 SetThreadPriority A1 CreatePipe 52 CloseHandle 2BF GlobalMemoryStatus 2EC InterlockedExchange 273 GetSystemInfo 218 GetModuleHandleW 214 GetModuleFileNameW 245 GetProcAddress 162 FreeLibrary 367 MultiByteToWideChar 202 GetLastError 511 WideCharToMultiByte 33E LoadLibraryExW 215 GetModuleHandleA C7 DebugBreak EE EnterCriticalSection 389 OutputDebugStringA 490 SetThreadAffinityMask 246 GetProcessAffinityMask Everything looks in order to me ...

It's currently #37 top crasher in 9.0b4.

Since it doesn't happen on my machine, and since it doesn't happen for all users (in which case it would be top #1), i suppose something else (AV software?) is triggering it. Question is: how can we narrow it down?

I don't think there's anything actionable here that I can do. Maybe chofman can try to glean something from the crash data?