Tested with 10.3.181.14 Crash 1 1. http://www.myspace.com/plies 2. Crash plugin-container.exe with corrupt heap during local testing. Crash 2 1. http://www.streamiz.com/film--streaming--page-13-lecteur-.html 2. sometimes crash with corrupt heap. This may be a loaded advertisement that is periodic. This may be related to bug 651575. I believe I did see an ad from ad.xtendmedia.com/ when testing this crash. Automation found: Operating system: Windows NT 5.1.2600 Service Pack 3 CPU: x86 GenuineIntel family 6 model 44 stepping 2 1 CPU Crash reason: EXCEPTION_ACCESS_VIOLATION_WRITE Crash address: 0x38 Assertion: Unknown assertion type 0x00000000 Thread 0 (crashed) 0 xul.dll!std::_Construct<MessageLoop::PendingTask,MessageLoop::PendingTask>(MessageLoop::PendingTask *,MessageLoop::PendingTask const &) [xmemory : 53 + 0x1 eip = 0x01d4edeb esp = 0x0012e738 ebp = 0x0012e744 ebx = 0x00000000 esi = 0x00000001 edi = 0x07734988 eax = 0x00000038 ecx = 0x045641e0 edx = 0x0012e7cc efl = 0x00210202 Found by: given as instruction pointer in context 1 xul.dll!std::allocator<MessageLoop::PendingTask>::construct(MessageLoop::PendingTask *,MessageLoop::PendingTask const &) [xmemory : 156 + 0xc] eip = 0x01d4c814 esp = 0x0012e74c ebp = 0x0012e758 Found by: call frame info 2 xul.dll!std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> >::push_back(MessageLoop::PendingTask const &) [deque : 826 + 0x2a] eip = 0x01d4bc46 esp = 0x0012e760 ebp = 0x0012e778 Found by: call frame info 3 xul.dll!std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > >::push(MessageLoop::PendingTask eip = 0x01d4ad73 esp = 0x0012e780 ebp = 0x0012e788 Found by: call frame info 4 xul.dll!MessageLoop::PostTask_Helper(tracked_objects::Location const &,Task *,int,bool) [message_loop.cc : 293 + 0x11] eip = 0x01d49f8f esp = 0x0012e790 ebp = 0x0012e7e4 Found by: call frame info 5 xul.dll!MessageLoop::PostTask(tracked_objects::Location const &,Task *) [message_loop.cc : 251 + 0x13] eip = 0x01d49e1b esp = 0x0012e7ec ebp = 0x0012e800 Found by: call frame info 6 xul.dll!mozilla::ipc::RPCChannel::EnqueuePendingMessages() [RPCChannel.cpp : 366 + 0x5c] eip = 0x01b3512a esp = 0x0012e808 ebp = 0x0012e844 Found by: call frame info original socorro report bp-f3ef37b3-08e9-4a9a-a7e7-19e0b2110513 Crash Report [@ UserCallWinProcCheckWow ]

tracking internally as #bug=2875277. we are looking into it...

was asked to close this based that Flash Player in not in the stack.

I'm very disappointed in your early dismissal. I normally don't include full stacks, but will if necessary to get your attention.

thanks. the full stack does help.

bob - dev is not finding the 2 files as informative. would you be able to attach the binary dump file? thank you.

I have the dmp file that Firefox produces and can attach that, but I doubt that is what they want. I bet they want the minidump created from windbg?

I reproduced it twice with the myspace url by going to the page and attaching windbg to the plugin-container process then reloading. The saved dmp files are 131M and 158M so I'm not sure how you would like to handle it. You should be able to easily reproduce with a debug Firefox build and windbg. In windbg it looks like: HEAP[plugin-container.exe]: Invalid Address specified to RtlValidateHeap( 03360000, 03D8FC40 ) (11f4.d0c): Break instruction exception - code 80000003 (first chance) eax=03d8fc38 ebx=03d8fc38 ecx=7c91d4fd edx=0012e2ce esi=03360000 edi=03360000 eip=7c90120e esp=0012e4d4 ebp=0012e4d8 iopl=0 nv up ei pl nz na po nc cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00200202 ntdll!DbgBreakPoint: 7c90120e cc int 3 WARNING: Stack unwind information not available. Following frames may be wrong. ntdll!DbgBreakPoint ntdll!RtlpNtMakeTemporaryKey+0x6b72 ntdll!RtlValidateHeap+0xe0 kernel32!HeapValidate+0x14 MSVCR80D!CrtIsValidHeapPointer+0x15a MSVCR80D!free_dbg+0x196 MSVCR80D!free_dbg+0x4e MSVCR80D!free+0xe mozalloc!moz_free(void * ptr = 0x03d8fc60)+0xd xul!operator delete(void * ptr = 0x03d8fc60)+0xd xul!std::allocator<std::_List_nod<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *>,std::allocator<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *> > >::_Node>::deallocate(struct std::_List_nod<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *>,std::allocator<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *> > >::_Node * _Ptr = 0x03d8fc60, unsigned int __formal = 1)+0x10 xul!std::list<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *>,std::allocator<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *> > >::erase(class std::list<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *>,std::allocator<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *> > >::_Iterator<1> _Where = class std::list<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *>,std::allocator<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *> > >::_Iterator<1>)+0xe6 xul!stdext::_Hash<stdext::_Hmap_traits<int,mozilla::ipc::RPCChannel::RPCListener *,stdext::hash_compare<int,std::less<int> >,std::allocator<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *> >,0> >::erase(class std::list<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *>,std::allocator<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *> > >::_Iterator<1> _Where = class std::list<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *>,std::allocator<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *> > >::_Iterator<1>)+0x8f xul!IDMap<mozilla::ipc::RPCChannel::RPCListener>::Remove(int id = -275)+0x9c xul!mozilla::plugins::PPluginModuleChild::Unregister(int aId = -275)+0x19 xul!mozilla::plugins::PPluginInstanceChild::Unregister(int aId = -275)+0x1e xul!mozilla::plugins::PStreamNotifyChild::Unregister(int aId = -275)+0x1e xul!mozilla::plugins::PStreamNotifyChild::DestroySubtree(mozilla::ipc::IProtocolManager<mozilla::ipc::RPCChannel::RPCListener>::ActorDestroyReason why = Deletion (1))+0x1f xul!mozilla::plugins::PStreamNotifyChild::OnMessageReceived(class IPC::Message * __msg = 0x0012e8e0)+0x33e xul!mozilla::plugins::PPluginModuleChild::OnMessageReceived(class IPC::Message * __msg = 0x0012e8e0)+0x62 It doesn't have flash on the stack however if that matters though I'm not sure how much to trust the stack especially since I don't know how to start plugin-container with the debugger already attached. I don't have a 3.5 debug build handy to try a non out of process plugin though.

http://www.pspiso.com/ - Automation reproduced crash with Windows XP Nightly debug build and Flash 10.3.181.22 but not with any other combination of Windows XP, Windows 7 and 2.0, beta, aurora, nightly. http://www.myspace.com/plies - Automation could not reproduce the crash with at all. Manually testing using the following steps on Windows XP: 1. Start MSVC 2. Start *DEBUG* build of Firefox 3. Navigate to cnn.com to activate plugin-container 4. Attach MSVC to plugin-container process 5. Load url 6. Shutdown http://www.myspace.com/plies Nightly - Normal http://www.myspace.com/plies Aurora - breakpoint ntdll.dll!7c90120e() [Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll] ntdll.dll!7c96ee31() ntdll.dll!7c96f26e() ntdll.dll!7c962fe0() ntdll.dll!7c90f65c() > msvcp80d.dll!_Mtxlock(_RTL_CRITICAL_SECTION * _Mtx=0x03340000) Line 45 C kernel32.dll!7c85f9a7() msvcr80d.dll!_CrtIsValidHeapPointer(const void * pUserData=0x03d6dc90) Line 2072 C++ msvcr80d.dll!_free_dbg_nolock(void * pUserData=0x03d6dc90, int nBlockUse=1) Line 1279 + 0x9 bytes C++ msvcr80d.dll!_free_dbg(void * pUserData=0x03d6dc90, int nBlockUse=1) Line 1220 + 0xd bytes C++ msvcr80d.dll!free(void * pUserData=0x03d6dc90) Line 1178 + 0xb bytes C++ mozalloc.dll!moz_free(void * ptr=0x03d6dc90) Line 94 + 0xa bytes C++ xul.dll!operator delete(void * ptr=0x03d6dc90) Line 253 + 0xa bytes C++ xul.dll!std::allocator<MessageLoop::PendingTask>::deallocate(MessageLoop::PendingTask * _Ptr=0x03d6dc90, unsigned int __formal=1) Line 141 + 0x9 bytes C++ xul.dll!std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> >::_Tidy() Line 1254 C++ xul.dll!std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> >::clear() Line 1081 C++ xul.dll!std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> >::operator=(const std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > & _Right=[0]()) Line 622 + 0x8 bytes C++ xul.dll!std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > >::operator=(const std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > & __that=[0]()) + 0x13 bytes C++ xul.dll!std::swap<std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > >(std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > & _Left=[0](), std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > & _Right=[0]()) Line 19 + 0xc bytes C++ xul.dll!MessageLoop::ReloadWorkQueue() Line 385 + 0x16 bytes C++ xul.dll!MessageLoop::DoWork() Line 437 C++ xul.dll!base::MessagePumpForIO::DoRunLoop() Line 454 + 0x17 bytes C++ xul.dll!base::MessagePumpWin::RunWithDispatcher(base::MessagePump::Delegate * delegate=0x03d2feb8, base::MessagePumpWin::Dispatcher * dispatcher=0x00000000) Line 54 C++ xul.dll!base::MessagePumpWin::Run(base::MessagePump::Delegate * delegate=0x03d2feb8) Line 78 + 0x15 bytes C++ xul.dll!MessageLoop::RunInternal() Line 219 C++ xul.dll!MessageLoop::RunHandler() Line 203 C++ xul.dll!MessageLoop::Run() Line 177 C++ xul.dll!base::Thread::ThreadMain() Line 159 C++ xul.dll!`anonymous namespace'::ThreadFunc(void * closure=0x0334eb38) Line 27 C++ kernel32.dll!7c80b729() xul.dll!mozilla::SVGLengthListSMILType::Add(nsSMILValue & aDest={...}, const nsSMILValue & aValueToAdd={...}, unsigned int aCount=2422607882) Line 183 + 0x3e bytes C++ xul.dll!nsDocShell::DoURILoad(nsIURI * aURI=0x296201dd, nsIURI * aReferrerURI=0x2b4555e4, int aSendReferrer=1298266322, nsISupports * aOwner=0xcf795128, const char * aTypeHint=0x904ee41d, nsIInputStream * aPostData=0xe021cd50, nsIInputStream * aHeadersData=0xa2b780c1, int aFirstParty=1654818883, nsIDocShell * * aDocShell=0xdde8de34, nsIRequest * * aRequest=0xd0e9a941, int aIsNewWindowTarget=1548465500, int aBypassClassifier=-1503536295, int aForceAllowCookies=-383164654) Line 8813 C++ continuing hits invalid heap assertion > msvcr80d.dll!_free_dbg_nolock(void * pUserData=0x03d6dc90, int nBlockUse=1) Line 1279 + 0x30 bytes C++ msvcr80d.dll!_free_dbg(void * pUserData=0x03d6dc90, int nBlockUse=1) Line 1220 + 0xd bytes C++ msvcr80d.dll!free(void * pUserData=0x03d6dc90) Line 1178 + 0xb bytes C++ mozalloc.dll!moz_free(void * ptr=0x03d6dc90) Line 94 + 0xa bytes C++ xul.dll!operator delete(void * ptr=0x03d6dc90) Line 253 + 0xa bytes C++ xul.dll!std::allocator<MessageLoop::PendingTask>::deallocate(MessageLoop::PendingTask * _Ptr=0x03d6dc90, unsigned int __formal=1) Line 141 + 0x9 bytes C++ xul.dll!std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> >::_Tidy() Line 1254 C++ xul.dll!std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> >::clear() Line 1081 C++ xul.dll!std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> >::operator=(const std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > & _Right=[0]()) Line 622 + 0x8 bytes C++ xul.dll!std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > >::operator=(const std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > & __that=[0]()) + 0x13 bytes C++ xul.dll!std::swap<std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > >(std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > & _Left=[0](), std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > & _Right=[0]()) Line 19 + 0xc bytes C++ xul.dll!MessageLoop::ReloadWorkQueue() Line 385 + 0x16 bytes C++ xul.dll!MessageLoop::DoWork() Line 437 C++ xul.dll!base::MessagePumpForIO::DoRunLoop() Line 454 + 0x17 bytes C++ xul.dll!base::MessagePumpWin::RunWithDispatcher(base::MessagePump::Delegate * delegate=0x03d2feb8, base::MessagePumpWin::Dispatcher * dispatcher=0x00000000) Line 54 C++ xul.dll!base::MessagePumpWin::Run(base::MessagePump::Delegate * delegate=0x03d2feb8) Line 78 + 0x15 bytes C++ xul.dll!MessageLoop::RunInternal() Line 219 C++ xul.dll!MessageLoop::RunHandler() Line 203 C++ xul.dll!MessageLoop::Run() Line 177 C++ xul.dll!base::Thread::ThreadMain() Line 159 C++ xul.dll!`anonymous namespace'::ThreadFunc(void * closure=0x0334eb38) Line 27 C++ kernel32.dll!7c80b729() [Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll] xul.dll!mozilla::SVGLengthListSMILType::Add(nsSMILValue & aDest={...}, const nsSMILValue & aValueToAdd={...}, unsigned int aCount=2422607882) Line 183 + 0x3e bytes C++ xul.dll!nsDocShell::DoURILoad(nsIURI * aURI=0x296201dd, nsIURI * aReferrerURI=0x2b4555e4, int aSendReferrer=1298266322, nsISupports * aOwner=0xcf795128, const char * aTypeHint=0x904ee41d, nsIInputStream * aPostData=0xe021cd50, nsIInputStream * aHeadersData=0xa2b780c1, int aFirstParty=1654818883, nsIDocShell * * aDocShell=0xdde8de34, nsIRequest * * aRequest=0xd0e9a941, int aIsNewWindowTarget=1548465500, int aBypassClassifier=-1503536295, int aForceAllowCookies=-383164654) Line 8813 C++ http://www.myspace.com/plies Beta - Normal http://www.myspace.com/plies 2.0.0 - breakpoint ntdll.dll!7c90120e() [Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll] ntdll.dll!7c96ee31() ntdll.dll!7c96f26e() ntdll.dll!7c962fe0() ntdll.dll!7c91005d() kernel32.dll!7c85f9a7() > msvcr80d.dll!_CrtIsValidHeapPointer(const void * pUserData=0x0419ace0) Line 2072 C++ msvcr80d.dll!_free_dbg_nolock(void * pUserData=0x0419ace0, int nBlockUse=1) Line 1279 + 0x9 bytes C++ msvcr80d.dll!_free_dbg(void * pUserData=0x0419ace0, int nBlockUse=1) Line 1220 + 0xd bytes C++ msvcr80d.dll!free(void * pUserData=0x0419ace0) Line 1178 + 0xb bytes C++ mozalloc.dll!moz_free(void * ptr=0x0419ace0) Line 92 + 0xa bytes C++ xul.dll!operator delete(void * ptr=0x0419ace0) Line 253 + 0xa bytes C++ xul.dll!std::allocator<MessageLoop::PendingTask *>::deallocate(MessageLoop::PendingTask * * _Ptr=0x0419ace0, unsigned int __formal=8) Line 141 + 0x9 bytes C++ xul.dll!std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> >::_Tidy() Line 1259 C++ xul.dll!std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> >::clear() Line 1081 C++ xul.dll!std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> >::operator=(const std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > & _Right=[0]()) Line 622 + 0x8 bytes C++ xul.dll!std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > >::operator=(const std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > & __that=[0]()) + 0x13 bytes C++ xul.dll!std::swap<std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > >(std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > & _Left=[0](), std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > & _Right=[0]()) Line 19 + 0xc bytes C++ xul.dll!MessageLoop::ReloadWorkQueue() Line 386 + 0x16 bytes C++ xul.dll!MessageLoop::DoWork() Line 438 C++ xul.dll!base::MessagePumpForIO::DoRunLoop() Line 454 + 0x17 bytes C++ xul.dll!base::MessagePumpWin::RunWithDispatcher(base::MessagePump::Delegate * delegate=0x0418feb8, base::MessagePumpWin::Dispatcher * dispatcher=0x00000000) Line 54 C++ xul.dll!base::MessagePumpWin::Run(base::MessagePump::Delegate * delegate=0x0418feb8) Line 78 + 0x15 bytes C++ xul.dll!MessageLoop::RunInternal() Line 220 C++ xul.dll!MessageLoop::RunHandler() Line 203 C++ xul.dll!MessageLoop::Run() Line 177 C++ xul.dll!base::Thread::ThreadMain() Line 159 C++ xul.dll!`anonymous namespace'::ThreadFunc(void * closure=0x039ce7d0) Line 27 C++ kernel32.dll!7c80b729() xul.dll!nsBindingManager::DropDocumentReference() Line 1690 + 0xe bytes C++ continuing hits invalid heap assertion > msvcr80d.dll!_free_dbg_nolock(void * pUserData=0x0419ace0, int nBlockUse=1) Line 1279 + 0x30 bytes C++ msvcr80d.dll!_free_dbg(void * pUserData=0x0419ace0, int nBlockUse=1) Line 1220 + 0xd bytes C++ msvcr80d.dll!free(void * pUserData=0x0419ace0) Line 1178 + 0xb bytes C++ mozalloc.dll!moz_free(void * ptr=0x0419ace0) Line 92 + 0xa bytes C++ xul.dll!operator delete(void * ptr=0x0419ace0) Line 253 + 0xa bytes C++ xul.dll!std::allocator<MessageLoop::PendingTask *>::deallocate(MessageLoop::PendingTask * * _Ptr=0x0419ace0, unsigned int __formal=8) Line 141 + 0x9 bytes C++ xul.dll!std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> >::_Tidy() Line 1259 C++ xul.dll!std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> >::clear() Line 1081 C++ xul.dll!std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> >::operator=(const std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > & _Right=[0]()) Line 622 + 0x8 bytes C++ xul.dll!std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > >::operator=(const std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > & __that=[0]()) + 0x13 bytes C++ xul.dll!std::swap<std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > >(std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > & _Left=[0](), std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > & _Right=[0]()) Line 19 + 0xc bytes C++ xul.dll!MessageLoop::ReloadWorkQueue() Line 386 + 0x16 bytes C++ xul.dll!MessageLoop::DoWork() Line 438 C++ xul.dll!base::MessagePumpForIO::DoRunLoop() Line 454 + 0x17 bytes C++ xul.dll!base::MessagePumpWin::RunWithDispatcher(base::MessagePump::Delegate * delegate=0x0418feb8, base::MessagePumpWin::Dispatcher * dispatcher=0x00000000) Line 54 C++ xul.dll!base::MessagePumpWin::Run(base::MessagePump::Delegate * delegate=0x0418feb8) Line 78 + 0x15 bytes C++ xul.dll!MessageLoop::RunInternal() Line 220 C++ xul.dll!MessageLoop::RunHandler() Line 203 C++ xul.dll!MessageLoop::Run() Line 177 C++ xul.dll!base::Thread::ThreadMain() Line 159 C++ xul.dll!`anonymous namespace'::ThreadFunc(void * closure=0x039ce7d0) Line 27 C++ kernel32.dll!7c80b729() [Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll] xul.dll!nsBindingManager::DropDocumentReference() Line 1690 + 0xe bytes C++ http://www.pspiso.com/ Nightly - Normal http://www.pspiso.com/ Aurora - Normal http://www.pspiso.com/ Beta - Normal http://www.pspiso.com/ 2.0.0 - Normal

Sitting on http://www.pspiso.com/ in Aurora I hit // In non-debug builds, this method is declared as an empty inline method. #ifndef NDEBUG void LockImpl::AssertAcquired() const { DCHECK(recursion_count_shadow_ > 0); => DCHECK_EQ(owning_thread_id_, PlatformThread::CurrentId()); } NPSWF32.dll!0429fc8f() [Frames below may be incorrect and/or missing, no symbols loaded for NPSWF32.dll] NPSWF32.dll!04526a8e() NPSWF32.dll!0415b2f3() NPSWF32.dll!0422f70f() NPSWF32.dll!0421b2e2() NPSWF32.dll!0422f8e3() NPSWF32.dll!0422ff33() NPSWF32.dll!043187ff() NPSWF32.dll!0423041d() NPSWF32.dll!042c1f28() NPSWF32.dll!0423060c() NPSWF32.dll!04230617() NPSWF32.dll!042305b1() NPSWF32.dll!042a185a() > xul.dll!LockImpl::AssertAcquired() Line 71 + 0x5 bytes C++ xul.dll!Lock::Release() Line 17 + 0xf bytes C++ cdcdcd00() Looks like uninitialized memory got called. Sorry but I don't have Flash symbols for 22 handy. I tried to reproduce by loading the page and letting it sit idle for a while and by reloading multiple times but could not.

This appears to be heap corruption. Probably none of the stacks are going to point to the actual problem: we need to find the point of corruption, which may only be possible using a valgrind-like tool or VMWare recording.

This appears to have been fixed in Flash 10.3.181.26 as I can not longer reproduce. Leaving confidential for now.

With Flash 10.3.183.5 http://www.blogdelnarco.com/2011/07/destazan-brutalmente-hombre-en-morelos.html#more Locally on Windows XP I see corrupted heap. In automation I saw the original signature. http://www.ceropeliculasmegavideo.net/search?updated-max=2011-07-17T18%25253A14%25253A01-04%25253A30%2526max-results=42%2526popening=2 On reload I hit bug 641226 xul.dll!mozilla::plugins::PPluginInstanceChild::FatalError(const char * const msg=0x026d70e4) Line 2367 + 0x18 bytes C++

we re-opened our internal bug, #2875277. it is currently under review. Reproduced an Access violation write on a XP VM with 10,3,183,5...

Jeff, blogdelnarco.com appears to have been taken down due to threats against bloggers in mexico. I retested all of the crashes I have seen with Flash 10.3.183.10. I'll detail the ones that result in debug crt assetions here. I just confirmed these on Windows XP and Firefox 7 from the beta channel. http://www.incontri-x-sesso.com/?id=1%25252525252526track=IT-Exit-Megasesso NSFW (mild). You may need to reload and/or mouse over the flash images to see the problem. http://www.outofaces.com/lucy-pinder/ NSFW (mild) http://www.plunderguide.com/aryane-steinkopf/aryane-steinkopf-4/ NSFW (mild) bug 587982 contains patches to force aborts when MS CRT debug assertions fire. It is helpful but not necessary. Debug builds of Firefox are necessary to show the debug MS CRT assertions and errors however.

Darin, Today, I retested the urls in this bug on Windows XP and Windows 7 with Flash 11.1.102.55 using debug builds of Firefox which include the patch from bug 587982 on Beta/9, Aurora/10, Nightly/11 and reproduced one ABORT: CRT ASSERT dbgheap.c(1279) : Assertion failed: _CrtIsValidHeapPointer(pUserData) on Aurora WIndows XP with signature mozalloc_abort(char const* const) | MSCRTReportHook moz_free operator delete(void*) mozilla::plugins::StreamNotifyChild::`vector deleting destructor'(unsigned int) + 0x1f mozilla::plugins::BrowserStreamChild::Deliver() NULL http://www.incontri-x-sesso.com/?id=1%25252525252526track=IT-Exit-Megasesso (NSFW) I have attached the full set of reproducible Flash crashes from the 11/26 Flash crash retest. Additional crashes and debug heap assertions have been seen since then, but this should be a good start. Note that some of these urls are extremely NSFW. Please use a debug build of Firefox on Windows XP with the appropriate patch from bug 587982. If you have any questions please contact me. Thanks for looking into this.

Start Visual Studio or your debugger of choice, then open this file in a debug build of Firefox. Once the page loads, attach the debugger to the plugin-container.exe process and wait for it to hit the CRT assertion. It may take some time, but it should hit the CRT assertion. I was able to hit the assertion fairly regularly up until mid-morning. I think you will need to leave it open until the ad rotation brings back the offending content. This loads SFW advertisements. Sorry, I haven't been able to get the actual flash file that asserts.

Of course, after being reproducible all last night and this morning I have been running the test page all day and not hit the assert once. I'll keep that one running but will start looking for another example....

This reproduces for me. It appears the number of simultaneous ads helps.

A related assertion that may be of interest is: ###!!! ASSERTION: nsChannelClassifier not thread-safe: '_mOwningThread.GetThread() == PR_GetCurrentThread()', file c:/work/mozilla/builds/nightly/mozilla/netwerk/base/src/nsChannelClassifier.cpp, line 57

as well as ###!!! ASSERTION: mozHunspellDirProvider not thread-safe: '_mOwningThread.GetThread() == PR_GetCurrentThread()', file c:/work/mozilla/builds/nightly/mozilla/extensions/spellcheck/hunspell/src/mozHunspellDirProvider.cpp, line 45 ###!!! ASSERTION: mozHunspellDirProvider not thread-safe: '_mOwningThread.GetThread() == PR_GetCurrentThread()', file c:/work/mozilla/builds/nightly/mozilla/extensions/spellcheck/hunspell/src/mozHunspellDirProvider.cpp, line 45 ###!!! ASSERTION: DirectoryProvider not thread-safe: '_mOwningThread.GetThread() == PR_GetCurrentThread()', file c:/work/mozilla/builds/nightly/mozilla/browser/components/dirprovider/DirectoryProvider.cpp, line 65 ###!!! ASSERTION: DirectoryProvider not thread-safe: '_mOwningThread.GetThread() == PR_GetCurrentThread()', file c:/work/mozilla/builds/nightly/mozilla/browser/components/dirprovider/DirectoryProvider.cpp, line 65

Johnny, can you look at this? I can reliably reproduce this on Windows XP but Adobe is having problems. The stacks I see don't have Flash on the stack though that is not a reliable indicator that is it not Flash, but it could just as easily be our fault. Start debug build of Firefox trunk Go to adobe.com/flashplayer to instantiate the plugin-container process. Attach Visual Studio's debugger to plugin-container.exe Open both test case attachments in bug 657588 in separate Windows. Assert (time may vary depending on ad rotations)

I think the only way we're going to make progress on this is with either record-replay or a purify build, since this involves heap corruption which happens before the crash. I don't have access to either of those tools right now. Whoever does those runs will need Flash PDB files.

We have purify on sm-purify01 but I find it exceedingly difficult to get it to do anything useful with Firefox. There is a record-replay vm set up in the office right?

Yes, we have a record and replay box set up in the office. No flash symbols, but a good first step would be to find out whether it's us or flash that's corrupting memory here, if it's flash, we can work on getting access to the necessary symbols on the replay box. Kyle has had good success with the replay tools before. Kyle, can you take a stab at this? If not, please re-assign back to smadayag until we find a better owner. If we could get a stack that shows where the corruption is coming from then we can find the right person to work on that, whether it's someone at Mozilla or someone at Adobe. Bob, is this only happening in WinXP? I believe we have Win7 installed in the replay VM, but I also believe it's possible to record and do replay debugging in a WinXP VM. IIRC we even used to have one set up, but I'm not sure if we do at this point.

Yes, this is XP only.

I've been reading up on this assertion and it appears one of the cases where this may occur is when memory is allocated in one dll and then passed to another dll where each dll uses a different crt. Especially problematic is the case where a dll frees heap allocated in a different dll. http://msdn.microsoft.com/en-us/library/ms235460%28v=vs.80%29.aspx I assume the flash dll is using a different crt than our debug build and wonder if that may be the cause of the CRT Assert? Even in a release build we would be using different crt I think since a release build of Firefox will be using a patched version of the crt that includes jemalloc support while Flash uses something different I presume.

(In reply to Bob Clary [:bc:] from comment #28) > I assume the flash dll is using a different crt than our debug build and > wonder if that may be the cause of the CRT Assert? Even in a release build > we would be using different crt I think since a release build of Firefox > will be using a patched version of the crt that includes jemalloc support > while Flash uses something different I presume. I'm hardly an NPAPI expert but this is what NPN_MemAlloc is for, right? dumpbin.exe -IMPORTS NPSWF32.dll shows no load time dependencies on any CRT. I think this is more likely to be actual memory corruption than a heap mismatch.

I've got an XP VM set up for record and replay. Attempting to reproduce.

No luck with the testcase, and bc had no luck with the other URLs he tried manually. He's going to have the automation retest the URLs and we'll go from there.

bc found a URL that asserted repeatedly for him, and I was able to capture the assertion in the replay setup.

Unfortunately I haven't made much progress determining what exactly is going wrong in the recording. I have some other stuff I need to finish up by the end of the quarter so I'm not going to get back to this for a week or two.

This is a bug in Gecko. More details and a patch in a few minutes.

I managed to get this in a replay machine and isolate the code causing the memory corruption. After sitting down with bent and getting some lessons on how all this IPC plugins stuff works, we determined that the problem is the following: At http://hg.mozilla.org/mozilla-central/annotate/e0d9c8ddd5bd/dom/plugins/ipc/PluginModuleChild.cpp#l1067 we create a StreamNotifyChild object and pass it to CallPStreamNotifyConstructor. We end up at http://hg.mozilla.org/mozilla-central/annotate/e0d9c8ddd5bd/dom/plugins/ipc/PluginInstanceParent.cpp#l440 in the parent process. This function ensures that if the child was not told to delete its stream notify actor and the constructor failed, that the child is told to delete its stream notify actor. It does not, however, ensure that if the child is told to delete its stream notify actor that an error code is returned. Thus we return from the ctor with NPERR_ERROR_OK but the stream notify actor is already gone. We return to the child side, where we see 1085 if (NPERR_NO_ERROR == err) { 1086 // If NPN_PostURLNotify fails, the parent will immediately send us 1087 // a PStreamNotifyDestructor, which should not call NPP_URLNotify. 1088 sn->SetValid(aNotifyData); 1089 } Since err is NPERR_NO_ERR we write to the freed memory and cause the heap corruption that later causes the assertions/crashes. This patch ensures that if the stream notify actor was destroyed that we return an error code (NPERR_GENERIC_ERR) and thus avoid the heap corruption.

bc is going to rerun automation with this patch and see if he still sees any of these crashes/asserts.

Comment on attachment 595229 [details] [diff] [review] Patch This does not appear to be the patch you meant to attach.

And I attached the wrong file.

This appears to have helped. Reproducing the heap corruption on nightly has been difficult so the fact I did not see it after this patch is not a guarantee. But this also eliminated ( I think ) the related FatalError crash on Nightly in bug 641226. It would really really help to make sure if we can get this on both aurora and beta asap.

From my read of the patch this looks low risk, but looks can of course be deceiving. Would you all mind providing a risk evaluation?

Plugin code is kind of hairy, but this is pretty low risk. I suppose there is potential for this to cause us to return an error to plugins where they weren't getting it before, but at the moment instead of getting that error they're just crashing (or worse!).

I think this is about as safe as a plugin bug can be.

Comment on attachment 595432 [details] [diff] [review] Patch [Triage Comment] Given the low-risk evaluation, approving for Aurora 12 and Beta 11.

https://hg.mozilla.org/releases/mozilla-aurora/rev/46a2e9a276f1 https://hg.mozilla.org/releases/mozilla-beta/rev/cea20ca3ed3c

We also want this on 10esr and probably on 1.9.2 as well (the code is the same there).

I rebuilt beta and aurora and retested with urls from the last week and was not able to reproduce the abort or the related crashes. Thanks Kyle, you rock!

Please land on mozilla-esr10 as soon as possible. For more information on the landing process, please see https://wiki.mozilla.org/Release_Management/ESR_Landing_Process

[Triage comment] This bug is being tracked for landing on ESR branch. Please land patches on http://hg.mozilla.org/releases/mozilla-esr10/by Thursday March 1, 2012 in order to be ready for go-to-build on Friday March 2, 2012. See https://wiki.mozilla.org/Release_Management/ESR_Landing_Process for more information.

Comment on attachment 595432 [details] [diff] [review] Patch Approved for 1.9.2.28, a=dveditz for release-drivers

https://hg.mozilla.org/releases/mozilla-esr10/rev/83533cc02c57 http://hg.mozilla.org/releases/mozilla-1.9.2/rev/9b893007fa4e

(In reply to Bob Clary [:bc:] from comment #48) > I rebuilt beta and aurora and retested with urls from the last week and was > not able to reproduce the abort or the related crashes. Thanks Kyle, you > rock! Was this with Firefox 11 and 12 then? I'm trying to figure out if we need to verify it in those branches or if you already did so, Bob? Flags for those releases are still marked as "fixed" and not "verfied" so I am asking.

I'm unable to reproduce the crash using Flash 11.1.102.63 so I can't verify if this is fixed for Firefox 10.0.3esr. Does this require a particular version of Flash?

I've been trying to reproduce the issue using affected builds and Flash version 10.3.181.14, but I haven't been able to. I don't have a build environment handy to check with the more detailed instructions. Bob, could you help us verify this on your environment?

This wasn't a problem in flash but in our code and only appeared with debug builds on windows xp. I've already verified this for Beta/11, Aurora/12, and Nightly/13. I haven't seen the ms crt debug asserts in some time. I don't have a 1.9.2 or 10 environment available but the patch is so simple that I don't expect them to be any different. Juan, I assume 10.3.181.14 means you were testing with Mac OS X 10.5?

I was testing on XP. I got an older version of Flash (the one mentioned in comment #0) from Adobe's archives.

Have you been trying to verify this with debug builds?

No, I was feeling lucky thinking I could reproduce the crash without having to go use debug builds, and that's why I asked for bc's help. I'll report back when after trying debug builds.

Tried the esr10 debug build from February 22nd, 2012 with Flash 10.3.181.14 on Windows XP and I can't reproduce the original crash -- therefor, I am unable to verify the fix.

I was able to reproduce it with ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2012-02-25-mozilla-esr10-debug/ and NSFW http://www.incontri-x-sesso.com/?id=1%25252525252526track=IT-Exit-Megasesso I was not able to reproduce with ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-esr10/firefox-10.0.4esrpre.en-US.win32.zip Anthony + Juan: You can do which ever qa stamps you need for esr.

Thanks Bob, based on your testing I'm going to call this verified for ESR. If you have a chance could you quickly verify for Firefox 3.6.28 as well?

I couldn't reproduce the debug assertion or crash with 1.9.2. Btw in comment 63 I incorrectly verified with a non debug build. I retested with a debug build and verified 10esr did not assert/crash.

(In reply to Bob Clary [:bc:] from comment #65) > I couldn't reproduce the debug assertion or crash with 1.9.2. With a 1.9.2 build from before or after this was fixed in 3.6.28?

either

(In reply to Bob Clary [:bc:] from comment #67) > either Is there anything we can do to verify this fixed for Firefox 3.6.28 then? I'm guessing no since it's not reproducible in the first place.

Not easily that I can think of. I could try to manually test the list of urls where I have seen this before but that is a sink of time and not worth the effort imo.

(In reply to Bob Clary [:bc:] from comment #69) > Not easily that I can think of. I could try to manually test the list of > urls where I have seen this before but that is a sink of time and not worth > the effort imo. Given that, QA won't spend any more time on this either. It's been verified everywhere else, so I have a fair bit of confidence in the assumption that this is okay for 3.6.28.