It's #2 top browser crasher in today's build. It first appeared in 16.0a1/20120703110846. The regression range is: http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=e61399f31505&tochange=1d370ca5bb8d The culprit is bug 769048. Signature mozilla::ipc::RPCChannel::EnteredCxxStack() | mozilla::ipc::RPCChannel::CxxStackFrame::CxxStackFrame(mozilla::ipc::RPCChannel&, mozilla::ipc::RPCChannel::Direction, IPC::Message const*) | mozilla::plugins::PPluginInstanceParent::CallUpdateWindow() More Reports Search UUID 61e301d6-8e29-49c3-a257-9f6792120704 Date Processed 2012-07-04 07:19:46 Uptime 1267 Last Crash 1.5 days before submission Install Age 6.7 hours since version was first installed. Install Time 2012-07-04 00:39:21 Product Firefox Version 16.0a1 Build ID 20120703110846 Release Channel nightly OS Windows NT OS Version 6.1.7601 Service Pack 1 Build Architecture x86 Build Architecture Info GenuineIntel family 6 model 37 stepping 5 Crash Reason EXCEPTION_ACCESS_VIOLATION_READ Crash Address 0x0 App Notes AdapterVendorID: 0x8086, AdapterDeviceID: 0x0046, AdapterSubsysID: 04881025, AdapterDriverVersion: 8.15.10.2622 Has dual GPUs. GPU #2: AdapterVendorID2: 0x10de, AdapterDeviceID2: 0x0df4, AdapterSubsysID2: 04881025, AdapterDriverVersion2: 8.17.13.142D2D? D2D+ DWrite? DWrite+ D3D10 Layers? D3D10 Layers+ EMCheckCompatibility True Adapter Vendor ID 0x8086 Adapter Device ID 0x0046 Total Virtual Memory 4294836224 Available Virtual Memory 3400708096 System Memory Use Percentage 42 Available Page File 5826461696 Available Physical Memory 2269347840 Frame Module Signature Source 0 xul.dll mozilla::ipc::RPCChannel::EnteredCxxStack obj-firefox/dist/include/mozilla/ipc/RPCChannel.h:202 1 xul.dll mozilla::ipc::RPCChannel::CxxStackFrame::CxxStackFrame obj-firefox/dist/include/mozilla/ipc/RPCChannel.h:255 2 xul.dll mozilla::ipc::RPCChannel::Call ipc/glue/RPCChannel.cpp:136 3 xul.dll mozilla::plugins::PPluginInstanceParent::CallUpdateWindow obj-firefox/ipc/ipdl/PPluginInstanceParent.cpp:1050 4 xul.dll nsWindow::OnPaint widget/windows/nsWindowGfx.cpp:225 5 xul.dll nsWindow::ProcessMessage widget/windows/nsWindow.cpp:4727 6 xul.dll nsCOMPtr_base::assign_from_qi obj-firefox/xpcom/build/nsCOMPtr.cpp:65 7 xul.dll nsWindow::WindowProcInternal widget/windows/nsWindow.cpp:4314 8 xul.dll CallWindowProcCrashProtected xpcom/base/nsCrashOnException.cpp:32 9 xul.dll xul.dll@0x1d6d4f 10 user32.dll InternalCallWinProc 11 xul.dll CallWindowProcCrashProtected xpcom/base/nsCrashOnException.cpp:38 12 xul.dll CallWindowProcCrashProtected xpcom/base/nsCrashOnException.cpp:38 13 user32.dll CallWindowProcAorW 14 user32.dll CallWindowProcW 15 xul.dll mozilla::plugins::PluginInstanceParent::PluginWindowHookProc dom/plugins/ipc/PluginInstanceParent.cpp:1857 16 user32.dll InternalCallWinProc 17 user32.dll UserCallWinProcCheckWow 18 user32.dll CallWindowProcAorW 19 user32.dll CallWindowProcW 20 xul.dll PluginWndProcInternal dom/plugins/base/nsPluginNativeWindowWin.cpp:325 21 xul.dll CallWindowProcCrashProtected xpcom/base/nsCrashOnException.cpp:32 22 xul.dll PluginWndProc dom/plugins/base/nsPluginNativeWindowWin.cpp:354 23 user32.dll InternalCallWinProc 24 user32.dll UserCallWinProcCheckWow 25 user32.dll DispatchClientMessage 26 user32.dll __fnDWORD 27 ntdll.dll KiUserCallbackDispatcher 28 ntdll.dll KiUserApcDispatcher 29 user32.dll DispatchMessageW 30 xul.dll nsAppShell::ProcessNextNativeEvent widget/windows/nsAppShell.cpp:322 31 xul.dll nsBaseAppShell::OnProcessNextEvent widget/xpwidgets/nsBaseAppShell.cpp:280 ... More reports at: https://crash-stats.mozilla.com/report/list?signature=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3AEnteredCxxStack%28%29+|+mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26%2C+mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C+IPC%3A%3AMessage+const*%29+|+mozilla%3A%3Aplugins%3A%3APPluginInstanceParent%3A%3ACallUpdateWindow%28%29

I get this crash with the testcase from bug 763237 using bsmedberg's special build from bug 769048.

I add another signature with a stack that slightly differs and appeared at the same time: Frame Module Signature Source 0 xul.dll mozilla::ipc::RPCChannel::EnteredCxxStack obj-firefox/dist/include/mozilla/ipc/RPCChannel.h:202 1 xul.dll mozilla::ipc::RPCChannel::CxxStackFrame::CxxStackFrame obj-firefox/dist/include/mozilla/ipc/RPCChannel.h:255 2 xul.dll mozilla::ipc::RPCChannel::Call ipc/glue/RPCChannel.cpp:136 3 xul.dll mozilla::plugins::PPluginModuleParent::CallNPP_ClearSiteData obj-firefox/ipc/ipdl/PPluginModuleParent.cpp:513 4 xul.dll mozilla::plugins::PluginModuleParent::NPP_ClearSiteData dom/plugins/ipc/PluginModuleParent.cpp:919 5 xul.dll nsPluginHost::ClearSiteData dom/plugins/base/nsPluginHost.cpp:1787 6 xul.dll NS_InvokeByIndex_P xpcom/reflect/xptcall/src/md/win32/xptcinvoke.cpp:70 7 xul.dll XPCWrappedNative::CallMethod js/xpconnect/src/XPCWrappedNative.cpp:2371 8 xul.dll XPC_WN_CallMethod js/xpconnect/src/XPCWrappedNativeJSOps.cpp:1474 ... More reports at: https://crash-stats.mozilla.com/report/list?signature=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3AEnteredCxxStack%28%29+|+mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26%2C+mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C+IPC%3A%3AMessage+const*%29+|+mozilla%3A%3Aplugins%3A%3APPluginModuleParent%3A%3ACallNPP_ClearSiteData%28nsC...

Although I can verify in local testing that this seems to fix the crash issues, I don't have a good way of automated testing this because the crash injector is limited to Flash subprocesses.

Comment on attachment 639419 [details] [diff] [review] Close the IPC channel correctly so that it reports an error, rev. 1 I don't like the fact that normal shutdown vs. abnormal shutdown causes such an (apparent) behavioral difference in PluginInstanceParent, but this code is delicate and untested enough a workaround is probably a good practical option. >diff --git a/ipc/glue/AsyncChannel.h b/ipc/glue/AsyncChannel.h >+ // Force the channel to behave as if a channel error occurred. Valid >+ // for process links only, not thread links. >+ void ForceError(); >+ Call this |CloseWithError()|. r=me with that.

https://hg.mozilla.org/releases/mozilla-aurora/rev/e807b189b32a (a=akeybl in bug 769048)

This crash doesn't reproduce anymore on any of the builds with the fix: https://crash-stats.mozilla.com/report/list?query_search=signature&query_type=contains&reason_type=contains&range_value=4&range_unit=weeks&hang_type=any&process_type=any&signature=mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3AEnteredCxxStack%28%29%20%7C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ACxxStackFrame%3A%3ACxxStackFrame%28mozilla%3A%3Aipc%3A%3ARPCChannel%26%2C%20mozilla%3A%3Aipc%3A%3ARPCChannel%3A%3ADirection%2C%20IPC%3A%3AMessage%20const%2A%29%20%7C%20mozilla%3A%3Aplugins%3A%3APPluginInstanceParent%3A%3ACallUpdateWindow%28%29