[Tracking Requested - why for this release]: This bug was filed from the Socorro interface and is report bp-9ec87c58-ab7c-45ed-869d-194b12160224. ============================================================= Crashing Thread (0) Frame Module Signature Source 0 xul.dll mozilla::plugins::PluginModuleChild::NPN_CreateObject(_NPP*, NPClass*) dom/plugins/ipc/PluginModuleChild.cpp 1 xul.dll mozilla::plugins::PluginScriptableObjectChild::InitializeProxy() dom/plugins/ipc/PluginScriptableObjectChild.cpp 2 xul.dll mozilla::plugins::PluginInstanceChild::RecvPPluginScriptableObjectConstructor(mozilla::plugins::PPluginScriptableObjectChild*) dom/plugins/ipc/PluginInstanceChild.cpp 3 xul.dll mozilla::plugins::PPluginInstanceChild::OnMessageReceived(IPC::Message const&) obj-firefox/ipc/ipdl/PPluginInstanceChild.cpp 4 xul.dll mozilla::plugins::PPluginModuleChild::OnMessageReceived(IPC::Message const&) obj-firefox/ipc/ipdl/PPluginModuleChild.cpp 5 xul.dll mozilla::ipc::MessageChannel::DispatchMessageW(IPC::Message const&) ipc/glue/MessageChannel.cpp 6 xul.dll mozilla::ipc::MessageChannel::Call(IPC::Message*, IPC::Message*) ipc/glue/MessageChannel.cpp 7 xul.dll mozilla::plugins::PPluginScriptableObjectChild::CallNPN_Evaluate(nsCString const&, mozilla::plugins::Variant*, bool*) obj-firefox/ipc/ipdl/PPluginScriptableObjectChild.cpp 8 xul.dll mozilla::plugins::PluginScriptableObjectChild::Evaluate(_NPString*, _NPVariant*) dom/plugins/ipc/PluginScriptableObjectChild.cpp 9 shell32.dll IViewSettings_IsGrouped(IViewSettings*) 10 npswf32_20_0_0_306.dll F1759705523_______________________________________________________________________________________________________ F_715140683__________________________________________________________________:788 11 npswf32_20_0_0_306.dll F1560148926__________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ F462186421___________________________________________________________:879 12 npswf32_20_0_0_306.dll F_957036607____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ F_298184567___________________________________________________________________:556 13 npswf32_20_0_0_306.dll F1926476525___________________________________________________________________________ F1049082337__________________________________________________________________:408 14 npswf32_20_0_0_306.dll F_1513036030________________________________________ F_1654095196__________________________________________________________________:47 15 npswf32_20_0_0_306.dll F_652032984_____________________________________________________ F_1951567228__________________________________________________________:261 16 npswf32_20_0_0_306.dll F1607135317_____________________________________________________________________ F_851861807__________________________________________________________:139 17 npswf32_20_0_0_306.dll F2166389_____________________________________________________________________ F_851861807__________________________________________________________:576 18 npswf32_20_0_0_306.dll F_917831355____________________________________________ F_851861807__________________________________________________________:493 19 npswf32_20_0_0_306.dll F1315696776________________________________ F_851861807__________________________________________________________:444 20 npswf32_20_0_0_306.dll F_1428703866________________________________ F_766591945____________________________________________________________________:203 21 npswf32_20_0_0_306.dll F845925699_____________________________________ F1677514683__________________________________________________________________________________:104 22 npswf32_20_0_0_306.dll F_274593382________________________________ F209679109___________________________________________________________________________________:1738 23 npswf32_20_0_0_306.dll F_305312235__________________________________________ F209679109___________________________________________________________________________________:1019 24 user32.dll _InternalCallWinProc 25 user32.dll UserCallWinProcCheckWow 26 user32.dll DispatchMessageWorker 27 user32.dll DispatchMessageW 28 xul.dll base::MessagePumpForUI::DoRunLoop() ipc/chromium/src/base/message_pump_win.cc 29 xul.dll base::MessagePumpWin::RunWithDispatcher(base::MessagePump::Delegate*, base::MessagePumpWin::Dispatcher*) ipc/chromium/src/base/message_pump_win.cc 30 xul.dll base::MessagePumpWin::Run(base::MessagePump::Delegate*) ipc/chromium/src/base/message_pump_win.h 31 xul.dll MessageLoop::RunInternal() ipc/chromium/src/base/message_loop.cc 32 xul.dll MessageLoop::RunHandler() ipc/chromium/src/base/message_loop.cc 33 xul.dll MessageLoop::Run() ipc/chromium/src/base/message_loop.cc 34 xul.dll XRE_InitChildProcess toolkit/xre/nsEmbedFunctions.cpp 35 plugin-container.exe content_process_main(int, char** const) ipc/contentproc/plugin-container.cpp 36 plugin-container.exe wmain toolkit/xre/nsWindowsWMain.cpp 37 plugin-container.exe __tmainCRTStartup f:/dd/vctools/crt/crtw32/startup/crt0.c:255 38 kernel32.dll BaseThreadInitThunk 39 ntdll.dll __RtlUserThreadStart 40 ntdll.dll _RtlUserThreadStart this plugin crash (with various versions of flash) is significantly rising since february 23 across channels: https://crash-stats.mozilla.com/signature/?date=%3E%3D2016-02-18&signature=mozilla%3A%3Aplugins%3A%3APluginModuleChild%3A%3ANPN_CreateObject&page=1#graphs the signature is making up nearly 20% of plugin crashes on release now.

This is weird. Here: http://hg.mozilla.org/releases/mozilla-release/annotate/60e96806ff1c/dom/plugins/ipc/PluginModuleChild.cpp#l2233 We're hitting a null `i` which means that aNPP->ndata is null, which most-likly means we've already hit PluginInstanceChild::Destroy() for that instance. The situation/stack is: <Flash> NPN_Evaluate/PluginScriptableObjectChild::Evaluate <IPC stuff> PluginInstanceChild::RecvPPluginScriptableObjectConstructor PluginScriptableObjectChild::InitializeProxy PluginScriptableObjectChild::CreateProxyObject We're pretty clearly creating a scriptable object for an plugin instance which is being destroyed, but perhaps isn't all the way dead yet. There are two possibilities I can think of: #1: At the top of the stack, Flash is calling NPN_Evaluate on an object associated with a dead instance. #2: At the top of the stack, the instance is still alive. Inside of the NPN_Evaluate evaluation, we destroy the instance. I don't think this should be possible because plugin stack guards should prevent this, but those have been known to fail before. Kairo, are there any interesting URLs or comments that might point us to STR? Nominating for tracking based on a release regression.

The URLs point to Facebook games mostly: 408 https://apps.fb.miniclip.com/arcade/play/8-ball-pool-multiplayer/2471/?fb_source=bookmark&ref=bookmarks&count=0&fb_bmpos=_0 336 https://facebook2.poker.zynga.com/poker/?fb_source=bookmark&ref=bookmarks&count=0&fb_bmpos=_0 122 https://facebook2.poker.zynga.com/poker/?fb_source=bookmark&ref=bookmarks&count=1&fb_bmpos=_1 102 https://apps.fb.miniclip.com/arcade/play/8-ball-pool-multiplayer/2471/?fb_source=sidebar_bookmark 68 https://society.rolltower.com/?fb_source=bookmark&ref=bookmarks&count=0&fb_bmpos=_0 64 https://facebook2.poker.zynga.com/poker/ 55 https://apps.fb.miniclip.com/arcade/play/8-ball-pool-multiplayer/2471/?fb_source=canvas_bookmark 41 https://apps.fb.miniclip.com/arcade/play/8-ball-pool-multiplayer/2471/ 40 https://facebook2.poker.zynga.com/poker/?fb_source=canvas_bookmark 35 https://facebook2.poker.zynga.com/poker/?fb_source=sidebar_bookmark 33 https://apps.fb.miniclip.com/arcade/play/8-ball-pool-multiplayer/2471/?fb_source=bookmark&ref=bookmarks&count=1&fb_bmpos=_1 29 https://facebook2.poker.zynga.com/poker/?fb_source=bookmark&ref=bookmarks&count=3&fb_bmpos=_3 29 https://facebook2.poker.zynga.com/poker/?fb_source=bookmark&ref=bookmarks&count=2&fb_bmpos=_2 24 https://facebook2.poker.zynga.com/poker/?fb_source=bookmark_favorites&ref=bookmarks&count=0&fb_bmpos=_0 23 https://facebook2.poker.zynga.com/poker/index.php?src_track_str=Poker+%25CLIENT_SN%25+Canvas+Other+%25ACTION%25+o%3AHome%3Aswf_refresh%3A2009-04-01 21 https://apps.fb.miniclip.com/arcade/play/8-ball-pool-multiplayer/2471/?fb_source=search&ref=ts&fref=ts 17 https://apps.fb.miniclip.com/arcade/play/8-ball-pool-multiplayer/2471/?fb_source=bookmark_favorites&ref=bookmarks&count=0&fb_bmpos=_0 14 https://facebook2.poker.zynga.com/poker/index.php?src_track_str=Poker+%25CLIENT_SN%25+request+Other+%25ACTION%25+o%3AHandleRequestFailed%3AdeleteRequest%3A2011-12-19 13 https://facebook2.poker.zynga.com/poker/?fb_source=bookmark&ref=bookmarks&count=1&fb_bmpos=_1# 12 https://society.rolltower.com/?fb_source=canvas_bookmark 12 https://facebook2.poker.zynga.com/poker/?fb_source=bookmark&ref=bookmarks&count=4&fb_bmpos=_4 10 https://apps.fb.miniclip.com/arcade/play/8-ball-pool-multiplayer/2471/?fb_source=bookmark&ref=bookmarks&count=2&fb_bmpos=_2 10 https://facebook2.poker.zynga.com/poker/?fb_source=bookmark_favorites&ref=bookmarks&count=1&fb_bmpos=_1 [...]

Comments are very angry about crashing repeatedly all the time, and about the only hint the give to a specific site or game is a high number of people talking about Farmville 2 crashing.

Could I get some testing around Farmville 2 and some of the other arcade games listed to see if we can come up with reliable STR? With that it will be fairly easy to diagnose.

Oh, and the comments complain about crashing in the middle of playing, so it's not startup or shutdown of the plugin, I guess.

Affected versions go back to at least Firefox 38 ESR and Flash 16, though the most recent releases Firefox 44.0.2 and Flash 20.0.0.306 are predominant.

marcia used to do a lot of farmville testing a few years ago when we had problems there. she might have documented some test scenarios that we could re-use. we also got help from a general call out to all mozillians to go play farmville looking for crashes.

http://stackoverflow.com/questions/8743269/how-to-stress-and-load-test-flash-games-like-mafia-wars-farmville also suggests test clients its best to use the lowest powered systems you can find to surface bugs. stats say 50% of the crashes are happening on windows 7 and maybe hardware that is 2009-2012 vintage.

(In reply to chris hofmann from comment #8) > http://stackoverflow.com/questions/8743269/how-to-stress-and-load-test-flash- > games-like-mafia-wars-farmville also suggests test clients its best to use > the lowest powered systems you can find to surface bugs. stats say 50% of > the crashes are happening on windows 7 and maybe hardware that is 2009-2012 > vintage. Another thing that can help (in the absence of vintage hardware) is to boot the test OS with a configuration that artificially limits the number of cores.

Update: Martin connected me to folks in Zynga. They are experiencing a flash v20 memory consumption issue. From David dschreck@zynga.com: "FarmVille 2 is currently being affected by a Flash plugin issue related to memory consumption (seen in flash v20.) If there are players crashing in Farm2 there may be some crossover in user reports."

possible related bug where we run out of memory and then crash with facebook/zynga? involved because our estimation of allocation is incorrect. https://bugzilla.mozilla.org/show_bug.cgi?id=1229384 If they fix the problem on the flash end we might see both signatures decline.

I tried to reproduce the crash on Firefox 44.0.2 with Shockwave Flash 20.0.0.306, but without success. I tried on Windows 10 x32, Windows 8 x32, Windows 7 x32 /x64. Even when the CPU and Memory consumption reached high of 99%, the plugin did not crash. There were times it appeared to freeze up but I could not get it to crash. The only time when a plugin crash occurred was on Firefox 38 and the steps were: Version 38.0 Build ID 20150508094354 User Agent Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0 1. Open Firefox and Login to Facebook, 2. Select Play Game FarmVille2, move around a little in game and then close the game and the browser 3. Wait and then repeat This occurred on the 3rd interval: Play/Close, Play/Close, Play...

On the recent weekend, this spiked to the same values as the weekend before, it's lower on work days as people probably play less. This is an ongoing grave issue and is probably losing users for those games and possibly Firefox as well.

Tracking in case there is something we could do

Zynga was unable to repro these crashes in their test environment. They suspect a third party ad could be causing issues, likely adap.tv. I will try to see if there is a way to contact these guys.

I was able to contact folks from adap.tv. I have shared the URLs and top locales that are impacted.

Aaron, folks from adap.tv are asking for a flash related stack trace that might help them pinpoint this to a specific ad? Can we deduce that from our crash reports? Thanks!

Currently #11 top crasher in beta 46, #5 top crasher in the beta e10s experiment.

(In reply to Ritu Kothari (:ritu) from comment #17) > Aaron, folks from adap.tv are asking for a flash related stack trace that > might help them pinpoint this to a specific ad? Can we deduce that from our > crash reports? Thanks! I am unaware of anything that we could give them -- AFAIK, outside of crash report URLs we don't really collect anything that could allow us to deduce such things.

The theory is that this plugin has been destroyed and then a scriptable call is being made to it. If that's true, this patch should prevent those crashes. I'd like to land it to try to prove or disprove that theory.

also, if you revisit the crash graph from comment #0 it looks like on 2016-03-30 the crashes dropped from ~3000 per day to currently ~600 without us doing anything yet. maybe we can take a look how the affected urls changed since then...

Comment on attachment 8737266 [details] [diff] [review] NPP_null_on_destroy.patch >diff --git a/dom/plugins/ipc/PluginInstanceChild.h b/dom/plugins/ipc/PluginInstanceChild.h >--- a/dom/plugins/ipc/PluginInstanceChild.h >+++ b/dom/plugins/ipc/PluginInstanceChild.h >@@ -226,16 +226,18 @@ public: > const nsCString& mimeType, > const bool& seekable, > uint16_t* stype); > > bool Initialize(); > > NPP GetNPP() > { >+ if (mDestroyed) >+ return nullptr; I don't think this will work properly. There are various actions taken in the body of PluginInstanceChild::Destroy() after we set mDestroyed which could cause serious and unpredictable misbehavior. For instance if there are pending network streams the act of destroying them calls in using this NPP. >diff --git a/dom/plugins/ipc/PluginModuleChild.cpp b/dom/plugins/ipc/PluginModuleChild.cpp > NPObject* > PluginModuleChild::NPN_CreateObject(NPP aNPP, NPClass* aClass) > { > PLUGIN_LOG_DEBUG_FUNCTION; > ENSURE_PLUGIN_THREAD(nullptr); > >+ if (!aNPP) { >+ return nullptr; >+ } Because of the above, the NPP check here isn't going to be useful, but we could perhaps add an mDestroyed check to PluginInstanceChild::RecvPPluginScriptableObjectConstructor. We should assert that mDestroyed is false, because this is still evidence of a logic error somewhere, but we could bail. I'm worried that this will still cause a crash when we try to use this object and it hasn't been properly initialized, so we'd be moving this around without fixing it. I'm trying a little patch which won't fix the crash but will move it to a much more useful spot asserting that we don't destroy while there are scriptable methods on the stack. I'd like to try that to see if it shows something useful.

ok, reassigning to you. One note though, I do think the aNPP null check would work. aNPP is passed into NPN_CreateObject() from PluginScriptableObjectChild::CreateProxyObject(), which gets that value from PluginInstanceChild::GetNPP(), where I added the mDestroyed check.

The aNPP check would work if the first hunk was ok. But it's not, because it will break normal shutdown.

Review commit: https://reviewboard.mozilla.org/r/45289/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/45289/

Comment on attachment 8739487 [details] MozReview Request: Bug 1252152 - Make plugin instances destroyed while that instance is on the stack crash earlier and more usefully, r?jimm https://reviewboard.mozilla.org/r/45289/#review41817

No nightly crashes since the 20160411140819 build. Looks like this is fixed. Feel confide3nt uplifting this?

This patch isn't going to "fix" anything, it should just move the crash around to a more useful spot. I'm monitoring https://crash-stats.mozilla.com/search/?release_channel=nightly&signature=~NS_DebugBreak&process_type=plugin&_facets=signature&_facets=abort_message&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=abort_message#crash-reports but I haven't seen the "new" crash yet. This was always low-volume on nightly. I don't want to uplift this to 46, because we don't know whether it will break things unexpectedly and cause more crashes. I'll probably request 47 uplift so that we have a full 6-week cycle to monitor beta.

merge day, did you want to uplift this to 47?

Comment on attachment 8739487 [details] MozReview Request: Bug 1252152 - Make plugin instances destroyed while that instance is on the stack crash earlier and more usefully, r?jimm Approval Request Comment [Feature/regressing bug #]: crash with unknown cause [User impact if declined]: This patch will not fix the crash, but should make it more reproducible/diagnosible [Describe test coverage new/current, TreeHerder]: Landed, nothing appears to have gotten worse on nightly [Risks and why]: It's possible that this will increase the # of crashes. That's why I want a full beta cycle for testing. [String/UUID change made/needed]: None

This landed in Fx48 according to comment 28.

Comment on attachment 8739487 [details] MozReview Request: Bug 1252152 - Make plugin instances destroyed while that instance is on the stack crash earlier and more usefully, r?jimm Improving crash diagnostics, Beta47+

Crash volume for signature 'mozilla::plugins::PluginModuleChild::NPN_CreateObject': - nightly (50): 34 - aurora (49): 188 - esr (45): 498 Affected platforms: Windows, Mac OS X, Linux

Crash volume for signature 'mozilla::plugins::PluginModuleChild::NPN_CreateObject': - nightly (version 51): 12 crashes from 2016-08-01. - aurora (version 50): 115 crashes from 2016-08-01. - beta (version 49): 1681 crashes from 2016-08-02. - release (version 48): 4695 crashes from 2016-07-25. - esr (version 45): 721 crashes from 2016-05-02. Crash volume on the last weeks (Week N is from 08-22 to 08-28): W. N-1 W. N-2 W. N-3 - nightly 8 2 0 - aurora 53 30 14 - beta 390 716 427 - release 1483 1512 766 - esr 59 51 54 Affected platforms: Windows, Mac OS X, Linux Crash rank on the last 7 days: Browser Content Plugin - nightly #79 - aurora #647 #12 - beta #6 - release #4 - esr #15

Crash volume for signature 'mozilla::plugins::PluginModuleChild::NPN_CreateObject': - nightly (version 52): 1 crash from 2016-09-19. - aurora (version 51): 8 crashes from 2016-09-19. - beta (version 50): 102 crashes from 2016-09-20. - release (version 49): 2719 crashes from 2016-09-05. - esr (version 45): 1015 crashes from 2016-06-01. Crash volume on the last weeks (Week N is from 10-03 to 10-09): W. N-1 W. N-2 - nightly 1 0 - aurora 7 1 - beta 75 27 - release 2071 648 - esr 91 78 Affected platforms: Windows, Mac OS X, Linux Crash rank on the last 7 days: Browser Content Plugin - nightly #154 - aurora #49 - beta #15 - release #4 - esr #9

I just debugged one of these (https://crash-stats.mozilla.com/report/index/a81d373c-5962-47be-9c6e-7621d2170123): xul.dll!mozilla::plugins::PluginModuleChild::NPN_CreateObject(_NPP * aNPP, NPClass * aClass) Line 2211 C++ xul.dll!mozilla::plugins::PluginScriptableObjectChild::InitializeProxy() Line 568 C++ xul.dll!mozilla::plugins::PluginInstanceChild::RecvPPluginScriptableObjectConstructor(mozilla::plugins::PPluginScriptableObjectChild * aActor) Line 2733 C++ xul.dll!mozilla::plugins::PPluginInstanceChild::OnMessageReceived(const IPC::Message & msg__) Line 2699 C++ xul.dll!mozilla::plugins::PPluginModuleChild::OnMessageReceived(const IPC::Message & msg__) Line 747 C++ xul.dll!mozilla::ipc::MessageChannel::DispatchAsyncMessage(const IPC::Message & aMsg) Line 1661 C++ xul.dll!mozilla::ipc::MessageChannel::DispatchMessageW(IPC::Message && aMsg) Line 1602 C++ xul.dll!mozilla::ipc::MessageChannel::Call(IPC::Message * aMsg, IPC::Message * aReply) Line 1365 C++ xul.dll!mozilla::plugins::PPluginScriptableObjectChild::CallNPN_Evaluate(const nsCString & aScript, mozilla::plugins::Variant * aResult, bool * aSuccess) Line 105 C++ xul.dll!mozilla::plugins::PluginScriptableObjectChild::Evaluate(_NPString * aScript, _NPVariant * aResult) Line 1191 C++ xul.dll!mozilla::plugins::child::_evaluate(_NPP * aNPP, NPObject * aObject, _NPString * aScript, _NPVariant * aResult) Line 1429 C++ NPSWF32_24_0_0_194.dll!6758196e() Unknown ... native event loop calling into Flash In NPN_CreateObject, aNPP is non-null, but PluginInstanceChild* i = InstCast(aNPP) `i` is null. Which is pretty weird because that comes from this code in CreateProxyObject: NPObject* npobject = PluginModuleChild::sBrowserFuncs.createobject(mInstance->GetNPP(), proxyClass); The object being created at the top of the stack belongs to a different plugin instance than the one at the bottom of the stack. Presumably the one at the top of the stack has already been destroyed, which would trigger assertions in debug builds but there are very few release asserts in this code. NPP.ndata is only nulled out at PluginInstanceChild::Destroy at http://searchfox.org/mozilla-central/rev/02a56df6474a97cf84d94bbcfaa126979970905d/dom/plugins/ipc/PluginInstanceChild.cpp#4466

Too late for firefox 52, mass-wontfix.