Filed by: philringnalda@gmail.com https://treeherder.mozilla.org/logviewer.html#?job_id=30455054&repo=mozilla-inbound http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-win32-debug/1466572563/mozilla-inbound_win7_ix-debug_test-mochitest-e10s-3-bm126-tests1-windows-build104.txt.gz

jimm, check out the weird stack of the plugin process at the time of this hang. Does this look familiar to you? We appear to be stuck waiting on a lock from within LdrpModuleCacheLoadNotification->EtwEventEnabled which all feels really strange. Thread 3 and Thread 5 of that same process are also in EtwEventEnabled locks. 23:25:54 INFO - 0 ntdll.dll!KiFastSystemCallRet + 0x0 23:25:54 INFO - eip = 0x772c70b4 esp = 0x001cea14 ebp = 0x001cea78 ebx = 0x00000000 23:25:54 INFO - esi = 0x77357340 edi = 0x00000000 eax = 0x00002e00 ecx = 0x67b1ee78 23:25:54 INFO - edx = 0x00002e2e efl = 0x00000217 23:25:54 INFO - Found by: given as instruction pointer in context 23:25:54 INFO - 1 ntdll.dll!NtWaitForSingleObject + 0xc 23:25:54 INFO - eip = 0x772c6a24 esp = 0x001cea18 ebp = 0x001cea78 23:25:54 INFO - Found by: call frame info 23:25:54 INFO - 2 ntdll.dll!RtlpWaitOnCriticalSection + 0xc4 23:25:54 INFO - eip = 0x772b2264 esp = 0x001cea1c ebp = 0x001cea78 23:25:54 INFO - Found by: call frame info 23:25:54 INFO - 3 ntdll.dll!EtwEventEnabled + 0xd9 23:25:54 INFO - eip = 0x772b2148 esp = 0x001cea80 ebp = 0x001ceaa0 ebx = 0x00000001 23:25:54 INFO - Found by: call frame info 23:25:54 INFO - 4 ntdll.dll!LdrpModuleCacheLoadNotification + 0x1d 23:25:54 INFO - eip = 0x772dfea3 esp = 0x001ceaa8 ebp = 0x001cec0c ebx = 0x00000001 23:25:54 INFO - Found by: call frame info 23:25:54 INFO - 5 ntdll.dll!LdrLoadDll + 0x74 23:25:54 INFO - eip = 0x772e232c esp = 0x001cec14 ebp = 0x001cec40 23:25:54 INFO - Found by: call frame info 23:25:54 INFO - 6 mozglue.dll!`anonymous namespace'::patched_LdrLoadDll [WindowsDllBlocklist.cpp:7b03f23fdde5 : 749 + 0x1e] 23:25:54 INFO - eip = 0x691a3502 esp = 0x001cec48 ebp = 0x001ced2c ebx = 0x67b1ee78 23:25:54 INFO - Found by: call frame info 23:25:54 INFO - 7 KERNELBASE.dll!FreeLibrary + 0xb4 23:25:54 INFO - eip = 0x755a88ee esp = 0x001ced34 ebp = 0x001ced64 23:25:54 INFO - Found by: call frame info 23:25:54 INFO - 8 nptest.dll + 0x5e486 23:25:54 INFO - eip = 0x67afe486 esp = 0x001ced6c ebp = 0x001ced84 23:25:54 INFO - Found by: previous frame's frame pointer 23:25:54 INFO - 9 nptest.dll + 0x5e3db 23:25:54 INFO - eip = 0x67afe3db esp = 0x001ced8c ebp = 0x001ced9c 23:25:54 INFO - Found by: previous frame's frame pointer 23:25:54 INFO - 10 nptest.dll + 0x5e286 23:25:54 INFO - eip = 0x67afe286 esp = 0x001ceda4 ebp = 0x001cede4 23:25:54 INFO - Found by: previous frame's frame pointer 23:25:54 INFO - 11 nptest.dll + 0x4ea65 23:25:54 INFO - eip = 0x67aeea65 esp = 0x001cedec ebp = 0x001cedf8 23:25:54 INFO - Found by: previous frame's frame pointer 23:25:54 INFO - 12 nptest.dll + 0x4eb04 23:25:54 INFO - eip = 0x67aeeb04 esp = 0x001cee00 ebp = 0x001cee44 23:25:54 INFO - Found by: previous frame's frame pointer 23:25:54 INFO - 13 nptest.dll + 0xd97e 23:25:54 INFO - eip = 0x67aad97e esp = 0x001cee4c ebp = 0x001cee54 23:25:54 INFO - Found by: previous frame's frame pointer 23:25:54 INFO - 14 nptest.dll + 0x3bad 23:25:54 INFO - eip = 0x67aa3bad esp = 0x001cee5c ebp = 0x001cee74 23:25:54 INFO - Found by: previous frame's frame pointer 23:25:54 INFO - 15 nptest.dll + 0x5709 23:25:54 INFO - eip = 0x67aa5709 esp = 0x001cee7c ebp = 0x001ceec8 23:25:54 INFO - Found by: previous frame's frame pointer 23:25:54 INFO - 16 nptest.dll + 0x34d9 23:25:54 INFO - eip = 0x67aa34d9 esp = 0x001ceed0 ebp = 0x001ceee8 23:25:54 INFO - Found by: previous frame's frame pointer 23:25:54 INFO - 17 nptest.dll + 0xa9bd 23:25:54 INFO - eip = 0x67aaa9bd esp = 0x001ceef0 ebp = 0x001cef00 23:25:54 INFO - Found by: previous frame's frame pointer 23:25:54 INFO - 18 nptest.dll + 0x87e2 23:25:54 INFO - eip = 0x67aa87e2 esp = 0x001cef08 ebp = 0x001cef10 23:25:54 INFO - Found by: previous frame's frame pointer 23:25:54 INFO - 19 nptest.dll + 0x37ac 23:25:54 INFO - eip = 0x67aa37ac esp = 0x001cef18 ebp = 0x001cef28 23:25:54 INFO - Found by: previous frame's frame pointer 23:25:54 INFO - 20 nptest.dll + 0x37df 23:25:54 INFO - eip = 0x67aa37df esp = 0x001cef30 ebp = 0x001cef48 23:25:54 INFO - Found by: previous frame's frame pointer 23:25:54 INFO - 21 nptest.dll + 0x3b01 23:25:54 INFO - eip = 0x67aa3b01 esp = 0x001cef50 ebp = 0x001cefdc 23:25:54 INFO - Found by: previous frame's frame pointer 23:25:54 INFO - 22 xul.dll!mozilla::plugins::PluginInstanceChild::DoNPP_New() [PluginInstanceChild.cpp:7b03f23fdde5 : 267 + 0x3b] 23:25:54 INFO - eip = 0x5f73f9b5 esp = 0x001cefe4 ebp = 0x001cf02c 23:25:54 INFO - Found by: previous frame's frame pointer

This is happening a lot on a bunch of different win32 tests. It would be spectacularly helpful to the fixing of this bug to either have: 1) the hang minidumps being collected here and native symbols including nptest.pdb or 2) nptest symbols feeding minidump-stackwalk (bug 1262337) Philor, is this happening mainly/only on AWS/EC2 instances, and not our older hardware testers?

Three instances (counting the dupes), all on hardware (machines named "t-w732-ix-*" are hardware, " t-w732-spot-*" are AWS, and browser-chrome is not yet running in AWS, unless maybe it's running on try). Personally, what I would say is happening here is that we added a pretty easy two or three click way to file intermittents from treeherder, and so rather than just ignoring the hundreds of unfiled expressions of the utter brokenness of plugin tests on Windows, we're, or perhaps I'm, now filing them. My gut feeling is that this is just more of bug 1262336, but one of the problems with having left bug 1262336 alone for several months now is that we're pretty unlikely to notice any new bustage in plugins on Windows, because we expect plugin tests on Windows to be broken.

(In reply to Benjamin Smedberg [:bsmedberg] from comment #4) > This is happening a lot on a bunch of different win32 tests. It would be > spectacularly helpful to the fixing of this bug to either have: > > 1) the hang minidumps being collected here and native symbols including > nptest.pdb > or > 2) nptest symbols feeding minidump-stackwalk (bug 1262337) > > Philor, is this happening mainly/only on AWS/EC2 instances, and not our > older hardware testers? It certainly would be nice to see what nptest is up to here. Looks likw this happens in NPP_New on a simple streaming test case. As for EtwEventEnabled, that appears to have something to do with Windows event tracing. Maybe AWS has tracing turned on and this triggers a deadlock?