Closed Bug 840165 Opened 12 years ago Closed 11 years ago

Intermittent Windows mochitest-1,2,3 hangs on Shutdown "TEST-UNEXPECTED-FAIL | Shutdown | application timed out after 330 seconds with no output"

Categories

(Core :: General, defect)

Product:
Core
Component:
General
Platform:
x86
Windows 7
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED DUPLICATE of bug 918759

People

(Reporter: RyanVM, Unassigned)

References

Details

(Keywords: crash, hang)

While we first started seeing these hangs beginning last night, retriggers have shown this to occur as far back as when bug 835698 re-landed on Friday. However, the hang still happens even after backing it out. An example of the hang is shown below: https://tbpl.mozilla.org/php/getParsedLog.php?id=19626178&tree=Mozilla-Inbound Rev3 WINNT 6.1 mozilla-inbound debug test mochitest-1 on 2013-02-10 17:39:22 PST for push a4ce303cca2f slave: talos-r3-w7-092 Operating system: Windows NT 6.1.7600 CPU: x86 GenuineIntel family 6 model 23 stepping 10 2 CPUs Crash reason: EXCEPTION_ACCESS_VIOLATION_WRITE Crash address: 0x0 Thread 26 (crashed) 0 crashinjectdll.dll!CrashingThread(void *) [crashinjectdll.cpp:a4ce303cca2f : 17 + 0x0] eip = 0x69c41019 esp = 0x0616f898 ebp = 0x0616f8a0 ebx = 0x00000000 esi = 0x00000000 edi = 0x00000000 eax = 0x76bf1162 ecx = 0x00000000 edx = 0x69c4100a efl = 0x00010246 Found by: given as instruction pointer in context 1 kernel32.dll + 0x51173 eip = 0x76bf1174 esp = 0x0616f89c ebp = 0x0616f8a0 Found by: call frame info 2 ntdll.dll + 0x5b3f4 eip = 0x77b0b3f5 esp = 0x0616f8a8 ebp = 0x0616f8e0 Found by: previous frame's frame pointer 3 ntdll.dll + 0x5b3c7 eip = 0x77b0b3c8 esp = 0x0616f8e8 ebp = 0x0616f8f8 Found by: previous frame's frame pointer Thread 0 0 ntdll.dll + 0x464f4 eip = 0x77af64f4 esp = 0x001ff68c ebp = 0x001ff6f8 ebx = 0x00000000 esi = 0x000000e0 edi = 0x00000000 eax = 0x00000008 ecx = 0x00000000 edx = 0x00000000 efl = 0x00200246 Found by: given as instruction pointer in context 1 kernel32.dll + 0x4f002 eip = 0x76bef003 esp = 0x001ff700 ebp = 0x001ff710 Found by: previous frame's frame pointer 2 kernel32.dll + 0x4efb1 eip = 0x76beefb2 esp = 0x001ff718 ebp = 0x001ff724 Found by: previous frame's frame pointer 3 nspr4.dll!_PR_MD_WAIT_CV [w95cv.c:a4ce303cca2f : 248 + 0xd] eip = 0x7344d1d9 esp = 0x001ff72c ebp = 0x0121be54 Found by: previous frame's frame pointer 4 nspr4.dll!_PR_WaitCondVar [prucv.c:a4ce303cca2f : 172 + 0x27] eip = 0x734485bc esp = 0x001ff748 ebp = 0x001ff798 Found by: call frame info 5 nspr4.dll!PR_Wait [prmon.c:a4ce303cca2f : 152 + 0x19] eip = 0x73448323 esp = 0x001ff760 ebp = 0x001ff798 Found by: call frame info 6 xul.dll!mozilla::ReentrantMonitor::Wait(unsigned int) [BlockingResourceBase.cpp:a4ce303cca2f : 313 + 0x1a] eip = 0x6bfd9832 esp = 0x001ff77c ebp = 0x001ff798 Found by: call frame info 7 xul.dll!nsEventQueue::GetEvent(bool,nsIRunnable * *) [nsEventQueue.cpp:a4ce303cca2f : 58 + 0x9] eip = 0x6c0299c6 esp = 0x001ff7a0 ebp = 0x001ff7b4 Found by: call frame info 8 xul.dll!nsThread::ProcessNextEvent(bool,bool *) [nsThread.cpp:a4ce303cca2f : 619 + 0x2d] eip = 0x6c02bca2 esp = 0x001ff7bc ebp = 0x001ff7ec Found by: call frame info 9 xul.dll!NS_ProcessNextEvent_P(nsIThread *,bool) [nsThreadUtils.cpp:a4ce303cca2f : 238 + 0xc] eip = 0x6bfd751f esp = 0x001ff7f4 ebp = 0x001ff800 Found by: call frame info 10 xul.dll!nsThread::Shutdown() [nsThread.cpp:a4ce303cca2f : 474 + 0x9] eip = 0x6c02c258 esp = 0x001ff808 ebp = 0x001ff830 Found by: call frame info 11 xul.dll!nsThreadManager::Shutdown() [nsThreadManager.cpp:a4ce303cca2f : 128 + 0x5] eip = 0x6c02d83a esp = 0x001ff838 ebp = 0x001ff858 Found by: call frame info 12 xul.dll!mozilla::ShutdownXPCOM(nsIServiceManager *) [nsXPComInit.cpp:a4ce303cca2f : 577 + 0x9] eip = 0x6bfdec3b esp = 0x001ff860 ebp = 0x001ff884 Found by: call frame info 13 xul.dll!ScopedXPCOMStartup::~ScopedXPCOMStartup() [nsAppRunner.cpp:a4ce303cca2f : 1125 + 0x6] eip = 0x6ab45faf esp = 0x001ff88c ebp = 0x001ff8a0 Found by: call frame info 14 xul.dll!XREMain::XRE_main(int,char * * const,nsXREAppData const *) [nsAppRunner.cpp:a4ce303cca2f : 3924 + 0xd] eip = 0x6ab4cccb esp = 0x001ff8a8 ebp = 0x001ff8b8 Found by: call frame info 15 xul.dll!XRE_main [nsAppRunner.cpp:a4ce303cca2f : 4102 + 0x11] eip = 0x6ab4ceca esp = 0x001ff8c0 ebp = 0x001ff9c8 Found by: call frame info 16 firefox.exe!do_main [nsBrowserApp.cpp:a4ce303cca2f : 185 + 0x11] eip = 0x01362194 esp = 0x001ff9d0 ebp = 0x001ffb50 Found by: call frame info 17 firefox.exe!NS_internal_main(int,char * *) [nsBrowserApp.cpp:a4ce303cca2f : 377 + 0xc] eip = 0x01362391 esp = 0x001ffb58 ebp = 0x001ffbe4 Found by: call frame info 18 firefox.exe!wmain [nsWindowsWMain.cpp:a4ce303cca2f : 105 + 0x6] eip = 0x013624e3 esp = 0x001ffbec ebp = 0x001ffc18 Found by: call frame info 19 firefox.exe!__tmainCRTStartup [crtexe.c : 552 + 0x18] eip = 0x013640ff esp = 0x001ffc20 ebp = 0x001ffc68 Found by: call frame info 20 firefox.exe!wmainCRTStartup [crtexe.c : 370 + 0x4] eip = 0x01363f2f esp = 0x001ffc70 ebp = 0x001ffc70 Found by: call frame info 21 kernel32.dll + 0x51173 eip = 0x76bf1174 esp = 0x001ffc78 ebp = 0x001ffc7c Found by: call frame info 22 ntdll.dll + 0x5b3f4 eip = 0x77b0b3f5 esp = 0x001ffc84 ebp = 0x001ffcbc Found by: previous frame's frame pointer 23 ntdll.dll + 0x5b3c7 eip = 0x77b0b3c8 esp = 0x001ffcc4 ebp = 0x001ffcd4 Found by: previous frame's frame pointer
Not sure why the top crash frame fallback (after "Shutdown" is ignored) isn't working; adding more logging in a soon to be attached patch in bug 793159.
https://tbpl.mozilla.org/php/getParsedLog.php?id=19672091&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=19672237&tree=Mozilla-Inbound
Summary: Intermittent Windows mochitest-1 hangs on Shutdown | application timed out after 330 seconds with no output | application crashed [@ CrashingThread(void *)] → Intermittent Windows mochitest-1,3 hangs on Shutdown | application timed out after 330 seconds with no output | application crashed [@ CrashingThread(void *)]
Depends on: 841786
seems this also hit now the win8 ix test maschines 14:59:51 INFO - PROCESS-CRASH | Shutdown | application crashed [@ CrashingThread(void *)] 14:59:51 INFO - Crash dump filename: c:\users\cltbld~1.t-w\appdata\local\temp\tmpq8bm5i\minidumps\36d1060f-79d7-4825-9b0a-c72b3fdaf65f.dmp 14:59:51 INFO - Operating system: Windows NT 14:59:51 INFO - 6.2.9200 14:59:51 INFO - CPU: x86 14:59:51 INFO - GenuineIntel family 6 model 30 stepping 5 14:59:51 INFO - 8 CPUs 14:59:51 INFO - Crash reason: EXCEPTION_ACCESS_VIOLATION_WRITE 14:59:51 INFO - Crash address: 0x0 14:59:51 INFO - Thread 35 (crashed) 14:59:51 INFO - 0 crashinjectdll.dll!CrashingThread(void *) [crashinjectdll.cpp:fc0b4114994c : 17 + 0x0] 14:59:51 INFO - eip = 0x6c1a1019 esp = 0x066ff9ec ebp = 0x066ff9f4 ebx = 0x00000000 14:59:51 INFO - esi = 0x00000000 edi = 0x00000000 eax = 0x76cd8535 ecx = 0x00000000 14:59:51 INFO - edx = 0x6c1a100a efl = 0x00010246 14:59:51 INFO - Found by: given as instruction pointer in context 14:59:51 INFO - 1 kernel32.dll + 0x28542 14:59:51 INFO - eip = 0x76cd8543 esp = 0x066ff9f0 ebp = 0x066ff9f4 14:59:51 INFO - Found by: call frame info 14:59:51 INFO - 2 ntdll.dll + 0x5ac68 14:59:51 INFO - eip = 0x779fac69 esp = 0x066ff9fc ebp = 0x066ffa38 14:59:51 INFO - Found by: previous frame's frame pointer 14:59:51 INFO - 3 ntdll.dll + 0x5ac3b 14:59:51 INFO - eip = 0x779fac3c esp = 0x066ffa40 ebp = 0x066ffa50 14:59:51 INFO - Found by: previous frame's frame pointer 14:59:51 INFO - Thread 0 14:59:51 INFO - 0 ntdll.dll + 0x40da8 14:59:51 INFO - eip = 0x779e0da8 esp = 0x00bced04 ebp = 0x00bced6c ebx = 0x00000000 14:59:51 INFO - esi = 0x00bced48 edi = 0x00000164 eax = 0x00000001 ecx = 0x00000000 14:59:51 INFO - edx = 0x00000000 efl = 0x00200202 14:59:51 INFO - Found by: given as instruction pointer in context 14:59:51 INFO - 1 KERNELBASE.dll + 0x10b3 14:59:51 INFO - eip = 0x773f10b4 esp = 0x00bced74 ebp = 0x00bced80 14:59:51 INFO - Found by: previous frame's frame pointer 14:59:51 INFO - 2 nspr4.dll!_PR_MD_WAIT_CV [w95cv.c:fc0b4114994c : 248 + 0xd] 14:59:51 INFO - eip = 0x7022d1d9 esp = 0x00bced88 ebp = 0x026f39e4 14:59:51 INFO - Found by: previous frame's frame pointer 14:59:51 INFO - 3 nspr4.dll!_PR_WaitCondVar [prucv.c:fc0b4114994c : 172 + 0x27] 14:59:51 INFO - eip = 0x702285bc esp = 0x00bceda4 ebp = 0x00bcedf4 14:59:51 INFO - Found by: call frame info 14:59:51 INFO - 4 nspr4.dll!PR_Wait [prmon.c:fc0b4114994c : 152 + 0x19]
Summary: Intermittent Windows mochitest-1,3 hangs on Shutdown | application timed out after 330 seconds with no output | application crashed [@ CrashingThread(void *)] → Intermittent Windows mochitest-1,2,3 hangs on Shutdown | application timed out after 330 seconds with no output | application crashed [@ CrashingThread(void *)]
TBPL now supports whole line matching, even for items on the blacklist (ie "Shutdown"), so the string "TEST-UNEXPECTED-FAIL | Shutdown | application timed out after 330 seconds with no output" will match - so we can remove the hang signature from the summary and avoid yet more false positives.
Summary: Intermittent Windows mochitest-1,2,3 hangs on Shutdown | application timed out after 330 seconds with no output | application crashed [@ CrashingThread(void *)] → Intermittent Windows mochitest-1,2,3 hangs on Shutdown "TEST-UNEXPECTED-FAIL | Shutdown | application timed out after 330 seconds with no output"
Meet the new shutdown hang bug, same as the old shutdown hang bug.
Status: NEW → RESOLVED
Closed: 11 years ago
Keywords: intermittent-failure
Resolution: --- → DUPLICATE
You need to log in before you can comment on or make changes to this bug.