Going out on a limb that this is related to bug 848423. This is only on inbound (so far) and all the instances are from after it landed. https://tbpl.mozilla.org/php/getParsedLog.php?id=20432737&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=20422286&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=20427264&tree=Mozilla-Inbound Rev3 WINNT 5.1 mozilla-inbound pgo test mochitest-2 on 2013-03-07 12:50:47 PST for push 908a5ff75e5e slave: talos-r3-xp-023 13:12:57 INFO - PROCESS-CRASH | /tests/dom/media/tests/mochitest/test_peerConnection_basicAudioVideo.html | application crashed [@ nsRunnable::Release()] 13:12:57 INFO - Crash dump filename: c:\docume~1\cltbld\locals~1\temp\tmpuucmmz\minidumps\4b38ac37-b53d-4459-912e-402812a31329.dmp 13:12:57 INFO - Operating system: Windows NT 13:12:57 INFO - 5.1.2600 Service Pack 2 13:12:57 INFO - CPU: x86 13:12:57 INFO - GenuineIntel family 6 model 23 stepping 10 13:12:57 INFO - 2 CPUs 13:12:57 INFO - Crash reason: EXCEPTION_ACCESS_VIOLATION_WRITE 13:12:57 INFO - Crash address: 0x5 13:12:57 INFO - Thread 6 (crashed) 13:12:57 INFO - 0 xul.dll!nsRunnable::Release() [nsThreadUtils.cpp:908a5ff75e5e : 31 + 0xd] 13:12:57 INFO - eip = 0x015723bd esp = 0x03bcfdd4 ebp = 0x03bcfdd8 ebx = 0x013fd101 13:12:57 INFO - esi = 0x00000001 edi = 0x00000001 eax = 0xffffffff ecx = 0x06fae910 13:12:57 INFO - edx = 0x00000005 efl = 0x00010286 13:12:57 INFO - Found by: given as instruction pointer in context 13:12:57 INFO - 1 xul.dll!mozilla::TransportFlow::Release() [transportflow.h:908a5ff75e5e : 65 + 0x21] 13:12:57 INFO - eip = 0x01c1a0ee esp = 0x03bcfde0 ebp = 0x03bcfde8 13:12:57 INFO - Found by: call frame info 13:12:57 INFO - 2 xul.dll!nsRefPtr<mozilla::TransportFlow>::~nsRefPtr<mozilla::TransportFlow>() [nsAutoPtr.h:908a5ff75e5e : 880 + 0x5] 13:12:57 INFO - eip = 0x01c7345e esp = 0x03bcfdf0 ebp = 0x03bcfdfc 13:12:57 INFO - Found by: call frame info 13:12:57 INFO - 3 xul.dll!mozilla::runnable_args_nm_1<void (*)(nsRefPtr<mozilla::TransportFlow>),already_AddRefed<mozilla::TransportFlow> >::Run() [runnable_utils_generated.h:908a5ff75e5e : 85 + 0xc] 13:12:57 INFO - eip = 0x01cbaa3b esp = 0x03bcfdf8 ebp = 0x03bcfdfc 13:12:57 INFO - Found by: call frame info 13:12:57 INFO - 4 xul.dll!nsThread::ProcessNextEvent(bool,bool *) [nsThread.cpp:908a5ff75e5e : 627 + 0x13] 13:12:57 INFO - eip = 0x0157a596 esp = 0x03bcfe04 ebp = 0x03bcfe70 13:12:57 INFO - Found by: call frame info 13:12:57 INFO - 5 xul.dll!NS_ProcessNextEvent_P(nsIThread *,bool) [nsThreadUtils.cpp:908a5ff75e5e : 238 + 0x1e] 13:12:57 INFO - eip = 0x015563bd esp = 0x03bcfe78 ebp = 0x03bcfe84 13:12:57 INFO - Found by: call frame info 13:12:57 INFO - 6 xul.dll!nsSocketTransportService::Run() [nsSocketTransportService2.cpp:908a5ff75e5e : 649 + 0x7] 13:12:57 INFO - eip = 0x0152bd08 esp = 0x03bcfe8c ebp = 0x03bcfea8 13:12:57 INFO - Found by: call frame info 07:45:17 INFO - PROCESS-CRASH | /tests/dom/media/tests/mochitest/test_peerConnection_basicAudioVideoCombined.html | application crashed [@ nsRefPtr<nsIDOMEventListener>::~nsRefPtr<nsIDOMEventListener>()] 07:45:17 INFO - Crash dump filename: c:\docume~1\cltbld\locals~1\temp\tmpqg98xo\minidumps\e8c8ddb4-4b0b-4af1-b135-32497e88e362.dmp 07:45:17 INFO - Operating system: Windows NT 07:45:17 INFO - 5.1.2600 Service Pack 2 07:45:17 INFO - CPU: x86 07:45:17 INFO - GenuineIntel family 6 model 23 stepping 10 07:45:17 INFO - 2 CPUs 07:45:17 INFO - Crash reason: EXCEPTION_ACCESS_VIOLATION_READ 07:45:17 INFO - Crash address: 0x4c191008 07:45:17 INFO - Thread 0 (crashed) 07:45:17 INFO - 0 xul.dll!nsRefPtr<nsIDOMEventListener>::~nsRefPtr<nsIDOMEventListener>() [nsAutoPtr.h:1768185637fd : 880 + 0x2] 07:45:17 INFO - eip = 0x01533868 esp = 0x0012e444 ebp = 0x0012e454 ebx = 0x0c4aa840 07:45:17 INFO - esi = 0x0c4aa860 edi = 0x00000001 eax = 0x0998f5ff ecx = 0x4c191000 07:45:17 INFO - edx = 0x80000008 efl = 0x00210206 07:45:17 INFO - Found by: given as instruction pointer in context 07:45:17 INFO - 1 xul.dll!UndisplayedNode::`scalar deleting destructor'(unsigned int) + 0x35 07:45:17 INFO - eip = 0x01428e64 esp = 0x0012e448 ebp = 0x0012e454 07:45:17 INFO - Found by: call frame info 07:45:17 INFO - 2 xul.dll!RemoveUndisplayedEntry [nsFrameManager.cpp:1768185637fd : 2005 + 0x29] 07:45:17 INFO - eip = 0x01456641 esp = 0x0012e45c ebp = 0x0012e470 07:45:17 INFO - Found by: call frame info 07:45:17 INFO - 3 plds4.dll!PL_HashTableEnumerateEntries [plhash.c:1768185637fd : 374 + 0xa] 07:45:17 INFO - eip = 0x003e16dd esp = 0x0012e478 ebp = 0x0bf35380 07:45:17 INFO - Found by: call frame info 07:45:17 INFO - 4 xul.dll!nsFrameManager::Destroy() [nsFrameManager.cpp:1768185637fd : 224 + 0x17] 07:45:17 INFO - eip = 0x0140ee0c esp = 0x0012e4a4 ebp = 0x0012e4c4 07:45:17 INFO - Found by: call frame info 07:45:17 INFO - 5 xul.dll!nsCSSFrameConstructor::WillDestroyFrameTree() [nsCSSFrameConstructor.cpp:1768185637fd : 8696 + 0x6] 07:45:17 INFO - eip = 0x015d6e71 esp = 0x0012e4bc ebp = 0x0012e4c4 07:45:17 INFO - Found by: call frame info 07:45:17 INFO - 6 xul.dll!PresShell::Destroy() [nsPresShell.cpp:1768185637fd : 1086 + 0x7] 07:45:17 INFO - eip = 0x01447dbc esp = 0x0012e4cc ebp = 0x0012e4e4 07:45:17 INFO - Found by: call frame info 07:45:17 INFO - 7 xul.dll!nsDocumentViewer::DestroyPresShell() [nsDocumentViewer.cpp:1768185637fd : 4366 + 0x6] 07:45:17 INFO - eip = 0x01586347 esp = 0x0012e4ec ebp = 0x0012e500 07:45:17 INFO - Found by: call frame info 10:41:02 INFO - PROCESS-CRASH | /tests/dom/media/tests/mochitest/test_peerConnection_basicVideo.html | application crashed [@ mozilla::MediaStreamGraphImpl::ExtractPendingInput(mozilla::SourceMediaStream *,__int64,bool *)] 10:41:02 INFO - Crash dump filename: c:\docume~1\cltbld\locals~1\temp\tmp4xf3v0\minidumps\cfb31ecf-da3e-43db-a569-564602a175bb.dmp 10:41:02 INFO - Operating system: Windows NT 10:41:02 INFO - 5.1.2600 Service Pack 2 10:41:02 INFO - CPU: x86 10:41:02 INFO - GenuineIntel family 6 model 23 stepping 10 10:41:02 INFO - 2 CPUs 10:41:02 INFO - Crash reason: EXCEPTION_ACCESS_VIOLATION_READ 10:41:02 INFO - Crash address: 0x8 10:41:02 INFO - Thread 43 (crashed) 10:41:02 INFO - 0 xul.dll!mozilla::MediaStreamGraphImpl::ExtractPendingInput(mozilla::SourceMediaStream *,__int64,bool *) [MediaStreamGraph.cpp:814a0c94b215 : 141 + 0x25] 10:41:02 INFO - eip = 0x02071ada esp = 0x09fffdd8 ebp = 0x09fffdfc ebx = 0x058fe110 10:41:02 INFO - esi = 0x0ab29fa8 edi = 0x093121c0 eax = 0x00000000 ecx = 0x0ac6839c 10:41:02 INFO - edx = 0x00000001 efl = 0x00010246 10:41:02 INFO - Found by: given as instruction pointer in context 10:41:02 INFO - 1 xul.dll!mozilla::MediaStreamGraphImpl::RunThread() [MediaStreamGraph.cpp:814a0c94b215 : 956 + 0x13] 10:41:02 INFO - eip = 0x02162dd6 esp = 0x09fffe04 ebp = 0x09fffe9c 10:41:02 INFO - Found by: call frame info 10:41:02 INFO - 2 xul.dll!mozilla::`anonymous namespace'::MediaStreamGraphThreadRunnable::Run() [MediaStreamGraph.cpp:814a0c94b215 : 1104 + 0xa] 10:41:02 INFO - eip = 0x02163504 esp = 0x09fffea4 ebp = 0x09fffea8 10:41:02 INFO - Found by: call frame info 10:41:02 INFO - 3 xul.dll!nsThread::ProcessNextEvent(bool,bool *) [nsThread.cpp:814a0c94b215 : 627 + 0x13] 10:41:02 INFO - eip = 0x014de4f6 esp = 0x09fffeb0 ebp = 0x09ffff1c 10:41:02 INFO - Found by: call frame info 10:41:02 INFO - 4 xul.dll!nsThread::ThreadFunc(void *) [nsThread.cpp:814a0c94b215 : 265 + 0x1b] 10:41:02 INFO - eip = 0x0157f247 esp = 0x09ffff24 ebp = 0x09ffff44 10:41:02 INFO - Found by: call frame info 10:41:02 INFO - 5 nspr4.dll!_PR_NativeRunThread [pruthr.c:814a0c94b215 : 395 + 0x8] 10:41:02 INFO - eip = 0x003a2f4f esp = 0x09ffff4c ebp = 0x09ffff68 10:41:02 INFO - Found by: call frame info 10:41:02 INFO - 6 nspr4.dll!pr_root [w95thred.c:814a0c94b215 : 90 + 0xc] 10:41:02 INFO - eip = 0x003a44ed esp = 0x09ffff70 ebp = 0x09ffffa8 10:41:02 INFO - Found by: call frame info

All PGO builds too, goodie.

Bug 848423 backed out for now. I'll resolve this once I can confirm with PGO builds that this is gone. https://hg.mozilla.org/integration/mozilla-inbound/rev/1d028bd354d0

https://tbpl.mozilla.org/php/getParsedLog.php?id=20427264&tree=Mozilla-Inbound is from before bug 848423 landed (and stuck), so I guess that makes it unlikely to be the cause.

Further from RyanVM: "My guess is that whatever broke it is in here somewhere: https://hg.mozilla.org/mozilla-central/pushloghtml?startID=24405&endID=24406"

Okay, a search through m-i for changes in this range that impacted files in the media subtree narrows us down: Bug 839677 - Remove problematic gsm_msg_queue and use gsm_msgq instead Bug 828917 - move critsect to cover thread existence test (upstream issue 1465) r=derf Bug 837035 - Support DataChannels with SDP Negotiation r=jesup Bug 836824 - Make libsoundtouch build on FreeBSD and Mac without command-line tools. r=khuey The following changesets were backed out, so they aren't suspects: Bug 848423 - Change nearly all WebRTC instances of linked_ptr<> to nsRefPtr<> r=ehugg Bug 839650 - Add debugs to MediaStreamGraph to ease investigation of issues in the future r=roc

A quick rundown of the suspect bugs: Bug 839677: This patch simply takes two variables that were the same value and collapses them to one variable. It is almost certainly not the cause. Bug 828917: Jesup has indicated that he "can guarantee" it's not this bug. Bug 837035: Given the number of changes in here, it's *possible* that it could be the root; however, I don't see anything specific to call termination that would be particularly suspect. Bug 836824: This rather small patch would seem to only impact CPU feature detection. With this list of suspects, I'd start with Bug 837035 as the most likely candidate.

I triggered some more test runs. Maybe it'll help? https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=908a5ff75e5e

The wild set of random crash locations speaks to some sort of memory error whomping different things. An asan run wouldn't be a bad idea, though the interaction with PGO could be a problem. How frequent/easy is it to trigger? Is it amenable to a small amount of bisection? (painful on PGO, true)

Seems to be near-perma fail on the Windows PGO builds.

So, backing out bug 837035, bug 828917, and bug 839677 did indeed make this go away. Bug 837035 was the first backout and the Win PGO M2 runs were clean. I will reland bug 828917 and bug 839677 in the morning when I can trigger PGO on them and report back.

Fixed by backing out bug 837035.