(no longer active) Reporter
	Description • 11 years ago

It turns out that the cubeb callbacks are received until way late during shutdown, and our deadlock handler cannot handle the case where it gets called after it has been shut down, so we just crash.  See this gdb log:

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x0000000000000008
[Switching to process 71269 thread 0x12007]
0x00000001040899c9 in mozilla::DeadlockDetector<mozilla::BlockingResourceBase::DeadlockDetectorEntry>::CheckAcquisition (this=0x0, aLast=0x0, aProposed=0x113c12b00, aCallContext=@0x1220805c0) at DeadlockDetector.h:402
402	        PRAutoLock _(mLock);
(gdb) bt
#0  0x00000001040899c9 in mozilla::DeadlockDetector<mozilla::BlockingResourceBase::DeadlockDetectorEntry>::CheckAcquisition (this=0x0, aLast=0x0, aProposed=0x113c12b00, aCallContext=@0x1220805c0) at DeadlockDetector.h:402
#1  0x0000000104088b7c in mozilla::BlockingResourceBase::CheckAcquire (this=0x11fec74e0, aCallContext=@0x1220805c0) at BlockingResourceBase.cpp:105
#2  0x000000010408923a in mozilla::Mutex::Lock (this=0x11fec74e0) at BlockingResourceBase.cpp:227
#3  0x00000001025e4675 in mozilla::Monitor::Lock (this=0x11fec74e0) at Monitor.h:37
#4  0x00000001025e4763 in mozilla::MonitorAutoLock::MonitorAutoLock (this=0x1220806a0, aMonitor=@0x11fec74e0) at Monitor.h:92
#5  0x00000001025e397d in mozilla::MonitorAutoLock::MonitorAutoLock (this=0x1220806a0, aMonitor=@0x11fec74e0) at Monitor.h:93
#6  0x00000001025e30b5 in mozilla::BufferedAudioStream::DataCallback (this=0x11fec7450, aBuffer=0x12418b000, aFrames=557) at /Users/ehsanakhgari/moz/mozilla-central/content/media/AudioStream.cpp:837
#7  0x00000001025e39dc in mozilla::BufferedAudioStream::DataCallback_S (aThis=0x11fec7450, aBuffer=0x12418b000, aFrames=557) at /Users/ehsanakhgari/moz/mozilla-central/content/media/AudioStream.cpp:334
#8  0x00000001049ad544 in audiounit_output_callback (user_ptr=0x12415e5e0, flags=0x1220807ec, tstamp=0x122f62160, bus=0, nframes=557, bufs=0x124198e20) at /Users/ehsanakhgari/moz/mozilla-central/media/libcubeb/src/cubeb_audiounit.c:65
#9  0x000000012428c987 in AUInputElement::PullInput ()
#10 0x000000012428c050 in AUInputFormatConverter2::InputProc ()
#11 0x00007fff8fc8efe1 in AudioConverterChain::CallInputProc ()
#12 0x00007fff8fc8ed04 in AudioConverterChain::FillBufferFromInputProc ()
#13 0x00007fff8fc8ec92 in BufferedAudioConverter::GetInputBytes ()
#14 0x00007fff8fc90b5e in Resampler2Wrapper::RenderOutput ()
#15 0x00007fff8fc8e930 in BufferedAudioConverter::FillBuffer ()
#16 0x00007fff8fc8ea99 in AudioConverterChain::RenderOutput ()
#17 0x00007fff8fc8e930 in BufferedAudioConverter::FillBuffer ()
#18 0x00007fff8fc8e59a in AudioConverterFillComplexBuffer ()
#19 0x000000012428bf64 in AUInputFormatConverter2::PullAndConvertInput ()
#20 0x000000012428bbd6 in AUConverterBase::RenderBus ()
#21 0x0000000124289671 in AUBase::DoRenderBus ()
#22 0x0000000124287dbd in AUBase::DoRender ()
#23 0x000000012428ee81 in AUHAL::AUIOProc ()
#24 0x00007fff917b6bad in HALC_ProxyIOContext::IOWorkLoop ()
#25 0x00007fff917b61a1 in HALC_ProxyIOContext::IOThreadEntry ()
#26 0x00007fff917b6069 in HALB_IOThread::Entry ()
#27 0x00007fff87ecf7a2 in _pthread_start ()
#28 0x00007fff87ebc1e1 in thread_start ()
(gdb) thread 1
[Switching to thread 1 (process 71269), "com.apple.main-thread"]
0x00007fff87f4340b in __ultoa ()
(gdb) bt 
#0  0x00007fff87f4340b in __ultoa ()
#1  0x00007fff87f44f54 in __vfprintf ()
#2  0x00007fff87f432fb in vfprintf_l ()
#3  0x00007fff87f38cbf in fprintf ()
#4  0x0000000104141562 in BloatEntry::Dump (this=0x1108b27a0, i=724, out=0x7fff75aeabc0, type=ALL_STATS) at /Users/ehsanakhgari/moz/mozilla-central/xpcom/base/nsTraceRefcntImpl.cpp:374
#5  0x000000010413e833 in nsTraceRefcntImpl::DumpStatistics (type=ALL_STATS, out=0x7fff75aeabc0) at /Users/ehsanakhgari/moz/mozilla-central/xpcom/base/nsTraceRefcntImpl.cpp:522
#6  0x000000010413ed69 in mozilla::LogTerm () at /Users/ehsanakhgari/moz/mozilla-central/xpcom/base/nsTraceRefcntImpl.cpp:945
#7  0x000000010413ecd9 in NS_LogTerm () at /Users/ehsanakhgari/moz/mozilla-central/xpcom/base/nsTraceRefcntImpl.cpp:917
#8  0x00000001000073fb in NS_LogTerm () at /Users/ehsanakhgari/moz/mozilla-central/xpcom/glue/standalone/nsXPCOMGlue.cpp:851
#9  0x000000010000189c in main (argc=6, argv=0x7fff5fbff308) at /Users/ehsanakhgari/moz/mozilla-central/browser/app/nsBrowserApp.cpp:578

Note that the main thread is way past XPCOM shutdown at this point.

Ben, do you think it's acceptable for me to fix the deadlock detector to just ignore the cases where sDeadlockDetector is null?

	(no longer active) Reporter
	Comment 1 • 11 years ago

(Note that this doesn't matter in non-debug builds, but I see this crash a hundred times every day and am fed up with it!)

	(no longer active) Reporter
	Comment 2 • 11 years ago

Hmm, so it seems to me like this will only happen if we leak some AudioStream objects.  Looking through the lifetime management of the MediaStreamGraphImpl object (it's really hard to understand exactly what's happening here), if we fall into this code path <http://mxr.mozilla.org/mozilla-central/source/content/media/MediaStreamGraph.cpp?rev=59e93bff76d4#1251>, we will leak the MSG object and all of its members, including probably some AudioStream objects, which means that they won't destroy their cubeb stream, which means that the data callbacks will run through shutdown, in which case bad things will happen...

	Robert O'Callahan (:roc) (email my personal email if necessary)
	Comment 3 • 11 years ago

I'm afraid the forced-shutdown MSG shutdown case is complicated. The comments here may help: http://dxr.mozilla.org/mozilla-central/content/media/MediaStreamGraphImpl.h#l423

I don't know exactly what's going on in your case, but I expect the graph to enter the  LIFECYCLE_WAITING_FOR_STREAM_DESTRUCTION state and before or after that point, all DOMMediaStreams should be cycle-collected. As their destructors run, we should be calling MediaStream::Destroy on all MediaStreams. The last MediaStream::Destroy should actually destroy the MediaStreamGraphImpl (on the main thread!).

What I don't understand is why we deleted the BufferedAudioStream object and allowed its cubeb callback to keep running. Isn't that a bug in BufferedAudioStream?

	(no longer active) Reporter
	Comment 4 • 11 years ago

(In reply to comment #3)
> What I don't understand is why we deleted the BufferedAudioStream object and
> allowed its cubeb callback to keep running. Isn't that a bug in
> BufferedAudioStream?

Could be, I don't know a lot about that code.

	(no longer active) Reporter
	Comment 5 • 11 years ago

Mass moving Web Audio bugs to the Web Audio component.  Filter on duckityduck.

	(no longer active) Reporter
	Comment 6 • 11 years ago

I haven't seen this for quite some time...