Closed Bug 864709 Opened 11 years ago Closed 11 years ago

Intermittent webspeech/recognition/test/test_abort.html,test_audio_capture_error.html | application crashed [@ jemalloc_crash] or [@ PR_Lock] or [@ ntdll.dll + 0x4f592], both with mozilla::dom::ScriptProcessorNodeEngine::ProduceAudioBlock() further down

Categories

(Core :: Web Audio, defect)

Product:
Core
Component:
Web Audio
Platform:
ARM
Android
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

()

Status:
RESOLVED FIXED

People

(Reporter: gfritzsche, Assigned: ehsan.akhgari)

References

Details

(Keywords: crash, intermittent-failure)

Crash Data

Attachments

(2 files)

Part 1: Hold a strong ref to the AudioNodeStream before calling ProduceOutput on it
(deleted), patch
padenot
: review+
Details | Diff | Splinter Review
Part 2: Protect AudioNodeEngine::mNode using a lock
(deleted), patch
padenot
: review+
Details | Diff | Splinter Review 
https://tbpl.mozilla.org/php/getParsedLog.php?id=22129455&tree=Mozilla-Inbound&full=1

Android Armv6 Tegra 250 mozilla-inbound opt test mochitest-2 on 2013-04-23 02:45:20 PDT for push 5de9e9db23c5

28148 INFO TEST-INFO | /tests/content/media/webspeech/recognition/test/test_abort.html | requesting EVENT_START
28149 INFO TEST-INFO | /tests/content/media/webspeech/recognition/test/test_abort.html | requesting EVENT_AUDIO_DATA
28150 INFO TEST-PASS | /tests/content/media/webspeech/recognition/test/test_abort.html | received event start
28151 INFO TEST-PASS | /tests/content/media/webspeech/recognition/test/test_abort.html | received event audiostart
INFO | automation.py | Application ran for: 0:15:22.748048
INFO | zombiecheck | Reading PID log: /tmp/tmpM6i3iepidlog
mozcrash INFO | Downloading symbols from: http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android-armv6/1366705961/fennec-23.0a1.en-US.android-arm-armv6.crashreporter-symbols.zip
Downloading symbols from: http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android-armv6/1366705961/fennec-23.0a1.en-US.android-arm-armv6.crashreporter-symbols.zip
PROCESS-CRASH | /tests/content/media/webspeech/recognition/test/test_abort.html | application crashed [@ jemalloc_crash]
Crash dump filename: /tmp/tmpcxD3O5/0d97f57b-0b75-a79d-7227713f-5a2733ee.dmp
Operating system: Android
                  0.0.0 Linux 2.6.32.9-00002-gd8084dc-dirty #1 SMP PREEMPT Wed Feb 2 11:32:06 PST 2011 armv7l nvidia/harmony/harmony/harmony:2.2/FRF91/20110202.102810:eng/test-keys
CPU: arm
     2 CPUs

Crash reason:  SIGSEGV
Crash address: 0x0

Thread 29 (crashed)
 0  libmozglue.so!jemalloc_crash [jemalloc.c:5de9e9db23c5 : 1590 + 0x8]
     r4 = 0x4ea74040    r5 = 0x00000c00    r6 = 0x59757000    r7 = 0x4ea74358
     r8 = 0x00000080    r9 = 0x4ea74044   r10 = 0x59700000    fp = 0x00000001
     sp = 0x5cdbfc78    lr = 0x80c08800    pc = 0x80c06e94
    Found by: given as instruction pointer in context
 1  libmozglue.so!arena_dalloc [jemalloc.c:5de9e9db23c5 : 4608 + 0x2]
     sp = 0x5cdbfc80    pc = 0x80c08800
    Found by: stack scanning
 2  libxul.so!mozilla::dom::ScriptProcessorNodeEngine::ProduceAudioBlock(mozilla::AudioNodeStream*, mozilla::AudioChunk const&, mozilla::AudioChunk*, bool*) [mozalloc.h:5de9e9db23c5 : 225 + 0x2]
     sp = 0x5cdbfca0    pc = 0x539b76e4
    Found by: stack scanning
 3  libxul.so!mozilla::AudioNodeStream::ProduceOutput(long long, long long) [AudioNodeStream.cpp:5de9e9db23c5 : 295 + 0x16]
     sp = 0x5cdbfd10    pc = 0x5398f86c
    Found by: stack scanning
 4  libmozglue.so!__gnu_ldivmod_helper [bpabi.c : 41 + 0x2]
     sp = 0x5cdbfd48    pc = 0x80c260a8
    Found by: stack scanning
 5  libmozglue.so!__gnu_ldivmod_helper [bpabi.c : 41 + 0x2]
     sp = 0x5cdbfd78    pc = 0x80c260a8
    Found by: stack scanning
 6  libxul.so!mozilla::MediaStreamGraphImpl::ProduceDataForStreamsBlockByBlock(unsigned int, long long, long long) [MediaStreamGraph.cpp:5de9e9db23c5 : 930 + 0x16]
     sp = 0x5cdbfda0    pc = 0x53996de0
    Found by: stack scanning
 7  libxul.so!mozilla::MediaStreamGraphImpl::RunThread() [MediaStreamGraph.cpp:5de9e9db23c5 : 1001 + 0x12]
     sp = 0x5cdbfde8    pc = 0x5399c464
    Found by: stack scanning
 8  libxul.so!mozilla::::MediaStreamGraphThreadRunnable::Run [MediaStreamGraph.cpp:5de9e9db23c5 : 1126 + 0x6]
     sp = 0x5cdbfe60    pc = 0x5399c724
    Found by: stack scanning
 9  libxul.so!mozilla::MediaStreamGraphImpl::RunThread() [MediaStreamGraph.cpp:5de9e9db23c5 : 1049 + 0x22]
     sp = 0x5cdbfe64    pc = 0x5399c718
    Found by: stack scanning
10  libxul.so!nsThread::ProcessNextEvent(bool, bool*) [nsThread.cpp:5de9e9db23c5 : 627 + 0xa]
     sp = 0x5cdbfe68    pc = 0x54224244
    Found by: stack scanning
11  libnss3.so!pt_AttachThread [ptthread.c : 283 + 0x12]
     sp = 0x5cdbfe88    pc = 0x50cf3208
    Found by: stack scanning
12  libnss3.so!pt_AttachThread [ptthread.c : 283 + 0x12]
     sp = 0x5cdbfea0    pc = 0x50cf3208
    Found by: stack scanning
13  libxul.so!NS_ProcessNextEvent(nsIThread*, bool) [nsThreadUtils.cpp:5de9e9db23c5 : 238 + 0x12]
     sp = 0x5cdbfea8    pc = 0x541f3374
    Found by: stack scanning
14  libxul.so!nsThread::ThreadFunc(void*) [nsThread.cpp:5de9e9db23c5 : 265 + 0xa]
     sp = 0x5cdbfeb8    pc = 0x54223ea4
    Found by: stack scanning
15  libnss3.so!_pt_root [ptthread.c : 191 + 0xa]
     sp = 0x5cdbfed8    pc = 0x50cf32e0
    Found by: stack scanning
16  libnss3.so!pt_AttachThread [ptthread.c : 283 + 0x12]
     sp = 0x5cdbfee4    pc = 0x50cf3208
    Found by: stack scanning
17  libc.so + 0x110a6
     sp = 0x5cdbfef0    pc = 0xafd110a8
    Found by: stack scanning
18  libc.so + 0x42326
     sp = 0x5cdbfef4    pc = 0xafd42328
    Found by: stack scanning
19  libc.so + 0x10c3a
     sp = 0x5cdbff00    pc = 0xafd10c3c
    Found by: stack scanning
Crash Signature: [@ jemalloc_crash] → [@ jemalloc_crash | arena_dalloc | mozilla::dom::ScriptProcessorNodeEngine::ProduceAudioBlock(mozilla::AudioNodeStream*, mozilla::AudioChunk const&, mozilla::AudioChunk*, bool*)]
Summary: Intermittent content/media/webspeech/recognition/test/test_abort.html | application crashed [@ mozilla::BlockingResourceBase::CheckAcquire(mozilla::CallStack const &)] → Intermittent content/media/webspeech/recognition/test/test_abort.html | application crashed [@ jemalloc_crash ]
Summary: Intermittent content/media/webspeech/recognition/test/test_abort.html | application crashed [@ jemalloc_crash ] → Intermittent content/media/webspeech/recognition/test/test_abort.html | application crashed [@ jemalloc_crash]
https://tbpl.mozilla.org/php/getParsedLog.php?id=22132030&tree=Mozilla-Central

PROCESS-CRASH | /tests/content/media/webspeech/recognition/test/test_abort.html | application crashed [@ PR_Lock]
Crash dump filename: /tmp/tmpgaXS35/235e99ae-1de6-e190-4789a59f-306360d9.dmp
Operating system: Android
                  0.0.0 Linux 3.2.0+ #2 SMP PREEMPT Thu Nov 29 08:06:57 EST 2012 armv7l pandaboard/pandaboard/pandaboard:4.0.4/IMM76I/5:eng/test-keys
CPU: arm
     2 CPUs

Crash reason:  SIGSEGV
Crash address: 0x3c

Thread 38 (crashed)
 0  libnss3.so!PR_Lock [ptsynch.c : 184 + 0x2]
     r4 = 0x00000000    r5 = 0x680d03c0    r6 = 0x6da56c98    r7 = 0x680d0384
     r8 = 0x67f6d000    r9 = 0x637d13e8   r10 = 0x00000002    fp = 0x00000001
     sp = 0x6da56c88    lr = 0x5f2dc1a5    pc = 0x5f2dc1a6
    Found by: given as instruction pointer in context
 1  libxul.so!mozilla::dom::ScriptProcessorNodeEngine::ProduceAudioBlock(mozilla::AudioNodeStream*, mozilla::AudioChunk const&, mozilla::AudioChunk*, bool*) [Mutex.h:acf388eaf9e9 : 74 + 0x7]
     sp = 0x6da56c90    pc = 0x625b98d1
    Found by: stack scanning
 2  libxul.so!mozilla::dom::AudioBufferSourceNodeEngine::ProduceAudioBlock(mozilla::AudioNodeStream*, mozilla::AudioChunk const&, mozilla::AudioChunk*, bool*) [AudioBufferSourceNode.cpp:acf388eaf9e9 : 383 + 0xf]
     sp = 0x6da56ca0    pc = 0x625b491d
    Found by: stack scanning
 3  libxul.so!mozilla::AudioNodeStream::EnsureTrack() [AudioNodeStream.cpp:acf388eaf9e9 : 166 + 0x7]
     sp = 0x6da56cd0    pc = 0x6259f3c7
    Found by: stack scanning
 4  libxul.so!mozilla::dom::ScriptProcessorNodeEngine::SendBuffersToMainThread(mozilla::AudioNodeStream*) [ScriptProcessorNode.cpp:acf388eaf9e9 : 323 + 0x9]
     sp = 0x6da56ce8    pc = 0x625b980d
    Found by: stack scanning
 5  libxul.so!mozilla::AudioNodeStream::ProduceOutput(long long, long long) [AudioNodeStream.cpp:acf388eaf9e9 : 295 + 0xb]
     sp = 0x6da56d00    pc = 0x6259f5d7
    Found by: stack scanning
 6  libxul.so!mozilla::MediaStreamGraphImpl::RecomputeBlockingAt(nsTArray<mozilla::MediaStream*> const&, long long, long long, long long*) [MediaStreamGraph.cpp:acf388eaf9e9 : 658 + 0xf]
     sp = 0x6da56d08    pc = 0x625a78fd
    Found by: stack scanning
 7  libmozglue.so!__gnu_ldivmod_helper [bpabi.c : 41 + 0x3]
     sp = 0x6da56d38    pc = 0x5bcb200f
    Found by: stack scanning
 8  libmozglue.so!__gnu_ldivmod_helper [bpabi.c : 41 + 0x3]
     sp = 0x6da56d68    pc = 0x5bcb200f
    Found by: stack scanning
Crash Signature: [@ jemalloc_crash | arena_dalloc | mozilla::dom::ScriptProcessorNodeEngine::ProduceAudioBlock(mozilla::AudioNodeStream*, mozilla::AudioChunk const&, mozilla::AudioChunk*, bool*)] → [@ jemalloc_crash | arena_dalloc | mozilla::dom::ScriptProcessorNodeEngine::ProduceAudioBlock(mozilla::AudioNodeStream*, mozilla::AudioChunk const&, mozilla::AudioChunk*, bool*)] [@ PR_Lock | mozilla::dom::ScriptProcessorNodeEngine::ProduceAudioBlock(moz…
Summary: Intermittent content/media/webspeech/recognition/test/test_abort.html | application crashed [@ jemalloc_crash] → Intermittent content/media/webspeech/recognition/test/test_abort.html | application crashed [@ jemalloc_crash] or [@ PR_Lock], both with mozilla::dom::ScriptProcessorNodeEngine::ProduceAudioBlock() further down stack
https://tbpl.mozilla.org/php/getParsedLog.php?id=22133558&tree=Mozilla-Inbound
Summary: Intermittent content/media/webspeech/recognition/test/test_abort.html | application crashed [@ jemalloc_crash] or [@ PR_Lock], both with mozilla::dom::ScriptProcessorNodeEngine::ProduceAudioBlock() further down stack → Intermittent content/media/webspeech/recognition/test/test_abort.html | application crashed [@ jemalloc_crash] or [@ PR_Lock] or [@ ntdll.dll + 0x4f592], both with mozilla::dom::ScriptProcessorNodeEngine::ProduceAudioBlock() further down stack
Summary: Intermittent content/media/webspeech/recognition/test/test_abort.html | application crashed [@ jemalloc_crash] or [@ PR_Lock] or [@ ntdll.dll + 0x4f592], both with mozilla::dom::ScriptProcessorNodeEngine::ProduceAudioBlock() further down stack → Intermittent webspeech/recognition/test/test_abort.html,test_audio_capture_error.html | application crashed [@ jemalloc_crash] or [@ PR_Lock] or [@ ntdll.dll + 0x4f592], both with mozilla::dom::ScriptProcessorNodeEngine::ProduceAudioBlock() further down
Whatever someone did to ScriptProcessorNodeEngine::ProduceAudioBlock, please back it out forthwith, before we ragekill all of your tests.
Blocks: 865081
(In reply to Phil Ringnalda (:philor) from comment #14)
> Whatever someone did to ScriptProcessorNodeEngine::ProduceAudioBlock, please
> back it out forthwith, before we ragekill all of your tests.

Ehsan just landed this class in bug 834513, it looks like.
Flags: needinfo?(ehsan)
Assignee: nobody → ehsan
So one threading race condition that I can see here is that when we call MediaStreamGraphImpl::ProduceDataForStreamsBlockByBlock, we do not hold a strong reference to the media stream, so we have no guarantee for the stream (and therefore the engine) to not die when we're in a call to ProduceAudioBlock, if the cycle collector frees the node object on the main thread.

The other race condition is in AudioNode::DestroyMediaStream, when we clear the node reference on the engine object.  Code such as ScriptProcessorNode::ProduceAudioBlock first checks mNode to not be null, and then goes ahead and accesses things off of it, such as mSharedBuffers (which is really a pointer to an object on the node.)  We probably need to add a lock for mNode accesses to the engine and hold it in ProdudeAudioBlock implementations which rely on being able to access mNode.
Blocks: webaudio
Flags: needinfo?(ehsan)
Attachment #741347 - Flags: review?(paul) → review+

        Attachment #741366 -
        Flags: review?(paul)

    
    

    
  
I believe that these two patches should fix the crashes we're seeing here, but it's best to leave the bug open to make sure.
Whiteboard: [leave open]

        Attachment #741366 -
        Flags: review?(paul) → review+
Blocks: 865004

    
    

    
      
  

    
    

    
      
  

    
    

    
  
Note that bug 865004 should also help in fixing these kinds of crashes.

    
    

    
  
I'll call this fixed.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED

    
  
Whiteboard: [leave open]

    
    

    
      
  

    
    

    
  
Mass moving Web Audio bugs to the Web Audio component.  Filter on duckityduck.
Component: Video/Audio → Web Audio
Depends on: 956611

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: