* STR ** Enable MediaCodec by setting "media.omx.async.enabled" to true ** open video app and play the video clip in the attachment Video app will crash and here is the back trace. That's because mCodec in frame 5(MediaCodecProxy.cpp:337) was released by binder thread. #0 pthread_mutex_lock_impl (mutex=mutex@entry=0x24) at bionic/libc/bionic/pthread.c:588 #1 0xb6eec938 in pthread_mutex_lock (mutex=mutex@entry=0x24) at bionic/libc/bionic/pthread.c:672 #2 0xb6f019ec in pthread_rwlock_timedrdlock (rwlock=0x24, abs_timeout=0x0) at bionic/libc/bionic/pthread-rwlocks.c:228 #3 0xb55dc2ee in android::RWLock::readLock (this=this@entry=0x24) at ../../../../../flame-kk/system/core/include/utils/RWLock.h:105 #4 0xb55dc7a2 in AutoRLock (rwlock=..., this=0xb2339a44) at ../../../../../flame-kk/system/core/include/utils/RWLock.h:62 #5 android::MediaCodecProxy::dequeueOutputBuffer (this=0x0, aIndex=aIndex@entry=0xb2339a78, aOffset=aOffset@entry=0xb2339a7c, aSize=aSize@entry=0xb2339a80, aPresentationTimeUs=aPresentationTimeUs@entry=0xb2339a88, aFlags=aFlags@entry=0xb2339a84, aTimeoutUs=0) at ../../../../../gecko-dev/dom/media/omx/MediaCodecProxy.cpp:337 #6 0xb55de7b0 in EnsureCodecFormatParsed (aTrack=..., this=0xb1f1f800) at ../../../../../gecko-dev/dom/media/omx/MediaCodecReader.cpp:1903 #7 mozilla::MediaCodecReader::EnsureCodecFormatParsed (this=0xb1f1f800, aTrack=...) at ../../../../../gecko-dev/dom/media/omx/MediaCodecReader.cpp:1885 #8 0xb55de848 in UpdateVideoInfo (this=0xb1f1f800) at ../../../../../gecko-dev/dom/media/omx/MediaCodecReader.cpp:1626 #9 mozilla::MediaCodecReader::UpdateVideoInfo (this=0xb1f1f800) at ../../../../../gecko-dev/dom/media/omx/MediaCodecReader.cpp:1592 #10 0xb55e02b6 in ReadMetadata (aTags=0xb1f21008, aInfo=0xb2339bac, this=0xb1f1f800) at ../../../../../gecko-dev/dom/media/omx/MediaCodecReader.cpp:718 #11 mozilla::MediaCodecReader::ReadMetadata (this=0xb1f1f800, aInfo=0xb2339bac, aTags=0xb1f21008) at ../../../../../gecko-dev/dom/media/omx/MediaCodecReader.cpp:684 #12 0xb557a6ba in mozilla::MediaDecoderStateMachine::DecodeMetadata (this=this@entry=0xb1f21000) at ../../../../gecko-dev/dom/media/MediaDecoderStateMachine.cpp:2210 #13 0xb557a84a in mozilla::MediaDecoderStateMachine::CallDecodeMetadata (this=0xb1f21000) at ../../../../gecko-dev/dom/media/MediaDecoderStateMachine.cpp:2186 #14 0xb4d1af1c in nsRunnableMethodImpl<nsresult (nsMemoryReporterManager::*)(), void, true>::Run (this=<optimized out>) at ../../dist/include/nsThreadUtils.h:386 #15 0xb557f724 in mozilla::MediaTaskQueue::Runner::Run (this=0xb1fc8530) at ../../../../gecko-dev/dom/media/MediaTaskQueue.cpp:230 #16 0xb4d42ed6 in nsThreadPool::Run (this=0xb1fc96f0) at ../../../../gecko-dev/xpcom/threads/nsThreadPool.cpp:225 #17 0xb4d40f88 in nsThread::ProcessNextEvent (this=0xb1f44970, aMayWait=<optimized out>, aResult=0xb2339d27) at ../../../../gecko-dev/xpcom/threads/nsThread.cpp:855 #18 0xb4d4e314 in NS_ProcessNextEvent (aThread=<optimized out>, aMayWait=aMayWait@entry=false) at /Volumes/ssd/gecko-dev/xpcom/glue/nsThreadUtils.cpp:265 #19 0xb4e8e81c in mozilla::ipc::MessagePumpForNonMainThreads::Run (this=0xb1f46760, aDelegate=0xb2db4980) at ../../../../gecko-dev/ipc/glue/MessagePump.cpp:339 #20 0xb4e82ed4 in MessageLoop::RunInternal (this=this@entry=0xb2db4980) at ../../../../gecko-dev/ipc/chromium/src/base/message_loop.cc:233 #21 0xb4e82f88 in RunHandler (this=0xb2db4980) at ../../../../gecko-dev/ipc/chromium/src/base/message_loop.cc:226 #22 MessageLoop::Run (this=this@entry=0xb2db4980) at ../../../../gecko-dev/ipc/chromium/src/base/message_loop.cc:200 #23 0xb4d412e6 in nsThread::ThreadFunc (aArg=0xb1f44970) at ../../../../gecko-dev/xpcom/threads/nsThread.cpp:356 #24 0xb69bba72 in _pt_root (arg=0xb28cff00) at ../../../../../../gecko-dev/nsprpub/pr/src/pthreads/ptthread.c:212 #25 0xb6eeb22c in __thread_entry (func=0xb69bb9d9 <_pt_root>, arg=0xb28cff00, tls=0xb2339dd0) at bionic/libc/bionic/pthread_create.cpp:105 #26 0xb6eeb3c4 in pthread_create (thread_out=0xb237bac4, attr=<optimized out>, start_routine=0xb69bb9d9 <_pt_root>, arg=0x78) at bionic/libc/bionic/pthread_create.cpp:224 #27 0x00000000 in ?? ()

The crash root cause is that when the codecReserved is called after video hw instance is created on binder thread, we try to configure the video hw codec by ConfigureMediaCodec(). In this case, the configuration failed then we call DestroyMediaCodec() directly. At then same time, the other thread Decoder thread is in a while loop EnsureCodecFormatParsed() access the codec we just destroyed.

In this patch, I made some code flow, threading change. When ConfigureMediaCodec failed, we need to call DestroyMediaCodec to release the HW resource. Before this change, these two functions are invoked on binder thread, and there is a deadlock issue at MediaCodecProxy::~MediaCodecProxy()::releaseCodec(); (Not figure out the root cause of deadlock). So I move ConfigureMediaCodec and DestroyMediaCodec to our own TaskQueue/DecodeTaskQueue. Then, since the ConfigureMediaCodec must be called after codecReserved, that means I need to refact the MediaCodecReader::ReadMetadata(), make it always in the waiting_resource state at the first ReadMetadata(). This patch works for the crash issue, but it reveals another bug, the dormant and the ReadMedata in MDSM. We release the lock the call mReader->ReadMetadata https://dxr.mozilla.org/mozilla-central/source/dom/media/MediaDecoderStateMachine.cpp#2169 Then the document's activity change https://dxr.mozilla.org/mozilla-central/source/dom/media/MediaDecoderStateMachine.cpp#1456 => MDSM state: Metadata -> dormant -> None (but the decoder thread is still in MDSM::DecodeMetadata function)

1. Move ConfigureMediaCodec from binder thread to decode tread to fix crash problem. 2. Remove MediaCodecReader::MessageHandler class because we don't need it for now.

try server: https://treeherder.mozilla.org/#/jobs?repo=try&revision=88f8eee0d9f9

r=sotaro