Need to load this testcase over http to see instant crash. I think your libvpx copy is old, it is best to update it since we have found several others that were fixed recently. ==15811==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x61f00006e29c at pc 0x7f617614b7d9 bp 0x7f614e38ab30 sp 0x7f614e38ab28 WRITE of size 4 at 0x61f00006e29c thread T22 (MediaPl~back #2) #0 0x7f617614b7d8 in resize_context_buffers media/libvpx/vp9/decoder/vp9_decodeframe.c:681:26 #1 0x7f617614ae89 in setup_frame_size media/libvpx/vp9/decoder/vp9_decodeframe.c:721:3 #2 0x7f61761402e7 in vp9_decode_frame media/libvpx/vp9/decoder/vp9_decodeframe.c:1368:5 #3 0x7f617615e171 in vp9_receive_compressed_data media/libvpx/vp9/decoder/vp9_decoder.c:356:3 #4 0x7f61762ec37b in frame_worker_hook media/libvpx/vp9/vp9_dx_iface.c:322:7 #5 0x7f61760faca6 in execute media/libvpx/vp9/common/vp9_thread.c:134:27 #6 0x7f61762eba0d in decode_one media/libvpx/vp9/vp9_dx_iface.c:493:5 #7 0x7f61762e5437 in decoder_decode media/libvpx/vp9/vp9_dx_iface.c:686:37 #8 0x7f61762ed2f7 in vpx_codec_decode media/libvpx/vpx/src/vpx_decoder.c:122:11 #9 0x7f6173224e5f in mozilla::SoftwareWebMVideoDecoder::DecodeVideoFrame(bool&, long) dom/media/webm/SoftwareWebMVideoDecoder.cpp:149:7 #10 0x7f6172dedc19 in mozilla::MediaDecoderReader::RequestVideoData(bool, long) dom/media/MediaDecoderReader.cpp:277:10 #11 0x7f6172ee1a6d in mozilla::detail::MethodCallWithTwoArgs<mozilla::MediaPromise<nsRefPtr<mozilla::VideoData>, mozilla::MediaDecoderReader::NotDecodedReason, true>, mozilla::MediaDecoderReader, bool, long>::Invoke() objdir-ff-asan/dist/include/MediaPromise.h:902:52 #12 0x7f6172ee1d83 in mozilla::detail::ProxyRunnable<mozilla::MediaPromise<nsRefPtr<mozilla::VideoData>, mozilla::MediaDecoderReader::NotDecodedReason, true> >::Run() objdir-ff-asan/dist/include/MediaPromise.h:919:31 #13 0x7f6172d9d12c in mozilla::AutoTaskDispatcher::TaskGroupRunnable::Run() dom/media/TaskDispatcher.h:181:11 #14 0x7f6172f295b9 in mozilla::MediaTaskQueue::Runner::Run() dom/media/MediaTaskQueue.cpp:256:5 #15 0x7f616e4c353e in nsThreadPool::Run() xpcom/threads/nsThreadPool.cpp:221:7 #16 0x7f616e4c3b1c in non-virtual thunk to nsThreadPool::Run() xpcom/threads/nsThreadPool.cpp:151:15 #17 0x7f616e4bd1b6 in nsThread::ProcessNextEvent(bool, bool*) xpcom/threads/nsThread.cpp:848:7 #18 0x7f616e53419c in NS_ProcessNextEvent(nsIThread*, bool) xpcom/glue/nsThreadUtils.cpp:265:10 #19 0x7f616ede4906 in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) ipc/glue/MessagePump.cpp:326:20 #20 0x7f616ed6e6a1 in MessageLoop::Run() ipc/chromium/src/base/message_loop.cc:234:3 #21 0x7f616e4b9c51 in nsThread::ThreadFunc(void*) xpcom/threads/nsThread.cpp:360:5 #22 0x7f617bca0ffa in _pt_root nsprpub/pr/src/pthreads/ptthread.c:212:5 #23 0x7f617c2e5181 in start_thread /build/buildd/eglibc-2.19/nptl/pthread_create.c:312 0x61f00006e29f is located 0 bytes to the right of 3103-byte region [0x61f00006d680,0x61f00006e29f) allocated by thread T24 (MediaPl~back #3) here: #0 0x4b6338 in __interceptor_malloc _asan_rtl_ #1 0x7f61762ef186 in vpx_calloc media/libvpx/vpx_mem/vpx_mem.c:126:10 #2 0x7f61762e446a in decoder_decode media/libvpx/vp9/vp9_dx_iface.c:372:36 #3 0x7f61762ed2f7 in vpx_codec_decode media/libvpx/vpx/src/vpx_decoder.c:122:11 #4 0x7f6173224e5f in mozilla::SoftwareWebMVideoDecoder::DecodeVideoFrame(bool&, long) dom/media/webm/SoftwareWebMVideoDecoder.cpp:149:7 #5 0x7f6172dedc19 in mozilla::MediaDecoderReader::RequestVideoData(bool, long) dom/media/MediaDecoderReader.cpp:277:10 #6 0x7f6172ee1a6d in mozilla::detail::MethodCallWithTwoArgs<mozilla::MediaPromise<nsRefPtr<mozilla::VideoData>, mozilla::MediaDecoderReader::NotDecodedReason, true>, mozilla::MediaDecoderReader, bool, long>::Invoke() objdir-ff-asan/dist/include/MediaPromise.h:902:52 #7 0x7f6172ee1d83 in mozilla::detail::ProxyRunnable<mozilla::MediaPromise<nsRefPtr<mozilla::VideoData>, mozilla::MediaDecoderReader::NotDecodedReason, true> >::Run() objdir-ff-asan/dist/include/MediaPromise.h:919:31 #8 0x7f6172d9d12c in mozilla::AutoTaskDispatcher::TaskGroupRunnable::Run() dom/media/TaskDispatcher.h:181:11 #9 0x7f6172f295b9 in mozilla::MediaTaskQueue::Runner::Run() dom/media/MediaTaskQueue.cpp:256:5 #10 0x7f616e4c353e in nsThreadPool::Run() xpcom/threads/nsThreadPool.cpp:221:7 #11 0x7f616e4c3b1c in non-virtual thunk to nsThreadPool::Run() xpcom/threads/nsThreadPool.cpp:151:15 #12 0x7f616e4bd1b6 in nsThread::ProcessNextEvent(bool, bool*) xpcom/threads/nsThread.cpp:848:7 #13 0x7f616e53419c in NS_ProcessNextEvent(nsIThread*, bool) xpcom/glue/nsThreadUtils.cpp:265:10 #14 0x7f616ede4a82 in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) ipc/glue/MessagePump.cpp:355:5 #15 0x7f616ed6e6a1 in MessageLoop::Run() ipc/chromium/src/base/message_loop.cc:234:3 #16 0x7f616e4b9c51 in nsThread::ThreadFunc(void*) xpcom/threads/nsThread.cpp:360:5 #17 0x7f617bca0ffa in _pt_root nsprpub/pr/src/pthreads/ptthread.c:212:5 #18 0x7f617c2e5181 in start_thread /build/buildd/eglibc-2.19/nptl/pthread_create.c:312 Thread T22 (MediaPl~back #2) created by T0 (Web Content) here: #0 0x430269 in pthread_create _asan_rtl_ #1 0x7f617bc9ddbf in _PR_CreateThread nsprpub/pr/src/pthreads/ptthread.c:453:14 #2 0x7f617bc9d9ea in PR_CreateThread nsprpub/pr/src/pthreads/ptthread.c:544:12 #3 0x7f616e4bb016 in nsThread::Init() xpcom/threads/nsThread.cpp:470:19 #4 0x7f616e4c0d1f in nsThreadManager::NewThread(unsigned int, unsigned int, nsIThread**) xpcom/threads/nsThreadManager.cpp:253:17 #5 0x7f616e4c248d in nsThreadPool::PutEvent(nsIRunnable*) xpcom/threads/nsThreadPool.cpp:102:3 #6 0x7f616e4c3f8a in nsThreadPool::Dispatch(nsIRunnable*, unsigned int) xpcom/threads/nsThreadPool.cpp:262:5 #7 0x7f6172f27dce in mozilla::MediaTaskQueue::DispatchLocked(already_AddRefed<nsIRunnable>, mozilla::MediaTaskQueue::DispatchMode, mozilla::AbstractThread::DispatchFailureHandling, mozilla::AbstractThread::DispatchReason) dom/media/MediaTaskQueue.cpp:65:17 #8 0x7f6172e8b4ff in mozilla::MediaTaskQueue::Dispatch(already_AddRefed<nsIRunnable>, mozilla::AbstractThread::DispatchFailureHandling, mozilla::AbstractThread::DispatchReason) objdir-ff-asan/dist/include/MediaTaskQueue.h:52:19 #9 0x7f6172d9bcc8 in mozilla::AutoTaskDispatcher::~AutoTaskDispatcher() dom/media/TaskDispatcher.h:233:5 #10 0x7f6172d9b8d7 in mozilla::XPCOMThreadWrapper::FireTailDispatcher() objdir-ff-asan/dist/include/mozilla/Maybe.h:373:7 #11 0x7f6172d9df20 in nsRunnableMethodImpl<void (mozilla::XPCOMThreadWrapper::*)(), true>::Run() objdir-ff-asan/dist/include/nsThreadUtils.h:618:5 #12 0x7f6173e669c1 in nsBaseAppShell::RunSyncSectionsInternal(bool, unsigned int) widget/nsBaseAppShell.cpp:376:7 #13 0x7f6173e67c9e in non-virtual thunk to nsBaseAppShell::AfterProcessNextEvent(nsIThreadInternal*, unsigned int, bool) widget/nsBaseAppShell.h:95:7 #14 0x7f616e4bd613 in nsThread::ProcessNextEvent(bool, bool*) xpcom/threads/nsThread.cpp:862:5 #15 0x7f616e53419c in NS_ProcessNextEvent(nsIThread*, bool) xpcom/glue/nsThreadUtils.cpp:265:10 #16 0x7f616ede3b2e in mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) ipc/glue/MessagePump.cpp:95:21 #17 0x7f616ed6e6a1 in MessageLoop::Run() ipc/chromium/src/base/message_loop.cc:234:3 #18 0x7f6173e6524f in nsBaseAppShell::Run() widget/nsBaseAppShell.cpp:165:3 #19 0x7f6175d451f3 in XRE_RunAppShell toolkit/xre/nsEmbedFunctions.cpp:778:12 #20 0x7f616ed6e6a1 in MessageLoop::Run() ipc/chromium/src/base/message_loop.cc:234:3 #21 0x7f6175d44717 in XRE_InitChildProcess toolkit/xre/nsEmbedFunctions.cpp:614:7 #22 0x4dbbf2 in content_process_main(int, char**) ipc/contentproc/plugin-container.cpp:236:19 #23 0x7f616b8c2ec4 in __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287 Thread T24 (MediaPl~back #3) created by T22 (MediaPl~back #2) here: #0 0x430269 in pthread_create _asan_rtl_ #1 0x7f617bc9ddbf in _PR_CreateThread nsprpub/pr/src/pthreads/ptthread.c:453:14 #2 0x7f617bc9d9ea in PR_CreateThread nsprpub/pr/src/pthreads/ptthread.c:544:12 #3 0x7f616e4bb016 in nsThread::Init() xpcom/threads/nsThread.cpp:470:19 #4 0x7f616e4c0d1f in nsThreadManager::NewThread(unsigned int, unsigned int, nsIThread**) xpcom/threads/nsThreadManager.cpp:253:17 #5 0x7f616e4c248d in nsThreadPool::PutEvent(nsIRunnable*) xpcom/threads/nsThreadPool.cpp:102:3 #6 0x7f616e4c3f8a in nsThreadPool::Dispatch(nsIRunnable*, unsigned int) xpcom/threads/nsThreadPool.cpp:262:5 #7 0x7f6172f27dce in mozilla::MediaTaskQueue::DispatchLocked(already_AddRefed<nsIRunnable>, mozilla::MediaTaskQueue::DispatchMode, mozilla::AbstractThread::DispatchFailureHandling, mozilla::AbstractThread::DispatchReason) dom/media/MediaTaskQueue.cpp:65:17 #8 0x7f6172e8b4ff in mozilla::MediaTaskQueue::Dispatch(already_AddRefed<nsIRunnable>, mozilla::AbstractThread::DispatchFailureHandling, mozilla::AbstractThread::DispatchReason) objdir-ff-asan/dist/include/MediaTaskQueue.h:52:19 #9 0x7f6172d9bcc8 in mozilla::AutoTaskDispatcher::~AutoTaskDispatcher() dom/media/TaskDispatcher.h:233:5 #10 0x7f6172f29980 in mozilla::MediaTaskQueue::Runner::Run() objdir-ff-asan/dist/include/MediaTaskQueue.h:152:5 #11 0x7f616e4c353e in nsThreadPool::Run() xpcom/threads/nsThreadPool.cpp:221:7 #12 0x7f616e4c3b1c in non-virtual thunk to nsThreadPool::Run() xpcom/threads/nsThreadPool.cpp:151:15 #13 0x7f616e4bd1b6 in nsThread::ProcessNextEvent(bool, bool*) xpcom/threads/nsThread.cpp:848:7 #14 0x7f616e53419c in NS_ProcessNextEvent(nsIThread*, bool) xpcom/glue/nsThreadUtils.cpp:265:10 #15 0x7f616ede4906 in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) ipc/glue/MessagePump.cpp:326:20 #16 0x7f616ed6e6a1 in MessageLoop::Run() ipc/chromium/src/base/message_loop.cc:234:3 #17 0x7f616e4b9c51 in nsThread::ThreadFunc(void*) xpcom/threads/nsThread.cpp:360:5 #18 0x7f617bca0ffa in _pt_root nsprpub/pr/src/pthreads/ptthread.c:212:5 #19 0x7f617c2e5181 in start_thread /build/buildd/eglibc-2.19/nptl/pthread_create.c:312 Shadow bytes around the buggy address: 0x0c3e80005c00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c3e80005c10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c3e80005c20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c3e80005c30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c3e80005c40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x0c3e80005c50: 00 00 00[07]fa fa fa fa fa fa fa fa fa fa fa fa 0x0c3e80005c60: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c3e80005c70: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c3e80005c80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c3e80005c90: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c3e80005ca0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Heap right redzone: fb Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack partial redzone: f4 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==15811==ABORTING

(In reply to Abhishek Arya from comment #0) > Need to load this testcase over http to see instant crash. I think your > libvpx copy is old, it is best to update it since we have found several > others that were fixed recently. The work to update libvpx to v1.4 is going on in bug 1178215. Is that recent enough, or have you found and fixed crashers like this even more recently than that?

(In reply to Timothy B. Terriberry (:derf) from comment #1) > (In reply to Abhishek Arya from comment #0) > > Need to load this testcase over http to see instant crash. I think your > > libvpx copy is old, it is best to update it since we have found several > > others that were fixed recently. > > The work to update libvpx to v1.4 is going on in bug 1178215. Is that recent > enough, or have you found and fixed crashers like this even more recently > than that? Sorry, I don't know if libvpx 1.4 is recent enough. Please try these repros in an addresssanitizer build before and after the update. these should reproduce easily.

Nightly has been on 1.4.0 for almost two weeks, since bug 1151175 landed. Bug 1178215 is an update to a post-1.4.0 snapshot.

Abhishek: we landed the libvpx snapshot today, can you still reproduce this in tomorrow's (7/2) nightly?

(In reply to Daniel Veditz [:dveditz] from comment #4) > Abhishek: we landed the libvpx snapshot today, can you still reproduce this > in tomorrow's (7/2) nightly? Latest snapshot update (from Bug 1178215) has fixed this. It no longer reproduces.

rillian/derf, can we close this then?

I think we can, but I'll have to rely on rillian to mark affected versions (if any).

Are we going to take this snapshot on older releases or is this sec-critical going to ride the trains?

I agree we should uplift the libvpx update. Chrome is shipping it too, so it should be safe enough. I'll nominate as soon as I can. Do I need sec-approval for that? Seems less disclosing to just request uplift the upstream update bugs, but how do I signal the sec-critical dependency in the public request? Is referencing this opaque bug sufficient?

I wouldn't reference this bug as that's a red flag. I'd just say "We need to take this for security and stability" and if release management has email, you can tell them. Heck, needinfo? mean when you put the uplift requests in and I'll approve it.

Great, thanks.

Uplift of bug 1178215 to aurora and beta should address this issue for those branches. ESR38 may not be vulnerable. I uses a different libvpx snapshot with substantially different code. ESR31 is different again, but less so. I've been unable to do an asan build of the esr38 branch to verify. Abhishek, are you able to reproduce with an esr38 build?

I was able to get an asan build from try. I cannot reproduce the overflow report there. https://treeherder.mozilla.org/#/jobs?repo=try&revision=d3a81aeeaa17

I cannot reproduce with an asan build of esr31 either. https://treeherder.mozilla.org/#/jobs?repo=try&revision=1a19765b7b27

While going through verification with fx42, I ran into the following crash which I can reproduce pretty reliably using the STR mentioned below. This appears to be a shutdown issue and not related to this particular problem but just making sure. Andrew, might taking a quick look? Does this warrant a new bug? Used the following m-c asan build: - http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64-asan/1437991336/ STR: - open test.html from comment #0 via HTTP 5 times via a e10s window (leave opened) - open test.html from comment #0 via HTTP 5 times via a non-e10s window - select the hamburger menu and quit fx, wait a minute or so and you'll get the crash in the terminal ASAN:SIGSEGV ================================================================= ==4657==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7fbc38b1dae5 sp 0x7fbbe2650e20 bp 0x7fbbe2650e30 T76) #0 0x7fbc38b1dae4 in RunWatchdog nsTerminator.cpp:151 #1 0x7fbc3ff764b5 in _pt_root ptthread.c:212 #2 0x7fbc4349e181 in start_thread pthread_create.c:312 (discriminator 2) #3 0x7fbc4259f47c in clone clone.S:111 AddressSanitizer can not provide additional info. SUMMARY: AddressSanitizer: SEGV ??:0 ?? Thread T76 (Shutdow~minator) created by T0 here: #0 0x45eae5 in __interceptor_pthread_create _asan_rtl_ #1 0x7fbc3ff72e3d in _PR_CreateThread ptthread.c:453 #2 0x7fbc3ff729ba in PR_CreateThread ptthread.c:544 #3 0x7fbc38b1d324 in CreateSystemThread nsTerminator.cpp:69 #4 0x7fbc38b1e193 in Observe nsTerminator.cpp:439 #5 0x7fbc3166ea4c in NotifyObservers nsObserverList.cpp:113 #6 0x7fbc38a5c567 in Quit nsAppStartup.cpp:458 #7 0x7fbc38a5e4ae in ExitLastWindowClosingSurvivalArea nsAppStartup.cpp:532 #8 0x7fbc38a5e57c in non-virtual thunk to nsAppStartup::Observe(nsISupports*, char const*, char16_t const*) Unified_cpp_components_startup0.cpp:738 #9 0x7fbc3166ea4c in NotifyObservers nsObserverList.cpp:113 #10 0x7fbc382fbd42 in Destroy nsXULWindow.cpp:518 #11 0x7fbc382d275b in Destroy nsWebShellWindow.cpp:779 #12 0x7fbc33796b5e in ReallyCloseWindow nsGlobalWindow.cpp:8897 #13 0x7fbc337da618 in Run nsGlobalWindow.cpp:8671 #14 0x7fbc317242e7 in ProcessNextEvent nsThread.cpp:867 #15 0x7fbc3179301a in NS_ProcessNextEvent nsThreadUtils.cpp:277 #16 0x7fbc31fffe99 in Run MessagePump.cpp:95 #17 0x7fbc31f8cd0c in RunInternal message_loop.cc:234 #18 0x7fbc36cf7427 in Run nsBaseAppShell.cpp:165 #19 0x7fbc38a5b5c8 in Run nsAppStartup.cpp:280 #20 0x7fbc38b648e7 in XRE_mainRun nsAppRunner.cpp:4288 #21 0x7fbc38b65945 in XRE_main nsAppRunner.cpp:4385 #22 0x7fbc38b667c5 in XRE_main nsAppRunner.cpp:4474 #23 0x48a6e4 in do_main nsBrowserApp.cpp:212 #24 0x7fbc424c6ec4 in __libc_start_main libc-start.c:287 ==4657==ABORTING

(In reply to Kamil Jozwiak [:kjozwiak] from comment #15) > Andrew, might taking a quick look? Does this warrant a new bug? That's not really a bug. We intentionally crash when shutdown takes too long, and ASan shutdown is slow. If you run into this frequently, we'll want to file a bug about disabling the shutdown terminator.

Thanks Andrew. Reproduced the original issue using the following m-c build: - http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64-asan/1435355074/ Placed the files from webmt.zip (comment #0) on my personal server and loaded "test.html" via HTTP which reproduced the crash instantly in both non-e10s and e10s. Went through verification using the following builds: - http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64-asan/1437991336/ - http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-aurora-linux64-asan/1438018185/ - http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-beta-linux64-asan/1438018542/ Test Cases Used: - opened test.html via HTTP several times in new windows via non-e10s and e10s - opened test.html via HTTP several times in new tabs via non-e10s and e10s - opened test.html via HTTP several times in Private Browsing via non-e10s and e10s