Intermittent gtest | application crashed [@ FrameListener::OnVideoFrameConverted(webrtc::VideoFrame const&)]
Categories
(Core :: WebRTC: Audio/Video, defect, P2)
Tracking
()
People
(Reporter: intermittent-bug-filer, Assigned: pehrsons)
References
Details
(Keywords: crash, csectype-uaf, intermittent-failure, Whiteboard: [test only])
Crash Data
Attachments
(3 files, 3 obsolete files)
Filed by: csabou [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=345772107&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/L_QI7PqoQN2KkiuzpYcj3Q/runs/0/artifacts/public/logs/live_backing.log
[task 2021-07-20T17:30:45.860Z] 17:30:45 INFO - TEST-PASS | libvpx.test_cases | test completed (time: 12ms)
[task 2021-07-20T17:30:45.860Z] 17:30:45 INFO - TEST-START | VideoFrameConverterTest.BasicConversion
[task 2021-07-20T17:30:45.861Z] 17:30:45 INFO - mozilla::detail::MutexImpl::~MutexImpl: pthread_mutex_destroy failed: Device or resource busy
[task 2021-07-20T17:30:45.863Z] 17:30:45 INFO - ExceptionHandler::GenerateDump cloned child ExceptionHandler::WaitForContinueSignal waiting for continue signal...
[task 2021-07-20T17:30:45.863Z] 17:30:45 INFO - 18071
[task 2021-07-20T17:30:45.864Z] 17:30:45 INFO - ExceptionHandler::SendContinueSignalToChild sent continue signal to child
[task 2021-07-20T17:30:45.984Z] 17:30:45 INFO - gtest INFO | gtest | process wait complete, returncode=-11
[task 2021-07-20T17:30:45.984Z] 17:30:45 INFO - mozcrash checking /builds/worker/workspace/build/tests/gtest for minidumps...
[task 2021-07-20T17:30:45.985Z] 17:30:45 INFO - mozcrash INFO | Downloading symbols from: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/M4TxeE4qSrGA4chclPSEcA/artifacts/public/build/target.crashreporter-symbols.zip
[task 2021-07-20T17:30:51.514Z] 17:30:51 INFO - mozcrash INFO | Copy/paste: /builds/worker/fetches/minidump_stackwalk/minidump_stackwalk /builds/worker/workspace/build/tests/gtest/47104658-7a67-3c86-1a9e-9d75707f5920.dmp /tmp/tmp9s9sbg6f
[task 2021-07-20T17:30:56.940Z] 17:30:56 INFO - mozcrash INFO | Saved minidump as /builds/worker/workspace/build/blobber_upload_dir/47104658-7a67-3c86-1a9e-9d75707f5920.dmp
[task 2021-07-20T17:30:56.941Z] 17:30:56 INFO - mozcrash INFO | Saved app info as /builds/worker/workspace/build/blobber_upload_dir/47104658-7a67-3c86-1a9e-9d75707f5920.extra
[task 2021-07-20T17:30:56.946Z] 17:30:56 WARNING - PROCESS-CRASH | gtest | application crashed [@ FrameListener::OnVideoFrameConverted(webrtc::VideoFrame const&)]
[task 2021-07-20T17:30:56.946Z] 17:30:56 INFO - Crash dump filename: /builds/worker/workspace/build/tests/gtest/47104658-7a67-3c86-1a9e-9d75707f5920.dmp
[task 2021-07-20T17:30:56.946Z] 17:30:56 INFO - Mozilla crash reason: MOZ_CRASH(mozilla::detail::MutexImpl::~MutexImpl: pthread_mutex_destroy failed)
[task 2021-07-20T17:30:56.946Z] 17:30:56 INFO - Operating system: Linux
[task 2021-07-20T17:30:56.947Z] 17:30:56 INFO - 0.0.0 Linux 4.4.0-1014-aws #14taskcluster1-Ubuntu SMP Tue Apr 3 10:27:00 UTC 2018 x86_64
[task 2021-07-20T17:30:56.947Z] 17:30:56 INFO - CPU: x86
[task 2021-07-20T17:30:56.947Z] 17:30:56 INFO - AuthenticAMD family 23 model 1 stepping 2
[task 2021-07-20T17:30:56.947Z] 17:30:56 INFO - 4 CPUs
[task 2021-07-20T17:30:56.947Z] 17:30:56 INFO - GPU: UNKNOWN
[task 2021-07-20T17:30:56.948Z] 17:30:56 INFO - Crash reason: SIGSEGV /SEGV_MAPERR
[task 2021-07-20T17:30:56.948Z] 17:30:56 INFO - Crash address: 0x0
[task 2021-07-20T17:30:56.948Z] 17:30:56 INFO - Process uptime: not available
[task 2021-07-20T17:30:56.948Z] 17:30:56 INFO - Thread 63 (crashed) 0 libxul.so!webrtc::VideoFrame::VideoFrame(webrtc::VideoFrame const&) [video_frame.cc:a8a4dfcadce5e10500f492b55421700d78e5b9a1 : 41 + 0x28]
[task 2021-07-20T17:30:56.948Z] 17:30:56 INFO - eip = 0xe9a5d2d8 esp = 0xe0a6cb70 ebp = 0xe0a6cb88 ebx = 0xf0d01000
[task 2021-07-20T17:30:56.948Z] 17:30:56 INFO - esi = 0xf86c15e0 edi = 0xe684f560 eax = 0xe5e5e5e5 ecx = 0x00000000
[task 2021-07-20T17:30:56.948Z] 17:30:56 INFO - edx = 0xe68e3040 efl = 0x00210282
[task 2021-07-20T17:30:56.948Z] 17:30:56 INFO - Found by: given as instruction pointer in context
[task 2021-07-20T17:30:56.948Z] 17:30:56 INFO - 1 libxul.so!FrameListener::OnVideoFrameConverted(webrtc::VideoFrame const&) [TestVideoFrameConverter.cpp:a8a4dfcadce5e10500f492b55421700d78e5b9a1 : 67 + 0x488]
[task 2021-07-20T17:30:56.948Z] 17:30:56 INFO - eip = 0xec6a2420 esp = 0xe0a6cb90 ebp = 0xe0a6cc28 ebx = 0xf0d01000
[task 2021-07-20T17:30:56.948Z] 17:30:56 INFO - esi = 0xf86c15e0 edi = 0xe684f560
[task 2021-07-20T17:30:56.948Z] 17:30:56 INFO - Found by: call frame info
task 2021-07-20T17:30:56.948Z] 17:30:56 INFO - 2 libxul.so!mozilla::VideoFrameConverter::SameFrameTick(nsITimer*, void*) [VideoFrameConverter.h:a8a4dfcadce5e10500f492b55421700d78e5b9a1 : 236 + 0x12]
[task 2021-07-20T17:30:56.949Z] 17:30:56 INFO - eip = 0xe73c74d3 esp = 0xe0a6cc30 ebp = 0xe0a6cc48 ebx = 0xf0d01000
[task 2021-07-20T17:30:56.949Z] 17:30:56 INFO - esi = 0x00000000 edi = 0x00000000
[task 2021-07-20T17:30:56.949Z] 17:30:56 INFO - Found by: call frame info
[task 2021-07-20T17:30:56.949Z] 17:30:56 INFO - 3 libxul.so!nsTimerImpl::Fire(int) [nsTimerImpl.cpp:a8a4dfcadce5e10500f492b55421700d78e5b9a1 : 618 + 0x17]
[task 2021-07-20T17:30:56.949Z] 17:30:56 INFO - eip = 0xecb8aa01 esp = 0xe0a6cc50 ebp = 0xe0a6ccf8 ebx = 0xf0d01000
[task 2021-07-20T17:30:56.949Z] 17:30:56 INFO - esi = 0x00000000 edi = 0xf987f100
[task 2021-07-20T17:30:56.949Z] 17:30:56 INFO - Found by: call frame info
[task 2021-07-20T17:30:56.949Z] 17:30:56 INFO - 4 libxul.so!nsTimerEvent::Run() [TimerThread.cpp:a8a4dfcadce5e10500f492b55421700d78e5b9a1 : 248 + 0x12]
[task 2021-07-20T17:30:56.949Z] 17:30:56 INFO - eip = 0xecb796b4 esp = 0xe0a6cd00 ebp = 0xe0a6cd68 ebx = 0xf0d01000
[task 2021-07-20T17:30:56.949Z] 17:30:56 INFO - esi = 0xf1060678 edi = 0xf1060678
[task 2021-07-20T17:30:56.950Z] 17:30:56 INFO - Found by: call frame info
[task 2021-07-20T17:30:56.950Z] 17:30:56 INFO - 5 libxul.so!mozilla::TaskQueue::Runner::Run() [TaskQueue.cpp:a8a4dfcadce5e10500f492b55421700d78e5b9a1 : 208 + 0x8]
[task 2021-07-20T17:30:56.950Z] 17:30:56 INFO - eip = 0xecb76f1c esp = 0xe0a6cd70 ebp = 0xe0a6cdc8 ebx = 0xf0d01000
[task 2021-07-20T17:30:56.950Z] 17:30:56 INFO - esi = 0xf1060678 edi = 0x00000000
[task 2021-07-20T17:30:56.950Z] 17:30:56 INFO - Found by: call frame info
[task 2021-07-20T17:30:56.950Z] 17:30:56 INFO - 6 libxul.so!nsThreadPool::Run() [nsThreadPool.cpp:a8a4dfcadce5e10500f492b55421700d78e5b9a1 : 303 + 0x8]
[task 2021-07-20T17:30:56.950Z] 17:30:56 INFO - eip = 0xecb81394 esp = 0xe0a6cdd0 ebp = 0xe0a6cf58 ebx = 0xf0d01000
[task 2021-07-20T17:30:56.950Z] 17:30:56 INFO - esi = 0xe6971380 edi = 0x00000000
[task 2021-07-20T17:30:56.950Z] 17:30:56 INFO - Found by: call frame info
[task 2021-07-20T17:30:56.951Z] 17:30:56 INFO - 7 libxul.so!nsThread::ProcessNextEvent(bool, bool*) [nsThread.cpp:a8a4dfcadce5e10500f492b55421700d78e5b9a1 : 1142 + 0x18]
[task 2021-07-20T17:30:56.951Z] 17:30:56 INFO - eip = 0xeb2cfdc4 esp = 0xe0a6cf60 ebp = 0xe0a6d158 ebx = 0xf0d01000
[task 2021-07-20T17:30:56.951Z] 17:30:56 INFO - esi = 0xe0a6d0d0 edi = 0xf9866040
[task 2021-07-20T17:30:56.951Z] 17:30:56 INFO - Found by: call frame info
[task 2021-07-20T17:30:56.951Z] 17:30:56 INFO - 8 libxul.so!mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) [MessagePump.cpp:a8a4dfcadce5e10500f492b55421700d78e5b9a1 : 300 + 0x30]
[task 2021-07-20T17:30:56.951Z] 17:30:56 INFO - eip = 0xecd4fc46 esp = 0xe0a6d160 ebp = 0xe0a6d1b8 ebx = 0xf0d01000
[task 2021-07-20T17:30:56.951Z] 17:30:56 INFO - esi = 0xe68e35b0 edi = 0xe0a6d220
[task 2021-07-20T17:30:56.951Z] 17:30:56 INFO - Found by: call frame info
[task 2021-07-20T17:30:56.951Z] 17:30:56 INFO - 9 libxul.so!MessageLoop::Run() [message_loop.cc:a8a4dfcadce5e10500f492b55421700d78e5b9a1 : 306 + 0xc]
[task 2021-07-20T17:30:56.951Z] 17:30:56 INFO - eip = 0xecd2880f esp = 0xe0a6d1c0 ebp = 0xe0a6d1f8 ebx = 0xf0d01000
[task 2021-07-20T17:30:56.952Z] 17:30:56 INFO - esi = 0xe0a6d1d8 edi = 0xe0a6d220
[task 2021-07-20T17:30:56.952Z] 17:30:56 INFO - Found by: call frame info
[task 2021-07-20T17:30:56.952Z] 17:30:56 INFO - 10 libxul.so!nsThread::ThreadFunc(void*) [nsThread.cpp:a8a4dfcadce5e10500f492b55421700d78e5b9a1 : 390 + 0x8]
[task 2021-07-20T17:30:56.952Z] 17:30:56 INFO - eip = 0xecb7ad8e esp = 0xe0a6d200 ebp = 0xe0a6d318 ebx = 0xf0d01000
[task 2021-07-20T17:30:56.952Z] 17:30:56 INFO - esi = 0xf9866040 edi = 0xe0a6d220
[task 2021-07-20T17:30:56.952Z] 17:30:56 INFO - Found by: call frame info
[task 2021-07-20T17:30:56.952Z] 17:30:56 INFO - 11 libnspr4.so!_pt_root [ptthread.c:a8a4dfcadce5e10500f492b55421700d78e5b9a1 : 201 + 0x9]
[task 2021-07-20T17:30:56.952Z] 17:30:56 INFO - eip = 0xf67e4892 esp = 0xe0a6d320 ebp = 0xe0a6d358 ebx = 0xf67fc000
[task 2021-07-20T17:30:56.952Z] 17:30:56 INFO - esi = 0xf49c5958 edi = 0xf985c340
[task 2021-07-20T17:30:56.952Z] 17:30:56 INFO - Found by: call frame info
[task 2021-07-20T17:30:56.953Z] 17:30:56 INFO - 12 libpthread.so.0 + 0x63bd
[task 2021-07-20T17:30:56.953Z] 17:30:56 INFO - eip = 0xf76a53bd esp = 0xe0a6d360 ebp = 0xe0a6d428 ebx = 0x00000000
[task 2021-07-20T17:30:56.953Z] 17:30:56 INFO - esi = 0xe0a6db40 edi = 0xf76bb000
[task 2021-07-20T17:30:56.953Z] 17:30:56 INFO - Found by: call frame info
[task 2021-07-20T17:30:56.953Z] 17:30:56 INFO - 13 libc.so.6 + 0xf8fe6
[task 2021-07-20T17:30:56.954Z] 17:30:56 INFO - eip = 0xf7307fe6 esp = 0xe0a6d430 ebp = 0x00000000
[task 2021-07-20T17:30:56.954Z] 17:30:56 INFO - Found by: previous frame's frame pointer
Updated•3 years ago
|
Comment 1•3 years ago
|
||
This is a gtest, so maybe sec-high is not accurate. The top frames of the stack don't look unlike what I'd expect from a browser stack, FWIW.
Assignee | ||
Comment 2•3 years ago
|
||
This looks to me like a UAF in VideoFrameConverterTest
showing in the mMonitor
member, i.e., test-only.
What I don't understand is how the crashing stack leads to ~MutexImpl()
. Assuming the nature of a UAF leads us there.
What I see as a plausible order of events is:
- The gtest runs to completion on main thread.
- The test fixture TearDown method runs on main thread,
VideoFrameConverter::Shutdown
dispatches a clean up task to its worker thread, holding a strong ref to the converter itself. - The test fixture is destroyed on main thread.
- On the worker thread, before the clean up task has run, the same frame timer ticks and runs its handler, calling into the test fixture's FrameListener.
mTest
is the destroyed test fixture. UAF.
The non-test equivalent to the raw ptr causing problems above is not as naive.
Assignee | ||
Updated•3 years ago
|
Assignee | ||
Comment 3•3 years ago
|
||
Assignee | ||
Comment 4•3 years ago
|
||
Assignee | ||
Comment 5•3 years ago
|
||
Assignee | ||
Comment 6•3 years ago
|
||
Comment on attachment 9247963 [details]
Bug 1721443 - Use higher order functions for getting frames in TestVideoFrameConverter. r?bwc
Security Approval Request
- How easily could an exploit be constructed based on the patch?: So this is a gtest-only bug. How does that affect the rating? Docs were not explicit on this.
- Do comments in the patch, the check-in comment, or tests included in the patch paint a bulls-eye on the security problem?: No
- Which older supported branches are affected by this flaw?: all
- If not all supported branches, which bug introduced the flaw?: None
- Do you have backports for the affected branches?: No
- If not, how different, hard to create, and risky will they be?: Older branches than 85 will require more backports. Easiest would be to skip esr78.
- How likely is this patch to cause regressions; how much testing does it need?: not likely
Assignee | ||
Updated•3 years ago
|
Comment 7•3 years ago
|
||
If this is test only, then it doesn't need to be a sec bug, and doesn't need sec-approval. Thanks for your caution, though.
Updated•3 years ago
|
Updated•3 years ago
|
Assignee | ||
Comment 8•3 years ago
|
||
Assignee | ||
Comment 9•3 years ago
|
||
Assignee | ||
Comment 10•3 years ago
|
||
Updated•3 years ago
|
Updated•3 years ago
|
Updated•3 years ago
|
Comment 11•3 years ago
|
||
Comment 12•3 years ago
|
||
Backed out for causing bustages on TestVideoFrameConverter.cpp. CLOSED TREE
Backout link : https://hg.mozilla.org/integration/autoland/rev/bdcaa953b2cb5bd5ea0e8b61f31cdf56541bff59
Link to failure log : https://treeherder.mozilla.org/logviewer?job_id=356346266&repo=autoland&lineNumber=33797
Assignee | ||
Updated•3 years ago
|
Comment 13•3 years ago
|
||
Comment 14•3 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/d70290bc3fb5
https://hg.mozilla.org/mozilla-central/rev/a3eea7436bc2
https://hg.mozilla.org/mozilla-central/rev/c38ef0933c9c
Updated•3 years ago
|
Description
•