Closed Bug 1806224 Opened 2 years ago Closed 2 years ago

hanging at shutdown; attempting crash report (fatal error)

Categories

(Core :: IPC, defect)

defect

Tracking

()

RESOLVED FIXED
110 Branch
Tracking Status
firefox-esr102 --- unaffected
firefox108 --- unaffected
firefox109 --- unaffected
firefox110 --- fixed

People

(Reporter: tsmith, Assigned: jld)

References

(Blocks 1 open bug, Regression)

Details

(Keywords: regression)

Attachments

(1 file)

First found while fuzzing 20221214-7ff758e0d08b (--enable-address-sanitizer --enable-fuzzing)

There seems to be a large spike in SIGABRT reports in the last few days. I don't have a reliable reduced test case at the moment but I did managed to get a Pernosco session: https://pernos.co/debug/TUxtQ8Y5KTB_yLD_xQgPjQ/index.html

stderr from the reports all contain:

[Parent 473, IPC I/O Parent] WARNING: Process 654 may be hanging at shutdown; will wait for up to 8000ms: file /builds/worker/checkouts/gecko/ipc/chromium/src/chrome/common/process_watcher_posix_sigchld.cc:175
[Parent 473, IPC I/O Parent] WARNING: Process 654 hanging at shutdown; attempting crash report (fatal error).: file /builds/worker/checkouts/gecko/ipc/chromium/src/chrome/common/process_watcher_posix_sigchld.cc:198
AddressSanitizer:DEADLYSIGNAL
[Parent 473, IPC I/O Parent] WARNING: process 654 exited with status 77: file /builds/worker/checkouts/gecko/ipc/chromium/src/base/process_util_posix.cc:262

The stacks are mostly variations of:

==377148==ERROR: AddressSanitizer: ABRT on unknown address 0x03e80005c06c (pc 0x55d40e820177 bp 0x6240000fc000 sp 0x7ffc595583c8 T0)
    #0 0x55d40e820177 in __sanitizer::internal_munmap(void*, unsigned long) /builds/worker/fetches/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_linux.cpp:219:3
    #1 0x55d40e821b0b in __sanitizer::UnmapOrDie(void*, unsigned long) /builds/worker/fetches/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_posix.cpp:59:14
    #2 0x55d40e78889c in __sanitizer::Quarantine<__asan::QuarantineCallback, __asan::AsanChunk>::DoRecycle(__sanitizer::QuarantineCache<__asan::QuarantineCallback>*, __asan::QuarantineCallback) /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_quarantine.h:193:12
    #3 0x55d40e788474 in __sanitizer::Quarantine<__asan::QuarantineCallback, __asan::AsanChunk>::Recycle(unsigned long, __asan::QuarantineCallback) /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_quarantine.h:181:5
    #4 0x55d40e78a667 in Put /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_quarantine.h:112:7
    #5 0x55d40e78a667 in __asan::Allocator::QuarantineChunk(__asan::AsanChunk*, void*, __sanitizer::BufferedStackTrace*) /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/asan_allocator.cpp:654:18
    #6 0x55d40e807666 in free /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:53:3
    #7 0x7f029bd81929 in finalize /gecko/js/src/vm/JSObject-inl.h:99:12
    #8 0x7f029bd81929 in unsigned long js::gc::Arena::finalize<JSObject>(JS::GCContext*, js::gc::AllocKind, unsigned long) /gecko/js/src/gc/Sweeping.cpp:132:10
    #9 0x7f029bd81152 in bool FinalizeTypedArenas<JSObject>(JS::GCContext*, js::gc::ArenaList&, js::gc::SortedArenaList&, js::gc::AllocKind, js::SliceBudget&) /gecko/js/src/gc/Sweeping.cpp:204:29
    #10 0x7f029bd60489 in js::gc::GCRuntime::backgroundFinalize(JS::GCContext*, JS::Zone*, js::gc::AllocKind, js::gc::Arena**) /gecko/js/src/gc/Sweeping.cpp:274:3
    #11 0x7f029bd640d0 in js::gc::GCRuntime::sweepBackgroundThings(js::gc::ZoneList&) /gecko/js/src/gc/Sweeping.cpp:352:9
    #12 0x7f029bd64858 in js::gc::GCRuntime::sweepFromBackgroundThread(js::AutoLockHelperThreadState&) /gecko/js/src/gc/Sweeping.cpp:429:5
    #13 0x7f029bcfa4fa in js::GCParallelTask::runTask(JS::GCContext*, js::AutoLockHelperThreadState&) /gecko/js/src/gc/GCParallelTask.cpp:202:3
    #14 0x7f029bcf98a7 in js::GCParallelTask::runFromMainThread() /gecko/js/src/gc/GCParallelTask.cpp:157:3
    #15 0x7f029bd64631 in js::gc::GCRuntime::queueZonesAndStartBackgroundSweep(js::gc::ZoneList&&) /gecko/js/src/gc/Sweeping.cpp:412:15
    #16 0x7f029bd70530 in js::gc::GCRuntime::endSweepingSweepGroup(JS::GCContext*, js::SliceBudget&) /gecko/js/src/gc/Sweeping.cpp:1671:3
    #17 0x7f029bd91142 in sweepaction::SweepActionSequence::run(js::gc::SweepAction::Args&) /gecko/js/src/gc/Sweeping.cpp:2128:23
    #18 0x7f029bd8abc3 in sweepaction::SweepActionForEach<js::gc::SweepGroupsIter, JSRuntime*>::run(js::gc::SweepAction::Args&) /gecko/js/src/gc/Sweeping.cpp:2163:19
    #19 0x7f029bd759d8 in js::gc::GCRuntime::performSweepActions(js::SliceBudget&) /gecko/js/src/gc/Sweeping.cpp:2305:53
    #20 0x7f029bccd529 in js::gc::GCRuntime::incrementalSlice(js::SliceBudget&, JS::GCReason, bool) /gecko/js/src/gc/GC.cpp:3611:11
    #21 0x7f029bcd2521 in js::gc::GCRuntime::gcCycle(bool, js::SliceBudget const&, JS::GCReason) /gecko/js/src/gc/GC.cpp:4123:3
    #22 0x7f029bcd3d00 in js::gc::GCRuntime::collect(bool, js::SliceBudget const&, JS::GCReason) /gecko/js/src/gc/GC.cpp:4311:9
    #23 0x7f029bc9b5db in js::gc::GCRuntime::gc(JS::GCOptions, JS::GCReason) /gecko/js/src/gc/GC.cpp:4388:3
    #24 0x7f028d1e8954 in nsCycleCollector::FixGrayBits(bool, TimeLog&) /gecko/xpcom/base/nsCycleCollector.cpp
    #25 0x7f028d1ea396 in nsCycleCollector::BeginCollection(mozilla::CCReason, ccIsManual, nsICycleCollectorListener*) /gecko/xpcom/base/nsCycleCollector.cpp:3605:3
    #26 0x7f028d1e99f1 in nsCycleCollector::Collect(mozilla::CCReason, ccIsManual, js::SliceBudget&, nsICycleCollectorListener*, bool) /gecko/xpcom/base/nsCycleCollector.cpp:3437:9
    #27 0x7f028d1e934b in nsCycleCollector::ShutdownCollect() /gecko/xpcom/base/nsCycleCollector.cpp:3376:20
    #28 0x7f028d1eb876 in nsCycleCollector::Shutdown(bool) /gecko/xpcom/base/nsCycleCollector.cpp:3672:5
    #29 0x7f028d1ed742 in nsCycleCollector_shutdown(bool) /gecko/xpcom/base/nsCycleCollector.cpp:3996:18
    #30 0x7f028d42dc6c in mozilla::ShutdownXPCOM(nsIServiceManager*) /gecko/xpcom/build/XPCOMInit.cpp:673:3
    #31 0x7f029ac354fd in XRE_InitChildProcess(int, char**, XREChildData const*) /gecko/toolkit/xre/nsEmbedFunctions.cpp:680:16
    #32 0x55d40e844454 in content_process_main(mozilla::Bootstrap*, int, char**) /gecko/browser/app/../../ipc/contentproc/plugin-container.cpp:57:28
    #33 0x55d40e844917 in main /gecko/browser/app/nsBrowserApp.cpp:359:18
    #34 0x7f02afe70082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16
    #35 0x55d40e782ed8 in _start (/home/worker/builds/m-c-20221216093922-fuzzing-asan-opt/firefox+0x111ed8) (BuildId: b0226006fcdaca0d284803f8ca2b65238d7abda4)

Set release status flags based on info from the regressing bug 1793525

:jld, since you are the author of the regressor, bug 1793525, could you take a look? Also, could you set the severity field?

For more information, please visit auto_nag documentation.

The fuzzers are reporting this frequently.

Summary: hanging at shutdown → hanging at shutdown; attempting crash report (fatal error)

This looks like a false positive, so we'll probably need to increase the timeouts for ASan/TSan like we did for Windows ccov builds.

Interestingly, I haven't gotten any reports (yet) like this for non-fuzzing tests on those builds; maybe the fuzzing can cause atypical usage patterns that result in needing more time to clean up the heap on shutdown or something.

Assignee: nobody → jld
Status: NEW → ASSIGNED
Pushed by jedavis@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/74cdcae1f76a Increase IPC child hang shutdown timeout for sanitizer builds. r=ipc-reviewers,mccr8
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → 110 Branch
Flags: needinfo?(jld)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: