Closed Bug 1541928 (ls-shutdown-hangs) Opened 6 years ago Closed 5 years ago

Crash in [@ mozilla::dom::(anonymous namespace)::QuotaClient::ShutdownTimedOut]

Categories

(Core :: Storage: localStorage & sessionStorage, defect, P1)

Product:
Core
Component:
Storage: localStorage & sessionStorage
Platform:
Unspecified
macOS
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla71
Tracking Flags:
Tracking Status
firefox-esr60 --- unaffected
firefox-esr68 --- disabled
firefox67 --- unaffected
firefox67.0.1 --- unaffected
firefox68 + disabled
firefox69 + disabled
firefox70 + fixed
firefox71 --- fixed

People

(Reporter: marcia, Assigned: janv)

References

(Blocks 1 open bug)

Details

(Keywords: crash, regression)

Crash Data

This bug is for crash report bp-a96fa52f-3190-4f3e-99a0-efc020190404.

Two crashes (One Mac and one Linux) in Build 20190404063228 - Moz crash reason MOZ_CRASH(LocalStorage shutdown timed out). Maybe expected due to Bug 1539750.

Top 10 frames of crashing thread:

0 XUL mozilla::dom:: dom/localstorage/ActorsParent.cpp:8114
1 XUL mozilla::dom:: const dom/localstorage/ActorsParent.cpp:8061
2 XUL mozilla::dom:: dom/localstorage/ActorsParent.cpp:8061
3 XUL nsTimerEvent::Run xpcom/threads/nsTimerImpl.cpp:559
4 XUL nsThread::ProcessNextEvent xpcom/threads/nsThread.cpp:1180
5 XUL NS_ProcessNextEvent xpcom/threads/nsThreadUtils.cpp:486
6 XUL mozilla::dom:: xpcom/threads/nsThreadUtils.h:348
7 XUL mozilla::dom::quota::QuotaManager::Shutdown dom/quota/ActorsParent.cpp:3344
8 XUL mozilla::dom::quota::RecvShutdownQuotaManager dom/quota/ActorsParent.cpp:2958
9 XUL mozilla::ipc::BackgroundParentImpl::RecvShutdownQuotaManager ipc/glue/BackgroundParentImpl.cpp:973
Blocks: 1540402
Crash Signature: [@ mozilla::dom::(anonymous namespace)::QuotaClient::ShutdownTimedOut] → [@ mozilla::dom::(anonymous namespace)::QuotaClient::ShutdownTimedOut] [@ mozilla::dom::`anonymous namespace'::QuotaClient::ShutdownTimedOut]

P2, given it can take long time to get rid of this completely.

Priority: -- → P2

Could this be a dupe of bug 1541776?

No, that one tracks shutdown hangs caused by IndexedDB. The one here, tracks new LocalStorage.

Blocks: 1539835

Next step here is to add more debugging information to crash annotations.

Crash Signature: [@ mozilla::dom::(anonymous namespace)::QuotaClient::ShutdownTimedOut] [@ mozilla::dom::`anonymous namespace'::QuotaClient::ShutdownTimedOut] → [@ mozilla::dom::(anonymous namespace)::QuotaClient::ShutdownTimedOut] [@ mozilla::dom::`anonymous namespace'::QuotaClient::ShutdownTimedOut] [@ mozilla::dom::`anonymous namespace'::QuotaClient::ShutdownWorkThreads::<T>::__invoke]

This crash is pretty visible in 68 beta - 170 crashes in one of the signatures in b4.

Depends on: 1556730
Alias: ls-shutdown-hangs
Crash Signature: [@ mozilla::dom::(anonymous namespace)::QuotaClient::ShutdownTimedOut] [@ mozilla::dom::`anonymous namespace'::QuotaClient::ShutdownTimedOut] [@ mozilla::dom::`anonymous namespace'::QuotaClient::ShutdownWorkThreads::<T>::__invoke] → [@ mozilla::dom::(anonymous namespace)::QuotaClient::ShutdownTimedOut] [@ mozilla::dom::(anonymous namespace)::QuotaClient::ShutdownWorkThreads::$_14::__invoke] [@ mozilla::dom::`anonymous namespace'::QuotaClient::ShutdownTimedOut] [@ mozilla::dom::`an…
Depends on: 1542572

According to:
https://crash-stats.mozilla.org/report/index/85c4699d-f196-4cc6-b7f1-0b79d0190609#allthreads
the LS thread has WaitForUnlockNotify in the stack trace. That can cause content process hangs like in bug 1554321.
It seems it can also block shutdown.

:asuth already investigated this in bug 1554321 and I was able to simulate it with my local build.
I'm going to file a new bug (with a patch) for that, so it should be fixed very soon.

Depends on: 1558112
Depends on: 1559029
Priority: P2 → P1
Assignee: nobody → jvarga
Status: NEW → ASSIGNED

Ok, we have zero crashes (LocalStorage shutdown hangs) for last 3 nightly builds. It seems bug 1559029 was causing most of the problems.

(In reply to Jan Varga [:janv] from comment #7)

Ok, we have zero crashes (LocalStorage shutdown hangs) for last 3 nightly builds. It seems bug 1559029 was causing most of the problems.

Jan - There are still a handful of crashes recently on 69 nightly, but I cannot tell whether they are the LocalStorage shutdown hangs or not. For instance, https://crash-stats.mozilla.org/report/index/fa86228d-f955-4cde-93cc-d1cb10190626 seems to have some of that at the top of the stack.

Yes, there are still some, I'll be adding more debugging info for that which should help with fixing remaining crashes/shutdown hangs.

This should go away for 68 when bug 1562942 lands in RC2.

tracking-firefox68: --- → +
tracking-firefox69: --- → +
Depends on: 1562942

Hi Jan, this signature is still sticking around in the 69 betas. Any thoughts on how to proceed here given that we're midway through the Beta cycle?

Flags: needinfo?(jvarga)

We plan to disable LSNG in 69 in bug 1570644.

Flags: needinfo?(jvarga)
Depends on: 1570644

Jan is working on another high volume crash. This is in the queue after that.

Depends on: 1581127

It seems bug 1581127 helped with reducing LS shutdown hangs.

Yeah, the statistics look better, but there are still some unexplained shutdown blockers. I'm going to add more debugging info for that.

Depends on: 1584323
Crash Signature: [@ mozilla::dom::(anonymous namespace)::QuotaClient::ShutdownTimedOut] [@ mozilla::dom::(anonymous namespace)::QuotaClient::ShutdownWorkThreads::$_14::__invoke] [@ mozilla::dom::`anonymous namespace'::QuotaClient::ShutdownTimedOut] [@ mozilla::dom::`an… → [@ mozilla::dom::(anonymous namespace)::QuotaClient::ShutdownTimedOut] [@ mozilla::dom::(anonymous namespace)::QuotaClient::ShutdownWorkThreads::$_14::__invoke] [@ mozilla::dom::(anonymous namespace)::QuotaClient::ShutdownWorkThreads::$_15::__invoke] […

Crash volume is still very high for beta, is there anything else you can investigate here?

status-firefox71: --- → affected
Flags: needinfo?(jvarga)

I'm working on a fix. We had to add more debugging information to crash stats to see what remaining problem is. The fix should be ready this week.

Flags: needinfo?(jvarga)
Depends on: 1587931
Depends on: 1586315

Marking as fixed given low frequency. Remaining issues are tracked in bug 1588498.

Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
status-firefox70: affected → fixed
status-firefox71: affected → fixed
Target Milestone: --- → mozilla71
Crash Signature: namespace)::QuotaClient::ShutdownWorkThreads::$_15::__invoke] [@ mozilla::dom::`anonymous namespace'::QuotaClient::ShutdownTimedOut] [@ mozilla::dom::`anonymous namespace'::QuotaClient::ShutdownWorkThreads::<T>::__invoke] → namespace)::QuotaClient::ShutdownWorkThreads::$_15::__invoke] [@ mozilla::dom::`anonymous namespace'::QuotaClient::ShutdownTimedOut] [@ mozilla::dom::`anonymous namespace'::QuotaClient::ShutdownWorkThreads::<T>::__invoke] [@ static void mozilla::dom::…
You need to log in before you can comment on or make changes to this bug.