Filed by: hskupin [at] gmail.com https://treeherder.mozilla.org/logviewer.html#?job_id=130580283&repo=try https://queue.taskcluster.net/v1/task/HXxEKU5oSySyl0JxLAHB-A/runs/0/artifacts/public/logs/live_backing.log [task 2017-09-13T08:51:32.790132Z] 08:51:32 INFO - Crash reason: SIGSEGV [task 2017-09-13T08:51:32.790853Z] 08:51:32 INFO - Crash address: 0x0 [task 2017-09-13T08:51:32.791383Z] 08:51:32 INFO - Process uptime: not available [task 2017-09-13T08:51:32.791673Z] 08:51:32 INFO - [task 2017-09-13T08:51:32.792249Z] 08:51:32 INFO - Thread 27 (crashed) [task 2017-09-13T08:51:32.792934Z] 08:51:32 INFO - 0 libxul.so!RunWatchdog [nsTerminator.cpp:a73cc4e08bf5 : 160 + 0x0] [task 2017-09-13T08:51:32.793289Z] 08:51:32 INFO - rax = 0x0000000000000000 rdx = 0x0000000000000000 [task 2017-09-13T08:51:32.793860Z] 08:51:32 INFO - rcx = 0x00007ff28e0692ad rbx = 0x00007ff24f2674e8 [task 2017-09-13T08:51:32.796891Z] 08:51:32 INFO - rsi = 0x00007ff28e338770 rdi = 0x00007ff28e337540 [task 2017-09-13T08:51:32.796959Z] 08:51:32 INFO - rbp = 0x00007ff24a465ec0 rsp = 0x00007ff24a465eb0 [task 2017-09-13T08:51:32.797064Z] 08:51:32 INFO - r8 = 0x00007ff28e338770 r9 = 0x00007ff24a466700 [task 2017-09-13T08:51:32.797125Z] 08:51:32 INFO - r10 = 0x0000000000000012 r11 = 0x0000000000000000 [task 2017-09-13T08:51:32.797232Z] 08:51:32 INFO - r12 = 0x000000000000003f r13 = 0x00000000000014c2 [task 2017-09-13T08:51:32.798058Z] 08:51:32 INFO - r14 = 0x00007ff24a466700 r15 = 0x00007ff24a466670 [task 2017-09-13T08:51:32.798109Z] 08:51:32 INFO - rip = 0x00007ff27f753590 [task 2017-09-13T08:51:32.798202Z] 08:51:32 INFO - Found by: given as instruction pointer in context [task 2017-09-13T08:51:32.798258Z] 08:51:32 INFO - 1 libnspr4.so!_pt_root [ptthread.c:a73cc4e08bf5 : 216 + 0x7] [task 2017-09-13T08:51:32.798312Z] 08:51:32 INFO - rbx = 0x00007ff250094800 rbp = 0x00007ff24a465f10 [task 2017-09-13T08:51:32.799102Z] 08:51:32 INFO - rsp = 0x00007ff24a465ed0 r12 = 0x0000000000000001 [task 2017-09-13T08:51:32.799891Z] 08:51:32 INFO - r13 = 0x00000000000014c2 r14 = 0x00007ff24a466700 [task 2017-09-13T08:51:32.800712Z] 08:51:32 INFO - r15 = 0x00007ff24a466670 rip = 0x00007ff28d8d16e6 [task 2017-09-13T08:51:32.801419Z] 08:51:32 INFO - Found by: call frame info [task 2017-09-13T08:51:32.802104Z] 08:51:32 INFO - 2 libpthread-2.23.so + 0x76ba [task 2017-09-13T08:51:32.802868Z] 08:51:32 INFO - rbx = 0x0000000000000000 rbp = 0x0000000000000000 [task 2017-09-13T08:51:32.803407Z] 08:51:32 INFO - rsp = 0x00007ff24a465f20 r12 = 0x0000000000000000 [task 2017-09-13T08:51:32.804057Z] 08:51:32 INFO - r13 = 0x00007ffd1916420f r14 = 0x00007ff24a4669c0 [task 2017-09-13T08:51:32.804703Z] 08:51:32 INFO - r15 = 0x00007ffd191642a0 rip = 0x00007ff28eff06ba [task 2017-09-13T08:51:32.805379Z] 08:51:32 INFO - Found by: call frame info [task 2017-09-13T08:51:32.806027Z] 08:51:32 INFO - 3 libc-2.23.so + 0x1073dd [task 2017-09-13T08:51:32.806717Z] 08:51:32 INFO - rsp = 0x00007ff24a465fc0 rip = 0x00007ff28e0793dd [task 2017-09-13T08:51:32.806915Z] 08:51:32 INFO - Found by: stack scanning Looks like there is something blocking us from shutdown, and Firefox gets killed.

I see the following in the log: [task 2017-09-15T08:17:53.934Z] 08:17:53 INFO - 1505463473929 Marionette DEBUG Received observer notification "xpcom-shutdown" [task 2017-09-15T08:17:53.955Z] 08:17:53 INFO - [Parent 4712, Main Thread] WARNING: 'NS_FAILED(rr->RetargetDeliveryTo(sts))', file /builds/worker/workspace/build/src/dom/fetch/FetchDriver.cpp, line 661 [task 2017-09-15T08:17:53.956Z] 08:17:53 INFO - [Parent 4712, Main Thread] WARNING: 'NS_FAILED(rv)', file /builds/worker/workspace/build/src/dom/fetch/FetchConsumer.cpp, line 516 [task 2017-09-15T08:17:53.957Z] 08:17:53 INFO - [Parent 4712, Main Thread] WARNING: Retargeting failed: file /builds/worker/workspace/build/src/dom/fetch/FetchConsumer.cpp, line 517 [task 2017-09-15T08:17:54.038Z] 08:17:54 INFO - [Parent 4712, Main Thread] WARNING: A runnable was posted to a worker that is already shutting down!: file /builds/worker/workspace/build/src/dom/workers/WorkerPrivate.cpp, line 2958 [task 2017-09-15T08:17:54.040Z] 08:17:54 INFO - [Parent 4712, Main Thread] WARNING: Could not dispatch ConsumeBodyRunnable: file /builds/worker/workspace/build/src/dom/fetch/FetchConsumer.cpp, line 224 [task 2017-09-15T08:18:21.499Z] 08:18:21 INFO - JavaScript error: resource://gre/modules/osfile/osfile_async_front.jsm, line 410: Error: OS.File has been shut down. Rejecting post to stat [task 2017-09-15T08:18:57.334Z] 08:18:57 INFO - Hit MOZ_CRASH(Shutdown too long, probably frozen, causing a crash.) at /builds/worker/workspace/build/src/toolkit/components/terminator/nsTerminator.cpp:160 [task 2017-09-15T08:18:57.336Z] 08:18:57 INFO - #01: ???[/builds/worker/workspace/build/application/firefox/libnspr4.so +0x286e6] [task 2017-09-15T08:18:57.337Z] 08:18:57 INFO - #02: ???[/lib/x86_64-linux-gnu/libpthread.so.0 +0x76ba] [task 2017-09-15T08:18:57.338Z] 08:18:57 INFO - #03: clone[/lib/x86_64-linux-gnu/libc.so.6 +0x1073dd] [task 2017-09-15T08:18:57.340Z] 08:18:57 INFO - #04: ??? (???:???) Andrea, could this be one more issue with FetchConsumer and which is causing a shutdown hang?

Those hangs started on September 13th on autoland if it helps.

This seems more related to worker shutting down. I'm planning to change this part soon.

Hey baku, this is currently a P5 but triage managers felt it might be more important. Can you update the priority and let us know when you expect to get to this?

I would say P3. overholt?

Since it's something baku's likely to work on in the next few months, let's go with P2. (The component here seems odd)

This bug looks like a dupe of bug 1425323 now. Here a link to the stack: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=151425042&lineNumber=47965 Andrea, I assume sometimes in December the quota manager got moved from the main thread to it's own thread?

> Andrea, I assume sometimes in December the quota manager got moved from the > main thread to it's own thread? Yes, but the shutdown notifications are received on the main-thread. I agree. it's a dup of bug 1425323.