Open Bug 1719400 Opened 3 years ago Updated 3 years ago

Intermittent Fission OOM - PROCESS-CRASH | Last test finished | application crashed [@ _pt_root] MOZ_CRASH(Shutdown hanging after all known phases and workers finished.)

Categories

(Core :: Graphics: Layers, defect, P3)

Product:
Core
Component:
Graphics: Layers
Platform:
Unspecified
All
Type:
defect

Tracking

()

Project Flags:
Fission Milestone Future
Tracking Flags:
Tracking Status
firefox-esr78 --- unaffected
firefox-esr91 --- unaffected
firefox90 --- unaffected
firefox91 --- unaffected
firefox92 --- affected
firefox93 --- affected
firefox94 --- affected
firefox95 --- affected

People

(Reporter: intermittent-bug-filer, Unassigned, NeedInfo)

References

Details

(Keywords: crash, intermittent-failure, Whiteboard: [stockwell unknown])

Crash Data

Filed by: abutkovits [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=344528817&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/YkdxiZaiRjSJGeN2gC4W9A/runs/0/artifacts/public/logs/live_backing.log

[task 2021-07-07T01:28:10.780Z] 01:28:10    ERROR - TEST-UNEXPECTED-FAIL | Last test finished | application terminated with exit code 1
[task 2021-07-07T01:28:10.781Z] 01:28:10     INFO - runtests.py | Application ran for: 0:18:43.947924
[task 2021-07-07T01:28:10.781Z] 01:28:10     INFO - zombiecheck | Reading PID log: /var/folders/9y/g98kq9jn4gg8dkkd4tldh4dh000014/T/tmpuheev59ipidlog
[task 2021-07-07T01:28:10.781Z] 01:28:10     INFO - ==> process 3985 launched child process 3986
[task 2021-07-07T01:28:10.782Z] 01:28:10     INFO - ==> process 3985 launched child process 3987
[task 2021-07-07T01:28:10.782Z] 01:28:10     INFO - ==> process 3985 launched child process 3988
[task 2021-07-07T01:28:10.782Z] 01:28:10     INFO - ==> process 3985 launched child process 3989
[task 2021-07-07T01:28:10.783Z] 01:28:10     INFO - ==> process 3985 launched child process 3990
[task 2021-07-07T01:28:10.783Z] 01:28:10     INFO - ==> process 3985 launched child process 3995
[task 2021-07-07T01:28:10.784Z] 01:28:10     INFO - ==> process 3985 launched child process 3996
[task 2021-07-07T01:28:10.784Z] 01:28:10     INFO - ==> process 3985 launched child process 3997
[task 2021-07-07T01:28:10.784Z] 01:28:10     INFO - ==> process 3985 launched child process 3998
[task 2021-07-07T01:28:10.784Z] 01:28:10     INFO - ==> process 3985 launched child process 3999
[task 2021-07-07T01:28:10.785Z] 01:28:10     INFO - ==> process 3985 launched child process 4000
[task 2021-07-07T01:28:10.785Z] 01:28:10     INFO - ==> process 3985 launched child process 4001
[task 2021-07-07T01:28:10.786Z] 01:28:10     INFO - ==> process 3985 launched child process 4002
[task 2021-07-07T01:28:10.786Z] 01:28:10     INFO - ==> process 3985 launched child process 4003
[task 2021-07-07T01:28:10.786Z] 01:28:10     INFO - ==> process 3985 launched child process 4004
[task 2021-07-07T01:28:10.787Z] 01:28:10     INFO - ==> process 3985 launched child process 4005
[task 2021-07-07T01:28:10.787Z] 01:28:10     INFO - ==> process 3985 launched child process 4006
[task 2021-07-07T01:28:10.787Z] 01:28:10     INFO - ==> process 3985 launched child process 4007
[task 2021-07-07T01:28:10.788Z] 01:28:10     INFO - ==> process 3985 launched child process 4008
[task 2021-07-07T01:28:10.788Z] 01:28:10     INFO - ==> process 3985 launched child process 4009
[task 2021-07-07T01:28:10.788Z] 01:28:10     INFO - ==> process 3985 launched child process 4010
[task 2021-07-07T01:28:10.788Z] 01:28:10     INFO - ==> process 3985 launched child process 4011
[task 2021-07-07T01:28:10.789Z] 01:28:10     INFO - ==> process 3985 launched child process 4012
[task 2021-07-07T01:28:10.789Z] 01:28:10     INFO - ==> process 3985 launched child process 4013
[task 2021-07-07T01:28:10.790Z] 01:28:10     INFO - ==> process 3985 launched child process 4018
[task 2021-07-07T01:28:10.790Z] 01:28:10     INFO - ==> process 3985 launched child process 4019
[task 2021-07-07T01:28:10.790Z] 01:28:10     INFO - ==> process 3985 launched child process 4020
[task 2021-07-07T01:28:10.790Z] 01:28:10     INFO - ==> process 3985 launched child process 4021
[task 2021-07-07T01:28:10.791Z] 01:28:10     INFO - ==> process 3985 launched child process 4022
[task 2021-07-07T01:28:10.791Z] 01:28:10     INFO - ==> process 3985 launched child process 4023
[task 2021-07-07T01:28:10.791Z] 01:28:10     INFO - ==> process 3985 launched child process 4051
[task 2021-07-07T01:28:10.792Z] 01:28:10     INFO - ==> process 3985 launched child process 4059
[task 2021-07-07T01:28:10.792Z] 01:28:10     INFO - ==> process 3985 launched child process 4068
[task 2021-07-07T01:28:10.792Z] 01:28:10     INFO - ==> process 3985 launched child process 4077
[task 2021-07-07T01:28:10.793Z] 01:28:10     INFO - ==> process 3985 launched child process 4096
[task 2021-07-07T01:28:10.793Z] 01:28:10     INFO - ==> process 3985 launched child process 4101
[task 2021-07-07T01:28:10.793Z] 01:28:10     INFO - ==> process 3985 launched child process 4106
[task 2021-07-07T01:28:10.793Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 4096
[task 2021-07-07T01:28:10.794Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 4101
[task 2021-07-07T01:28:10.794Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 4106
[task 2021-07-07T01:28:10.795Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 3986
[task 2021-07-07T01:28:10.795Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 3987
[task 2021-07-07T01:28:10.795Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 3988
[task 2021-07-07T01:28:10.796Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 3989
[task 2021-07-07T01:28:10.796Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 3990
[task 2021-07-07T01:28:10.796Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 3995
[task 2021-07-07T01:28:10.796Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 3996
[task 2021-07-07T01:28:10.797Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 3997
[task 2021-07-07T01:28:10.797Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 3998
[task 2021-07-07T01:28:10.797Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 3999
[task 2021-07-07T01:28:10.797Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 4000
[task 2021-07-07T01:28:10.798Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 4001
[task 2021-07-07T01:28:10.798Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 4002
[task 2021-07-07T01:28:10.798Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 4003
[task 2021-07-07T01:28:10.798Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 4004
[task 2021-07-07T01:28:10.799Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 4005
[task 2021-07-07T01:28:10.799Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 4006
[task 2021-07-07T01:28:10.799Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 4007
[task 2021-07-07T01:28:10.800Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 4008
[task 2021-07-07T01:28:10.800Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 4009
[task 2021-07-07T01:28:10.800Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 4010
[task 2021-07-07T01:28:10.800Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 4011
[task 2021-07-07T01:28:10.801Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 4012
[task 2021-07-07T01:28:10.801Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 4013
[task 2021-07-07T01:28:10.801Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 4018
[task 2021-07-07T01:28:10.801Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 4019
[task 2021-07-07T01:28:10.802Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 4020
[task 2021-07-07T01:28:10.802Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 4021
[task 2021-07-07T01:28:10.802Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 4022
[task 2021-07-07T01:28:10.803Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 4023
[task 2021-07-07T01:28:10.803Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 4051
[task 2021-07-07T01:28:10.803Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 4059
[task 2021-07-07T01:28:10.803Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 4068
[task 2021-07-07T01:28:10.804Z] 01:28:10     INFO - zombiecheck | Checking for orphan process with PID: 4077
[task 2021-07-07T01:28:10.804Z] 01:28:10     INFO - mozcrash Copy/paste: /opt/worker/tasks/task_162561859219897/fetches/minidump_stackwalk/minidump_stackwalk /var/folders/9y/g98kq9jn4gg8dkkd4tldh4dh000014/T/tmpdepyhnwx.mozrunner/minidumps/44574F58-5128-4E1D-AF37-4FE517E1E674.dmp /opt/worker/tasks/task_162561859219897/build/symbols
[task 2021-07-07T01:28:17.249Z] 01:28:17     INFO - mozcrash Saved minidump as /opt/worker/tasks/task_162561859219897/build/blobber_upload_dir/44574F58-5128-4E1D-AF37-4FE517E1E674.dmp
[task 2021-07-07T01:28:17.250Z] 01:28:17     INFO - mozcrash Saved app info as /opt/worker/tasks/task_162561859219897/build/blobber_upload_dir/44574F58-5128-4E1D-AF37-4FE517E1E674.extra
[task 2021-07-07T01:28:17.450Z] 01:28:17     INFO - PROCESS-CRASH | Last test finished | application crashed [@ _pt_root]
[task 2021-07-07T01:28:17.450Z] 01:28:17     INFO - Mozilla crash reason: MOZ_CRASH(Shutdown hanging after all known phases and workers finished.)
[task 2021-07-07T01:28:17.450Z] 01:28:17     INFO - Crash dump filename: /var/folders/9y/g98kq9jn4gg8dkkd4tldh4dh000014/T/tmpdepyhnwx.mozrunner/minidumps/44574F58-5128-4E1D-AF37-4FE517E1E674.dmp
[task 2021-07-07T01:28:17.450Z] 01:28:17     INFO - Operating system: Mac OS X
[task 2021-07-07T01:28:17.450Z] 01:28:17     INFO -                   10.15.7 19H524
[task 2021-07-07T01:28:17.450Z] 01:28:17     INFO - CPU: amd64
[task 2021-07-07T01:28:17.450Z] 01:28:17     INFO -      family 6 model 158 stepping 10
[task 2021-07-07T01:28:17.450Z] 01:28:17     INFO -      12 CPUs
[task 2021-07-07T01:28:17.450Z] 01:28:17     INFO - 
[task 2021-07-07T01:28:17.450Z] 01:28:17     INFO - GPU: UNKNOWN
[task 2021-07-07T01:28:17.450Z] 01:28:17     INFO - 
[task 2021-07-07T01:28:17.450Z] 01:28:17     INFO - Crash reason:  EXC_BAD_ACCESS / KERN_INVALID_ADDRESS
[task 2021-07-07T01:28:17.450Z] 01:28:17     INFO - Crash address: 0x0
[task 2021-07-07T01:28:17.450Z] 01:28:17     INFO - Process uptime: 288 seconds
[task 2021-07-07T01:28:17.450Z] 01:28:17     INFO - 
[task 2021-07-07T01:28:17.450Z] 01:28:17     INFO - Thread 48 (crashed) - Shutdown Hang Terminator 0  XUL!mozilla::(anonymous namespace)::RunWatchdog(void*) [nsTerminator.cpp:e806428569824e8b75664663ad5289ad5c6a3c31 : 246 + 0x22]
[task 2021-07-07T01:28:17.450Z] 01:28:17     INFO -     rax = 0x000000010dea3934   rdx = 0x0000000000000000
[task 2021-07-07T01:28:17.451Z] 01:28:17     INFO -     rcx = 0x0000000100ac63d8   rbx = 0x0000000000000276
[task 2021-07-07T01:28:17.451Z] 01:28:17     INFO -     rsi = 0x00000000000120a8   rdi = 0x00007fff96522ca8
[task 2021-07-07T01:28:17.451Z] 01:28:17     INFO -     rbp = 0x000070000cdc3f70   rsp = 0x000070000cdc3f50
[task 2021-07-07T01:28:17.451Z] 01:28:17     INFO -      r8 = 0x00000000000130a8    r9 = 0x0000000000000000
[task 2021-07-07T01:28:17.451Z] 01:28:17     INFO -     r10 = 0x00007fff96522cc8   r11 = 0x00007fff96522cc0
[task 2021-07-07T01:28:17.451Z] 01:28:17     INFO -     r12 = 0x00000001194481b0   r13 = 0x0000000000000000
[task 2021-07-07T01:28:17.451Z] 01:28:17     INFO -     r14 = 0x0000000100d0f378   r15 = 0x0000000000000007
[task 2021-07-07T01:28:17.451Z] 01:28:17     INFO -     rip = 0x00000001098cf96c
[task 2021-07-07T01:28:17.451Z] 01:28:17     INFO -     Found by: given as instruction pointer in context
[task 2021-07-07T01:28:17.451Z] 01:28:17     INFO -  1  libnss3.dylib!_pt_root [ptthread.c:e806428569824e8b75664663ad5289ad5c6a3c31 : 201 + 0xa]
[task 2021-07-07T01:28:17.451Z] 01:28:17     INFO -     rbx = 0x0000000000000276   rbp = 0x000070000cdc3fb0
[task 2021-07-07T01:28:17.451Z] 01:28:17     INFO -     rsp = 0x000070000cdc3f80   r12 = 0x00000001194481b0
[task 2021-07-07T01:28:17.451Z] 01:28:17     INFO -     r13 = 0x0000000000000000   r14 = 0x0000000000000000
[task 2021-07-07T01:28:17.451Z] 01:28:17     INFO -     r15 = 0x0000000000000007   rip = 0x0000000100f883c9
[task 2021-07-07T01:28:17.451Z] 01:28:17     INFO -     Found by: call frame info
[task 2021-07-07T01:28:17.452Z] 01:28:17     INFO -  2  libsystem_pthread.dylib!_pthread_start + 0x94
[task 2021-07-07T01:28:17.452Z] 01:28:17     INFO -     rbx = 0x6e776f6474756853   rbp = 0x000070000cdc3fd0
[task 2021-07-07T01:28:17.452Z] 01:28:17     INFO -     rsp = 0x000070000cdc3fc0   r12 = 0x00000001194481b0
[task 2021-07-07T01:28:17.452Z] 01:28:17     INFO -     r13 = 0x0000000000000000   r14 = 0x0000000000000000
[task 2021-07-07T01:28:17.452Z] 01:28:17     INFO -     r15 = 0x0000000a0000001f   rip = 0x00007fff6fe7b109
[task 2021-07-07T01:28:17.452Z] 01:28:17     INFO -     Found by: call frame info
[task 2021-07-07T01:28:17.452Z] 01:28:17     INFO -  3  libsystem_pthread.dylib!thread_start + 0xf
[task 2021-07-07T01:28:17.452Z] 01:28:17     INFO -     rbx = 0x6e776f6474756853   rbp = 0x000070000cdc3ff0
[task 2021-07-07T01:28:17.452Z] 01:28:17     INFO -     rsp = 0x000070000cdc3fe0   r12 = 0x0000000000000000
[task 2021-07-07T01:28:17.452Z] 01:28:17     INFO -     r13 = 0x0000000000000000   r14 = 0x0000000000000000
[task 2021-07-07T01:28:17.452Z] 01:28:17     INFO -     r15 = 0x0000000a0000001f   rip = 0x00007fff6fe76b8b
[task 2021-07-07T01:28:17.452Z] 01:28:17     INFO -     Found by: call frame info

Reported after toolkit/components/url-classifier/tests/mochitest/test_socialtracking_annotate.html

Component: Safe Browsing → IPC
Product: Toolkit → Core
Summary: Intermittent PROCESS-CRASH | Last test finished | application crashed [@ _pt_root] → Intermittent Fission OOM - PROCESS-CRASH | Last test finished | application crashed [@ _pt_root]

There are 39 total failures in the last 7 days on

[task 2021-07-11T10:41:31.612Z] 10:41:31 INFO - TEST-START | toolkit/components/url-classifier/tests/mochitest/test_socialtracking_annotate.html
[task 2021-07-11T10:41:31.636Z] 10:41:31 INFO - GECKO(12482) | [Child 12566, Main Thread] WARNING: Extra shutdown CC: 'i < NORMAL_SHUTDOWN_COLLECTIONS', file /builds/worker/checkouts/gecko/xpcom/base/nsCycleCollector.cpp:3357
[task 2021-07-11T10:41:31.647Z] 10:41:31 INFO - GECKO(12482) | [Child 12566, Main Thread] WARNING: NS_ENSURE_TRUE(Preferences::InitStaticMembers()) failed: file /builds/worker/checkouts/gecko/modules/libpref/Preferences.cpp:4438
[task 2021-07-11T10:41:35.031Z] 10:41:35 INFO - GECKO(12482) | [Parent 12482, Main Thread] WARNING: WebProgress Ignored: no longer current window global: file /builds/worker/checkouts/gecko/dom/ipc/BrowserParent.cpp:2928
[task 2021-07-11T10:41:36.200Z] 10:41:36 INFO - GECKO(12482) | ### XPCOM_MEM_BLOAT_LOG defined -- logging bloat/leaks to /var/folders/dv/hnm8h5w55bv3mzsfk0_zk888000014/T/tmpybstp5m1.mozrunner/runtests_leaks_tab_pid12605.log
[task 2021-07-11T10:41:36.201Z] 10:41:36 INFO - GECKO(12482) | [12605, Main Thread] WARNING: XPCOM_MEM_BLOAT_LOG is set, disabling native allocations.: file /builds/worker/checkouts/gecko/tools/profiler/core/platform.cpp:248
[task 2021-07-11T10:41:38.887Z] 10:41:38 INFO - GECKO(12482) | [Child 12500, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:527
[task 2021-07-11T10:41:38.887Z] 10:41:38 INFO - GECKO(12482) | [Child 12500, Main Thread] WARNING: '!ipcActor->SendAccumulateChildHistograms(histogramsToSend)', file /builds/worker/checkouts/gecko/toolkit/components/telemetry/core/ipc/TelemetryIPCAccumulator.cpp:279
[task 2021-07-11T10:41:38.888Z] 10:41:38 INFO - GECKO(12482) | [Child 12500, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:527
[task 2021-07-11T10:41:38.888Z] 10:41:38 INFO - GECKO(12482) | [Child 12500, Main Thread] WARNING: '!ipcActor->SendRecordDiscardedData(discardedData)', file /builds/worker/checkouts/gecko/toolkit/components/telemetry/core/ipc/TelemetryIPCAccumulator.cpp:298
[task 2021-07-11T10:41:48.874Z] 10:41:48 INFO - GECKO(12482) | [Child 12500, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:527
[task 2021-07-11T10:41:48.874Z] 10:41:48 INFO - GECKO(12482) | [Child 12500, Main Thread] WARNING: '!ipcActor->SendAccumulateChildHistograms(histogramsToSend)', file /builds/worker/checkouts/gecko/toolkit/components/telemetry/core/ipc/TelemetryIPCAccumulator.cpp:279
[task 2021-07-11T10:41:48.875Z] 10:41:48 INFO - GECKO(12482) | [Child 12500, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:527
[task 2021-07-11T10:41:48.875Z] 10:41:48 INFO - GECKO(12482) | [Child 12500, Main Thread] WARNING: '!ipcActor->SendRecordDiscardedData(discardedData)', file /builds/worker/checkouts/gecko/toolkit/components/telemetry/core/ipc/TelemetryIPCAccumulator.cpp:298
[task 2021-07-11T10:41:52.647Z] 10:41:52 INFO - GECKO(12482) | MEMORY STAT | vsize 6631MB | residentFast 86MB | heapAllocated 7MB
[task 2021-07-11T10:41:55.411Z] 10:41:55 INFO - TEST-OK | toolkit/components/url-classifier/tests/mochitest/test_socialtracking_annotate.html | took 23798ms
[task 2021-07-11T10:41:58.748Z] 10:41:58 INFO - Error: Unable to restore focus, expect failures and timeouts.
[task 2021-07-11T10:41:58.755Z] 10:41:58 INFO - GECKO(12482) | [Parent 12482, Main Thread] WARNING: Received load event on unbridged BrowserParent!: file /builds/worker/checkouts/gecko/dom/ipc/BrowserParent.cpp:3898
[task 2021-07-11T10:41:58.756Z] 10:41:58 INFO - GECKO(12482) | [Parent 12482, Main Thread] WARNING: Received load event on unbridged BrowserParent!: file /builds/worker/checkouts/gecko/dom/ipc/BrowserParent.cpp:3898
[task 2021-07-11T10:41:58.759Z] 10:41:58 INFO - GECKO(12482) | [Child 12575, Main Thread] WARNING: nsAppShell::Exit() called redundantly: file /builds/worker/checkouts/gecko/widget/cocoa/nsAppShell.mm:764
[task 2021-07-11T10:41:58.774Z] 10:41:58 INFO - TEST-START | Shutdown

[task 2021-07-11T10:58:59.738Z] 10:58:59 INFO - PROCESS-CRASH | Last test finished | application crashed [@ _pt_root]
[task 2021-07-11T10:58:59.738Z] 10:58:59 INFO - Mozilla crash reason: MOZ_CRASH(Shutdown hanging after all known phases and workers finished.)
[task 2021-07-11T10:58:59.738Z] 10:58:59 INFO - Crash dump filename: /var/folders/dv/hnm8h5w55bv3mzsfk0_zk888000014/T/tmpybstp5m1.mozrunner/minidumps/D3A279EB-E43C-415D-9B28-8D583B7BE7E1.dmp
[task 2021-07-11T10:58:59.738Z] 10:58:59 INFO - Operating system: Mac OS X
[task 2021-07-11T10:58:59.738Z] 10:58:59 INFO - 10.15.7 19H524
[task 2021-07-11T10:58:59.738Z] 10:58:59 INFO - CPU: amd64
[task 2021-07-11T10:58:59.738Z] 10:58:59 INFO - family 6 model 158 stepping 10
[task 2021-07-11T10:58:59.738Z] 10:58:59 INFO - 12 CPUs
[task 2021-07-11T10:58:59.738Z] 10:58:59 INFO -
[task 2021-07-11T10:58:59.738Z] 10:58:59 INFO - GPU: UNKNOWN
[task 2021-07-11T10:58:59.738Z] 10:58:59 INFO -
[task 2021-07-11T10:58:59.738Z] 10:58:59 INFO - Crash reason: EXC_BAD_ACCESS / KERN_INVALID_ADDRESS
[task 2021-07-11T10:58:59.738Z] 10:58:59 INFO - Crash address: 0x0
[task 2021-07-11T10:58:59.738Z] 10:58:59 INFO - Process uptime: 291 seconds
[task 2021-07-11T10:58:59.738Z] 10:58:59 INFO -
[task 2021-07-11T10:58:59.738Z] 10:58:59 INFO - Thread 48 (crashed) - Shutdown Hang Terminator 0 XUL!mozilla::(anonymous namespace)::RunWatchdog(void*) [nsTerminator.cpp:e5591aaa2cdc8ace99686b1e09bcd5d4c7c39fa6 : 246 + 0x22]
[task 2021-07-11T10:58:59.739Z] 10:58:59 INFO - rax = 0x000000011cbee924 rdx = 0x0000000000000000
[task 2021-07-11T10:58:59.739Z] 10:58:59 INFO - rcx = 0x0000000106e5c3d8 rbx = 0x0000000000000276
[task 2021-07-11T10:58:59.739Z] 10:58:59 INFO - rsi = 0x00000000000120a8 rdi = 0x00007fff8e86aca8
[task 2021-07-11T10:58:59.739Z] 10:58:59 INFO - rbp = 0x0000700006c55f70 rsp = 0x0000700006c55f50
[task 2021-07-11T10:58:59.739Z] 10:58:59 INFO - r8 = 0x00000000000130a8 r9 = 0x0000000000000000
[task 2021-07-11T10:58:59.739Z] 10:58:59 INFO - r10 = 0x00007fff8e86acc8 r11 = 0x00007fff8e86acc0
[task 2021-07-11T10:58:59.739Z] 10:58:59 INFO - r12 = 0x0000000123fe9d90 r13 = 0x0000000000000000
[task 2021-07-11T10:58:59.739Z] 10:58:59 INFO - r14 = 0x000000010700f450 r15 = 0x0000000000000007
[task 2021-07-11T10:58:59.739Z] 10:58:59 INFO - rip = 0x000000011861936c
[task 2021-07-11T10:58:59.739Z] 10:58:59 INFO - Found by: given as instruction pointer in context
[task 2021-07-11T10:58:59.739Z] 10:58:59 INFO - 1 libnss3.dylib!_pt_root [ptthread.c:e5591aaa2cdc8ace99686b1e09bcd5d4c7c39fa6 : 201 + 0xa]
[task 2021-07-11T10:58:59.739Z] 10:58:59 INFO - rbx = 0x0000000000000276 rbp = 0x0000700006c55fb0
[task 2021-07-11T10:58:59.739Z] 10:58:59 INFO - rsp = 0x0000700006c55f80 r12 = 0x0000000123fe9d90
[task 2021-07-11T10:58:59.739Z] 10:58:59 INFO - r13 = 0x0000000000000000 r14 = 0x0000000000000000
[task 2021-07-11T10:58:59.739Z] 10:58:59 INFO - r15 = 0x0000000000000007 rip = 0x00000001072883c9
[task 2021-07-11T10:58:59.739Z] 10:58:59 INFO - Found by: call frame info
[task 2021-07-11T10:58:59.739Z] 10:58:59 INFO - 2 libsystem_pthread.dylib!_pthread_start + 0x94
[task 2021-07-11T10:58:59.739Z] 10:58:59 INFO - rbx = 0x6e776f6474756853 rbp = 0x0000700006c55fd0
[task 2021-07-11T10:58:59.739Z] 10:58:59 INFO - rsp = 0x0000700006c55fc0 r12 = 0x0000000123fe9d90
[task 2021-07-11T10:58:59.739Z] 10:58:59 INFO - r13 = 0x0000000000000000 r14 = 0x0000000000000000
[task 2021-07-11T10:58:59.739Z] 10:58:59 INFO - r15 = 0x0000000a0000001f rip = 0x00007fff681c3109
[task 2021-07-11T10:58:59.740Z] 10:58:59 INFO - Found by: call frame info
[task 2021-07-11T10:58:59.740Z] 10:58:59 INFO - 3 libsystem_pthread.dylib!thread_start + 0xf
[task 2021-07-11T10:58:59.740Z] 10:58:59 INFO - rbx = 0x6e776f6474756853 rbp = 0x0000700006c55ff0
[task 2021-07-11T10:58:59.740Z] 10:58:59 INFO - rsp = 0x0000700006c55fe0 r12 = 0x0000000000000000
[task 2021-07-11T10:58:59.740Z] 10:58:59 INFO - r13 = 0x0000000000000000 r14 = 0x0000000000000000
[task 2021-07-11T10:58:59.740Z] 10:58:59 INFO - r15 = 0x0000000a0000001f rip = 0x00007fff681beb8b
[task 2021-07-11T10:58:59.740Z] 10:58:59 INFO - Found by: call frame info

Jim, can you please assign someone to take a look?

Flags: needinfo?(jmathies)
Whiteboard: [stockwell needswork:owner]

@ Nical, could these test hangs be caused by opaque pass bug 1717592, which landed on July 5?

This is a watchdog timeout after a test hang. The first test failures were on July 6. Did some big graphics changes land on July 5 or 6?

Here is the changelog from July 5 through July 6:

https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=095ea66cb9a1dc25f2abb79087dddf6b1b74102f&tochange=5b4463c3dc93e0becc8ab691915dc6ca9da928f5

Linux semi-headless mode on July 6, but I don't think it's related to this regression because we see these test failures on both Linux and macOS.

Fission Milestone: --- → Future
status-firefox90: --- → unaffected
status-firefox91: --- → unaffected
status-firefox92: --- → affected
status-firefox-esr78: --- → unaffected
Flags: needinfo?(nical.bugzilla)

@ Nical, could these test hangs be caused by opaque pass bug 1717592, which landed on July 5?

No that patch only changed some details about how some things are drawn.

The CompositorThreadHolder::Shutdown stuff spins the event loop until some shutdown logic happening asynchronously in other threads is complete. The other threads all appear to be sitting idle in their event loops (no deadlock as far as I can see) so something is holding to a RefPtr<CompositorThreadHolder> (it's when the last one goes away that we let the rest of the main thread's shutdown continue).

Flags: needinfo?(nical.bugzilla)

Update:
There have been 37 failures within the last 7 days:
• 17 failures on Linux 18.04 x64 debug
• 5 failures on Linux 18.04 x64 WebRender debug
• 15 failures on OS X 10.15 WebRender debug
Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=345587870&repo=autoland&lineNumber=4550

@ Emilio, do you think your PresShell active changes in bug 1717983 might cause intermittent compositor shutdown hangs in Fission tests on Linux and macOS? Nical's comment 6 has more details about the hanging threads.

100% of these test hangs are Fission tests.

The first failing changeset was e806428569824e8b75664663ad5289ad5c6a3c31, which hit five test hangs on autoland on 2021-07-06, but those changes don't look suspicious because they are related to Pocket and the New Tab:

https://treeherder.mozilla.org/jobs?repo=autoland&revision=e806428569824e8b75664663ad5289ad5c6a3c31&selectedTaskRun=OM1bM2hCS5a_tlDj1Rut0Q.0

Here is the mozilla-central pushlog that contains changeset e806428569824e8b75664663ad5289ad5c6a3c31 plus a few pushes before it:

https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=095ea66cb9a1dc25f2abb79087dddf6b1b74102f&tochange=e806428569824e8b75664663ad5289ad5c6a3c31

Flags: needinfo?(emilio)
OS: Unspecified → All
Summary: Intermittent Fission OOM - PROCESS-CRASH | Last test finished | application crashed [@ _pt_root] → Intermittent Fission OOM - PROCESS-CRASH | Last test finished | application crashed [@ _pt_root] MOZ_CRASH(Shutdown hanging after all known phases and workers finished.)

I don't think so, my patch only propagates activeness to in-process subdocuments properly, and they don't keep the compositor around in any way that the parent document wouldn't afaict.

Flags: needinfo?(emilio)

Update:
There have been 38 failures within the last 7 days:
• 5 failures on Linux 18.04 x64 debug
• 23 failures on Linux 18.04 x64 WebRender debug
• 10 failures on OS X 10.15 WebRender debug
Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=347009473&repo=mozilla-central&lineNumber=13408

Update:
There have been 39 failures within the last 7 days:
• 34 failures on Linux 18.04 x64 WebRender debug
• 5 failures on OS X 10.15 WebRender debug
Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=347752013&repo=autoland&lineNumber=30807

<...>
[task 2021-08-07T17:31:04.031Z] 17:31:04     INFO - GECKO(15806) | [Child 16823, Main Thread] WARNING: '!ipcActor->SendRecordDiscardedData(discardedData)', file /builds/worker/checkouts/gecko/toolkit/components/telemetry/core/ipc/TelemetryIPCAccumulator.cpp:298
[task 2021-08-07T17:31:13.925Z] 17:31:13     INFO - GECKO(15806) | out of memory: 0x0000000028700000 bytes requested
[task 2021-08-07T17:31:13.925Z] 17:31:13     INFO - GECKO(15806) | Hit MOZ_CRASH() at /builds/worker/checkouts/gecko/memory/mozalloc/mozalloc_abort.cpp:33
[task 2021-08-07T17:31:14.009Z] 17:31:14     INFO - GECKO(15806) | #01: mozalloc_abort [memory/mozalloc/mozalloc_abort.cpp:33]
[task 2021-08-07T17:31:14.011Z] 17:31:14     INFO - GECKO(15806) | #02: mozalloc_handle_oom(unsigned long) [memory/mozalloc/mozalloc_oom.cpp:51]
[task 2021-08-07T17:31:14.012Z] 17:31:14     INFO - GECKO(15806) | #03: moz_xstrdup [memory/mozalloc/mozalloc.cpp:80]
[task 2021-08-07T17:31:14.012Z] 17:31:14     INFO - GECKO(15806) | #04: nsTArrayInfallibleAllocator::ResultTypeProxy nsTArray_base<nsTArrayInfallibleAllocator, nsTArray_RelocateUsingMemutils>::EnsureCapacity<nsTArrayInfallibleAllocator>(unsigned long, unsigned long) [xpcom/ds/nsTArray-inl.h:223]
[task 2021-08-07T17:31:14.014Z] 17:31:14     INFO - GECKO(15806) | #05: mozilla::net::ChannelEventQueue::RunOrEnqueue(mozilla::net::ChannelEvent*, bool) [netwerk/ipc/ChannelEventQueue.h:218]
[task 2021-08-07T17:31:14.014Z] 17:31:14     INFO - GECKO(15806) | #06: mozilla::net::HttpChannelChild::DoNotifyListener() [netwerk/protocol/http/HttpChannelChild.cpp:1213]
[task 2021-08-07T17:31:14.014Z] 17:31:14     INFO - GECKO(15806) | #07: mozilla::detail::RunnableMethodImpl<RefPtr<mozilla::net::HttpChannelChild>, void (mozilla::net::HttpChannelChild::*)(), true, (mozilla::RunnableKind)0, >::Run() [xpcom/threads/nsThreadUtils.h:1204]
[task 2021-08-07T17:31:14.015Z] 17:31:14     INFO - GECKO(15806) | #08: mozilla::RunnableTask::Run() [xpcom/threads/TaskController.cpp:503]
[task 2021-08-07T17:31:14.016Z] 17:31:14     INFO - GECKO(15806) | #09: mozilla::TaskController::DoExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) [xpcom/threads/TaskController.cpp:805]
[task 2021-08-07T17:31:14.017Z] 17:31:14     INFO - GECKO(15806) | #10: mozilla::TaskController::ExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) [xpcom/threads/TaskController.cpp:641]
[task 2021-08-07T17:31:14.018Z] 17:31:14     INFO - GECKO(15806) | #11: mozilla::TaskController::ProcessPendingMTTask(bool) [xpcom/threads/TaskController.cpp:425]
[task 2021-08-07T17:31:14.020Z] 17:31:14     INFO - GECKO(15806) | #12: mozilla::detail::RunnableFunction<mozilla::TaskController::InitializeInternal()::$_0>::Run() [xpcom/threads/nsThreadUtils.h:533]
[task 2021-08-07T17:31:14.021Z] 17:31:14     INFO - GECKO(15806) | #13: nsThread::ProcessNextEvent(bool, bool*) [xpcom/threads/nsThread.cpp:1152]
[task 2021-08-07T17:31:14.022Z] 17:31:14     INFO - GECKO(15806) | #14: NS_ProcessNextEvent(nsIThread*, bool) [xpcom/threads/nsThreadUtils.cpp:466]
[task 2021-08-07T17:31:14.023Z] 17:31:14     INFO - GECKO(15806) | #15: mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) [ipc/glue/MessagePump.cpp:85]
[task 2021-08-07T17:31:14.024Z] 17:31:14     INFO - GECKO(15806) | #16: MessageLoop::RunInternal() [ipc/chromium/src/base/message_loop.cc:331]
[task 2021-08-07T17:31:14.026Z] 17:31:14     INFO - GECKO(15806) | #17: MessageLoop::Run() [ipc/chromium/src/base/message_loop.cc:307]
[task 2021-08-07T17:31:14.027Z] 17:31:14     INFO - GECKO(15806) | #18: nsBaseAppShell::Run() [widget/nsBaseAppShell.cpp:139]
[task 2021-08-07T17:31:14.028Z] 17:31:14     INFO - GECKO(15806) | #19: XRE_RunAppShell() [toolkit/xre/nsEmbedFunctions.cpp:917]
[task 2021-08-07T17:31:14.030Z] 17:31:14     INFO - GECKO(15806) | #20: mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate*) [ipc/glue/MessagePump.cpp:235]
[task 2021-08-07T17:31:14.031Z] 17:31:14     INFO - GECKO(15806) | #21: MessageLoop::RunInternal() [ipc/chromium/src/base/message_loop.cc:331]
[task 2021-08-07T17:31:14.032Z] 17:31:14     INFO - GECKO(15806) | #22: MessageLoop::Run() [ipc/chromium/src/base/message_loop.cc:307]
[task 2021-08-07T17:31:14.034Z] 17:31:14     INFO - GECKO(15806) | #23: XRE_InitChildProcess(int, char**, XREChildData const*) [toolkit/xre/nsEmbedFunctions.cpp:749]
[task 2021-08-07T17:31:14.036Z] 17:31:14     INFO - GECKO(15806) | #24: content_process_main(mozilla::Bootstrap*, int, char**) [ipc/contentproc/plugin-container.cpp:58]
[task 2021-08-07T17:31:14.037Z] 17:31:14     INFO - GECKO(15806) | #25: main [browser/app/nsBrowserApp.cpp:327]
[task 2021-08-07T17:31:14.038Z] 17:31:14     INFO - GECKO(15806) | #26: __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6 + 0x21b97]
[task 2021-08-07T17:31:14.039Z] 17:31:14     INFO - GECKO(15806) | #27: ??? [/builds/worker/workspace/build/application/firefox/firefox-bin + 0x413b9]
[task 2021-08-07T17:31:14.040Z] 17:31:14     INFO - GECKO(15806) | #28: ??? (???:???)
[task 2021-08-07T17:31:14.277Z] 17:31:14     INFO - TEST-INFO | Main app process: exit 11
[task 2021-08-07T17:31:14.278Z] 17:31:14     INFO - TEST-INFO | Confirming we saw 110 DOCSHELL created and 109 destroyed log strings.
[task 2021-08-07T17:31:14.279Z] 17:31:14     INFO - TEST-INFO | Confirming we saw 298 DOMWINDOW created and 286 destroyed log strings.
[task 2021-08-07T17:31:14.279Z] 17:31:14     INFO - TEST-INFO | toolkit/components/places/tests/browser/browser_bug1601563.js | This test created 1 hidden docshell(s)
[task 2021-08-07T17:31:14.280Z] 17:31:14     INFO - TEST-INFO | toolkit/components/places/tests/browser/browser_bug399606.js | This test created 1 hidden docshell(s)
[task 2021-08-07T17:31:14.281Z] 17:31:14     INFO - TEST-INFO | toolkit/components/places/tests/browser/browser_bug461710.js | This test created 1 hidden docshell(s)
[task 2021-08-07T17:31:14.282Z] 17:31:14     INFO - TEST-INFO | toolkit/components/places/tests/browser/browser_bug646422.js | This test created 1 hidden docshell(s)
[task 2021-08-07T17:31:14.283Z] 17:31:14     INFO - TEST-INFO | toolkit/components/places/tests/browser/browser_bug680727.js | This test created 1 hidden docshell(s)
[task 2021-08-07T17:31:14.283Z] 17:31:14     INFO - TEST-INFO | toolkit/components/places/tests/browser/browser_double_redirect.js | This test created 1 hidden docshell(s)
[task 2021-08-07T17:31:14.284Z] 17:31:14     INFO - TEST-INFO | toolkit/components/places/tests/browser/browser_favicon_privatebrowsing_perwindowpb.js | This test created 1 hidden docshell(s)
[task 2021-08-07T17:31:14.288Z] 17:31:14     INFO - TEST-INFO | toolkit/components/places/tests/browser/browser_history_post.js | This test created 1 hidden docshell(s)
[task 2021-08-07T17:31:14.288Z] 17:31:14     INFO - TEST-INFO | toolkit/components/places/tests/browser/browser_multi_redirect_frecency.js | This test created 1 hidden docshell(s)
[task 2021-08-07T17:31:14.288Z] 17:31:14     INFO - TEST-INFO | toolkit/components/places/tests/browser/browser_notfound.js | This test created 1 hidden docshell(s)
[task 2021-08-07T17:31:14.289Z] 17:31:14     INFO - TEST-INFO | toolkit/components/places/tests/browser/browser_onvisit_title_null_for_navigation.js | This test created 1 hidden docshell(s)
[task 2021-08-07T17:31:14.289Z] 17:31:14     INFO - TEST-INFO | toolkit/components/places/tests/browser/browser_redirect.js | This test created 1 hidden docshell(s)
[task 2021-08-07T17:31:14.289Z] 17:31:14     INFO - TEST-INFO | toolkit/components/places/tests/browser/browser_redirect_self.js | This test created 1 hidden docshell(s)
[task 2021-08-07T17:31:14.290Z] 17:31:14     INFO - TEST-INFO | toolkit/components/places/tests/browser/browser_settitle.js | This test created 1 hidden docshell(s)
[task 2021-08-07T17:31:14.292Z] 17:31:14     INFO - TEST-INFO | toolkit/components/places/tests/browser/browser_visited_notfound.js | This test created 1 hidden docshell(s)
[task 2021-08-07T17:31:14.292Z] 17:31:14     INFO - TEST-INFO | toolkit/components/places/tests/browser/browser_visituri.js | This test created 1 hidden docshell(s)
[task 2021-08-07T17:31:14.292Z] 17:31:14     INFO - TEST-INFO | toolkit/components/places/tests/browser/browser_visituri_nohistory.js | This test created 1 hidden docshell(s)
[task 2021-08-07T17:31:14.292Z] 17:31:14     INFO - TEST-INFO | toolkit/components/places/tests/browser/browser_visituri_privatebrowsing_perwindowpb.js | This test created 1 hidden docshell(s)
[task 2021-08-07T17:31:14.293Z] 17:31:14     INFO - Buffered messages finished
[task 2021-08-07T17:31:14.293Z] 17:31:14    ERROR - TEST-UNEXPECTED-FAIL | Last test finished | application terminated with exit code 11
[task 2021-08-07T17:31:14.293Z] 17:31:14     INFO - runtests.py | Application ran for: 0:14:21.776261
[task 2021-08-07T17:31:14.293Z] 17:31:14     INFO - zombiecheck | Reading PID log: /tmp/tmpe_p91aptpidlog
[task 2021-08-07T17:31:14.294Z] 17:31:14     INFO - ==> process 15806 launched child process 15824
<...>
[task 2021-08-07T17:31:14.348Z] 17:31:14     INFO - zombiecheck | Checking for orphan process with PID: 17023
[task 2021-08-07T17:31:14.349Z] 17:31:14     INFO - mozcrash Copy/paste: /builds/worker/fetches/minidump_stackwalk/minidump_stackwalk /tmp/tmphpe6zosg.mozrunner/minidumps/4430e0d1-f608-70f4-ee10-b17200b1736d.dmp /builds/worker/workspace/build/symbols
[task 2021-08-07T17:31:20.653Z] 17:31:20     INFO - mozcrash Saved minidump as /builds/worker/workspace/build/blobber_upload_dir/4430e0d1-f608-70f4-ee10-b17200b1736d.dmp
[task 2021-08-07T17:31:20.655Z] 17:31:20     INFO - mozcrash Saved app info as /builds/worker/workspace/build/blobber_upload_dir/4430e0d1-f608-70f4-ee10-b17200b1736d.extra
[task 2021-08-07T17:31:20.777Z] 17:31:20     INFO - PROCESS-CRASH | Last test finished | application crashed [@ _pt_root]
[task 2021-08-07T17:31:20.778Z] 17:31:20     INFO - Mozilla crash reason: MOZ_CRASH(Shutdown hanging after all known phases and workers finished.)
[task 2021-08-07T17:31:20.778Z] 17:31:20     INFO - Crash dump filename: /tmp/tmphpe6zosg.mozrunner/minidumps/4430e0d1-f608-70f4-ee10-b17200b1736d.dmp
[task 2021-08-07T17:31:20.778Z] 17:31:20     INFO - Operating system: Linux
[task 2021-08-07T17:31:20.778Z] 17:31:20     INFO -                   0.0.0 Linux 4.4.0-1014-aws #14taskcluster1-Ubuntu SMP Tue Apr 3 10:27:00 UTC 2018 x86_64
[task 2021-08-07T17:31:20.779Z] 17:31:20     INFO - CPU: amd64
[task 2021-08-07T17:31:20.779Z] 17:31:20     INFO -      family 6 model 85 stepping 7
[task 2021-08-07T17:31:20.780Z] 17:31:20     INFO -      2 CPUs
[task 2021-08-07T17:31:20.780Z] 17:31:20     INFO - 
[task 2021-08-07T17:31:20.780Z] 17:31:20     INFO - GPU: UNKNOWN
[task 2021-08-07T17:31:20.781Z] 17:31:20     INFO - 
[task 2021-08-07T17:31:20.781Z] 17:31:20     INFO - Crash reason:  SIGSEGV /SEGV_MAPERR
[task 2021-08-07T17:31:20.781Z] 17:31:20     INFO - Crash address: 0x0
[task 2021-08-07T17:31:20.781Z] 17:31:20     INFO - Process uptime: not available
[task 2021-08-07T17:31:20.782Z] 17:31:20     INFO - 
[task 2021-08-07T17:31:20.782Z] 17:31:20     INFO - Thread 37 (crashed) 0  libxul.so!mozilla::(anonymous namespace)::RunWatchdog(void*) [nsTerminator.cpp:a1d9ddb4fb7941b513a163f408f588c966515dc3 : 246 + 0x22]
[task 2021-08-07T17:31:20.782Z] 17:31:20     INFO -     rax = 0x00007fa4c8ea48c0   rdx = 0x0000000000000000
[task 2021-08-07T17:31:20.782Z] 17:31:20     INFO -     rcx = 0x0000561e42db3660   rbx = 0x0000000000000276
[task 2021-08-07T17:31:20.782Z] 17:31:20     INFO -     rsi = 0x00007fa4dc9958b0   rdi = 0x00007fa4dc994680
[task 2021-08-07T17:31:20.782Z] 17:31:20     INFO -     rbp = 0x00007fa4b79f7e60   rsp = 0x00007fa4b79f7e30
[task 2021-08-07T17:31:20.782Z] 17:31:20     INFO -      r8 = 0x00007fa4dc9958b0    r9 = 0x00007fa4b79f8700
[task 2021-08-07T17:31:20.783Z] 17:31:20     INFO -     r10 = 0x0000000000000000   r11 = 0x0000000000000000
[task 2021-08-07T17:31:20.783Z] 17:31:20     INFO -     r12 = 0x00007fa4b79f8630   r13 = 0x0000000000000007
[task 2021-08-07T17:31:20.783Z] 17:31:20     INFO -     r14 = 0x00007fa4b79f7e40   r15 = 0x00007fa4a61591c0
[task 2021-08-07T17:31:20.784Z] 17:31:20     INFO -     rip = 0x00007fa4cfd07eb2
[task 2021-08-07T17:31:20.784Z] 17:31:20     INFO -     Found by: given as instruction pointer in context
[task 2021-08-07T17:31:20.784Z] 17:31:20     INFO -  1  libnspr4.so!_pt_root [ptthread.c:a1d9ddb4fb7941b513a163f408f588c966515dc3 : 201 + 0x7]
[task 2021-08-07T17:31:20.784Z] 17:31:20     INFO -     rbx = 0x00007fa49dd36dc0   rbp = 0x00007fa4b79f7eb0
[task 2021-08-07T17:31:20.785Z] 17:31:20     INFO -     rsp = 0x00007fa4b79f7e70   r12 = 0x00007fa4b79f8630
[task 2021-08-07T17:31:20.785Z] 17:31:20     INFO -     r13 = 0x0000000000000007   r14 = 0x00000000000042a6
[task 2021-08-07T17:31:20.785Z] 17:31:20     INFO -     r15 = 0x00007fa4b79f8700   rip = 0x00007fa4dd8743f3
[task 2021-08-07T17:31:20.785Z] 17:31:20     INFO -     Found by: call frame info
[task 2021-08-07T17:31:20.785Z] 17:31:20     INFO -  2  libpthread.so.0 + 0x76db
[task 2021-08-07T17:31:20.785Z] 17:31:20     INFO -     rbx = 0x0000000000000000   rbp = 0x0000000000000000
[task 2021-08-07T17:31:20.786Z] 17:31:20     INFO -     rsp = 0x00007fa4b79f7ec0   r12 = 0x00007fa4b79f7f80
[task 2021-08-07T17:31:20.786Z] 17:31:20     INFO -     r13 = 0x0000000000000000   r14 = 0x00007fa49dd36dc0
[task 2021-08-07T17:31:20.786Z] 17:31:20     INFO -     r15 = 0x00007ffc58a56d50   rip = 0x00007fa4dd4e36db
[task 2021-08-07T17:31:20.786Z] 17:31:20     INFO -     Found by: call frame info
[task 2021-08-07T17:31:20.787Z] 17:31:20     INFO -  3  libc.so.6 + 0x121a3f
[task 2021-08-07T17:31:20.787Z] 17:31:20     INFO -     rsp = 0x00007fa4b79f7f80   rip = 0x00007fa4dc6c9a3f
[task 2021-08-07T17:31:20.787Z] 17:31:20     INFO -     Found by: stack scanning
[task 2021-08-07T17:31:20.787Z] 17:31:20     INFO - 
<...>

(In reply to Iulian Moraru from comment #15)

[task 2021-08-07T17:31:13.925Z] 17:31:13 INFO - GECKO(15806) | out of memory: 0x0000000028700000 bytes requested

That failure looks like there's more going on than just a shutdown hang.

Flags: needinfo?(jmathies)

There are 122 total failures in the last 7 days on

[task 2021-09-24T15:10:08.190Z] 15:10:08     INFO - TEST-START | browser/components/extensions/test/browser/browser_ext_windows_remove.js
[task 2021-09-24T15:10:08.230Z] 15:10:08     INFO - GECKO(3110) | [Child 3115: Main Thread]: I/DocShellAndDOMWindowLeak ++DOCSHELL 10869cc00 == 7 [pid = 3115] [id = 966]
[task 2021-09-24T15:10:08.231Z] 15:10:08     INFO - GECKO(3110) | [Child 3115: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 45 (108679c80) [pid = 3115] [serial = 2761] [outer = 0]
[task 2021-09-24T15:10:08.231Z] 15:10:08     INFO - GECKO(3110) | [Child 3115: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 46 (1086a0c00) [pid = 3115] [serial = 2762] [outer = 108679c80]
[task 2021-09-24T15:10:08.237Z] 15:10:08     INFO - GECKO(3110) | [Child 3115, Main Thread] WARNING: Fallback to BasicLayerManager: file /builds/worker/checkouts/gecko/dom/ipc/BrowserChild.cpp:2864

[task 2021-09-24T15:27:01.306Z] 15:27:01     INFO - PROCESS-CRASH | Last test finished | application crashed [@ _pt_root]
[task 2021-09-24T15:27:01.306Z] 15:27:01     INFO - Mozilla crash reason: MOZ_CRASH(Shutdown hanging after all known phases and workers finished.)
[task 2021-09-24T15:27:01.306Z] 15:27:01     INFO - Crash dump filename: /var/folders/sf/kf7kx8y520g2kgmq0hhfxzf8000014/T/tmpi3_kc8gx.mozrunner/minidumps/6BC58CA4-DF96-4541-9C04-955EE72C1DF0.dmp
[task 2021-09-24T15:27:01.306Z] 15:27:01     INFO - Operating system: Mac OS X
[task 2021-09-24T15:27:01.306Z] 15:27:01     INFO -                   10.15.7 19H524
[task 2021-09-24T15:27:01.306Z] 15:27:01     INFO - CPU: amd64
[task 2021-09-24T15:27:01.306Z] 15:27:01     INFO -      family 6 model 158 stepping 10
[task 2021-09-24T15:27:01.307Z] 15:27:01     INFO -      12 CPUs
[task 2021-09-24T15:27:01.307Z] 15:27:01     INFO - 
[task 2021-09-24T15:27:01.307Z] 15:27:01     INFO - GPU: UNKNOWN
[task 2021-09-24T15:27:01.307Z] 15:27:01     INFO - 
[task 2021-09-24T15:27:01.307Z] 15:27:01     INFO - Crash reason:  EXC_BAD_ACCESS / KERN_INVALID_ADDRESS
[task 2021-09-24T15:27:01.307Z] 15:27:01     INFO - Crash address: 0x0
[task 2021-09-24T15:27:01.307Z] 15:27:01     INFO - Process uptime: 969 seconds
[task 2021-09-24T15:27:01.307Z] 15:27:01     INFO - 
[task 2021-09-24T15:27:01.307Z] 15:27:01     INFO - Thread 46 tid 555787 (crashed) - Shutdown Hang Terminator 0  XUL!mozilla::(anonymous namespace)::RunWatchdog(void*) [nsTerminator.cpp:4e8c68a872fe1601f9f05a2ddc43b1335413a383 : 246 + 0x22]
[task 2021-09-24T15:27:01.307Z] 15:27:01     INFO -     rax = 0x000000011268280d   rdx = 0x0000000000000000
[task 2021-09-24T15:27:01.307Z] 15:27:01     INFO -     rcx = 0x0000000105b77458   rbx = 0x0000000000000276
[task 2021-09-24T15:27:01.307Z] 15:27:01     INFO -     rsi = 0x00000000000120a8   rdi = 0x00007fff8ecb0ca8
[task 2021-09-24T15:27:01.307Z] 15:27:01     INFO -     rbp = 0x00007000055aaf70   rsp = 0x00007000055aaf50
[task 2021-09-24T15:27:01.307Z] 15:27:01     INFO -      r8 = 0x00000000000130a8    r9 = 0x0000000000000000
[task 2021-09-24T15:27:01.307Z] 15:27:01     INFO -     r10 = 0x00007fff8ecb0cc8   r11 = 0x00007fff8ecb0cc0
[task 2021-09-24T15:27:01.307Z] 15:27:01     INFO -     r12 = 0x000000011c9d0c60   r13 = 0x0000000000000000
[task 2021-09-24T15:27:01.307Z] 15:27:01     INFO -     r14 = 0x00000001227a8f20   r15 = 0x0000000000000007
[task 2021-09-24T15:27:01.307Z] 15:27:01     INFO -     rip = 0x000000010e3be0dc
[task 2021-09-24T15:27:01.307Z] 15:27:01     INFO -     Found by: given as instruction pointer in context
[task 2021-09-24T15:27:01.307Z] 15:27:01     INFO -  1  libnss3.dylib!_pt_root [ptthread.c:4e8c68a872fe1601f9f05a2ddc43b1335413a383 : 201 + 0xa]
[task 2021-09-24T15:27:01.307Z] 15:27:01     INFO -     rbx = 0x00007000055ab000   rbp = 0x00007000055aafb0
[task 2021-09-24T15:27:01.307Z] 15:27:01     INFO -     rsp = 0x00007000055aaf80   r12 = 0x000000011c9d0c60
[task 2021-09-24T15:27:01.307Z] 15:27:01     INFO -     r13 = 0x0000000000000000   r14 = 0x00007000055ab000
[task 2021-09-24T15:27:01.308Z] 15:27:01     INFO -     r15 = 0x0000000000000007   rip = 0x0000000106009939
[task 2021-09-24T15:27:01.308Z] 15:27:01     INFO -     Found by: call frame info
[task 2021-09-24T15:27:01.308Z] 15:27:01     INFO -  2  libsystem_pthread.dylib!_pthread_start + 0x94
[task 2021-09-24T15:27:01.308Z] 15:27:01     INFO -     rbx = 0x00007000055ab000   rbp = 0x00007000055aafd0
[task 2021-09-24T15:27:01.308Z] 15:27:01     INFO -     rsp = 0x00007000055aafc0   r12 = 0x0000000000000000
[task 2021-09-24T15:27:01.308Z] 15:27:01     INFO -     r13 = 0x0000000000000000   r14 = 0x0000000000000000
[task 2021-09-24T15:27:01.308Z] 15:27:01     INFO -     r15 = 0x0000000000000000   rip = 0x00007fff68609109
[task 2021-09-24T15:27:01.308Z] 15:27:01     INFO -     Found by: call frame info
[task 2021-09-24T15:27:01.308Z] 15:27:01     INFO -  3  libsystem_pthread.dylib!thread_start + 0xf
[task 2021-09-24T15:27:01.308Z] 15:27:01     INFO -     rbx = 0x0000000000000000   rbp = 0x00007000055aaff0
[task 2021-09-24T15:27:01.308Z] 15:27:01     INFO -     rsp = 0x00007000055aafe0   r12 = 0x0000000000000000
[task 2021-09-24T15:27:01.308Z] 15:27:01     INFO -     r13 = 0x0000000000000000   r14 = 0x0000000000000000
[task 2021-09-24T15:27:01.308Z] 15:27:01     INFO -     r15 = 0x0000000000000000   rip = 0x00007fff68604b8b
[task 2021-09-24T15:27:01.308Z] 15:27:01     INFO -     Found by: call frame info
[task 2021-09-24T15:27:01.308Z] 15:27:01     INFO - 
[task 2021-09-24T15:27:01.308Z] 15:27:01     INFO - Thread 0 tid 775 - GeckoMain
[task 2021-09-24T15:27:01.308Z] 15:27:01     INFO -  0  libsystem_kernel.dylib!__psynch_cvwait + 0xa
[task 2021-09-24T15:27:01.308Z] 15:27:01     INFO -     rax = 0x0000000002000131   rdx = 0x0000000000005200
[task 2021-09-24T15:27:01.308Z] 15:27:01     INFO -     rcx = 0x00007ffeea147f18   rbx = 0x0000000000000002
[task 2021-09-24T15:27:01.308Z] 15:27:01     INFO -     rsi = 0x0000520100005300   rdi = 0x0000000117413530
[task 2021-09-24T15:27:01.308Z] 15:27:01     INFO -     rbp = 0x00007ffeea147fb0   rsp = 0x00007ffeea147f18
[task 2021-09-24T15:27:01.308Z] 15:27:01     INFO -      r8 = 0x0000000000000000    r9 = 0x00000000000000a4
[task 2021-09-24T15:27:01.308Z] 15:27:01     INFO -     r10 = 0x0000000000000000   r11 = 0x0000000000000202
[task 2021-09-24T15:27:01.308Z] 15:27:01     INFO -     r12 = 0x0000000117413530   r13 = 0x0000000000000016
[task 2021-09-24T15:27:01.309Z] 15:27:01     INFO -     r14 = 0x0000520100005300   r15 = 0x00000001087fcdc0
[task 2021-09-24T15:27:01.309Z] 15:27:01     INFO -     rip = 0x00007fff68548882
[task 2021-09-24T15:27:01.309Z] 15:27:01     INFO -     Found by: given as instruction pointer in context
[task 2021-09-24T15:27:01.309Z] 15:27:01     INFO -  1  libmozglue.dylib!mozilla::detail::ConditionVariableImpl::wait_for(mozilla::detail::MutexImpl&, mozilla::BaseTimeDuration<mozilla::TimeDurationValueCalculator> const&) [ConditionVariable_posix.cpp:4e8c68a872fe1601f9f05a2ddc43b1335413a383 : 115 + 0xb]
[task 2021-09-24T15:27:01.309Z] 15:27:01     INFO -     rbp = 0x00007ffeea148000   rsp = 0x00007ffeea147fc0
[task 2021-09-24T15:27:01.309Z] 15:27:01     INFO -     rip = 0x0000000105b2a0fd
[task 2021-09-24T15:27:01.309Z] 15:27:01     INFO -     Found by: previous frame's frame pointer
[task 2021-09-24T15:27:01.309Z] 15:27:01     INFO -  2  XUL!mozilla::OffTheBooksCondVar::Wait(mozilla::BaseTimeDuration<mozilla::TimeDurationValueCalculator>) [BlockingResourceBase.cpp:4e8c68a872fe1601f9f05a2ddc43b1335413a383 : 534 + 0x11]
[task 2021-09-24T15:27:01.309Z] 15:27:01     INFO -     rbx = 0x0000000117413510   rbp = 0x00007ffeea148050
[task 2021-09-24T15:27:01.309Z] 15:27:01     INFO -     rsp = 0x00007ffeea148010   r12 = 0x0000000000000000
[task 2021-09-24T15:27:01.309Z] 15:27:01     INFO -     r13 = 0x0000000105d49120   r14 = 0x0000000105d49120
[task 2021-09-24T15:27:01.309Z] 15:27:01     INFO -     r15 = 0x0000000117413501   rip = 0x00000001089b5c88
[task 2021-09-24T15:27:01.309Z] 15:27:01     INFO -     Found by: call frame info
[task 2021-09-24T15:27:01.309Z] 15:27:01     INFO -  3  XUL!mozilla::OffTheBooksCondVar::Wait() [BlockingResourceBase.cpp:4e8c68a872fe1601f9f05a2ddc43b1335413a383 : 514 + 0x5]
[task 2021-09-24T15:27:01.309Z] 15:27:01     INFO -     rbx = 0x0000000117413400   rbp = 0x00007ffeea148060
[task 2021-09-24T15:27:01.309Z] 15:27:01     INFO -     rsp = 0x00007ffeea148060   r12 = 0x0000000111f7359e
[task 2021-09-24T15:27:01.309Z] 15:27:01     INFO -     r13 = 0x00007ffeea148078   r14 = 0xaaaaaaaaaaaaaaaa
[task 2021-09-24T15:27:01.309Z] 15:27:01     INFO -     r15 = 0x0000000117413510   rip = 0x00000001089b5bb3
[task 2021-09-24T15:27:01.309Z] 15:27:01     INFO -     Found by: call frame info
[task 2021-09-24T15:27:01.309Z] 15:27:01     INFO -  4  XUL!mozilla::TaskController::GetRunnableForMTTask(bool) [TaskController.cpp:4e8c68a872fe1601f9f05a2ddc43b1335413a383 : 515 + 0x8]
[task 2021-09-24T15:27:01.309Z] 15:27:01     INFO -     rbx = 0x0000000117413400   rbp = 0x00007ffeea1480b0
[task 2021-09-24T15:27:01.309Z] 15:27:01     INFO -     rsp = 0x00007ffeea148070   r12 = 0x0000000111f7359e
[task 2021-09-24T15:27:01.309Z] 15:27:01     INFO -     r13 = 0x00007ffeea148078   r14 = 0xaaaaaaaaaaaaaaaa
[task 2021-09-24T15:27:01.309Z] 15:27:01     INFO -     r15 = 0x0000000117413510   rip = 0x00000001089c6e88
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     Found by: call frame info
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -  5  XUL!nsThread::ProcessNextEvent(bool, bool*) [nsThread.cpp:4e8c68a872fe1601f9f05a2ddc43b1335413a383 : 1082 + 0x19]
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     rbx = 0x0000000000000001   rbp = 0x00007ffeea1481d0
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     rsp = 0x00007ffeea1480c0   r12 = 0x0000000000000001
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     r13 = 0x0000000105d21320   r14 = 0x000000011cb66b01
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     r15 = 0x000000011e9a5601   rip = 0x00000001089d9f31
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     Found by: call frame info
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -  6  XUL!NS_ProcessNextEvent(nsIThread*, bool) [nsThreadUtils.cpp:4e8c68a872fe1601f9f05a2ddc43b1335413a383 : 466 + 0xd]
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     rbx = 0x0000000000000001   rbp = 0x00007ffeea148200
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     rsp = 0x00007ffeea1481e0   r12 = 0x000000011e9fd100
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     r13 = 0x0000000000000000   r14 = 0x000000011cb66b60
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     r15 = 0x000000011e9a5680   rip = 0x00000001089e0ddc
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     Found by: call frame info
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -  7  XUL!mozilla::layers::CompositorThreadHolder::Shutdown() [CompositorThread.cpp:4e8c68a872fe1601f9f05a2ddc43b1335413a383 : 141 + 0x47]
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     rbx = 0x0000000105d21320   rbp = 0x00007ffeea148250
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     rsp = 0x00007ffeea148210   r12 = 0x000000011e9fd100
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     r13 = 0x0000000000000000   r14 = 0x000000011cb66b60
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     r15 = 0x000000011e9a5680   rip = 0x0000000109baeca7
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     Found by: call frame info
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -  8  XUL!gfxPlatform::ShutdownLayersIPC() [gfxPlatform.cpp:4e8c68a872fe1601f9f05a2ddc43b1335413a383 : 1297 + 0x5]
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     rbx = 0x00007ffeea1482b0   rbp = 0x00007ffeea148280
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     rsp = 0x00007ffeea148260   r12 = 0x0000000105d8bc40
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     r13 = 0x0000000000000000   r14 = 0x0000000105d61908
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     r15 = 0x00007ffeea148450   rip = 0x0000000109c6a09f
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     Found by: call frame info
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -  9  XUL!mozilla::ShutdownXPCOM(nsIServiceManager*) [XPCOMInit.cpp:4e8c68a872fe1601f9f05a2ddc43b1335413a383 : 622 + 0x5]
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     rbx = 0x00007ffeea1482b0   rbp = 0x00007ffeea148320
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     rsp = 0x00007ffeea148290   r12 = 0x0000000105d8bc40
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     r13 = 0x0000000000000000   r14 = 0x0000000105d61908
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     r15 = 0x00007ffeea148450   rip = 0x0000000108a2837d
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     Found by: call frame info
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO - 10  XUL!ScopedXPCOMStartup::~ScopedXPCOMStartup() [nsAppRunner.cpp:4e8c68a872fe1601f9f05a2ddc43b1335413a383 : 1683 + 0x8]
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     rbx = 0xaaaaaaaaaaaaaaaa   rbp = 0x00007ffeea1483b0
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     rsp = 0x00007ffeea148330   r12 = 0x0000000105d8bc40
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     r13 = 0x0000000000000000   r14 = 0x0000000105d16500
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     r15 = 0x00007ffeea148450   rip = 0x000000010e3f8829
[task 2021-09-24T15:27:01.310Z] 15:27:01     INFO -     Found by: call frame info
[task 2021-09-24T15:27:01.311Z] 15:27:01     INFO - 11  XUL!XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) [nsAppRunner.cpp:4e8c68a872fe1601f9f05a2ddc43b1335413a383 : 5493 + 0x19]
[task 2021-09-24T15:27:01.311Z] 15:27:01     INFO -     rbx = 0x0000000105d16500   rbp = 0x00007ffeea148440
[task 2021-09-24T15:27:01.311Z] 15:27:01     INFO -     rsp = 0x00007ffeea1483c0   r12 = 0x0000000105d8bc40
[task 2021-09-24T15:27:01.311Z] 15:27:01     INFO -     r13 = 0x0000000000000000   r14 = 0x0000000000000000
[task 2021-09-24T15:27:01.311Z] 15:27:01     INFO -     r15 = 0x00007ffeea148450   rip = 0x000000010e4072a9
[task 2021-09-24T15:27:01.311Z] 15:27:01     INFO -     Found by: call frame info
[task 2021-09-24T15:27:01.311Z] 15:27:01     INFO - 12  XUL!XRE_main(int, char**, mozilla::BootstrapConfig const&) [nsAppRunner.cpp:4e8c68a872fe1601f9f05a2ddc43b1335413a383 : 5535 + 0x10]
[task 2021-09-24T15:27:01.311Z] 15:27:01     INFO -     rbx = 0x0000000000000005   rbp = 0x00007ffeea148580
[task 2021-09-24T15:27:01.311Z] 15:27:01     INFO -     rsp = 0x00007ffeea148450   r12 = 0x00007ffeea148450
[task 2021-09-24T15:27:01.311Z] 15:27:01     INFO -     r13 = 0x00007ffeea148a18   r14 = 0x00007ffeea1485c0
[task 2021-09-24T15:27:01.311Z] 15:27:01     INFO -     r15 = 0x00007ffeea148a18   rip = 0x000000010e407b25
[task 2021-09-24T15:27:01.311Z] 15:27:01     INFO -     Found by: call frame info
[task 2021-09-24T15:27:01.311Z] 15:27:01     INFO - 13  firefox!main [nsBrowserApp.cpp:4e8c68a872fe1601f9f05a2ddc43b1335413a383 : 386 + 0x29d]
[task 2021-09-24T15:27:01.311Z] 15:27:01     INFO -     rbx = 0x0000000105ab9331   rbp = 0x00007ffeea1489f0
[task 2021-09-24T15:27:01.311Z] 15:27:01     INFO -     rsp = 0x00007ffeea148590   r12 = 0x000002711f5b9e6d
[task 2021-09-24T15:27:01.311Z] 15:27:01     INFO -     r13 = 0x00007ffeea148a18   r14 = 0x00007ffeea148d31
[task 2021-09-24T15:27:01.311Z] 15:27:01     INFO -     r15 = 0x0000000000000005   rip = 0x0000000105ab72d7
[task 2021-09-24T15:27:01.311Z] 15:27:01     INFO -     Found by: call frame info
[task 2021-09-24T15:27:01.311Z] 15:27:01     INFO - 14  libdyld.dylib!start + 0x1
[task 2021-09-24T15:27:01.311Z] 15:27:01     INFO -     rbx = 0x0000000000000000   rbp = 0x00007ffeea148a08
[task 2021-09-24T15:27:01.311Z] 15:27:01     INFO -     rsp = 0x00007ffeea148a00   r12 = 0x0000000000000000
[task 2021-09-24T15:27:01.311Z] 15:27:01     INFO -     r13 = 0x0000000000000000   r14 = 0x0000000000000000
[task 2021-09-24T15:27:01.311Z] 15:27:01     INFO -     r15 = 0x0000000000000000   rip = 0x00007fff68404cc9
[task 2021-09-24T15:27:01.311Z] 15:27:01     INFO -     Found by: call frame info
[task 2021-09-24T15:27:01.311Z] 15:27:01     INFO - 15  libdyld.dylib!start + 0x1
[task 2021-09-24T15:27:01.311Z] 15:27:01     INFO -     rbx = 0x0000000000000000   rbp = 0x00007ffeea148a08
[task 2021-09-24T15:27:01.311Z] 15:27:01     INFO -     rsp = 0x00007ffeea148a08   r12 = 0x0000000000000000
[task 2021-09-24T15:27:01.311Z] 15:27:01     INFO -     r13 = 0x0000000000000000   r14 = 0x0000000000000000
[task 2021-09-24T15:27:01.311Z] 15:27:01     INFO -     r15 = 0x0000000000000000   rip = 0x00007fff68404cc9
[task 2021-09-24T15:27:01.312Z] 15:27:01     INFO -     Found by: call frame info

Jed, are you working on this?

Flags: needinfo?(jld)
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]

(In reply to Andreea Pavel [:apavel] from comment #29)

Jed, are you working on this?

Based on Jed's comment 2 and comment 15, I think this is a bug for the Graphics team. A process seems to be OOM'ing and then deadlocking in CompositorThreadHolder::Shutdown().

(In reply to Intermittent Failures Robot from comment #36)

109 failures in 3376 pushes (0.032 failures/push) were associated with this bug in the last 7 days.

This is the #7 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

Jim, this intermittent test failure (a shutdown hang) looks like a graphics issue. It started in July and then the failure volume increased after some spikes in September.

status-firefox95: --- → affected
Flags: needinfo?(jld) → needinfo?(jmathies)
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]

There are 125 total failures in the last 7 days on

  • macosx1015-64-qr debug
  • linux1804-64-qr debug
  • linux1804-64-ccov-qr opt

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=355054253&repo=mozilla-central&lineNumber=4239

[task 2021-10-17T02:12:32.655Z] 02:12:32     INFO - TEST-START | dom/fetch/tests/test_invalid_header_exception.html
[task 2021-10-17T02:12:32.696Z] 02:12:32     INFO - GECKO(2717) | [Child 2909, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:527
[task 2021-10-17T02:12:32.868Z] 02:12:32     INFO - GECKO(2717) | [Child 2909, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:527
[task 2021-10-17T02:12:32.924Z] 02:12:32     INFO - GECKO(2717) | [Child 2909, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:527
[task 2021-10-17T02:12:32.948Z] 02:12:32     INFO - GECKO(2717) | [Child 2909, Main Thread] WARNING: Extra shutdown CC: 'i < NORMAL_SHUTDOWN_COLLECTIONS', file /builds/worker/checkouts/gecko/xpcom/base/nsCycleCollector.cpp:3353
[task 2021-10-17T02:12:32.970Z] 02:12:32     INFO - GECKO(2717) | [Parent 2717, Main Thread] WARNING: WebProgress Ignored: no longer current window global: file /builds/worker/checkouts/gecko/dom/ipc/BrowserParent.cpp:2976
[task 2021-10-17T02:12:32.973Z] 02:12:32     INFO - GECKO(2717) | [Child 2909, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:527
[task 2021-10-17T02:12:32.987Z] 02:12:32     INFO - GECKO(2717) | [Child 2909, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:527
[task 2021-10-17T02:12:32.990Z] 02:12:32     INFO - GECKO(2717) | [Child 2909, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:527
[task 2021-10-17T02:12:33.005Z] 02:12:33     INFO - GECKO(2717) | ### XPCOM_MEM_BLOAT_LOG defined -- logging bloat/leaks to /tmp/tmpnyszzx14.mozrunner/runtests_leaks_tab_pid3010.log
[task 2021-10-17T02:12:33.008Z] 02:12:33     INFO - GECKO(2717) | [3010, Main Thread] WARNING: XPCOM_MEM_BLOAT_LOG is set, disabling native allocations.: file /builds/worker/checkouts/gecko/tools/profiler/core/platform.cpp:251
[task 2021-10-17T02:12:33.016Z] 02:12:33     INFO - GECKO(2717) | [Child 2909, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:527
[task 2021-10-17T02:12:33.030Z] 02:12:33     INFO - GECKO(2717) | [Child 2909, Main Thread] WARNING: NS_ENSURE_TRUE(Preferences::InitStaticMembers()) failed: file /builds/worker/checkouts/gecko/modules/libpref/Preferences.cpp:4439
[task 2021-10-17T02:12:33.111Z] 02:12:33     INFO - GECKO(2717) | [Child 3010, Main Thread] WARNING: could not set real-time limit in CubebUtils::InitLibrary: file /builds/worker/checkouts/gecko/dom/media/CubebUtils.cpp:639
[task 2021-10-17T02:12:33.123Z] 02:12:33     INFO - GECKO(2717) | [Child 2940, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:527
[task 2021-10-17T02:12:33.124Z] 02:12:33     INFO - GECKO(2717) | [Child 2940, Main Thread] WARNING: '!ipcActor->SendRecordDiscardedData(discardedData)', file /builds/worker/checkouts/gecko/toolkit/components/telemetry/core/ipc/TelemetryIPCAccumulator.cpp:298
[task 2021-10-17T02:12:33.238Z] 02:12:33     INFO - GECKO(2717) | ### XPCOM_MEM_BLOAT_LOG defined -- logging bloat/leaks to /tmp/tmpnyszzx14.mozrunner/runtests_leaks_tab_pid3016.log
[task 2021-10-17T02:12:33.265Z] 02:12:33     INFO - GECKO(2717) | [3016, Main Thread] WARNING: XPCOM_MEM_BLOAT_LOG is set, disabling native allocations.: file /builds/worker/checkouts/gecko/tools/profiler/core/platform.cpp:251
[task 2021-10-17T02:12:33.379Z] 02:12:33     INFO - GECKO(2717) | [Child 3016, Main Thread] WARNING: could not set real-time limit in CubebUtils::InitLibrary: file /builds/worker/checkouts/gecko/dom/media/CubebUtils.cpp:639
[task 2021-10-17T02:12:35.217Z] 02:12:35     INFO - GECKO(2717) | MEMORY STAT | vsize 2518MB | residentFast 136MB | heapAllocated 8MB
[task 2021-10-17T02:12:35.259Z] 02:12:35     INFO - TEST-OK | dom/fetch/tests/test_invalid_header_exception.html | took 2604ms
[task 2021-10-17T02:12:35.401Z] 02:12:35     INFO - GECKO(2717) | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2021-10-17T02:12:35.403Z] 02:12:35     INFO - GECKO(2717) | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2021-10-17T02:12:35.406Z] 02:12:35     INFO - GECKO(2717) | [Child 2943, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:527
[task 2021-10-17T02:12:35.472Z] 02:12:35     INFO - GECKO(2717) | [Child 2943, Main Thread] WARNING: IPC message discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:527
[task 2021-10-17T02:12:35.547Z] 02:12:35     INFO - TEST-START | Shutdown

[task 2021-10-17T02:25:35.664Z] 02:25:35     INFO - GECKO(2717) | out of memory: 0x0000000028700000 bytes requested
[task 2021-10-17T02:25:35.665Z] 02:25:35     INFO - GECKO(2717) | Hit MOZ_CRASH() at /builds/worker/checkouts/gecko/memory/mozalloc/mozalloc_abort.cpp:33
[task 2021-10-17T02:25:35.845Z] 02:25:35     INFO - GECKO(2717) | #01: mozalloc_abort [memory/mozalloc/mozalloc_abort.cpp:33]
[task 2021-10-17T02:25:35.846Z] 02:25:35     INFO - GECKO(2717) | #02: mozalloc_handle_oom(unsigned long) [memory/mozalloc/mozalloc_oom.cpp:51]
[task 2021-10-17T02:25:35.848Z] 02:25:35     INFO - GECKO(2717) | #03: moz_xstrdup [memory/mozalloc/mozalloc.cpp:80]
[task 2021-10-17T02:25:35.849Z] 02:25:35     INFO - GECKO(2717) | #04: nsTArrayInfallibleAllocator::ResultTypeProxy nsTArray_base<nsTArrayInfallibleAllocator, nsTArray_RelocateUsingMemutils>::EnsureCapacity<nsTArrayInfallibleAllocator>(unsigned long, unsigned long) [xpcom/ds/nsTArray-inl.h:223]
[task 2021-10-17T02:25:35.851Z] 02:25:35     INFO - GECKO(2717) | #05: mozilla::net::ChannelEventQueue::RunOrEnqueue(mozilla::net::ChannelEvent*, bool) [netwerk/ipc/ChannelEventQueue.h:218]
[task 2021-10-17T02:25:35.853Z] 02:25:35     INFO - GECKO(2717) | #06: mozilla::net::HttpChannelChild::DoNotifyListener() [netwerk/protocol/http/HttpChannelChild.cpp:1218]
[task 2021-10-17T02:25:35.854Z] 02:25:35     INFO - GECKO(2717) | #07: mozilla::detail::RunnableMethodImpl<RefPtr<mozilla::net::HttpChannelChild>, void (mozilla::net::HttpChannelChild::*)(), true, (mozilla::RunnableKind)0, >::Run() [xpcom/threads/nsThreadUtils.h:1203]
[task 2021-10-17T02:25:35.855Z] 02:25:35     INFO - GECKO(2717) | #08: mozilla::RunnableTask::Run() [xpcom/threads/TaskController.cpp:468]
[task 2021-10-17T02:25:35.856Z] 02:25:35     INFO - GECKO(2717) | #09: mozilla::TaskController::DoExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) [xpcom/threads/TaskController.cpp:770]
[task 2021-10-17T02:25:35.857Z] 02:25:35     INFO - GECKO(2717) | #10: mozilla::TaskController::ExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) [xpcom/threads/TaskController.cpp:606]
[task 2021-10-17T02:25:35.858Z] 02:25:35     INFO - GECKO(2717) | #11: mozilla::TaskController::ProcessPendingMTTask(bool) [xpcom/threads/TaskController.cpp:390]
[task 2021-10-17T02:25:35.859Z] 02:25:35     INFO - GECKO(2717) | #12: mozilla::detail::RunnableFunction<mozilla::TaskController::InitializeInternal()::$_0>::Run() [xpcom/threads/nsThreadUtils.h:532]
[task 2021-10-17T02:25:35.859Z] 02:25:35     INFO - GECKO(2717) | #13: nsThread::ProcessNextEvent(bool, bool*) [xpcom/threads/nsThread.cpp:1155]
[task 2021-10-17T02:25:35.860Z] 02:25:35     INFO - GECKO(2717) | #14: NS_ProcessNextEvent(nsIThread*, bool) [xpcom/threads/nsThreadUtils.cpp:467]
[task 2021-10-17T02:25:35.861Z] 02:25:35     INFO - GECKO(2717) | #15: mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) [ipc/glue/MessagePump.cpp:85]
[task 2021-10-17T02:25:35.862Z] 02:25:35     INFO - GECKO(2717) | #16: MessageLoop::RunInternal() [ipc/chromium/src/base/message_loop.cc:331]
[task 2021-10-17T02:25:35.863Z] 02:25:35     INFO - GECKO(2717) | #17: MessageLoop::Run() [ipc/chromium/src/base/message_loop.cc:307]
[task 2021-10-17T02:25:35.863Z] 02:25:35     INFO - GECKO(2717) | #18: nsBaseAppShell::Run() [widget/nsBaseAppShell.cpp:139]
[task 2021-10-17T02:25:35.864Z] 02:25:35     INFO - GECKO(2717) | #19: XRE_RunAppShell() [toolkit/xre/nsEmbedFunctions.cpp:917]
[task 2021-10-17T02:25:35.865Z] 02:25:35     INFO - GECKO(2717) | #20: mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate*) [ipc/glue/MessagePump.cpp:235]
[task 2021-10-17T02:25:35.866Z] 02:25:35     INFO - GECKO(2717) | #21: MessageLoop::RunInternal() [ipc/chromium/src/base/message_loop.cc:331]
[task 2021-10-17T02:25:35.866Z] 02:25:35     INFO - GECKO(2717) | #22: MessageLoop::Run() [ipc/chromium/src/base/message_loop.cc:307]
[task 2021-10-17T02:25:35.867Z] 02:25:35     INFO - GECKO(2717) | #23: XRE_InitChildProcess(int, char**, XREChildData const*) [toolkit/xre/nsEmbedFunctions.cpp:749]
[task 2021-10-17T02:25:35.868Z] 02:25:35     INFO - GECKO(2717) | #24: content_process_main(mozilla::Bootstrap*, int, char**) [ipc/contentproc/plugin-container.cpp:58]
[task 2021-10-17T02:25:35.869Z] 02:25:35     INFO - GECKO(2717) | #25: main [browser/app/nsBrowserApp.cpp:327]
[task 2021-10-17T02:25:35.869Z] 02:25:35     INFO - GECKO(2717) | #26: __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6 + 0x21b97]
[task 2021-10-17T02:25:35.870Z] 02:25:35     INFO - GECKO(2717) | #27: ??? [/builds/worker/workspace/build/application/firefox/firefox-bin + 0x42089]
[task 2021-10-17T02:25:35.871Z] 02:25:35     INFO - GECKO(2717) | #28: ??? (???:???)
[task 2021-10-17T02:25:36.086Z] 02:25:36     INFO - TEST-INFO | Main app process: exit 11
[task 2021-10-17T02:25:36.089Z] 02:25:36     INFO - Buffered messages finished
[task 2021-10-17T02:25:36.090Z] 02:25:36    ERROR - TEST-UNEXPECTED-FAIL | Last test finished | application terminated with exit code 11
[task 2021-10-17T02:25:36.094Z] 02:25:36     INFO - runtests.py | Application ran for: 0:13:18.845999

[task 2021-10-17T02:25:45.863Z] 02:25:45     INFO - PROCESS-CRASH | Last test finished | application crashed [@ _pt_root]
[task 2021-10-17T02:25:45.863Z] 02:25:45     INFO - Mozilla crash reason: MOZ_CRASH(Shutdown hanging after all known phases and workers finished.)
[task 2021-10-17T02:25:45.863Z] 02:25:45     INFO - Crash dump filename: /tmp/tmpnyszzx14.mozrunner/minidumps/4aab7ed2-6256-f409-b029-066d41325728.dmp
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO - Operating system: Linux
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -                   0.0.0 Linux 4.4.0-1014-aws #14taskcluster1-Ubuntu SMP Tue Apr 3 10:27:00 UTC 2018 x86_64
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO - CPU: amd64
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -      family 6 model 85 stepping 4
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -      2 CPUs
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO - 
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO - GPU: UNKNOWN
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO - 
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO - Crash reason:  SIGSEGV / SEGV_MAPERR
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO - Crash address: 0x0
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO - Process uptime: not available
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO - 
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO - Thread 39 tid 3051 (crashed) 0  libxul.so!mozilla::(anonymous namespace)::RunWatchdog(void*) [nsTerminator.cpp:e361209af61f51a3fa1dbf9ea8a2d276e22cc5bf : 256 + 0x22]
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -     rax = 0x00007f50d080b810   rdx = 0x0000000000000000
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -     rcx = 0x00005574bea3c780   rbx = 0x00007f50a02fee20
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -     rsi = 0x00007f50e44f88b0   rdi = 0x00007f50e44f7680
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -     rbp = 0x00007f50a02fee60   rsp = 0x00007f50a02fee20
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -      r8 = 0x00007f50e44f88b0    r9 = 0x00007f50a02ff700
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -     r10 = 0x0000000000000000   r11 = 0x0000000000000000
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -     r12 = 0x00007f50a02ff630   r13 = 0x0000000000000007
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -     r14 = 0x00007f50a02fee40   r15 = 0x00007f50a28c50b0
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -     rip = 0x00007f50d780147a
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -     Found by: given as instruction pointer in context
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -  1  libnspr4.so!_pt_root [ptthread.c:e361209af61f51a3fa1dbf9ea8a2d276e22cc5bf : 201 + 0x7]
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -     rbx = 0x00007f50a377b040   rbp = 0x00007f50a02feeb0
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -     rsp = 0x00007f50a02fee70   r12 = 0x00007f50a02ff630
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -     r13 = 0x0000000000000007   r14 = 0x0000000000000beb
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -     r15 = 0x00007f50a02ff700   rip = 0x00007f50e53d73f3
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -     Found by: call frame info
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -  2  libpthread.so.0 + 0x76db
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -     rbx = 0x0000000000000000   rbp = 0x0000000000000000
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -     rsp = 0x00007f50a02feec0   r12 = 0x00007f50a02fef80
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -     r13 = 0x0000000000000000   r14 = 0x00007f50a377b040
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -     r15 = 0x00007fffcbb462f0   rip = 0x00007f50e50466db
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -     Found by: call frame info
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -  3  libc.so.6 + 0x121a3f
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -     rsp = 0x00007f50a02fef80   rip = 0x00007f50e422ca3f
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -     Found by: stack scanning
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -  0  libpthread.so.0 + 0xd9f3
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -     rax = 0xfffffffffffffe00   rdx = 0x0000000000000000
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -     rcx = 0x00007f50e504c9f3   rbx = 0x00007f50cd55eb00
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -     rsi = 0x0000000000000080   rdi = 0x00007f50cd55eb2c
[task 2021-10-17T02:25:45.864Z] 02:25:45     INFO -     rbp = 0x00007f50cd55eb24   rsp = 0x00007fffcbb495f0
[task 2021-10-17T02:25:45.865Z] 02:25:45     INFO -      r8 = 0x0000000000000001    r9 = 0x0000000000000000
[task 2021-10-17T02:25:45.865Z] 02:25:45     INFO -     r10 = 0x0000000000000000   r11 = 0x0000000000000246
[task 2021-10-17T02:25:45.865Z] 02:25:45     INFO -     r12 = 0x00007f50cd55eb2c   r13 = 0x0000000000000000
[task 2021-10-17T02:25:45.865Z] 02:25:45     INFO -     r14 = 0x00007f50cd55ea00   r15 = 0x00000000000000c3
[task 2021-10-17T02:25:45.865Z] 02:25:45     INFO -     rip = 0x00007f50e504c9f3
[task 2021-10-17T02:25:45.865Z] 02:25:45     INFO -     Found by: given as instruction pointer in context
[task 2021-10-17T02:25:45.865Z] 02:25:45     INFO -  1  libgdk-3.so.0 + 0x67fd6
[task 2021-10-17T02:25:45.865Z] 02:25:45     INFO -     rsp = 0x00007fffcbb49630   rip = 0x00007f50e2775fd6
[task 2021-10-17T02:25:45.865Z] 02:25:45     INFO -     Found by: stack scanning
[task 2021-10-17T02:25:45.865Z] 02:25:45     INFO -  2  libpthread.so.0 + 0xd690
[task 2021-10-17T02:25:45.865Z] 02:25:45     INFO -     rsp = 0x00007fffcbb49638   rip = 0x00007f50e504c690
[task 2021-10-17T02:25:45.865Z] 02:25:45     INFO -     Found by: stack scanning
[task 2021-10-17T02:25:45.865Z] 02:25:45     INFO -  3  firefox-bin!mozilla::detail::ConditionVariableImpl::wait(mozilla::detail::MutexImpl&) [ConditionVariable_posix.cpp:e361209af61f51a3fa1dbf9ea8a2d276e22cc5bf : 108 + 0xb]
[task 2021-10-17T02:25:45.865Z] 02:25:45     INFO -     rsp = 0x00007fffcbb496c0   rip = 0x00005574bea03089
[task 2021-10-17T02:25:45.866Z] 02:25:45     INFO -     Found by: stack scanning
[task 2021-10-17T02:25:45.866Z] 02:25:45     INFO -  4  firefox-bin!mozilla::detail::ConditionVariableImpl::wait_for(mozilla::detail::MutexImpl&, mozilla::BaseTimeDuration<mozilla::TimeDurationValueCalculator> const&) [ConditionVariable_posix.cpp:e361209af61f51a3fa1dbf9ea8a2d276e22cc5bf : 115 + 0x8]
[task 2021-10-17T02:25:45.866Z] 02:25:45     INFO -     rsp = 0x00007fffcbb496e0   rip = 0x00005574bea03104
[task 2021-10-17T02:25:45.867Z] 02:25:45     INFO -     Found by: stack scanning
[task 2021-10-17T02:25:45.867Z] 02:25:45     INFO -  5  ld-linux-x86-64.so.2 + 0x138cc
[task 2021-10-17T02:25:45.867Z] 02:25:45     INFO -     rsp = 0x00007fffcbb49710   rip = 0x00007f50e52718cc
[task 2021-10-17T02:25:45.867Z] 02:25:45     INFO -     Found by: stack scanning
[task 2021-10-17T02:25:45.867Z] 02:25:45     INFO -  6  libxul.so!mozilla::OffTheBooksCondVar::Wait(mozilla::BaseTimeDuration<mozilla::TimeDurationValueCalculator>) [BlockingResourceBase.cpp:e361209af61f51a3fa1dbf9ea8a2d276e22cc5bf : 534 + 0x10]
[task 2021-10-17T02:25:45.867Z] 02:25:45     INFO -     rsp = 0x00007fffcbb49750   rip = 0x00007f50d35c118a
[task 2021-10-17T02:25:45.868Z] 02:25:45     INFO -     Found by: stack scanning
[task 2021-10-17T02:25:45.868Z] 02:25:45     INFO -  7  libxul.so!mozilla::OffTheBooksCondVar::Wait() [BlockingResourceBase.cpp:e361209af61f51a3fa1dbf9ea8a2d276e22cc5bf : 514 + 0x5]
[task 2021-10-17T02:25:45.868Z] 02:25:45     INFO -     rsp = 0x00007fffcbb497a0   rip = 0x00007f50d35c10df
[task 2021-10-17T02:25:45.868Z] 02:25:45     INFO -     Found by: stack scanning
[task 2021-10-17T02:25:45.868Z] 02:25:45     INFO -  8  libxul.so!mozilla::TaskController::GetRunnableForMTTask(bool) [TaskController.cpp:e361209af61f51a3fa1dbf9ea8a2d276e22cc5bf : 515 + 0x8]
[task 2021-10-17T02:25:45.868Z] 02:25:45     INFO -     rsp = 0x00007fffcbb497b0   rip = 0x00007f50d35cf0d7
[task 2021-10-17T02:25:45.868Z] 02:25:45     INFO -     Found by: stack scanning
[task 2021-10-17T02:25:45.869Z] 02:25:45     INFO -  9  libxul.so!nsThread::ProcessNextEvent(bool, bool*) [nsThread.cpp:e361209af61f51a3fa1dbf9ea8a2d276e22cc5bf : 1085 + 0x11]
[task 2021-10-17T02:25:45.869Z] 02:25:45     INFO -     rsp = 0x00007fffcbb49800   rip = 0x00007f50d35dca71
[task 2021-10-17T02:25:45.869Z] 02:25:45     INFO -     Found by: stack scanning
[task 2021-10-17T02:25:45.869Z] 02:25:45     INFO - 10  libxul.so!<.plt ELF section in libxul.so> + 0x56594a
[task 2021-10-17T02:25:45.869Z] 02:25:45     INFO -     rsp = 0x00007fffcbb49830   rip = 0x00007f50da4a00aa
[task 2021-10-17T02:25:45.869Z] 02:25:45     INFO -     Found by: stack scanning
[task 2021-10-17T02:25:45.869Z] 02:25:45     INFO - 11  ld-linux-x86-64.so.2 + 0x138cc
[task 2021-10-17T02:25:45.870Z] 02:25:45     INFO -     rsp = 0x00007fffcbb498c0   rip = 0x00007f50e52718cc
[task 2021-10-17T02:25:45.870Z] 02:25:45     INFO -     Found by: stack scanning
[task 2021-10-17T02:25:45.870Z] 02:25:45     INFO - 12  ld-linux-x86-64.so.2 + 0x19a58
[task 2021-10-17T02:25:45.870Z] 02:25:45     INFO -     rsp = 0x00007fffcbb498d0   rip = 0x00007f50e5277a58
[task 2021-10-17T02:25:45.870Z] 02:25:45     INFO -     Found by: stack scanning
[task 2021-10-17T02:25:45.870Z] 02:25:45     INFO - 13  libxul.so!NS_IsMainThread [nsThreadManager.cpp:e361209af61f51a3fa1dbf9ea8a2d276e22cc5bf : 224 + 0x10]
[task 2021-10-17T02:25:45.870Z] 02:25:45     INFO -     rsp = 0x00007fffcbb498e0   rip = 0x00007f50d35ceb2c
[task 2021-10-17T02:25:45.870Z] 02:25:45     INFO -     Found by: stack scanning
[task 2021-10-17T02:25:45.870Z] 02:25:45     INFO - 14  libxul.so!nsIXPConnect::XPConnect() [nsXPConnect.cpp:e361209af61f51a3fa1dbf9ea8a2d276e22cc5bf : 926 + 0x5]
[task 2021-10-17T02:25:45.870Z] 02:25:45     INFO -     rsp = 0x00007fffcbb498f0   rip = 0x00007f50d406d37f
[task 2021-10-17T02:25:45.870Z] 02:25:45     INFO -     Found by: stack scanning
[task 2021-10-17T02:25:45.871Z] 02:25:45     INFO - 15  libxul.so!XPCJSContext::RecordScriptActivity(bool) [XPCJSContext.cpp:e361209af61f51a3fa1dbf9ea8a2d276e22cc5bf : 551 + 0x5]
[task 2021-10-17T02:25:45.871Z] 02:25:45     INFO -     rsp = 0x00007fffcbb49900   rip = 0x00007f50d4040ec0
[task 2021-10-17T02:25:45.871Z] 02:25:45     INFO -     Found by: stack scanning
[task 2021-10-17T02:25:45.871Z] 02:25:45     INFO - 16  libxul.so!NS_ProcessNextEvent(nsIThread*, bool) [nsThreadUtils.cpp:e361209af61f51a3fa1dbf9ea8a2d276e22cc5bf : 467 + 0xc]
[task 2021-10-17T02:25:45.871Z] 02:25:45     INFO -     rsp = 0x00007fffcbb49940   rip = 0x00007f50d35e1850
[task 2021-10-17T02:25:45.871Z] 02:25:45     INFO -     Found by: stack scanning
[task 2021-10-17T02:25:45.871Z] 02:25:45     INFO - 17  libxul.so!mozilla::layers::CompositorThreadHolder::Shutdown() [CompositorThread.cpp:e361209af61f51a3fa1dbf9ea8a2d276e22cc5bf : 141 + 0x1fa]
[task 2021-10-17T02:25:45.871Z] 02:25:45     INFO -     rsp = 0x00007fffcbb49970   rip = 0x00007f50d4367688
[task 2021-10-17T02:25:45.871Z] 02:25:45     INFO -     Found by: stack scanning

Jim, are there any updates here?

Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]

There have been 112 total failures in the last 7 days, recent failure log.
Affected platforms are:

  • macosx1015-64-qr
  • linux1804-64-qr
  • linux1804-64-ccov-qr
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]

There have been 54 total failures in the last 7 days, recent failure log.
Affected platforms are:

  • linux1804-64-ccov-qr
  • linux1804-64-qr
  • macosx1015-64-qr
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]

Andrew, any thoughts here? Some sort of failure in proper shutdown after an OOM.

Blocks: gfx-triage
Flags: needinfo?(jmathies) → needinfo?(aosmond)
Priority: -- → P3

This appears to only happen on Linux and OSX. Not on Windows (or Android). Possibly related to not having a GPU process. Doesn't matter whether it is a debug build or opt although more seem common on debug builds. We don't explicitly close CompositorManagerParent protocols on debug builds, but I don't remember why:

https://searchfox.org/mozilla-central/rev/84c1031f19b2cf8221da0850983f8d0269f6efd1/gfx/layers/ipc/CompositorManagerParent.h#25
https://searchfox.org/mozilla-central/rev/84c1031f19b2cf8221da0850983f8d0269f6efd1/gfx/layers/ipc/CompositorManagerParent.cpp#206

I was able to reproduce quite readily in try. I'll add some debug prints and see if that helps to identify which is the culprit:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f612e6f46160733559411851ea851fd34eea8fb0

No longer blocks: gfx-triage
You need to log in before you can comment on or make changes to this bug.