Closed Bug 1594446 Opened 5 years ago Closed 5 years ago

Perma layout/base/crashtests/empty-mask.html | application crashed [@ mozalloc_handle_oom(unsigned int)]

Categories

(Core :: Graphics: WebRender, defect, P3)

Product:
Core
Component:
Graphics: WebRender
Platform:
ARM
Android
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla73
Tracking Flags:
Tracking Status
firefox-esr68 --- unaffected
firefox71 --- unaffected
firefox72 --- fixed
firefox73 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: jnicol)

References

(Blocks 1 open bug, Regression)

Details

(Keywords: crash, intermittent-failure, regression, Whiteboard: [stockwell fixed:patch])

Crash Data

Attachments

(1 file)

Bug 1594446 - Clamp scale at 32k to avoid excessively large visible regions r?mattwoodrow
(deleted), text/x-phabricator-request
jcristau
: approval-mozilla-beta+
Details

Filed by: malexandru [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=274841973&repo=autoland
Full log: https://queue.taskcluster.net/v1/task/JkqOvIZiSK6muiDHrynsog/runs/0/artifacts/public/logs/live_backing.log
Reftest URL: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/JkqOvIZiSK6muiDHrynsog/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

[task 2019-11-06T15:15:19.115Z] 15:14:49 INFO - REFTEST TEST-LOAD | http://10.7.205.205:8854/tests/layout/base/crashtests/empty-mask.html | 1908 / 3732 (51%)
[task 2019-11-06T15:15:19.115Z] 15:15:12 INFO - wait for org.mozilla.geckoview.test complete; top activity=com.bitbar.testdroid.monitor
[task 2019-11-06T15:15:19.115Z] 15:15:12 INFO - remoteautomation.py | Application ran for: 0:05:50.468563
[task 2019-11-06T15:15:19.115Z] 15:15:13 INFO - REFTEST INFO | Downloading symbols from: https://queue.taskcluster.net/v1/task/KVOfOuYoTiq-cnl1SRPDRw/artifacts/public/build/target.crashreporter-symbols.zip
[task 2019-11-06T15:15:19.115Z] 15:15:15 INFO - REFTEST INFO | Copy/paste: /builds/task_1573052777/workspace/build/linux64-minidump_stackwalk /tmp/tmpfK6Q7n/46659e71-4a46-4c91-462b-51259a8c9935.dmp /tmp/tmp1a6V5i
[task 2019-11-06T15:15:19.115Z] 15:15:19 INFO - REFTEST INFO | Saved minidump as /builds/task_1573052777/workspace/build/blobber_upload_dir/46659e71-4a46-4c91-462b-51259a8c9935.dmp
[task 2019-11-06T15:15:19.115Z] 15:15:19 INFO - REFTEST INFO | Saved app info as /builds/task_1573052777/workspace/build/blobber_upload_dir/46659e71-4a46-4c91-462b-51259a8c9935.extra
[task 2019-11-06T15:15:19.115Z] 15:15:19 WARNING - REFTEST PROCESS-CRASH | http://10.7.205.205:8854/tests/layout/base/crashtests/empty-mask.html | application crashed [@ mozalloc_handle_oom(unsigned int)]
[task 2019-11-06T15:15:19.115Z] 15:15:19 INFO - Crash dump filename: /tmp/tmpfK6Q7n/46659e71-4a46-4c91-462b-51259a8c9935.dmp
[task 2019-11-06T15:15:19.115Z] 15:15:19 INFO - Operating system: Android
[task 2019-11-06T15:15:19.115Z] 15:15:19 INFO - 0.0.0 Linux 4.4.56-g594d847d09a1 #1 SMP PREEMPT Thu Oct 26 22:34:08 UTC 2017 armv8l
[task 2019-11-06T15:15:19.115Z] 15:15:19 INFO - CPU: arm
[task 2019-11-06T15:15:19.115Z] 15:15:19 INFO - ARMv1 Qualcomm part(0x51008010) features: half,thumb,fastmult,vfpv2,edsp,neon,vfpv3,tls,vfpv4,idiva,idivt
[task 2019-11-06T15:15:19.115Z] 15:15:19 INFO - 8 CPUs
[task 2019-11-06T15:15:19.115Z] 15:15:19 INFO - GPU: UNKNOWN
[task 2019-11-06T15:15:19.115Z] 15:15:19 INFO - Crash reason: SIGSEGV /SEGV_MAPERR
[task 2019-11-06T15:15:19.115Z] 15:15:19 INFO - Crash address: 0x0
[task 2019-11-06T15:15:19.115Z] 15:15:19 INFO - Process uptime: not available
[task 2019-11-06T15:15:19.115Z] 15:15:19 INFO - Thread 55 (crashed)
[task 2019-11-06T15:15:19.115Z] 15:15:19 INFO - 0 libmozglue.so!mozalloc_abort [mozalloc_abort.cpp:d6709b4ccf48e8efa8a6efcd478f97c7ee4e2d48 : 33 + 0x10]
[task 2019-11-06T15:15:19.115Z] 15:15:19 INFO - r0 = 0x00000000 r1 = 0x00000021 r2 = 0x00430000 r3 = 0xefa3d3bc
[task 2019-11-06T15:15:19.115Z] 15:15:19 INFO - r4 = 0xbd6f85a0 r5 = 0xbd6f85a0 r6 = 0x00200000 r7 = 0xbd6f8598
[task 2019-11-06T15:15:19.115Z] 15:15:19 INFO - r8 = 0x00000000 r9 = 0x00000200 r10 = 0x0f000000 r12 = 0xf231bd5c
[task 2019-11-06T15:15:19.115Z] 15:15:19 INFO - fp = 0x00000004 sp = 0xbd6f8590 lr = 0xefa3d05d pc = 0xd460df44
[task 2019-11-06T15:15:19.115Z] 15:15:19 INFO - Found by: given as instruction pointer in context
[task 2019-11-06T15:15:19.115Z] 15:15:19 INFO - 1 libmozglue.so!mozalloc_handle_oom(unsigned int) [mozalloc_oom.cpp:d6709b4ccf48e8efa8a6efcd478f97c7ee4e2d48 : 51 + 0x3]
[task 2019-11-06T15:15:19.115Z] 15:15:19 INFO - r4 = 0x0000000f r5 = 0xbd6f85a0 r6 = 0x00200000 r7 = 0xbd6f85e0
[task 2019-11-06T15:15:19.115Z] 15:15:19 INFO - r8 = 0x00000000 r9 = 0x00000200 r10 = 0x0f000000 fp = 0x00000004
[task 2019-11-06T15:15:19.115Z] 15:15:19 INFO - sp = 0xbd6f85a0 lr = 0xd460e0c1 pc = 0xd460e0c1
[task 2019-11-06T15:15:19.115Z] 15:15:19 INFO - Found by: call frame info
[task 2019-11-06T15:15:19.115Z] 15:15:19 INFO - 2 libxul.so!gkrust_shared::oom_hook::hook [lib.rs:d6709b4ccf48e8efa8a6efcd478f97c7ee4e2d48 : 271 + 0x3]
[task 2019-11-06T15:15:19.115Z] 15:15:19 INFO - r4 = 0x40000000 r5 = 0xd43175ac r6 = 0x00200000 r7 = 0xbd6f85e8
[task 2019-11-06T15:15:19.115Z] 15:15:19 INFO - r8 = 0x00000000 r9 = 0x00000200 r10 = 0x0f000000 fp = 0x00000004
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - sp = 0xbd6f85e8 lr = 0xd22d3841 pc = 0xd22d3841
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - Found by: call frame info
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - 3 libxul.so!rust_oom [alloc.rs:625451e376bb2e5283fc4741caa0a3e8a2ca4d54 : 216 + 0x1]
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - r4 = 0x40000000 r5 = 0xd43175ac r6 = 0x00200000 r7 = 0xbd6f87d0
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - r8 = 0x00000000 r9 = 0x00000200 r10 = 0x0f000000 fp = 0x00000004
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - sp = 0xbd6f85f0 pc = 0xd28bc139
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - Found by: call frame info
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - 4 libxul.so!alloc::alloc::handle_alloc_error [alloc.rs:625451e376bb2e5283fc4741caa0a3e8a2ca4d54 : 248 + 0x3]
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - r4 = 0x40000000 r5 = 0xd43175ac r6 = 0x00200000 r7 = 0xbd6f87d0
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - r8 = 0x00000000 r9 = 0x00000200 r10 = 0x0f000000 fp = 0x00000004
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - sp = 0xbd6f85f8 pc = 0xd28c1d37
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - Found by: call frame info
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - 5 libxul.so!<core::iter::adapters::Map<I,F> as core::iter::traits::iterator::Iterator>::next [mod.rs:625451e376bb2e5283fc4741caa0a3e8a2ca4d54 : 603 + 0x1e9]
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - r4 = 0x40000000 r5 = 0xd43175ac r6 = 0x00200000 r7 = 0xbd6f87d0
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - r8 = 0x00000000 r9 = 0x00000200 r10 = 0x0f000000 fp = 0x00000004
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - sp = 0xbd6f8600 pc = 0xd24ed16d
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - Found by: call frame info
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - 6 libxul.so!webrender::render_backend::RenderBackend::process_api_msg [render_backend.rs:d6709b4ccf48e8efa8a6efcd478f97c7ee4e2d48 : 1231 + 0xcb]
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - r4 = 0xb2309ab4 r5 = 0xb2309ab4 r6 = 0x00000001 r7 = 0xbd6faa98
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - r8 = 0xb238d1b8 r9 = 0xbd6f9d38 r10 = 0xbd6f92e0 fp = 0x00000001
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - sp = 0xbd6f87d8 pc = 0xd24d920b
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - Found by: call frame info
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - 7 libxul.so!webrender::render_backend::RenderBackend::run [render_backend.rs:d6709b4ccf48e8efa8a6efcd478f97c7ee4e2d48 : 967 + 0x1f]
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - r4 = 0x00000000 r5 = 0xb2309a80 r6 = 0x00000000 r7 = 0xbd6fb148
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - r8 = 0x00000006 r9 = 0xb238d1b8 r10 = 0xbd6fbc00 fp = 0xbd6faee0
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - sp = 0xbd6faaa0 pc = 0xd24ca6d1
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - Found by: call frame info
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - 8 libxul.so!std::sys_common::backtrace::__rust_begin_short_backtrace [backtrace.rs:625451e376bb2e5283fc4741caa0a3e8a2ca4d54 : 77 + 0xe97]
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - r4 = 0xbd6fc098 r5 = 0xbd6fbd0c r6 = 0x00000000 r7 = 0xbd6fc990
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - r8 = 0xbd6fbbf0 r9 = 0xbd6fbc00 r10 = 0xbd6fbbd0 fp = 0x0000000c
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - sp = 0xbd6fb150 pc = 0xd24c5b5b
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - Found by: call frame info
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - 9 libxul.so!core::ops::function::FnOnce::call_once{{vtable.shim}} [function.rs:625451e376bb2e5283fc4741caa0a3e8a2ca4d54 : 235 + 0x47]
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - r4 = 0xbd6fd170 r5 = 0xbd6fc9a0 r6 = 0x00000000 r7 = 0xbd6fd168
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - r8 = 0xc005b800 r9 = 0xd357e008 r10 = 0xd28bcd05 fp = 0x00000002
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - sp = 0xbd6fc998 pc = 0xd24c4c13
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - Found by: call frame info
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - 10 libxul.so!<alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once [boxed.rs:625451e376bb2e5283fc4741caa0a3e8a2ca4d54 : 787 + 0x29]
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - r4 = 0x000007c0 r5 = 0xbd6fd170 r6 = 0xbd6fd930 r7 = 0xbd6fd940
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - r8 = 0xc005b800 r9 = 0xd357e008 r10 = 0xd28bcd05 fp = 0x00000002
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - sp = 0xbd6fd170 pc = 0xd28bb51d
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - Found by: call frame info
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - 11 libxul.so!std::sys::unix::thread::Thread::new::thread_start [thread.rs:625451e376bb2e5283fc4741caa0a3e8a2ca4d54 : 79 + 0x7]
[task 2019-11-06T15:15:19.118Z] 15:15:19 INFO - r4 = 0xc03fd9d8 r5 = 0xbd6fd970 r6 = 0xbd6fd970 r7 = 0x00000078
[task 2019-11-06T15:15:19.122Z] 15:15:19 INFO - r8 = 0x000017ba r9 = 0x000017f8 r10 = 0xd28bcd05 fp = 0x00000002
[task 2019-11-06T15:15:19.122Z] 15:15:19 INFO - sp = 0xbd6fd950 pc = 0xd28bcd11
[task 2019-11-06T15:15:19.122Z] 15:15:19 INFO - Found by: call frame info
[task 2019-11-06T15:15:19.122Z] 15:15:19 INFO - 12 libc.so + 0x47947
[task 2019-11-06T15:15:19.122Z] 15:15:19 INFO - r4 = 0xbd6fd970 r5 = 0xbd6fd970 r6 = 0xbd6fd970 r7 = 0x00000078
[task 2019-11-06T15:15:19.122Z] 15:15:19 INFO - r8 = 0x000017ba r9 = 0x000017f8 r10 = 0xd28bcd05 fp = 0x00000002
[task 2019-11-06T15:15:19.122Z] 15:15:19 INFO - sp = 0xbd6fd958 pc = 0xf22d1949
[task 2019-11-06T15:15:19.122Z] 15:15:19 INFO - Found by: call frame info
[task 2019-11-06T15:15:19.122Z] 15:15:19 INFO - 13 libc.so + 0x4792f

Looks like the oom happened in webrender.

Component: Layout → Graphics: WebRender
OS: Unspecified → Android
Hardware: Unspecified → ARM

There are 40 total failures in the last 7 days on android-hw-p2-8-0-arm7-api-16-qr opt

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=277114630&repo=autoland&lineNumber=10476

[task 2019-11-20T04:21:59.006Z] 04:21:29 INFO - REFTEST TEST-START | http://10.7.205.204:8854/tests/layout/base/crashtests/empty-mask.html
[task 2019-11-20T04:21:59.006Z] 04:21:29 INFO - REFTEST TEST-LOAD | http://10.7.205.204:8854/tests/layout/base/crashtests/empty-mask.html | 1906 / 3734 (51%)
[task 2019-11-20T04:21:59.006Z] 04:21:53 INFO - wait for org.mozilla.geckoview.test complete; top activity=com.bitbar.testdroid.monitor
[task 2019-11-20T04:21:59.006Z] 04:21:53 INFO - remoteautomation.py | Application ran for: 0:06:13.751502
[task 2019-11-20T04:21:59.006Z] 04:21:54 INFO - REFTEST INFO | Downloading symbols from: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/KyXEECv3SjGHDsB1Uu0nKQ/artifacts/public/build/target.crashreporter-symbols.zip
[task 2019-11-20T04:21:59.006Z] 04:21:55 INFO - REFTEST INFO | Copy/paste: /builds/task_1574222931/workspace/build/linux64-minidump_stackwalk /tmp/tmpryEK9v/0fed9131-993c-6285-4a45-5052e31a4c3f.dmp /tmp/tmpwt6Wm2
[task 2019-11-20T04:21:59.006Z] 04:21:59 INFO - REFTEST INFO | Saved minidump as /builds/task_1574222931/workspace/build/blobber_upload_dir/0fed9131-993c-6285-4a45-5052e31a4c3f.dmp
[task 2019-11-20T04:21:59.006Z] 04:21:59 INFO - REFTEST INFO | Saved app info as /builds/task_1574222931/workspace/build/blobber_upload_dir/0fed9131-993c-6285-4a45-5052e31a4c3f.extra
[task 2019-11-20T04:21:59.006Z] 04:21:59 WARNING - REFTEST PROCESS-CRASH | http://10.7.205.204:8854/tests/layout/base/crashtests/empty-mask.html | application crashed [@ mozalloc_handle_oom(unsigned int)]
[task 2019-11-20T04:21:59.006Z] 04:21:59 INFO - Crash dump filename: /tmp/tmpryEK9v/0fed9131-993c-6285-4a45-5052e31a4c3f.dmp
[task 2019-11-20T04:21:59.006Z] 04:21:59 INFO - Operating system: Android
[task 2019-11-20T04:21:59.006Z] 04:21:59 INFO - 0.0.0 Linux 4.4.56-g594d847d09a1 #1 SMP PREEMPT Thu Oct 26 22:34:08 UTC 2017 armv8l
[task 2019-11-20T04:21:59.006Z] 04:21:59 INFO - CPU: arm
[task 2019-11-20T04:21:59.006Z] 04:21:59 INFO - ARMv1 Qualcomm part(0x51008010) features: half,thumb,fastmult,vfpv2,edsp,neon,vfpv3,tls,vfpv4,idiva,idivt
[task 2019-11-20T04:21:59.006Z] 04:21:59 INFO - 8 CPUs
[task 2019-11-20T04:21:59.006Z] 04:21:59 INFO - GPU: UNKNOWN
[task 2019-11-20T04:21:59.006Z] 04:21:59 INFO - Crash reason: SIGSEGV /SEGV_MAPERR
[task 2019-11-20T04:21:59.006Z] 04:21:59 INFO - Crash address: 0x0
[task 2019-11-20T04:21:59.006Z] 04:21:59 INFO - Process uptime: not available
[task 2019-11-20T04:21:59.006Z] 04:21:59 INFO - Thread 55 (crashed)
[task 2019-11-20T04:21:59.006Z] 04:21:59 INFO - 0 libmozglue.so!mozalloc_abort [mozalloc_abort.cpp:4ec011830a0b2649d70b58d30e30edb8babfc6f1 : 33 + 0x10]
[task 2019-11-20T04:21:59.006Z] 04:21:59 INFO - r0 = 0x00000000 r1 = 0x00000021 r2 = 0x00430000 r3 = 0xec13b3bc
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - r4 = 0xb99fe458 r5 = 0xb99fe458 r6 = 0x00000004 r7 = 0xb99fe450
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - r8 = 0xb99fe5d8 r9 = 0x03c00000 r10 = 0x00400000 r12 = 0xed19bd5c
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - fp = 0x00000001 sp = 0xb99fe448 lr = 0xec13b05d pc = 0xd0d8cf7c
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - Found by: given as instruction pointer in context
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - 1 libmozglue.so!mozalloc_handle_oom(unsigned int) [mozalloc_oom.cpp:4ec011830a0b2649d70b58d30e30edb8babfc6f1 : 51 + 0x3]
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - r4 = 0x00000001 r5 = 0xb99fe458 r6 = 0x00000004 r7 = 0xb99fe498
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - r8 = 0xb99fe5d8 r9 = 0x03c00000 r10 = 0x00400000 fp = 0x00000001
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - sp = 0xb99fe458 lr = 0xd0d8d0f9 pc = 0xd0d8d0f9
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - Found by: call frame info
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - 2 libxul.so!gkrust_shared::oom_hook::hook [lib.rs:4ec011830a0b2649d70b58d30e30edb8babfc6f1 : 185 + 0x3]
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - r4 = 0x80c00000 r5 = 0x15000000 r6 = 0x00000004 r7 = 0xb99fe4a0
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - r8 = 0xb99fe5d8 r9 = 0x03c00000 r10 = 0x00400000 fp = 0x00000001
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - sp = 0xb99fe4a0 lr = 0xceb52921 pc = 0xceb52921
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - Found by: call frame info
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - 3 libxul.so!rust_oom [alloc.rs:625451e376bb2e5283fc4741caa0a3e8a2ca4d54 : 216 + 0x1]
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - r4 = 0x80c00000 r5 = 0x15000000 r6 = 0x00000004 r7 = 0xb99fe5c8
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - r8 = 0xb99fe5d8 r9 = 0x03c00000 r10 = 0x00400000 fp = 0x00000001
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - sp = 0xb99fe4a8 pc = 0xcf176c41
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - Found by: call frame info
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - 4 libxul.so!alloc::alloc::handle_alloc_error [alloc.rs:625451e376bb2e5283fc4741caa0a3e8a2ca4d54 : 248 + 0x3]
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - r4 = 0x80c00000 r5 = 0x15000000 r6 = 0x00000004 r7 = 0xb99fe5c8
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - r8 = 0xb99fe5d8 r9 = 0x03c00000 r10 = 0x00400000 fp = 0x00000001
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - sp = 0xb99fe4b0 pc = 0xcf17c83f
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - Found by: call frame info
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - 5 libxul.so!<webrender_bindings::moz2d_renderer::Moz2dBlobRasterizer as webrender_api::image::AsyncBlobImageRasterizer>::rasterize [moz2d_renderer.rs:4ec011830a0b2649d70b58d30e30edb8babfc6f1 : 532 + 0x1b]
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - r4 = 0x80c00000 r5 = 0x15000000 r6 = 0x00000004 r7 = 0xb99fe5c8
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - r8 = 0xb99fe5d8 r9 = 0x03c00000 r10 = 0x00400000 fp = 0x00000001
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - sp = 0xb99fe4b8 pc = 0xcecece47
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - Found by: call frame info
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - 6 libxul.so!webrender::scene_builder_thread::Transaction::rasterize_blobs [scene_builder_thread.rs:4ec011830a0b2649d70b58d30e30edb8babfc6f1 : 73 + 0x3]
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - r4 = 0xaf4dc2c0 r5 = 0xcececa91 r6 = 0x00000001 r7 = 0xb99fe610
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - r8 = 0xaf4dc2c0 r9 = 0xb7b8a490 r10 = 0x00000001 fp = 0x00000001
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - sp = 0xb99fe5d0 pc = 0xced4701d
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - Found by: call frame info
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - 7 libxul.so!std::sys_common::backtrace::__rust_begin_short_backtrace [backtrace.rs:625451e376bb2e5283fc4741caa0a3e8a2ca4d54 : 77 + 0xab]
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - r4 = 0xb99fe680 r5 = 0xb99fe730 r6 = 0x00000001 r7 = 0xb99fe8b0
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - r8 = 0xaf4dc2c0 r9 = 0xb7b8a490 r10 = 0x00000001 fp = 0x00000001
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - sp = 0xb99fe618 pc = 0xcee35e25
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - Found by: call frame info
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - 8 libxul.so!core::ops::function::FnOnce::call_once{{vtable.shim}} [function.rs:625451e376bb2e5283fc4741caa0a3e8a2ca4d54 : 235 + 0x57]
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - r4 = 0xb99fe900 r5 = 0xb99fe8c0 r6 = 0x00000000 r7 = 0xb99fe8f8
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - r8 = 0xb9d24a30 r9 = 0xcfe46edc r10 = 0xcf17780d fp = 0x00000002
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - sp = 0xb99fe8b8 pc = 0xcee35cff
[task 2019-11-20T04:21:59.010Z] 04:21:59 INFO - Found by: call frame info
[task 2019-11-20T04:21:59.017Z] 04:21:59 INFO - 9 libxul.so!<alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once [boxed.rs:625451e376bb2e5283fc4741caa0a3e8a2ca4d54 : 787 + 0x29]
[task 2019-11-20T04:21:59.017Z] 04:21:59 INFO - r4 = 0x0000002c r5 = 0xb99fe900 r6 = 0xb99fe930 r7 = 0xb99fe940
[task 2019-11-20T04:21:59.017Z] 04:21:59 INFO - r8 = 0xb9d24a30 r9 = 0xcfe46edc r10 = 0xcf17780d fp = 0x00000002
[task 2019-11-20T04:21:59.017Z] 04:21:59 INFO - sp = 0xb99fe900 pc = 0xcf176025
[task 2019-11-20T04:21:59.017Z] 04:21:59 INFO - Found by: call frame info
[task 2019-11-20T04:21:59.017Z] 04:21:59 INFO - 10 libxul.so!std::sys::unix::thread::Thread::new::thread_start [thread.rs:625451e376bb2e5283fc4741caa0a3e8a2ca4d54 : 79 + 0x7]
[task 2019-11-20T04:21:59.017Z] 04:21:59 INFO - r4 = 0xbc1ef9d8 r5 = 0xb99fe970 r6 = 0xb99fe970 r7 = 0x00000078
[task 2019-11-20T04:21:59.017Z] 04:21:59 INFO - r8 = 0x00001848 r9 = 0x00001886 r10 = 0xcf17780d fp = 0x00000002
[task 2019-11-20T04:21:59.017Z] 04:21:59 INFO - sp = 0xb99fe950 pc = 0xcf177819
[task 2019-11-20T04:21:59.017Z] 04:21:59 INFO - Found by: call frame info
[task 2019-11-20T04:21:59.017Z] 04:21:59 INFO - 11 libc.so + 0x47947
[task 2019-11-20T04:21:59.017Z] 04:21:59 INFO - r4 = 0xb99fe970 r5 = 0xb99fe970 r6 = 0xb99fe970 r7 = 0x00000078
[task 2019-11-20T04:21:59.017Z] 04:21:59 INFO - r8 = 0x00001848 r9 = 0x00001886 r10 = 0xcf17780d fp = 0x00000002
[task 2019-11-20T04:21:59.017Z] 04:21:59 INFO - sp = 0xb99fe958 pc = 0xed151949
[task 2019-11-20T04:21:59.017Z] 04:21:59 INFO - Found by: call frame info
[task 2019-11-20T04:21:59.017Z] 04:21:59 INFO - 12 libc.so + 0x4792f
[task 2019-11-20T04:21:59.017Z] 04:21:59 INFO - sp = 0xb99fe95c pc = 0xed151931
[task 2019-11-20T04:21:59.017Z] 04:21:59 INFO - Found by: stack scanning
[task 2019-11-20T04:21:59.017Z] 04:21:59 INFO - 13 libc.so + 0x1b381
[task 2019-11-20T04:21:59.017Z] 04:21:59 INFO - sp = 0xb99fe960 pc = 0xed125383
[task 2019-11-20T04:21:59.017Z] 04:21:59 INFO - Found by: stack scanning
[task 2019-11-20T04:21:59.017Z] 04:21:59 INFO - 14 libc.so + 0x4792f
[task 2019-11-20T04:21:59.017Z] 04:21:59 INFO - sp = 0xb99fe968 pc = 0xed151931
[task 2019-11-20T04:21:59.017Z] 04:21:59 INFO - Found by: stack scanning
[task 2019-11-20T04:21:59.017Z] 04:21:59 INFO - 15 libxul.so!std::sys::unix::thread::Thread::new + 0x1b3
[task 2019-11-20T04:21:59.017Z] 04:21:59 INFO - sp = 0xb99fe9a4 pc = 0xcf17780d
[task 2019-11-20T04:21:59.017Z] 04:21:59 INFO - Found by: stack scanning

Jessie can you assign someone to take a look?

Flags: needinfo?(jbonisteel)
Whiteboard: [stockwell needswork:owner]

Hey Kris - can we take a look and see if this would impact our Android MVP?

Blocks: wr-android-mvp
Flags: needinfo?(jbonisteel) → needinfo?(ktaeleman)
Priority: -- → P3

@jessie: It does not seem to occur frequent enough to block mvp, but I'm guessing there is a memory leak here after running a lot of tests/pages (1000+).

Flags: needinfo?(ktaeleman)
Blocks: wr-android
No longer blocks: wr-android-mvp

Jessie are you working on this?

Flags: needinfo?(jbonisteel)

Is the same signature as the crash you were looking at, Jamie?

Flags: needinfo?(jbonisteel) → needinfo?(jnicol)

Yes, this one looks like the same bug

Flags: needinfo?(jnicol)
Summary: Intermittent layout/base/crashtests/empty-mask.html | application crashed [@ mozalloc_handle_oom(unsigned int)] → Perma layout/base/crashtests/empty-mask.html | application crashed [@ mozalloc_handle_oom(unsigned int)]

Hi Brian, is this from Bug 1587142 ?

Flags: needinfo?(bgrinstead)

(In reply to Andreea Pavel [:apavel] from comment #11)

Hi Brian, is this from Bug 1587142 ?

I don't think so - that bug was only removing unrelated tests. It could have changed the order in which this test got run in relation with others in the directory which maybe triggered an existing bug?

Flags: needinfo?(bgrinstead)

Hi Jamie, Are you working on this? (Can I assign it to you?) Any idea what's going on here? This was flagged today as a major orange (which is why I'm following up). Thanks.

Flags: needinfo?(jnicol)

(In reply to Maire Reavy [:mreavy] Plz needinfo from comment #15)

Hi Jamie, Are you working on this? (Can I assign it to you?) Any idea what's going on here? This was flagged today as a major orange (which is why I'm following up). Thanks.

Hey Maire,
Yes, Jamie is looking into this. I'll let him update you on what is going on, but latest I heard was that he identified the change that started causing this and is investigating further.
I'll assign it to Jamie so it's clear this is being worked on.

Assignee: nobody → jnicol

There are 132 total failures in the last 30 days and 73 total failures in the last 7 on android-hw-p2-8-0-arm7-api-16-qr opt.

Kris, should this be disabled until there is a fix here?

Flags: needinfo?(ktaeleman)

I will take a look today if we can disable some of the tests while Jamie is looking for a fix.

Flags: needinfo?(ktaeleman)

Update on this:
It seems like no matter what test I disable, another test would crash. So disabling individual tests is not an option.
Still digging a bit further as I'm struggling with getting the callstacks resolved locally and want to make sure I'm looking at the same crash.

Nicolas, this is a regression from bug 1594446. Can you look into this, please?

Flags: needinfo?(nical.bugzilla)
Regressed by: 1587713
Keywords: regression

I'm looking in to this. The problem is that we have very large scale transforms, leading to us creating very large blob images. Since bug 1587713 we rasterize these all up-front causing us to OOM. On layers I believe we avoid this problem because we only rasterize the smaller critical display port at full resolution, and the larger display port in low-res, so it doesn't need as much memory. I am looking in to how best to limit the size.

Flags: needinfo?(jnicol)

The size of the visible region, for either a painted layer or a webrender blob
image, is calculated from the building rects of the contained display items, in
local-space. This should be restricted to the display port, to prevent the
visible regions growing too large leading to excessive memory usage.

For items within large scale transforms, the local-space visible region should
be very small. However, as we do not allow fractional sizes, the size of the
visible region will be rounded up to at least 1. This means that when we convert
the region back to screen-space, we are multiplying the extremely large scale by
at least one, rather than by a much smaller fraction. This can result in
incredibly large visible regions, and was causing OOM crashes.

To avoid this, we clamp the maximum chosen scale for these layers/blob images to
32k. Layers affected by this problem should have a visible region with
dimensions of 1 or 2, so this limits the resulting screen-space size for
those to an acceptable value. Layers with visible regions sized greater than
that should not have scales anywhere near this large, so will not be affected.

Pushed by jnicol@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/92a79c1e69d9
Clamp scale at 32k to avoid excessively large visible regions r=mattwoodrow
Flags: needinfo?(nical.bugzilla)

https://hg.mozilla.org/mozilla-central/rev/92a79c1e69d9

Status: NEW → RESOLVED
Closed: 5 years ago
status-firefox73: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla73

IIUC, I think we want this fix on Beta72 also? Please nominate the patch for approval if that's correct.

status-firefox71: --- → unaffected
status-firefox72: --- → affected
status-firefox-esr68: --- → unaffected
Flags: needinfo?(jnicol)

I think we do too, but was going to let it bake on nightly for a few days to make sure it's okay?

No failures since Dec 6th, and I'm not aware of any breakage. Let's uplift this!

Flags: needinfo?(jnicol)

Comment on attachment 9113249 [details]
Bug 1594446 - Clamp scale at 32k to avoid excessively large visible regions r?mattwoodrow

Beta/Release Uplift Approval Request

  • User impact if declined: OOM crashes on either webrender or non-webrender, most likely on Android but possibly other platforms too.
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: Yes
  • Needs manual test from QE?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): Touching FrameLayerBuilder always carries a bit of risk, but this only sets a very conservative scale limit: to such a high value that it should only affect sites we already perform incredibly poorly on.
  • String changes made/needed: N/A
Attachment #9113249 - Flags: approval-mozilla-beta?

Comment on attachment 9113249 [details]
Bug 1594446 - Clamp scale at 32k to avoid excessively large visible regions r?mattwoodrow

OOM fix, approved for 72.0b5

Attachment #9113249 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Crash Signature: [@ mozalloc_handle_oom(unsigned int)] → [@ mozalloc_handle_oom(unsigned int)] [@ webrender::device::gl::Device::new::{{closure}}]

https://hg.mozilla.org/releases/mozilla-beta/rev/d954a760349f

status-firefox72: affected → fixed
Crash Signature: [@ mozalloc_handle_oom(unsigned int)] [@ webrender::device::gl::Device::new::{{closure}}] → [@ mozalloc_handle_oom(unsigned int)] [@ webrender::device::gl::Device::new::{{closure}}]
Whiteboard: [stockwell disable-recommended] → [stockwell fixed:patch]
Has Regression Range: --- → yes
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: