See bug 1440545. https://treeherder.mozilla.org/logviewer.html#?job_id=163797965&repo=autoland https://queue.taskcluster.net/v1/task/M5rSMxdKT4aQGw9DxYuqkA/runs/0/artifacts/public/logs/live_backing.log [task 2018-02-22T22:25:55.304Z] 22:25:55 INFO - PROCESS-CRASH | awsy/test_memory_usage.py TestMemoryUsage.test_open_tabs | application crashed [@ 0x7f57a5a6f3dd] [task 2018-02-22T22:25:55.304Z] 22:25:55 INFO - Crash dump filename: /tmp/tmpj6ot_2.profile/minidumps/13ad11c2-02de-e8e8-24bb-b205d62a2f20.dmp [task 2018-02-22T22:25:55.304Z] 22:25:55 INFO - Operating system: Linux [task 2018-02-22T22:25:55.305Z] 22:25:55 INFO - 0.0.0 Linux 4.4.0-98-generic #121~14.04.1-Ubuntu SMP Wed Oct 11 11:54:55 UTC 2017 x86_64 [task 2018-02-22T22:25:55.305Z] 22:25:55 INFO - CPU: amd64 [task 2018-02-22T22:25:55.305Z] 22:25:55 INFO - family 6 model 62 stepping 4 [task 2018-02-22T22:25:55.305Z] 22:25:55 INFO - 4 CPUs [task 2018-02-22T22:25:55.306Z] 22:25:55 INFO - [task 2018-02-22T22:25:55.306Z] 22:25:55 INFO - GPU: UNKNOWN [task 2018-02-22T22:25:55.306Z] 22:25:55 INFO - [task 2018-02-22T22:25:55.306Z] 22:25:55 INFO - Crash reason: SIGSEGV [task 2018-02-22T22:25:55.307Z] 22:25:55 INFO - Crash address: 0x0 [task 2018-02-22T22:25:55.307Z] 22:25:55 INFO - Process uptime: not available [task 2018-02-22T22:25:55.307Z] 22:25:55 INFO - [task 2018-02-22T22:25:55.307Z] 22:25:55 INFO - Thread 0 (crashed) [task 2018-02-22T22:25:55.308Z] 22:25:55 INFO - 0 0x7f57a5a6f3dd [task 2018-02-22T22:25:55.308Z] 22:25:55 INFO - rax = 0xe5e5e5e5e5e5e5e5 rdx = 0x00007f5615fae2e0 [task 2018-02-22T22:25:55.308Z] 22:25:55 INFO - rcx = 0xe5e5e5e515151505 rbx = 0x00007f5778820288 [task 2018-02-22T22:25:55.308Z] 22:25:55 INFO - rsi = 0x0000000000007f51 rdi = 0x00007f5615f86800 [task 2018-02-22T22:25:55.309Z] 22:25:55 INFO - rbp = 0x00007f5615f86800 rsp = 0x00007fffcd6e4b70 [task 2018-02-22T22:25:55.309Z] 22:25:55 INFO - r8 = 0x00007f57778646c0 r9 = 0x00007f579c55c370 [task 2018-02-22T22:25:55.309Z] 22:25:55 INFO - r10 = 0x00000000000000db r11 = 0x00007f57a37a8dd0 [task 2018-02-22T22:25:55.310Z] 22:25:55 INFO - r12 = 0x00007f575c59a720 r13 = 0x0000000000007f51 [task 2018-02-22T22:25:55.310Z] 22:25:55 INFO - r14 = 0x0000000000007f51 r15 = 0x00007f575c538a00 [task 2018-02-22T22:25:55.310Z] 22:25:55 INFO - rip = 0x00007f57a5a6f3dd [task 2018-02-22T22:25:55.310Z] 22:25:55 INFO - Found by: given as instruction pointer in context [task 2018-02-22T22:25:55.311Z] 22:25:55 INFO - [task 2018-02-22T22:25:55.311Z] 22:25:55 INFO - Thread 1 [task 2018-02-22T22:25:55.311Z] 22:25:55 INFO - 0 0x7f57aa36f4d9 [task 2018-02-22T22:25:55.311Z] 22:25:55 INFO - rax = 0xfffffffffffffffc rdx = 0x0000000000000020 [task 2018-02-22T22:25:55.312Z] 22:25:55 INFO - rcx = 0x00007f57aa36f4d9 rbx = 0x0000000000000000 [task 2018-02-22T22:25:55.312Z] 22:25:55 INFO - rsi = 0x00007f57aa089200 rdi = 0x0000000000000011 [task 2018-02-22T22:25:55.312Z] 22:25:55 INFO - rbp = 0x00007f579823fba0 rsp = 0x00007f579823fb98 [task 2018-02-22T22:25:55.313Z] 22:25:55 INFO - r8 = 0x00007f57aa3cd1d0 r9 = 0x00007f579823fbf0 [task 2018-02-22T22:25:55.313Z] 22:25:55 INFO - r10 = 0x00000000ffffffff r11 = 0x0000000000000246 [task 2018-02-22T22:25:55.313Z] 22:25:55 INFO - r12 = 0x0000000000000000 r13 = 0x00007f57aa0f443c [task 2018-02-22T22:25:55.313Z] 22:25:55 INFO - r14 = 0x00007f57aa0f4430 r15 = 0x00007f57aa01e800 [task 2018-02-22T22:25:55.314Z] 22:25:55 INFO - rip = 0x00007f57aa36f4d9 [task 2018-02-22T22:25:55.314Z] 22:25:55 INFO - Found by: given as instruction pointer in context [task 2018-02-22T22:25:55.314Z] 22:25:55 INFO - 1 0x7f5799ec6349 [task 2018-02-22T22:25:55.315Z] 22:25:55 INFO - rbp = 0x00007f579823fbf0 rsp = 0x00007f579823fbb0 [task 2018-02-22T22:25:55.315Z] 22:25:55 INFO - rip = 0x00007f5799ec6349 [task 2018-02-22T22:25:55.315Z] 22:25:55 INFO - Found by: previous frame's frame pointer [task 2018-02-22T22:25:55.315Z] 22:25:55 INFO - 2 0x7f5799ec8c0d [task 2018-02-22T22:25:55.316Z] 22:25:55 INFO - rbp = 0x00007f579823fc80 rsp = 0x00007f579823fc00 [task 2018-02-22T22:25:55.316Z] 22:25:55 INFO - rip = 0x00007f5799ec8c0d [task 2018-02-22T22:25:55.316Z] 22:25:55 INFO - Found by: previous frame's frame pointer [task 2018-02-22T22:25:55.316Z] 22:25:55 INFO - 3 0x7f5799eb0972 [task 2018-02-22T22:25:55.317Z] 22:25:55 INFO - rbp = 0x00007f579823fcf0 rsp = 0x00007f579823fc90 [task 2018-02-22T22:25:55.317Z] 22:25:55 INFO - rip = 0x00007f5799eb0972 [task 2018-02-22T22:25:55.317Z] 22:25:55 INFO - Found by: previous frame's frame pointer [task 2018-02-22T22:25:55.317Z] 22:25:55 INFO - 4 0x7f5799eb2a42 [task 2018-02-22T22:25:55.318Z] 22:25:55 INFO - rbp = 0x00007f579823fd20 rsp = 0x00007f579823fd00 [task 2018-02-22T22:25:55.318Z] 22:25:55 INFO - rip = 0x00007f5799eb2a42 [task 2018-02-22T22:25:55.318Z] 22:25:55 INFO - Found by: previous frame's frame pointer [task 2018-02-22T22:25:55.319Z] 22:25:55 INFO - 5 0x7f5799ebed18 [task 2018-02-22T22:25:55.319Z] 22:25:55 INFO - rbp = 0x00007f579823ff00 rsp = 0x00007f579823fd30 [task 2018-02-22T22:25:55.319Z] 22:25:55 INFO - rip = 0x00007f5799ebed18 [task 2018-02-22T22:25:55.319Z] 22:25:55 INFO - Found by: previous frame's frame pointer [task 2018-02-22T22:25:55.320Z] 22:25:55 INFO - 6 0x7f5799eb048a [task 2018-02-22T22:25:55.320Z] 22:25:55 INFO - rbp = 0x00007f579823ff10 rsp = 0x00007f579823ff10 [task 2018-02-22T22:25:55.320Z] 22:25:55 INFO - rip = 0x00007f5799eb048a [task 2018-02-22T22:25:55.321Z] 22:25:55 INFO - Found by: previous frame's frame pointer

Awsy is basically permafailing on linux-qr and without stack information it's not easy to figure out what's going on. Is there something we can do here?

Recent example: https://taskcluster-artifacts.net/Qimv-GM_S-21fqlOdBfECw/0/public/logs/live_backing.log [task 2018-06-01T13:35:33.671Z] 13:35:33 INFO - Calling ['marionette', '--testvars=/builds/worker/workspace/build/tests/awsy/conf/testvars.json', '--testvars=/builds/worker/workspace/build/tests/awsy/runtime-testvars.json', '--log-raw=-', '--log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/marionette_errorsummary.log', '--binary=/builds/worker/workspace/build/application/firefox/firefox', '--profile=/builds/worker/workspace/build/profile', '--gecko-log=/builds/worker/workspace/build/blobber_upload_dir/gecko.log', '--symbols-path=https://queue.taskcluster.net/v1/task/CuHAL3TMQ8m1n5m2Tl91Gg/artifacts/public/build/target.crashreporter-symbols.zip', '--preferences=/builds/worker/workspace/build/tests/awsy/conf/prefs.json', '/builds/worker/workspace/build/tests/awsy/awsy/test_memory_usage.py'] with output_timeout 6500 It appears that we properly set the --symbols-path in https://searchfox.org/mozilla-central/source/testing/mozharness/scripts/awsy_script.py#174 [task 2018-06-01T13:55:50.817Z] 13:55:50 INFO - mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/CuHAL3TMQ8m1n5m2Tl91Gg/artifacts/public/build/target.crashreporter-symbols.zip [task 2018-06-01T13:55:56.847Z] 13:55:56 INFO - mozcrash Copy/paste: /builds/worker/workspace/build/linux64-minidump_stackwalk /tmp/tmpfa1W8e.profile/minidumps/6e2529ce-5fee-04a4-3b83-6a41e7452d5a.dmp /tmp/tmpbfp46y [task 2018-06-01T13:55:56.884Z] 13:55:56 INFO - mozcrash Saved minidump as /builds/worker/workspace/build/blobber_upload_dir/6e2529ce-5fee-04a4-3b83-6a41e7452d5a.dmp [task 2018-06-01T13:55:56.884Z] 13:55:56 INFO - mozcrash Saved app info as /builds/worker/workspace/build/blobber_upload_dir/6e2529ce-5fee-04a4-3b83-6a41e7452d5a.extra We can see that we hit: https://searchfox.org/mozilla-central/source/testing/mozbase/mozcrash/mozcrash/mozcrash.py#180 in CrashInfo._get_symbols which means we downloaded the symbols to a temporary path which seems ok unlike in bug 1276011. whimboo, ted: Any insights you can provide since you've seen similar things in the past?

This looks like some problem in minidump stackwalk maybe. At that level I cannot help and would have to defer to Ted.

(In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #1) > Awsy is basically permafailing on linux-qr and without stack information > it's not easy to figure out what's going on. Is there something we can do > here? Yeah this looks like a ted problem. In the meantime you could download the binary and symbols and fix the stacks locally or you could attempt to repro locally with |./mach awsy-test|.

I didn't look at the actual minidump, just the log, but the log doesn't show any module names, which means the minidump is probably busted. gsvelto might be able to diagnose, he's seen similar things while trying to update Breakpad.

Gabrielle - any ideas what is going on here?

I've looked into this a bit and I couldn't find any occurrence of this past the 26th of June. Recent crashes seem to have proper stack traces. I picked an older failure which exhibited the problem [1] but my analysis was rather inconclusive. The crash address falls within the libxul.so mapping but there are no symbols or functions there. In fact it seems that the code has jumped out of the .text section altogether. From what I can tell the problem isn't the lack of symbols, but the lack of valid code at the crashing point. [1] https://treeherder.mozilla.org/logviewer.html#?job_id=184986735&repo=mozilla-inbound&lineNumber=1279

Hm, ok. Let's close this bug then, if we get more crashes in bug 1440545 we can take a closer look.

We got more crashes sans symbolication in bug 1440545, can you take another look?

Tracking that as a more general issue in bug 1510574.

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #10) > Tracking that as a more general issue in bug 1510574. Bug 1510574 seems to have just started recently. But we've been getting these unsymbolicated stacks for a few weeks/months now in bug 1440545. Anyway we can reopen this bug again if resolving bug 1510574 doesn't fix it.