Closed Bug 1691022 Opened 4 years ago Closed 3 years ago

Stack traces on macOS/x86-64 seem to only use stack-scanning

Categories

(Toolkit :: Crash Reporting, defect)

x86_64
macOS
defect

Tracking

()

RESOLVED FIXED
91 Branch
Tracking Status
firefox91 --- fixed

People

(Reporter: gsvelto, Assigned: Gankra)

References

Details

Attachments

(1 file)

I stumbled upon this issue while looking at bug 1690753. The stack trace is fully within libxul and as such should be entirely obtained using CFI, instead the stack walker is falling back to stack scanning.

I opened the symbol files for a few builds and noticed that the CFI information appears to be very limited. A macOS/AArch64 build of libxul yields ~400K CFI directives in the .sym file, similarly a Windows/x86-64 one has ~360K; macOS/x86-64 builds have only ~80K CFI directives which suggests that a significant portion of them is missing.

Assignee: nobody → a.beingessner

Additional note: running objdump on the .dSYM files similarly only yield 7k lines of CIE info while my local builds have 80k lines. I'm assuming these dSYM files are the source the .sym files are built from, and that we're probably missing a compiler flag telling clang to generate that info (perhaps -funwind-tables is needed?).

Additional notes:

The .dSYM files are generates by llvm-dsymutil in the main build task (right before they're passed to dump_syms to generate the .sym files). e.g.

/builds/worker/workspace/obj-build/_virtualenvs/init_py3/bin/python -m mozbuild.action.dumpsymbols /builds/worker/workspace/obj-build/toolkit/library/build/XUL /builds/worker/workspace/obj-build/toolkit/library/build/XUL_syms.track --count-ctors
Running: /builds/worker/workspace/obj-build/_virtualenvs/init_py3/bin/python /builds/worker/checkouts/gecko/toolkit/crashreporter/tools/symbolstore.py -c -a x86_64 --vcs-info --install-manifest=/builds/worker/workspace/obj-build/_build_manifests/install/dist_include,/builds/worker/workspace/obj-build/dist/include -s /builds/worker/checkouts/gecko /builds/worker/fetches/dump_syms/dump_syms /builds/worker/workspace/obj-build/dist/crashreporter-symbols /builds/worker/workspace/obj-build/toolkit/library/build/XUL --count-ctors
Starting Mac pre-processing on file: /builds/worker/workspace/obj-build/toolkit/library/build/XUL
Running Mac pre-processing on file: /builds/worker/workspace/obj-build/toolkit/library/build/XUL

/builds/worker/checkouts/gecko/build/macosx/llvm-dsymutil --arch=x86_64 /builds/worker/workspace/obj-build/toolkit/library/build/XUL
Finished processing /builds/worker/workspace/obj-build/toolkit/library/build/XUL in 155.46s

Beginning work for file: /builds/worker/workspace/obj-build/toolkit/library/build/XUL
Processing file: /builds/worker/workspace/obj-build/toolkit/library/build/XUL

/builds/worker/fetches/dump_syms/dump_syms -a x86_64 --type macho -j 2 /builds/worker/workspace/obj-build/toolkit/library/build/XUL.dSYM /builds/worker/workspace/obj-build/toolkit/library/build/XUL
PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"name": "compiler_metrics", "subtests": [{"name": "num_static_constructors", "value": 101, "alertChangeType": "absolute", "alertThreshold": 3}]}]}
Finished processing /builds/worker/workspace/obj-build/toolkit/library/build/XUL in 35.82s

The -funwind-tables flag seems to be passed quite liberally throughout the build, need to further investigate why llvm-dsymutil isn't generating all the info we need (e.g. inspecting the files it takes as input, how they're generated, etc.)

The Breakpad dump_syms source elaborates on this:
https://chromium.googlesource.com/breakpad/breakpad/+/c484031f1f199ee53567241426efffee49008f82/src/tools/mac/dump_syms/dump_syms_tool.cc#113

Are you using your Rust dump_syms rewrite? If so you might need to ensure that you have the same logic.

Oh wow, I bet that's exactly it! Gonna work on this right away.

Nice:

https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/CFinj6sMQ1a_o28nW9lYUA/runs/0/artifacts/public/build/target.crashreporter-symbols-full.tar.zst

XUL has about 150k-180k CFI directives now (we still expect significantly less CFI lines because compact unwind info doesn't bother giving special directions in prologues/epilogues).

\o/

Pushed by abeingessner@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/6c35561239bd Update dump_syms for compact unwind info support. r=calixte
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → 91 Branch
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: