With a debug build of Firefox for Mac, both gdb and lldb are unable to locate NSS sources, so while I can step through Firefox / XUL source, if I enter an NSS function, I lose the source view. I've attached my debug mozconfig. With a similar debug build for Linux, I am able to source debug NSS, so it's something Mac specific. After some searching, I learned[1] that Macs rely on the Stabs object file locations in a binary or library to the read further debug info from them. For NSS, those all seem to point to temporary directories that no longer exist: $ nm -pa libnss3.dylib | grep OSO 00000000539886ef - 03 0001 OSO /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/security/build/tmplH_nUR/derdec.o 00000000539886ef - 03 0001 OSO /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/security/build/tmplH_nUR/derenc.o 00000000539886ef - 03 0001 OSO /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/security/build/tmplH_nUR/dersubr.o 00000000539886ef - 03 0001 OSO /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/security/build/tmplH_nUR/dertime.o 00000000539886ef - 03 0001 OSO /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/security/build/tmplH_nUR/errstrs.o [...many more lines...] For XUL, they point to files that do still exist: $ nm -pa XUL | grep OSO 00000000539886c9 - 03 0001 OSO /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/toolkit/library/build/../nsStaticXULComponents.o 0000000053988573 - 03 0001 OSO /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/toolkit/library/build/../../../media/kiss_fft/kiss_fft.o 0000000053988573 - 03 0001 OSO /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/toolkit/library/build/../../../media/kiss_fft/kiss_fftr.o 0000000053988574 - 03 0001 OSO /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/toolkit/library/build/../../../security/certverifier/Unified_cpp_certverifier0.o 00000000539886d2 - 03 0001 OSO /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/toolkit/library/build/../../../security/pkix/Unified_cpp_security_pkix0.o [1]: http://stackoverflow.com/questions/10044697/where-how-does-apples-gcc-store-dwarf-inside-an-executable#comment25100499_12827463

We do some funny things to build NSS, I'm not entirely surprised here. I haven't checked, but I'm pretty sure we have okay debug symbols for them in our crash reports (at least we used to). If you run dsymutil on libnss3.dylib, is it debuggable then?

No, that doesn't seem to change anything. It does produce the .dSYM directory, but source debugging still fails. dwarfdump of the .dSYM shows that it only seems to contain info about sqlite symbols. (After looking at the full output of nm -pa libnss3.dylib | grep OSO again, it seems the path to the one sqlite file is correct, so I'm guessing that's why those symbols do get picked up.) Also, it produces many warnings, as seen below, about the missing object files, so surely that is why it doesn't contain any of the NSS symbols: $ dsymutil libnss3.dylib warning: (x86_64) /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/security/build/tmplH_nUR/derdec.o unable to open object file warning: (x86_64) /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/security/build/tmplH_nUR/derenc.o unable to open object file warning: (x86_64) /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/security/build/tmplH_nUR/dersubr.o unable to open object file warning: (x86_64) /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/security/build/tmplH_nUR/dertime.o unable to open object file warning: (x86_64) /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/security/build/tmplH_nUR/errstrs.o unable to open object file [...many more lines...]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #1) > We do some funny things to build NSS, I'm not entirely surprised here. I > haven't checked, but I'm pretty sure we have okay debug symbols for them in > our crash reports (at least we used to). Actually, you should double check that, because it might not be the case. And if that's not the case, it would have been that way for as long as we've folded libraries.

Well, shit: [tmielczarek@symbols1.dmz.phx1 symbols_ffx]$ grep libnss3\.dylib\.sym firefox-33.0a1-Darwin-20140612030349-x86-macosx64-symbols.txt libnss3.dylib/570B8897922F38A7B4974CB950443A490/libnss3.dylib.sym libnss3.dylib/E5F4EB6CD046311DB263E3B5C8C5ECDF0/libnss3.dylib.sym libnss3.dylib/570B8897922F38A7B4974CB950443A490/libnss3.dylib.sym libnss3.dylib/E5F4EB6CD046311DB263E3B5C8C5ECDF0/libnss3.dylib.sym [tmielczarek@symbols1.dmz.phx1 symbols_ffx]$ head -n4 libnss3.dylib/570B8897922F38A7B4974CB950443A490/libnss3.dylib.sym MODULE mac x86 570B8897922F38A7B4974CB950443A490 libnss3.dylib FILE 0 /Developer/SDKs/MacOSX10.6.sdk/usr/include/secure/_string.h FILE 1 hg:hg.mozilla.org/mozilla-central:db/sqlite3/src/sqlite3.c:9e8e3e903484 FUNC 12dda0 18d 0 sqlite3_compileoption_used (no NSS source files listed at all)

Then it's been that way since bug 648407.

So, since Firefox 22. Amazing it wasn't found earlier.

Presumably, this is enough to fix this. It should be thoroughly tested, though, because it changes how the linkage is done.

Might not work, though. ISTR the reason i added that is that without, the linker just throws away nss.

The patch in attachment 8439633 [details] [diff] [review] makes the linker quite upset, claiming the NSS symbols are undefined. See attached log.

This fixes half the problem. About half of nspr remains broken, though.

Comment on attachment 8439906 [details] [diff] [review] Use object files next to archives when extracting them at link time, or in a subdirectory This change fixes NSS source debugging for me. Thanks for jumping on this so quickly!

Comment on attachment 8439906 [details] [diff] [review] Use object files next to archives when extracting them at link time, or in a subdirectory Review of attachment 8439906 [details] [diff] [review]: ----------------------------------------------------------------- ::: config/expandlibs_exec.py @@ +108,5 @@ > + # If the file extracted from the library also > + # exists in the directory containing the > + # library, or one of its subdirectories, use > + # that instead. > + maybe_obj = lookup(os.path.join(basedir, root[len(tmp)+1:]), f) I think I'd prefer os.path.relpath(root, tmp). ::: config/tests/unit-expandlibs.py @@ +249,5 @@ > def test_extract(self): > '''Test library extraction''' > # Divert subprocess.call > subprocess_call = subprocess.call > + subprocess_check_output = subprocess.check_output We have mock installed in the virtualenv, can you file a followup on using that here instead? Example: http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/unit-symbolstore.py#188 Docs: http://www.voidspace.org.uk/python/mock/ @@ +324,5 @@ > + else: > + dirname = os.path.dirname(f[len(self.tmpdir)+1:]) > + if base.endswith('f'): > + dirname = os.path.join(dirname, 'foo', 'bar') > + extracted_args.extend([self.tmpfile(dirname, Obj(base)), self.tmpfile(dirname, Obj(base + '2'))]) I kind of wish this was nicer but you wrote a test so I can't really complain.

Note I'm very likely to kill expand_libs soonish. Not sure it's worth filing bugs about its unit test at this point.

Let's give it a few days and uplift then. Flagging me to not forget.

And a fixup: https://hg.mozilla.org/integration/mozilla-inbound/rev/1d0eb8fb8d87

https://hg.mozilla.org/mozilla-central/rev/756d3d9456c3 https://hg.mozilla.org/mozilla-central/rev/1d0eb8fb8d87

Comment on attachment 8439906 [details] [diff] [review] Use object files next to archives when extracting them at link time, or in a subdirectory Approval Request Comment [Feature/regressing bug #]: A long time ago in bug 648407 [User impact if declined]: No NSS/NSPR symbols in crash reports. [Describe test coverage new/current, TBPL]: Landed on m-c a while ago, and verified to add the relevant symbols to debug symbols used by the crash report processing. [Risks and why]: Only risk is bug 1024620, which hasn't been clearly attributed to this bug. We'd watch perf numbers on aurora if this is a+ed. I'd request approval for beta as well if there wasn't this yet unidentified regression. [String/UUID change made/needed]: None

I don't this it's worth the risk of taking this change on beta or ESR 24. Note that 31 is the next ESR train so this issue will exist in ESR 31. We can consider taking this in ESR 31.1.0 if there is significant benefit.

Comment on attachment 8439906 [details] [diff] [review] Use object files next to archives when extracting them at link time, or in a subdirectory Aurora+. Please monitor this fix on Aurora over the next two weeks. I would like to identify any fallout before 32 moves to beta.