Seen while running the latest trunk on 10.5.8. STR: 1. Load ftp://ftp.mozilla.org/pub/firefox/nightly/10.0b6-candidates/build1/mac/en-US/ 2. Click on the Mac .dmg file in that directory 3. I get the attached screenshot

Unable to reproduce the crash using Mozilla/5.0 (Macintosh; Intel Mac OS X 10.5; rv:10.0) Gecko/20100101 Firefox/10.0 on the same machine (latest beta).

From the Mac console app: 1/25/12 11:56:53 AM [0x0-0x51051].org.mozilla.nightly ACR (Component): component init 1/25/12 11:56:53 AM [0x0-0x51051].org.mozilla.nightly *** nsHTMLElement::FinishDecoderSetup() mDecoder=0x23401560 stream=0x24d80000 src=http://videos-cdn.mozilla.net/serv/qa/qa-360p-mp4.theora.ogv 1/25/12 11:56:53 AM [0x0-0x51051].org.mozilla.nightly *** nsHTMLElement::FinishDecoderSetup() mDecoder=0x23401560 stream=0x24d80000 src=http://videos-cdn.mozilla.net/serv/qa/qa-360p-mp4.theora.ogv 1/25/12 11:57:09 AM [0x0-0x51051].org.mozilla.nightly dyld: Library not loaded: /usr/lib/libcrypto.0.9.8.dylib 1/25/12 11:57:09 AM [0x0-0x51051].org.mozilla.nightly Referenced from: /Users/mozilla/Desktop/trunk/Nightly.app/Contents/MacOS/crashreporter.app/Contents/MacOS/crashreporter 1/25/12 11:57:09 AM [0x0-0x51051].org.mozilla.nightly Reason: image not found

Comment on attachment 591574 [details] Crash reporter crashing The key bit here is: Library not loaded: /usr/lib/libcrypto.0.9.8.dylib Reason: image not found

So we're linking the crashreporter client to libcrypto: http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/client/Makefile.in#89 but I don't think it actually uses it directly. It just uses Cocoa classes to do the HTTP POST, I assume they'll load libcrypto to do SSL if need be. The Mac-specific client bits: http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/client/crashreporter_osx.mm http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/google-breakpad/src/common/mac/HTTPMultipartUpload.m

I'd guess the broken builds are built with the 10.6 SDK, not the 10.5 SDK. Let me see if I can confirm my hunch (by among other things finding the regression range). Here's Ted's Crashme extension to test with: http://ted.mielczarek.org/mozilla/crashme.html

I think that was bug 674655, but I don't see the changesets listed there.

Also your comment in bug 674655 comment 22 seems quite prescient now :)

Here's the regression range: firefox-2012-01-19-03-11-05-mozilla-central firefox-2012-01-20-03-11-25-mozilla-central > I think that was bug 674655 Actually it was bug 715397: Bug 715397 - ability to build m-c on 10.7 (lion). r=ted. author Rafael Ávila de Espíndola <respindola@mozilla.com> Wed Jan 18 15:35:28 2012 -0500 (at Wed Jan 18 15:35:28 2012 -0500) http://hg.mozilla.org/mozilla-central/rev/429503206f26

> Also your comment in bug 674655 comment 22 seems quite prescient now :) Yes, indeed. I suppose it's a good sign that this bug is the only major breakage discovered since the patch for bug 715397 was landed 3+ weeks ago. Or is it? :-) I think it's a major disaster for Breakpad to be broken on OS X 10.5, so I'm labeling this a "blocker".

STR (for OS X 10.5.8): 1) Run a recent mozilla-central nightly (2012-01-20 or later): 2) If you haven't already, install Ted Mielczarek's Crashme extension: http://ted.mielczarek.org/mozilla/crashme.html 3) Choose one of the crash options from Tools : Crash me! -- for example Null pointer deref! An Apple crash report will appear, with (more or less) the following contents: Process: crashreporter [1877] Path: /Users/smichaud/Desktop/FirefoxNightly 2012-02-13.app/Contents/MacOS/crashreporter.app/Contents/MacOS/crashreporter Identifier: crashreporter Version: ??? (???) Code Type: X86 (Native) Parent Process: firefox [1863] Interval Since Last Report: 84 sec Crashes Since Last Report: 1 Per-App Interval Since Last Report: 0 sec Per-App Crashes Since Last Report: 1 Date/Time: 2012-02-13 10:26:40.583 -0600 OS Version: Mac OS X 10.5.8 (9L30) Report Version: 6 Anonymous UUID: C9512736-EC85-4CD1-B209-AB5833DEE8E2 Exception Type: EXC_BREAKPOINT (SIGTRAP) Exception Codes: 0x0000000000000002, 0x0000000000000000 Crashed Thread: 0 Dyld Error Message: Library not loaded: /usr/lib/libcrypto.0.9.8.dylib Referenced from: /Users/smichaud/Desktop/FirefoxNightly 2012-02-13.app/Contents/MacOS/crashreporter.app/Contents/MacOS/crashreporter Reason: image not found The OS X 10.5.8 version of libcrypto.dylib is 0.9.7.

> I suppose it's a good sign that this bug is the only major breakage > discovered since the patch for bug 715397 was landed 3+ weeks ago. > Or is it? :-) What I meant is does anyone know of other kinds of breakage?

> So we're linking the crashreporter client to libcrypto: > http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/client/Makefile.in#89 > > but I don't think it actually uses it directly. So the obvious thing to try is to get rid of that line. I'll do that and see what happens.

Does the crashreporter client build if we drop the -lcrypto from the Makefile? If so that might be the simplest fix.

> Does the crashreporter client build if we drop the -lcrypto from the Makefile? Yes. I haven't yet gotten Breakpad to work in my builds, but that's probably because I'm not building correctly. I'll keep trying. I'll also do some tryserver builds. I've found another explicit link to libcryto.dylib for building dump_syms on the Mac. That isn't needed either (dump_syms builds fine without it). Shortly I'll post a patch and ask you to review it.

Stop explicitly linking libcrypto on the Mac, to avoid breakage on OS X 10.5 when building with the 10.6 SDK.

This bug also exists on the aurora branch -- it's reproducible with today's aurora nightly.

Comment on attachment 596724 [details] [diff] [review] Fix Review of attachment 596724 [details] [diff] [review]: ----------------------------------------------------------------- I think the dump_syms one is leftover from before we fixed bug 599475, which replaced use of libcrypto with Breakpad's internal MD5 implementation.

> I haven't yet gotten Breakpad to work in my builds, but that's > probably because I'm not building correctly. I'll keep trying. Breakpad now works fine (with my patch) on OS X 10.5.8, in a build made with the following mozconfig (which is the same as, or at least very close to, the mozconfig used for Intel-universal nightly builds): export CFLAGS="-gdwarf-2" export CXXFLAGS="-gdwarf-2" . $topsrcdir/browser/config/mozconfig . $topsrcdir/build/macosx/universal/mozconfig mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj-firefox-nightly mk_add_options MOZ_MAKE_FLAGS=-j4 mk_add_options AUTOCONF=autoconf213 ac_add_options --enable-application=browser ac_add_options --enable-tests ac_add_options --enable-codesighs ac_add_options --disable-install-strip ac_add_options --enable-js-diagnostics ac_add_options --with-macbundlename-prefix=Firefox # For NSS symbols export MOZ_DEBUG_SYMBOLS=1 ac_add_options --enable-debug-symbols="-gdwarf-2" # Needed to enable breakpad in application.ini export MOZILLA_OFFICIAL=1

Comment on attachment 596724 [details] [diff] [review] Fix Landed on mozilla-inbound: http://hg.mozilla.org/integration/mozilla-inbound/rev/8630cfc50df9

We should absolutely take this on aurora once we've verified that it fixes the problem in nightly.

This patch didn't make it into today's nightly. I'll test with tomorrow's, then (presuming it works as expected) I'll request approval for aurora.

Comment on attachment 596724 [details] [diff] [review] Fix Today's mozilla-central nightly works fine on OS X 10.5.8. You get the expected results from the STR in comment #10 -- Breakpad functions normally. On that strength I'm requesting approval to land on the aurora branch. This is a trivial patch, with almost zero risk, that fixes a very serious problem.

Comment on attachment 596724 [details] [diff] [review] Fix [Triage Comment] Low risk, prevents a blind spot in crash data. Approving for Aurora 12.

Comment on attachment 596724 [details] [diff] [review] Fix Landed on aurora: http://hg.mozilla.org/releases/mozilla-aurora/rev/dd496886b608

I have followed the steps from comment10 and no Apple crash report appears, only the Firefox crash reporter. It's enough to mark this as Verified on: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.5; rv:12.0) Gecko/20100101 Firefox/12.0 beta 3 ? Thanks

Yes, if the Firefox crash reporter appears then the bug is fixed.

Considering comment26 and also comment27, setting the resolution to Verified Fixed on Firefox 12 beta.

I have verified this using the steps from comment10 and no Mac crash reporter appeared. Mozilla/5.0 (Macintosh; Intel Mac OS X 10.5; rv:13.0) Gecko/20100101 Firefox/13.0 beta 2 Setting the resolution to Verified Fixed