User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 Build Identifier: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 Running the "firefox" Bash script on Mac OS X 10.6 (Snow Leopard) crashes with the following error message: dyld: Library not loaded: /usr/lib/libsqlite3.dylib Referenced from: /System/Library/Frameworks/Security.framework/Versions/A/Security Reason: Incompatible library version: Security requires version 9.0.0 or later, but libsqlite3.dylib provides version 1.0.0 This is because the shell script alters DYLD_LIBRARY_PATH such that the libsqlite3.dylib file in the Firefox directory takes precedence over the one in /usr/lib, and the Firefox version is incompatible with Snow Leopard. Executing the "firefox-bin" binary directly does not cause this problem, since the default DYLD_LIBRARY_PATH is being used. See http://support.mozilla.com/tiki-view_forum_thread.php?forumId=1&comments_parentId=432894 for some comments about this. It affects users trying to start the Profile Manager, and Selenium-RC. Reproducible: Always Steps to Reproduce: 1. Install Firefox 3.5.2 on Mac OS X 10.6. 2. Open the Terminal application. 3. Run the following command: "/Applications/Firefox.app/Contents/MacOS/firefox" Actual Results: Fails with the following message: dyld: Library not loaded: /usr/lib/libsqlite3.dylib Referenced from: /System/Library/Frameworks/Security.framework/Versions/A/Security Reason: Incompatible library version: Security requires version 9.0.0 or later, but libsqlite3.dylib provides version 1.0.0 Expected Results: Launches successfully. Either a newer version of libsqlite3.dylib must be included, or maybe none at all (use the version that comes with the OS). The problem can be worked around by manually replacing the /Applications/Firefox.app/Contents/MacOS/libsqlite3.dylib with the version in /usr/lib.

In case it helps, I confirmed that removing libsqlite3.dylib from the Firefox directly entirely will also fix the problem.

I verified that the same problem exists in Firefox 3.0.13.

Since running firefox-bin is the default, and we don't really use the shell script for much on mac, this isn't an especially important bug as filed. The snow leopard libsqlite is going to be incompatible with Firefox code in subtle and important ways. Removing the local copy is definitely the wrong solution and should not be recommended! We should rename our libsqlite, but that's hard partly because of the external NSS dependency on it. It sucks that Mac is shipping a system one, given the compatibility nightmares that sqlite is.

(In reply to comment #3) > We should rename our libsqlite, but that's hard partly because of the external > NSS dependency on it. It sucks that Mac is shipping a system one, given the > compatibility nightmares that sqlite is. As far as I know, they've shipped one in the past too. I don't really understand why it's suddenly an issue in 10.6.

Shawn, That's a good question. According to "otool -L", the version of libsqlite3.dylib that ships with Leopard is the same as the one in Snow Leopard ("compatibility version 9.0.0, current version 9.6.0"), although the actual files are not byte-for-byte the same. My guess, based on the error message, is that there is a new in-between dependency issue (via the Security framework) that wasn't there before.

(In reply to comment #3) > The snow leopard libsqlite is going to be incompatible with Firefox code in > subtle and important ways. Removing the local copy is definitely the wrong > solution and should not be recommended! If I understand correctly, when you execute firefox-bin (via double-clicking Firefox.app or via the command line), the OS version of libsqlite3.dylib is *already* being used. This is why Firefox doesn't crash. You only see the problem when you force the local copy to be used, via DYLD_LIBRARY_PATH.

That would be bad! We should check on that using DYLD_PRINT_LIBRARIES (see `man dyld` for details)

(In reply to comment #7) > That would be bad! We should check on that using DYLD_PRINT_LIBRARIES (see `man > dyld` for details) Interestingly, this shows both libraries being used, but the one in /usr/lib comes first. I'll attach the complete output.

I am not a hard-core Mac developer, but I may know what's going on. I tried "otool -L | grep sqlite" on the various executable files in the Firefox.app/Contents/MacOS directory -- namely, "firefox-bin", "XUL", and "*.dylib". Here is what I found: there are only two executables that link directly to sqlite. They are XUL and libsoftokn3.dylib. For each of these, otool shows the following output: @executable_path/libsqlite3.dylib (compatibility version 1.0.0, current version 1.0.0) So they are getting the local copy, no matter what (see `man dyld`). However, when DYLD_LIBRARY_PATH is set, then any code needed by the OS *also* ends up using the local copy, which is why the crash occurs. I don't know what the best solution is, but I believe this is the problem :-).

So if you don't use the bash script (which you don't need to on Mac) then you don't see this problem? My patch in bug 463605 (landed on trunk) makes us stop packaging that file anyway.

That is correct, the crash only happens when firefox-bin is launched by the bash script (or any other program that messes with DYLD_LIBRARY_PATH).

I am getting a very similar error building Mozilla on Snow Leopard.

The issue is with shlibsign. The script sign.sh has to update DYLD_LIBRARY_PATH, but then dyld incorrectly chooses Mozilla's sqlite library instead of the one in /usr/lib. Putting /usr/lib in DYLD_LIBRARY_PATH "fixes" the problem. I think the real fix involves renaming Moz's sqlite.

(In reply to comment #14) Instead of modifying sign.sh can you try a clean build. export NSS_USE_SYSTEM_SQLITE=1 I did not try to do a full mozilla build; I built only NSPR/NSS. Setting NSS_USE_SYSTEM_SQLITE will cause NSS to not build it's own version of sqlite and use the installed system version.

Dup of bug 511608?

If I fix bug 515002, fixing config.guess, we can't even build with the sqlite library incompatibility in 64-bit mode. This is the error output, dyld killing the build.

Builds on my old Core Duo Macbook also fail with the dyld error. Also I've seen that using the cross-compile instructions for i386 on a Core 2 will fail if you build a second time without making any changes to any files.

While this might not be important to fix, the official documentation for Mac OS X calls firefox instead of firefox-bin. http://kb.mozillazine.org/Profile_Manager#Accessing_the_Profile_Manager http://support.mozilla.com/en-US/kb/Managing+profiles

If we rename our sqlite, can we just provide a way to indicate the sqlite library name to NSS, so that everything is linked to libmozsqlite.dylib or whatever? Limi: so fix it, those are both wikis! (also, managing profiles is not intended to be first-class UI, and that support article notes it)

(Submitted a correction to the knowledge base, can't seem to register an account to fix the MozillaZine version, though)

(In reply to comment #21) > If we rename our sqlite, can we just provide a way to indicate the sqlite > library name to NSS, so that everything is linked to libmozsqlite.dylib or > whatever? Just need NSS to implement something to let us do that. I'm then worried about linking at runtime with two different versions of SQLite (this may not be a problem, but I don't know). I haven't had time to file a bug on NSS to do that yet.

This makes it impossible for me to run mochitests right now.

I suggest filing a different bug for mochitests: the fix there is much simpler, and just involves not setting DYLD_LIBRARY_PATH in any of the scripts or test automation.

Filed bug 516789 for the mochitest issue

This temporary patch works around the problems for me. We really need a real fix in the tree soon.

(In reply to comment #24) > This makes it impossible for me to run mochitests right now. Eh, this makes it impossible to build a product to be shipped on Snow Leopard and built on Snow Leopard too (and that should now be the default on Mac...) Bsmedberg, you said "this isn't an especially important bug as filed". From a technical point of view, I agree. From a product release point of view, that IS an important bug to fix...

"as filed" he meant "running the firefox shell script", which is not the way you need to run Firefox. The issues with building on 10.6 are important, yes. (but it's not like 10.6 is some sort of mandatory upgrade, you know.)

And it's impossible to run a XULRunner app with XULRunner… It's still possible with Firefox, but as glazou sais, it is needed to redistribute Xulrunner with Apps

Comment on attachment 403008 [details] [diff] [review] temp patch I'm pretty sure this patch breaks using system SQLite (used by linux distros).

Comment on attachment 403008 [details] [diff] [review] temp patch (In reply to comment #31) > (From update of attachment 403008 [details] [diff] [review]) > I'm pretty sure this patch breaks using system SQLite (used by linux distros). Of course, we don't actually support that, so I'm OK with a follow-up bug being filed and let a distro fix that. This doesn't actually work though because of NSS still, right?

It works on Leopard because NSS ends up linking to the system sqlite instead of the Mozilla one. But that's obviously not something that we can check into the tree.

(In reply to comment #33) > It works on Leopard because NSS ends up linking to the system sqlite instead of > the Mozilla one. But that's obviously not something that we can check into the > tree. Right, so it won't work on Windows. I have filed bug 519550 for the NSS bits of this.

(In reply to comment #32) > (From update of attachment 403008 [details] [diff] [review]) > (In reply to comment #31) > > (From update of attachment 403008 [details] [diff] [review] [details]) > > I'm pretty sure this patch breaks using system SQLite (used by linux distros). > Of course, we don't actually support that, so I'm OK with a follow-up bug being > filed and let a distro fix that. > Might be nice to let distro know in advance.

Quickly going through the patch, I see nothing that would break using system SQLite.

(In reply to comment #36) > Quickly going through the patch, I see nothing that would break using system > SQLite. won't we be linking with mozsqlite3 instead of sqlite3?

if test -z "$MOZ_NATIVE_SQLITE" then SQLITE_CFLAGS= - SQLITE_LIBS='$(call EXPAND_LIBNAME_PATH,sqlite3,$(DIST)/lib)' + SQLITE_LIBS='$(call EXPAND_LIBNAME_PATH,mozsqlite3,$(DIST)/lib)' else PKG_CHECK_MODULES(SQLITE, sqlite3 >= $SQLITE_VERSION) fi No, in the system sqlite case, we still check for libsqlite3.

Ah, right. We use SQLITE_LIBS everywhere instead of specifying sqlite3. Haven't heard anything from the NSS folks yet about this.

This is the last thing preventing people from doing native compiles on 10.6 now (64-bit).

Comment on attachment 403008 [details] [diff] [review] temp patch actually, this needs to have the removed-files.in change too.

We can't take my temp patch because it breaks some builds on systems that don't have sqlite (iirc). So we need my patch plus NSS modifications and the removed-files.in changes from comment 41.

This issue is very nasty for me as a user of Selenium-RC (automated website testing). Selenium-RC invokes Firefox via the shell script.

confirm this happens to me too on Mac OS X 10.6.1 with new firefox 3.5.4 See https://support.mozilla.com/en-US/forum/1/486263

I can also confirm that this fails using firefox 3.5.4

This bug needs no confirmation. Its status is NEW and we know it happens across branches.

This has been stalled with a partial patch for about three weeks. See c42. Selenium is a huge tool in the web testing world. This is going to start affecting our own Selenium test automation as we dramatically expend our web site automated tests and 10.6 becomes more important. Who on the NSS team do we need to reach out to?

(In reply to comment #48) > Who on the NSS team do we need to reach out to? This bug is blocked on bug 519550 (as noted in the dependency tree). Your best bet is to ask there.

(In reply to comment #44) > This issue is very nasty for me as a user of Selenium-RC (automated website > testing). Selenium-RC invokes Firefox via the shell script. This is written up in Selenium bug SRC-743 (http://jira.openqa.org/browse/SRC-743)

I think this should block release. We want to move to a future where we are going to run tests on builds we actually ship and this is going to block that effort.

I think Selenium should be able to fix their issue independently of us by: 1) Running firefox-bin, not firefox 2) Not setting DYLD_LIBRARY_PATH

This builds on Josh's patch + on top of my patch from bug 519550. It looks like it does the trick, I've pushed it to the try server for a sanity check. (I've only tested on OS X). I'll link the try server builds here when they finish, since I don't have 10.6.

Steps to Reproduce (in Mountain View QA Lab. Selenium RC has been installed here.): 1. On the 10.6 machine launch Terminal 2. enter: java -jar Downloads/selenium-remote-control-1.0.1/selenium-server-1.0.1/selenium-server.jar -interactive and hit return 3. enter: cmd=getNewBrowserSession&1=*firefox Desktop/Firefox3.5.2/Firefox.app/Contents/MacOS/firefox&2=http://www.mozilla.com and hit return You will see error message:firefox-bin quit unexpectedly.

We know exactly what the problem is, thanks. We don't need any more confirmation.

I asked Vishal to add his case since it was in-house. Sorry if it seemed excessive.

Comment on attachment 410087 [details] [diff] [review] change sqlite3 to mozsqlite3 Works perfectly for me. Fixes 32-bit and 64-bit builds on Mac OS X 10.6. Thanks Ted!

Is it possible for a user to apply this patch/patches "in the field" or even better to get a patched version of Firefox?

Ted, is renaming the library enough? If symbols overlap, we can run into very subtle and hard-to-diagnose bugs. What are the linkages on the sqlite symbols?

Joe: I can't say I know that much about how the OS X dynamic linker works. It would probably be good to figure that out. Also, here's the try server build: https://build.mozilla.org/tryserver-builds/tmielczarek@mozilla.com-try-4e83bd0c73ac/try-4e83bd0c73ac-macosx.dmg

These patches failed on Win32 on the try server. Looks like it hung executing shlibsign. I don't have a Win32 machine handy at the moment, if someone wants to test it there that would be cool.

Joe: after some research, it sounds like dyld is smarter than that: http://www.unumu.com/docs/oreilly/unix3/mac/ch05_03.htm "Mac OS X 10.1 introduced two-level namespaces, which the compiler uses by default. (Mac OS X 10.2 does not introduce any changes to two-level namespaces.) With this feature, you can link against version 1 of libfoo and libbar. The linker creates an application that knows logerror( ) lives in libfoo. So, even if a future version of libbar includes a logerror( ) function, your application will know which logerror( ) it should use."

(In reply to comment #59) > Is it possible for a user to apply this patch/patches "in the field" or even > better to get a patched version of Firefox? As a user there is no reason you should run into this. Simply run "firefox-bin" instead of "firefox" if you must run Firefox from the commandline.

(In reply to comment #64) > (In reply to comment #59) > > Is it possible for a user to apply this patch/patches "in the field" or even > > better to get a patched version of Firefox? > > As a user there is no reason you should run into this. Simply run "firefox-bin" > instead of "firefox" if you must run Firefox from the commandline. Much easier said than done. I'm not launching firefox directly from the command line, it's being launched by Selenium, and I haven't been able to figure out how to change the path it uses. That seems to be buried somewhere inside a Java jar file.

(In reply to comment #65) > Much easier said than done. I'm not launching firefox directly from the > command line, it's being launched by Selenium, and I haven't been able to > figure out how to change the path it uses. (In reply to comment #64) > As a user there is no reason you should run into this. Simply run "firefox-bin" > instead of "firefox" if you must run Firefox from the commandline. Rick, you do it with the browser_key setting (e.g. "*chrome /path/to/firefox"). However, Selenium refuses to run firefox-bin directly. I had to create a separate shell script to run it, so I don't think Ted's comment holds up.

Then that's a bug in Selenium. There's no reason it shouldn't be able to launch firefox-bin directly. Never fear, we will fix this bug anyway, but you should try to get the Selenium folks to fix their side as well.

I hope that the two-level namespace also works for references _inside_ libraries. Like if mozsqlite called sqlite_close(), is it guaranteed to always call mozsqlite.dyld:sqlite_close()?

It seems insane to assume otherwise.

(In reply to comment #67) > Then that's a bug in Selenium. There's no reason it shouldn't be able to launch > firefox-bin directly. Never fear, we will fix this bug anyway, but you should > try to get the Selenium folks to fix their side as well. If you launch firefox-bin directly from Selenium it picks up Firefox's re-set DYLD_LIBRARY_PATH and crashes with the same Security Framework libsqlite error. I'm not sure why, but it does - perhaps something to do with the firefox-bin.rsrc file? (clutching at straws a bit there, I'll admit)

(In reply to comment #68) > Like if mozsqlite called sqlite_close(), is it guaranteed to always > call mozsqlite.dyld:sqlite_close()? Yes, in my experience (when using a two-level namespace).

The upstream bug (http://jira.openqa.org/browse/SRC-743) indicates that Selenium is also setting DYLD_LIBRARY_PATH. Running 'firefox-bin' will not do so, AFAIK. So I think that Selenium bug will still need to be fixed even with this patch.

(Although with this patch you probably won't crash.)

(In reply to comment #62) > These patches failed on Win32 on the try server. Looks like it hung executing > shlibsign. I don't have a Win32 machine handy at the moment, if someone wants > to test it there that would be cool. My updated patch on bug 519550 should fix that. I missed a Windows case in nss/cmd, so it would make sense that shlibsign would fail!

Comment on attachment 410087 [details] [diff] [review] change sqlite3 to mozsqlite3 r=sdwilsh as long as you add the sqlite library to removed-files.in or whatever it's called these days.

Disclaimer - I'm out of my element in this discussion - but some of the commentary above is about how you would run into this bug and I think I have a new way not previously mentioned which may help with rating its importance. I think the same bug causes Firefox to crash when trying to use Cisco Systems Smart Tunnel (Clientless SSL VPN). After logging into the VPN website through a Firefox window and using a hyperlink to access Outlook Web Access I'm prompted to allow tunnel_mac.jnlp to start via Java Web Start. I then get the following message when just after it looks like a new window is trying to open: ***** Process: firefox-bin [3371] Path: /Applications/Firefox.app/Contents/MacOS/firefox-bin Identifier: firefox-bin Version: ??? (???) Code Type: X86 (Native) Parent Process: sh [3367] Date/Time: 2009-11-05 22:28:59.974 -0800 OS Version: Mac OS X 10.6.1 (10B504) Report Version: 6 Anonymous UUID: 8280FBAE-0B10-45E0-8CD7-EBFE406348D3 Exception Type: EXC_BREAKPOINT (SIGTRAP) Exception Codes: 0x0000000000000002, 0x0000000000000000 Crashed Thread: 0 Dyld Error Message: Library not loaded: /usr/lib/libsqlite3.dylib Referenced from: /System/Library/Frameworks/Security.framework/Versions/A/Security Reason: Incompatible library version: Security requires version 9.0.0 or later, but libsqlite3.dylib provides version 1.0.0 Model: iMac4,1, BootROM IM41.0055.B08, 2 processors, Intel Core Duo, 1.83 GHz, 2 GB, SMC 1.1f5 . . . *****

Added the old sqlite3 file to removed-files.in.

This doesn't block our product release. As I understand it, double clicking the Firefox.app icon works, and running the appropriate command in the shell (firefox-bin, not firefox) also works. Yes, it's causing problems with our unit test framework on 10.6 and with 64 bit builds, and thus it is wanted and once baked properly and tested I'll happily approve the patch (come find me on IRC or ring my phone, you'll find I'm happy to take your call) but I would still happily ship Firefox 3.6 to our hundreds of millions of users without it.

What's the status here. For anyone on 10.6 the tree is essentially permanently orange, something that we'd normally back things out for if we had actually had a tinderbox on 10.6. This is a severe problem for me. It used to be that I couldn't run mochitests. Comment 25 claims that that can be worked around (though I don't understand why we should spend time developing a workaround when it seems known what the actual problem is?) However I can no longer even start MinefieldDebug builds and am forced to use a tree that's several days old where the attached patch actually works.

(In reply to comment #79) Well I hope that it doesn't stay unfixed as long as https://bugs.launchpad.net/ubuntu/+source/grub/+bug/8497 Of course I don't thing that the incompatibility between grub and bios drive renumbering can be fixed, although we know that THIS bug can be.

(In reply to comment #79) > What's the status here. We are waiting on bug 519550 as noted by the dependency. It's an NSS dependency, and out of our control, hence the delay.

This is a problem we have to fix so can we just take the necessary fixes now even if that means taking a local, out-of-process patch to NSS and make sure to fix this the proper way when we can, certainly prior to any release?

Waiting isn't really ok here. If we had proper platform coverage we'd have an perma-orange tree right now. Renominating for blocking as currently the only way I can test patches is to push to try-server and see if it builds there, which is not an ok situation in case we urgently need to fix a security bug or last minute blocker.

Technically you can build with the patch here, that is what I have been doing, but it is annoying and not really acceptable, I agree.

Pulling tip, applying attachment 411453 [details] [diff] [review] and then running either mochitest or the firefox shellscript still gives me: dyld: Library not loaded: /usr/lib/libsqlite3.dylib Referenced from: /System/Library/Frameworks/Security.framework/Versions/A/Security Reason: Incompatible library version: Security requires version 9.0.0 or later, but libsqlite3.dylib provides version 1.0.0 running firefox-bin directly works fine though. So at least I can start the browser. Doing any mochitesting still puts me back several hours while pushing to tryserver though. Not really a workable solution if your patch requires writing a mochitest...

You also need the patch from bug 519550 (and you'll need to rebuild, maybe even clobber).

(In reply to comment #86) > You also need the patch from bug 519550 (and you'll need to rebuild, maybe even > clobber). Or alternatively: find $OBJDIR | grep libsqlite3.dylib | xargs rm

Not blocking as per comment 84 and comment 87, and the fact that we can happily ship a product without this fix. I'm by no means saying that we shouldn't fix it, nor that the fix shouldn't be given priority and development time (though it sounds like we need to ask nicely of the NSS folks, really) but I am saying that we can ship without it.

I updated the OSX build requirements page, please revert once this bug is fixed: https://developer.mozilla.org/En/Developer_Guide/Build_Instructions/Mac_OS_X_Prerequisites

Because people seemed not to know how to disable the setting of DYLD_LIBRARY_PATH in order to run mochitests on Snow Leopard, here's a patch to do that.

(In reply to comment #89) > I updated the OSX build requirements page, please revert once this bug is > fixed: > > https://developer.mozilla.org/En/Developer_Guide/Build_Instructions/Mac_OS_X_Prerequisites I spent a few hours last night trying to build from the instructions above and though the build succeeded, I still can't run my Selenium tests. Is there a pre-built version that is confirmed to work somewhere? If not, what is the latest version known to work with Snow Leopard and Selenium?

You'll need to take that up with the Selenium developers, the way Selenium invokes Firefox is not something we can change.

(In reply to comment #92) > You'll need to take that up with the Selenium developers, the way Selenium > invokes Firefox is not something we can change. OK. I was just trying to figure out if the steps for rebuilding Firefox (with the two specified patches) were supposed to be working with Selenium. For those that are still having issues with Selenium, I can confirm that I was able to get Firefox 3.5.5 to open and work properly with Selenium Server using the patch listed on their internal bug. I've added the simple set of steps that I used to make it work: http://jira.openqa.org/browse/SRC-743

I have this problem with Firefox 3.5.5 installed on a new MacBook Pro with Snow Leopard (with latest updates), and launching in the usual way, by double clicking on the Firefox.app entry in /Applications. If I go into the /Contents/MacOS and double click on firefox-bin, I can see the error message, including: Dyld Error Message: Library not loaded: /usr/lib/libsqlite3.dylib Referenced from: /System/Library/Frameworks/Security.framework/Versions/A/Security Reason: Incompatible library version: Security requires version 9.0.0 or later, but libsqlite3.dylib provides version 1.0.0 So, it seems that executing firefox-bin does have this problem, contrary to some of the above discussion.

(In reply to comment #94) > I have this problem with Firefox 3.5.5 installed on a new MacBook Pro with Snow > Leopard (with latest updates), and launching in the usual way, by double > clicking on the Firefox.app entry in /Applications. If I go into the > /Contents/MacOS and double click on firefox-bin, I can see the error message, > including: > Dyld Error Message: > Library not loaded: /usr/lib/libsqlite3.dylib > Referenced from: > /System/Library/Frameworks/Security.framework/Versions/A/Security > Reason: Incompatible library version: Security requires version 9.0.0 or > later, but libsqlite3.dylib provides version 1.0.0 > So, it seems that executing firefox-bin does have this problem, contrary to > some of the above discussion. Sorry. My error. That error message (above) does come from running "firefox". "firefox-bin" just crashes without showing any error message.

I implemented wtc's suggestion from bug 519550. This seems to WFM on OS X. I've pushed it to try server to make sure that I didn't break any other platforms.

Oops, this broke WinMo on the try server, it just needs to check for either Win32 or WinCE in that Makefile test.

Is this a perf and bloat issue on windows? If we load two libraries into memory that increases bloat and means more code cache misses.

We're not actually shipping two libraries, we're just trying to fool NSS into linking to our mozsqlite3 when it's looking for -lsqlite3.

Comment on attachment 416782 [details] [diff] [review] rename to mozsqlite3, use a hack to let NSS continue linking to -lsqlite3 but get mozsqlite3 r=sdwilsh

Comment on attachment 416782 [details] [diff] [review] rename to mozsqlite3, use a hack to let NSS continue linking to -lsqlite3 but get mozsqlite3 >+# This is awful, but NSS expects to link to "sqlite", and changing NSS >+# is like pulling teeth. This should read ..., and changing the NSS freebl and softoken libraries ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ is like pulling teeth. The rest of NSS isn't frozen for FIPS.

wtc: Josh tried this patch, and he's still hitting one problem. When NSS calls sign.sh, it passes $(DIST)/lib as one of the paths, and sign.sh adds that to DYLD_LIBRARY_PATH. http://mxr.mozilla.org/mozilla-central/source/security/nss/cmd/shlibsign/Makefile#118 http://mxr.mozilla.org/mozilla-central/source/security/nss/cmd/shlibsign/sign.sh#31 shlibsign then crashes because an OS X system library tries to load libsqlite3.dylib and finds our symlink instead.

This is an update I discussed with Ted that solves my problem, once. This will delete the symlink before it messes us up but it'll still be gone when you try to re-build NSS and thus NSS will fail until you re-build whatever makes the symlink. It is a start though. We'd need to add code that re-creates the link after we delete it. In any case, I think we can land the proper NSS fix now instead of this. It has r+ and we're OK to locally modify NSS.

Comment on attachment 411453 [details] [diff] [review] change sqlite3 to mozsqlite3 (+removed-files change) Yeah, apparently we can take this patch instead.

Comment on attachment 411453 [details] [diff] [review] change sqlite3 to mozsqlite3 (+removed-files change) r=sdwilsh

This means the MacOS users can kiss FIPS 140 compliance good bye. MoCo might as well remove the FIPS compliance options from Mac builds.

We intend to remove this and use an "official" fix from a merged branch before shipping anything. Such a merge will be a 1.9.3 branch blocker.

You understand that any "official" fix that is a change to the frozen FIPS code means approval by the FIPS lab. You ready to wait another year or two?

Whatever we end up doing by the time we ship, we need to fix this now. This has gone on for long enough. We can change the fix whenever, but not having a fix is not an option.

(In reply to comment #106) > MoCo might as well remove the FIPS compliance options from Mac builds. Sounds like a great idea to me, will you file the bug?

Includes another small fix in: db/sqlite3/src/sqlite.def

Ted, Josh: Although I had been working on a solution with you, I didn't know why you need to rename libsqlite3.dylib. Until last night. I recently bought a Mac mini, which comes with Snow Leopard. I built NSS standalone on the Mac mini for the first time last night. The build failed when the sign.sh script ran. I found that with DYLD_LIBRARY_PATH set to the $(DIST)/lib directory in the NSS source tree, even the cvs command crashed. In the standalone NSS build, the problem is compounded by the older copy of sqlite in NSS because it's missing the new sqlite3_open_v2 function. In any case, now that I fully understand the problem, I'd like to ask you if you considered building Mozilla's libsqlite3.dylib with -compatibility_version 9 -current_version 9. This change fixes the problem for a standalone NSS build (after updating NSS's sqlite to the latest version 3.6.21). Another solution (inferior but avoids changing the NSS softoken library's makefile) is to allow NSS to link with the system libsqlite3.dylib. Although this introduces code bloat, I believe this code bloat is inevitable when users run Firefox with libmozsqlite3.dylib on Snow Leopard because some system library that is linked with the Security Framework will bring the system libsqlite3.dylib into the process anyway.

I'm fine with any solution that works and I certainly prefer cleaner or more correct solutions. My immediate concern though is that we land something ASAP so I plan to land the currently reviewed changes as a temporary fix until we have something else that is actually ready to land. Thanks for your help Wan-Teh, looking forward to having this solved in a way that everyone is happy with.

pushed to mozilla-central http://hg.mozilla.org/mozilla-central/rev/3ff17b03644e

backed out due to Windows orange http://hg.mozilla.org/mozilla-central/rev/51b77a57960c

(In reply to comment #112) > In any case, now that I fully understand the problem, I'd like > to ask you if you considered building Mozilla's libsqlite3.dylib > with -compatibility_version 9 -current_version 9. This change > fixes the problem for a standalone NSS build (after updating > NSS's sqlite to the latest version 3.6.21). That is the workaround I used in Bug 511608. I think it is not good because then Security.framework may use Firefox libsqlite3.dylib. I guess renaming may avoid the code bloat. I'm not sure.

wtc: I'm not so concerned with the code bloat as I am with using a version of sqlite that we haven't tested with. Using the system sqlite means that NSS will be using a version of sqlite (especially on 10.6) that we haven't done any testing against. Using the Mozilla sqlite for a system library is equally scary, since Apple clearly hasn't tested their frameworks against our version.

Arg, I clobbered the tboxes when they failed but I had to run and failed to notice later on that the clobber actually worked. I'll re-land and clobber as soon as I can. FWIW, I couldn't reproduce any issues with testing using a clobber build on Windows 7.

Ted: The NSS packages in various Linux distributions are already using whatever versions of sqlite available on the system. So using the system sqlite on Mac OS X is not new for NSS. I am sure that sqlite has backward compatibility (otherwise it could have just changed its sqlite3_open function rather than adding sqlite3_open_v2), so as long as Mozilla's sqlite is newer than the sqlite in Mac OS X, it should be fine for the Security Framework to use Mozilla's sqlite. In any case, I won't tell you how you should fix this bug in your code. In standalone NSS build, I'll probably fix this bug by either building with the system sqlite or upgrading NSS's sqlite to the latest version and building it with -compatibility_version 9.

wtc, I would review a patch that picks up a newer version of sqlite with the -compability_version flag set. bob

pushed to mozilla-central again, seems to have stuck this time with a clobber http://hg.mozilla.org/mozilla-central/rev/a74853eba492

Just to check, does the installer needs updating too (one way or the other)? http://mxr.mozilla.org/comm-central/search?string=sqlite3&case=on&find=%2Fwindows%2Fnsis%2Fshared%5C.nsh%24

dmose: is this the bug you have to patch on each 1.9.1 release before you can ship Thunderbird 3.0.x?

Answering for dmose: no -- we have backported some of the mozStorage changes in 1.9.2 and had to layer on a fix for async storage calls. Nothing to do w/ this bug.

I had requested blocking-1.9.1 based on a support request, but now that I understand the problem in more detail, I don't think it should block. Removing the nomination.

(In reply to comment #125) > I had requested blocking-1.9.1 based on a support request, but now that I > understand the problem in more detail, I don't think it should block. Removing > the nomination. Well, dmose... That's a *severe* blocker for companies like mine using xulrunner to build apps for third-parties.

(In reply to comment #122) > Just to check, does the installer needs updating too (one way or the other)? > http://mxr.mozilla.org/comm-central/search?string=sqlite3&case=on&find=%2Fwindows%2Fnsis%2Fshared%5C.nsh%24 Eventually filed as bug 540379...

glazou: fair enough, I was speaking and acting purely from a Thunderbird perspective there. Feel free to renominate and make your case to the Gecko drivers...

I notice that this marked as wanted-1.9.2+. Josh, is this waiting on anything in particular to land there?

I don't know of anything holding this up.

(In reply to comment #130) > I don't know of anything holding this up. is this a straight up patch for 1.9.2?

That's my understanding. I'm not sure whether it needs explicit 1.9.2 approval, given that it's already marked wanted+, but it might.

While this page isn't up to date, I'm pretty sure it does need approval. (In this case, approval1.9.2.2+) https://wiki.mozilla.org/TreeRules

(In reply to comment #129) > I notice that this marked as wanted-1.9.2+. Josh, is this waiting on anything > in particular to land there? (In reply to comment #130) > I don't know of anything holding this up. Josh, are you planning on requesting approval for 1.9.2.whatever-we-are on the patch?

I am not planning on following up with a 1.9.2 fix but I'm not opposed to someone else doing so. I suggest you get approval before landing this.

I looked into backporting this whole bunch a little bit tonight, and it's not too bad except for all the OS/2 changes in bug 536451, which interact very badly with all the rewriting to isolate OS/2 in its own 8+3 cell in bug 514665.

I don't think OS/2 matters at all for the 1.9.2 branch.

http://hg.mozilla.org/comm-central/rev/07a296061f40 Port Bug 513747 (rename sqlite3 library to mozsqlite3 to fix compile problems and crashes on Mac OS X) to comm-central

All I did was a merge from the two patches for this bug, plus fixing merge conflicts.

Comment on attachment 468882 [details] [diff] [review] patch for 1.9.2 branch This broke my Windows build, no idea why: LINK : fatal error LNK1181: cannot open input file 'c:\verbosio\unstable\un-opt\dist\lib\sqlite3.lib' make[5]: *** [c:/verbosio/unstable/un-opt/nss/softokn/softokn3.dll] Error 157 make[4]: *** [libs] Error 2 make[3]: *** [libs] Error 2 make[2]: *** [libs_tier_toolkit] Error 2 make[1]: *** [tier_toolkit] Error 2 make: *** [default] Error 2

bug 519550 may not have landed on the 1.9.2 branch.

What Ted said. Alex, at a minimum you need attachment 417032 [details] [diff] [review] (or http://hg.mozilla.org/mozilla-central/rev/a74853eba492) here and *something* from bug 519550, either attachment 417036 [details] [diff] [review] / http://hg.mozilla.org/mozilla-central/rev/f6ee5a7df62f or whatever of the later patches is correct for the current version of NSS on 1.9.2. That error looks suspiciously like NSS is still trying to link to sqlite3, which means you don't have a fix from bug 519550. (Also, there shouldn't be "two patches for this bug", just the one here and one from the NSS bug 519550.) (I never got this finished when I started working because I couldn't correctly merge around all the OS/2 stuff that happened on m-c prior to these patches but after 1.9.2 branched, and ended up with broken Windows builds when I try-ed. And then I had more pressing things to focus on.)