It is time to upgrade the NSS version on the Mozilla trunk from NSS 3.10.2 to NSS 3.11. The difficulty of this NSS upgrade lies in the new freebl shared libraries/DLLs and the associated .chk files added in NSS 3.11. We need to search in the Mozilla source tree for these strings: "softokn" or "softokn3" "freebl_hybrid" or "freebl_hybrid_3" "freebl_pure32" or "freebl_pure32_3" Then do the following (each quoted string represents a shared library/DLL and a .chk file with platform-dependent file name prefix and suffix added) 1. HP-UX 32-bit PA-RISC: Change "freebl_hybrid_3" to "freebl_32fpu_3". Change "freebl_pure32_3" to "freebl_32int_3". 2. Solaris 32-bit SPARC: Change "freebl_hybrid_3" to "freebl_32fpu_3". Change "freebl_pure32_3" to "freebl_32int_3". Add "freebl_32int64_3". 3. Solaris 64-bit SPARC: Add "freebl_64fpu_3" and "freebl_64int_3". 4. All other platforms: Add "freebl3".

Wan-Teh: Firefox 1.5 will still work with NSS 3.11, correct?

First version of patch. Wan-Teh, is it correct to add the new libs to the link command line? I have not yet cared for the unix platform specific library additions.

Wolfgang: Firefox 1.5 is using NSS 3.10.2. Right now we only plan to integrate NSS 3.11 with the Mozilla trunk, targeting Firefox 2.0. Since NSS maintains backward compatibility, Firefox 1.5 will still work with NSS 3.11. Kai: don't add the new libs to the link command line. The new libs are all dynamically loaded with PR_LoadLibraryWithFlags calls at run time. (The "softokn3" library shouldn't be on the link command line either. But it's not necessary to fix that.)

While updating mozilla trunk to nss 3.11 we might want to have the patch from bug 101249 as well?

When I tried to apply patch v1 to the tip, it did not apply cleanly for security/manager/Makefile.in. Attached is the revised portion of the second chunk of that patch so it applies cleanly for me.

(In reply to comment #3) > Right now we only plan to integrate NSS 3.11 with the > Mozilla trunk, targeting Firefox 2.0. Firefox 2.0 will be released from the 1.8 branch, Firefox 3.0 from the current trunk, see http://wiki.mozilla.org/Global:1.9_Trunk_1.8_Branch_Plan

You're right Steffen, I think the new plan is to land on MOZILLA_1_8_BRANCH, too.

ccing mento and smfr for pending camino project file changes. please keep us in the loop, thanks!

Comment on attachment 206406 [details] [diff] [review] Patch v2 Kai, I didn't realize how much work this is until I reviewed this patch. Thanks a lot for your help. Most of this patch is good. I have some suggestions. I and II are especially important. I. Mac OS X related changes In three .pbxproj files, you indicated with question marks that you don't know what the magic hexadecimal strings for the new libfreebl3.dylib entries should be. Please ask the Mac developers for help. I did notice that there is a gap after the libssl3.dylib entries. So if I have to guess, I'd use libssl3.dylib's magic string for libfreebl3.dylib, and give libssl3.dylib the next magic string. (By observing the magic strings for the NSS .dylib's, you should be able to figure out what the next magic string should be.) In mozilla/config/prebind-address-table, you added > 0x0f000000 @executable_path/libldap50.dylib > 0x10000000 @executable_path/libxpcom_core.dylib >+0x11000000 @executable_path/libfreebl3.dylib We shouldn't need to prebind libfreebl3.dylib because it is always loaded with a PR_LoadLibrary call. I believe this is why libnssckbi.dylib, which is always loaded with PR_LoadLibrary, is not in the prebind-address-table file. Please ask a Mac developer to confirm my understanding. If we need to prebind libfreebl3.dylib, we should put it right after libsoftokn3.dylib; we can either assign 0x0b800000 to libfreebl3.dylib, or we need to adjust the addresses for all the .dylib's after libfreebl3.dylib. II. mozilla/directory/c-sdk changes These changes need to be reviewed and checked in by the module owners because that directory has restricted CVS checkin access. I also found that their ldapcsdk_50_client_branch seems quite old: http://lxr.mozilla.org/seamonkey/source/directory/c-sdk/component_versions.mk#27 27 # NSS - Network Security Services 28 NSSVERS = 3 29 NSS_RELEASE_TAG = NSS_3_3_2_RTM 30 #NSS_DYNAMIC_SOFTOKN = 1 Note that NSS_DYNAMIC_SOFTOKN is not set. This implies that the parts of the mozilla/directory/c-sdk makefiles you modified in this patch cannot possibly be used by the Mozilla clients (because they aren't doing anything with libsoftokn3.so, which is in the current NSS), and you should be able to build Mozilla clients without any changes to mozilla/directory/c-sdk. Please let me know if you can build Mozilla clients without changing mozilla/directory/c-sdk. It seems that mozilla/directory/c-sdk is designed so that one can build it with various versions of NSS. So, if we need to modify mozilla/directory/c-sdk, we should ask the module owners if they want our patch to preserve that. III. Other changes Your change to mozilla/nsprpub/pkg/linux/sun-nspr.spec should be removed from this patch. Please send an email to Christophe Ravel at Sun about this minor problem. In mozilla/security/manager/Makefile.in: The variable name HAVE_FREEBL_MODULES_3264 looks strange, but it's just a cosmetic issue. >+ifeq ($(OS_ARCH), HP-UX) >+ HAVE_FREEBL_MODULES_32 = 1 >+else >+ ifneq ($(OS_ARCH), SunOS) >+ HAVE_FREEBL_MODULES = 1 >+ else >+ ifeq ($(OS_TEST),i86pc) >+ HAVE_FREEBL_MODULES = 1 >+ else >+ ifeq ($(USE_64), 1) >+ HAVE_FREEBL_MODULES_64 = 1 >+ else >+ HAVE_FREEBL_MODULES_32 = 1 >+ HAVE_FREEBL_MODULES_3264 = 1 >+ endif >+ endif >+ endif > endif Rewrite the above as follows (fix the indentation as you see fit): # Default HAVE_FREEBL_MODULES = 1 # 32-bit HP-UX PA-RISC ifeq ($(OS_ARCH), HP-UX) ifneq ($(OS_TEST), ia64) ifndef HAVE_64BIT_OS HAVE_FREEBL_MODULES = HAVE_FREEBL_MODULES_32 = 1 endif endif endif # SunOS SPARC ifeq ($(OS_ARCH), SunOS) ifneq ($(OS_TEST), i86pc) ifdef HAVE_64BIT_OS HAVE_FREEBL_MODULES = HAVE_FREEBL_MODULES_64 = 1 else HAVE_FREEBL_MODULES = HAVE_FREEBL_MODULES_32 = 1 HAVE_FREEBL_MODULES_3264 = 1 endif endif endif Although this format is longer, it is easier to add a new platform to this format. Also note that I made the HP-UX case more specific (32-bit PA-RISC) and I use HAVE_64BIT_OS instead of USE_64. >+ifdef HAVE_FREEBL_MODULES >+FREEBL_LIB = $(DLL_PREFIX)freebl3$(DLL_SUFFIX) >+FREEBL_CHK = $(DLL_PREFIX)freebl3.chk > endif For consistency, please rename FREEBL_LIB as FREEBL_MODULE. In mozilla/xpinstall/packager/packages-static-unix: > ... > bin/libsoftokn3.chk > bin/libsoftokn3.so >+bin/libfreebl3.chk >+bin/libfreebl3.so > bin/libssl3.so > bin/chrome/pipnss.jar > bin/chrome/pippki.jar > ; These are required for solaris 32-bit builds >-bin/libfreebl_pure32_3.chk >-bin/libfreebl_pure32_3.so >-bin/libfreebl_hybrid_3.chk >-bin/libfreebl_hybrid_3.so >+bin/libfreebl_32fpu_3.chk >+bin/libfreebl_32fpu_3.so > ... Please remove the comment and list all the NSS files consecutively. In mozilla/xpinstall/packager/packages-unix: > ... > bin/libssl3.so > bin/chrome/pipnss.jar > bin/chrome/pippki.jar >-; These are required for solaris 32-bit builds >-bin/libfreebl_pure32_3.chk >-bin/libfreebl_pure32_3.so >-bin/libfreebl_hybrid_3.chk >-bin/libfreebl_hybrid_3.so >+bin/libfreebl3.chk >+bin/libfreebl3.so >+bin/libfreebl_32fpu_3.chk >+bin/libfreebl_32fpu_3.so > ... Please list all the NSS files consecutively. In mozilla/xulrunner/installer/mac/Makefile.in: > SOFTOKN = $(BINARY_DIR)/$(DLL_PREFIX)softokn3$(DLL_SUFFIX) >-FREEBL_HYBRID = $(BINARY_DIR)/$(DLL_PREFIX)freebl_hybrid_3$(DLL_SUFFIX) >-FREEBL_PURE = $(BINARY_DIR)/$(DLL_PREFIX)freebl_pure32_3$(DLL_SUFFIX) >+FREEBL = $(BINARY_DIR)/$(DLL_PREFIX)freebl3$(DLL_SUFFIX) >+FREEBL_32FPU = $(BINARY_DIR)/$(DLL_PREFIX)freebl_32fpu_3$(DLL_SUFFIX) >+FREEBL_32INT = $(BINARY_DIR)/$(DLL_PREFIX)freebl_32int_3$(DLL_SUFFIX) Remove the FREEBL_32FPU and FREEBL_32INT definitions. They are not used on the Mac, so we can use their default definitions in $(topsrcdir)/toolkit/mozapps/installer/package.mk, which this makefile includes.

(In reply to comment #10) > (From update of attachment 206406 [details] [diff] [review] [edit]) > Kai, > I. Mac OS X related changes > > In three .pbxproj files, you indicated with > question marks that you don't know what the > magic hexadecimal strings for the new > libfreebl3.dylib entries should be. Please > ask the Mac developers for help. I did notice > that there is a gap after the libssl3.dylib > entries. So if I have to guess, I'd use > libssl3.dylib's magic string for libfreebl3.dylib, > and give libssl3.dylib the next magic string. Don't do this by hand. You'll have to change the projects in Xcode to get this right. Hop onto #camino for help.

Mike, Simon: What is the purpose of filess mozilla/camino/Camino.xcode/project.pbxproj mozilla/embedding/browser/powerplant/PPEmbed.pbproj/project.pbxproj ? A) for linking the binary? or B) listing the files that should be packaged and shipped?

they are the XCode IDE project files, and serve both purposes (linking and packaging)

> In three .pbxproj files, you indicated with > question marks that you don't know what the > magic hexadecimal strings for the new > libfreebl3.dylib entries should be. Please > ask the Mac developers for help. I have removed these for now. Will ask Mac developers for help. > We shouldn't need to prebind libfreebl3.dylib > because it is always loaded with a PR_LoadLibrary > call. It sounds reasonable that we do not need to prebind the freebl library. I have removed the change for now. Will ask Mac developers for help. > Your change to mozilla/nsprpub/pkg/linux/sun-nspr.spec > should be removed from this patch. Please send an email > to Christophe Ravel at Sun about this minor problem. It was an accident that I had this change in my tree. Removed this change. > Rewrite the above as follows (fix the indentation > as you see fit): done > For consistency, please rename FREEBL_LIB as > FREEBL_MODULE. done > In mozilla/xpinstall/packager/packages-static-unix: > Please remove the comment and list all the NSS files > consecutively. done > In mozilla/xpinstall/packager/packages-unix: > Please list all the NSS files consecutively. done > In mozilla/xulrunner/installer/mac/Makefile.in: > Remove the FREEBL_32FPU and FREEBL_32INT definitions. done Now looking into directory/c-sdk

Mike, Simon: When landing this patch, two new files will be produced (library libfreebl3.dylib + special file libfreebl3.chk). With NSS 3.11, if these files are not present in the application directory, it will not start up, but will fail with an error message "can not initialize security component". I think that means the tests on Tinderbox will no longer run successfully. Therfore we should land the Mac project file changes at the same time. Simon, do you have a working environment with XCode? Could you please help us to produce a patch that adds these files to mozilla/camino/Camino.xcode/project.pbxproj mozilla/embedding/browser/powerplant/PPEmbed.pbproj/project.pbxproj mozilla/embedding/tests/cocoaEmbed/CocoaEmbed.pbproj/project.pbxproj ? Thanks.

> Note that NSS_DYNAMIC_SOFTOKN is not set. This implies > that the parts of the mozilla/directory/c-sdk makefiles > you modified in this patch cannot possibly be used by the > Mozilla clients (because they aren't doing anything with > libsoftokn3.so, which is in the current NSS), and you > should be able to build Mozilla clients without any > changes to mozilla/directory/c-sdk. Please let me know > if you can build Mozilla clients without changing > mozilla/directory/c-sdk. Wan-Teh, Mozilla builds fine without the changes to directory/c-sdk. Therefore I have removed those changes from the patch. (I will send the c-sdk patch portion to the owners of that directory, so they can decide whether they want the patch)

- addressed review comments as explained above - removed c-sdk changes, they are not necessary - integrated patch from bug 288647

(In reply to comment #15) > Mike, Simon: > > When landing this patch, two new files will be produced (library > libfreebl3.dylib + special file libfreebl3.chk). > > With NSS 3.11, if these files are not present in the application directory, it > will not start up, but will fail with an error message "can not initialize > security component". Do we really need the .chk file? We don't package libsoftokn3.chk now (maybe that's a bug).

> Do we really need the .chk file? We don't package libsoftokn3.chk now (maybe > that's a bug). Yes we do, and yes that's a bug. Mozilla's internal crypto component can operate in two different modes. The default, standard mode, which does not require the .chk file. A special FIPS mode, that can be enabled using preferences or by using edit/prefs/security devices/enable FIPS. To my knowledge, all members of US government agencies are required to use FIPS. So if you want to allow them to use your product, you should ship the .chk files.

The CocoaEmbed and PowerPlant projects have bitrotted already (and are not part of the build) so let's leave them for now.

Comment on attachment 206920 [details] [diff] [review] Patch v3 Kai, this patch is good. There is only one real problem. But I will also suggest a few style changes. 1. mozilla/security/manager/Makefile.in AND mozilla/security/manager/psm.pkg Optional: change "3264" to "32INT64". Optional: change "MODULE" to "LIB". For example, HAVE_FREEBL_MODULES -> HAVE_FREEBL_LIBS FREEBL_MODULE -> FREEBL_LIB 2. mozilla/xpinstall/packager/packages-static-unix AND mozilla/xpinstall/packager/packages-unix Please list the bin/libfreebl* files below bin/libsoftokn3.so because they are closely related. 3. mozilla/xulrunner/installer/mac/Makefile.in This is the only real problem in this patch. Please add the FREEBL definition (but not the FREEBL_32FPU and FREEBL_32INT definitions) in patch v2 (attachment 206406 [details] [diff] [review]) back.

> Optional: change "3264" to "32INT64". done > Optional: change "MODULE" to "LIB". For example, > HAVE_FREEBL_MODULES -> HAVE_FREEBL_LIBS > FREEBL_MODULE -> FREEBL_LIB done > Please list the bin/libfreebl* files below bin/libsoftokn3.so > because they are closely related. done > This is the only real problem in this patch. Please add > the FREEBL definition (but not the FREEBL_32FPU and > FREEBL_32INT definitions) in patch v2 (attachment 206406 [details] [diff] [review] [edit]) > back. done: SOFTOKN = $(BINARY_DIR)/$(DLL_PREFIX)softokn3$(DLL_SUFFIX) -FREEBL_HYBRID = $(BINARY_DIR)/$(DLL_PREFIX)freebl_hybrid_3$(DLL_SUFFIX) -FREEBL_PURE = $(BINARY_DIR)/$(DLL_PREFIX)freebl_pure32_3$(DLL_SUFFIX) +FREEBL = $(BINARY_DIR)/$(DLL_PREFIX)freebl3$(DLL_SUFFIX)

This new patch revision contains the change requested by Wan-Teh in bug 288647 comment 48.

Comment on attachment 207994 [details] [diff] [review] Patch v5 r=wtc. Benjamin, could you please review this patch? The following is a summary of the changes in this patch. Most of this patch deals with the new files in NSS 3.11. - On most platforms, there are two new files: libfreebl3.so/libfreebl3.dylib/freebl3.dll and libfreebl3.chk/freebl3.chk. - On Solaris SPARC and 32-bit HP-UX PA-RISC, there are more new files. They all have "freebl3" in their file names. - The *freebl3*.chk files are generated from the corresponding *freebl3*.so/.dylib/.dll files by running the "shlibsign" tool. In mozilla/security/manager/Makefile.in, this patch also contains the fix for bug 288647 to support building NSS with system NSPR. Look for the changes related to ABS_DIST, NSPR_INCLUDE_DIR, and NSPR_LIB_DIR.

The CVS checkin messages for "Patch v5" (attachment 207994 [details] [diff] [review]) and "Patch for Camino project on trunk" (attachment 206945 [details] [diff] [review]) on the Mozilla trunk (Gecko 1.9 alpha).

Since these patches have landed I've not been able to complete a debug build on Windows using MSYS. I've posted the log at: http://gavinsharp.com/tmp/log.xgt The fatal error is: LINK : fatal error LNK1104: cannot open file '\moz\mozilla\ff\dist\lib\secutil.lib' make[7]: *** [C:/moz-debug/mozilla/ff/nss/freebl/WIN95_SINGLE_SHLIB/freebl3.dll] Error 80

> cannot open file '\moz\mozilla\ff\dist\lib\secutil.lib' I don't have a windows system, but at least on Linux I still have libsecutil.a in dist/lib

(In reply to comment #28) > I don't have a windows system, but at least on Linux I still have libsecutil.a > in dist/lib I do too, strangely enough. It doesn't appear to be a permissions issue either. Maybe it is an MSYS issue?

I just learned what MSYS is, I wasn't aware before, it seems to be a secondary way to build on Windows. Wan-Teh, FYI: http://developer.mozilla.org/en/docs/Building_on_Windows_with_MSYS

MSYS build problems are not considered blockers, it is a tier-3 build configuration.

Not an MSYS problem, apparently. Bug 323645 filed.

*** Bug 323645 has been marked as a duplicate of this bug. ***

The win32 build problem is a combination of factors: using msys, with a non-standard mount point and MSVC. See bug 323645 for more info. This patch should fix the problem.

Comment on attachment 208691 [details] [diff] [review] Set ABS_DIST to win32 path for msys r=wtc. We should add the -a option to cygpath to more closely match the configure.in code that sets MOZ_BUILD_ROOT. configure.in also has special code for OS/2 VACPP that uses "pwd -D" to define MOZ_BUILD_ROOT. Should we have the equivalent code here? I know we only support OS/2 GCC builds now.

My trunk Seamonkey build attempt on FreeBSD5.4 ends like this the past couple of days (I did not try during the period 13-15 Jan): cd mangle; /usr/local/bin/gmake -j1 libs FreeBSD5.4_OPT.OBJ/shlibsign -v -i /scratch/glennrp/Mozilla/Mozilla-1.9a1-060117/mozilla/dist/lib/libsoftokn3.so Bus error (core dumped)

Comment on attachment 208691 [details] [diff] [review] Set ABS_DIST to win32 path for msys I checked in this patch by cls on the Mozilla trunk to fix the MSYS build failure. Checking in Makefile.in; /cvsroot/mozilla/security/manager/Makefile.in,v <-- Makefile.in new revision: 1.62; previous revision: 1.61 done

I opened a new bug, #323977, about the trouble on FreeBSD (comment #36)

Chris, what problem are you trying to avoid when not using the .chk files in the cross compile (is it that they don't exist?). The .chk files are required on all platforms if FIPS is ever turned on. They are platform independent and can be generated by in either the native or the cross environment (that is any platform can generate valid .chk files for any other platform as long is it has access to the shared libraries the .chk files validate). bob

It's simply that they don't exist. My understanding is that in order to build the .chk files, we would have to build complete host versions of DBM, NSPR & NSS to build a host version of shlibgen and we're not setup to do this.

Chris, Your understanding is correct - local host versions of NSPR/NSS are required in order to make chk files for the target platform. The chk files are a required part of NSS, so this needs to be added for the cross-compile build to be considered complete.

A better (but harder) solution is to ship the shlibsign tool and use it to generate the .chk files on the target platform during installation. Many NSS-based applications "post-process" the NSS libraries (for example, to strip the libraries of debug information or to rebase the libraries for faster application startup), so they already need to run shlibsign (during the build, on the host platform) to regenerate the .chk files.

Comment on attachment 208989 [details] [diff] [review] Do not attempt to install .chk files when cross-compiling I was about to write a patch just like this. We need this for Mac cross build/universal binary support (bug 322578).

The only problem I see with bundling shlibsign is that most of the post-processing occurs during the product's build & packaging cycle, not their install cycle. For embedders like minimo, the additional step to sign the libraries during the install phase could be restrictive. I checked in the patch to hold us over until we can figure this out. Checking in security/manager/Makefile.in; /cvsroot/mozilla/security/manager/Makefile.in,v <-- Makefile.in new revision: 1.63; previous revision: 1.62 done

Comment on attachment 207994 [details] [diff] [review] Patch v5 I have no clue who should be making this decision, let me ask drivers@mo.

Note that this patch does not contain any change to the Camino project, as Mark said in bug 325408 comment 5.

The patch for MOZILLA_1_8_BRANCH was checked in this afternoon.