Bug 611781 comment 4 suggests that there are decent wins to be had by building NSS with a minimal set of symbols that reflects what Firefox actually uses. (It would be nice to extend this to non-MOZ_FOLD_LIBS platforms, but the MOZ_FOLD_LIBS platforms are the interesting ones from a codesize perspective and are easier to do with a first cut anyway...)

Turns out that it's more convenient to just permit DATA and handle it "natively", so to speak.

The *-dummy.def files included in this patch were generated by: - Examining the set of all NSS symbols referenced by a Linux x86-64 libxul; - Adding in a few symbols from an Android libxul (since some things are done differently there); - Partitioning the symbols appropriately into separate files. I haven't compiled this on Windows yet, but I don't imagine the symbols will change too much. Happily, I don't think the Windows block just above the GCC_USE_GNU_LD block here has to change at all for this.

(In reply to Nathan Froyd (:froydnj) from comment #0) > (It would be nice to extend this to non-MOZ_FOLD_LIBS platforms, but the > MOZ_FOLD_LIBS platforms are the interesting ones from a codesize perspective > and are easier to do with a first cut anyway...) The reason there are non MOZ_FOLD_LIBS builds is because using Firefox on systems where Firefox's dependencies end up depending on NSS, and use Firefox's. For the same reason, hiding symbols on those builds is not possible.

Comment on attachment 8440908 [details] [diff] [review] part 2 - use hand-rolled .def files for linking NSS Review of attachment 8440908 [details] [diff] [review]: ----------------------------------------------------------------- rubberstamp on the contents of the .def files. As long as that builds :) ::: security/build/Makefile.in @@ +332,5 @@ > > # Add all static libraries for nss, smime, ssl and nssutil > SHARED_LIBRARY_LIBS = $(addprefix ../,$(NSS_STATIC_LIBS)) > > +nss_def_files := $(addprefix $(srcdir)/, \ you shouldn't need the addprefix anymore. @@ +333,5 @@ > # Add all static libraries for nss, smime, ssl and nssutil > SHARED_LIBRARY_LIBS = $(addprefix ../,$(NSS_STATIC_LIBS)) > > +nss_def_files := $(addprefix $(srcdir)/, \ > + nss-dummy.def \ Those don't really characterize as dummy files :) s/-dummy// Note I'm not entirely sure there's much benefit from keeping separate files, except if you want to write scripts that would check them against the ones in the nss source, but even that i'm not convinced it's worth doing (build will fail in obvious and straightworward ways if there are new symbols that need exporting).

I tried to build Firefox with these with tests enabled on Windows and linking failed due to several missing symbols that are used in the test code. IMO, it would be *much* better to just have 1 file that lists all the exports from libnss3, instead of the multiple files that mirror the original DEF files. When I tried to modify your patch to fix the above issue, it was unnecessarily time-consuming to figure out which file to edit, even though I have a lot of experience with NSS. I also seemed to have trouble getting "mach build" to notice changes to the "dummy" files. Please make sure the dependencies are set correctly so that a change to any dummy file causes libnss to get relinked. It would be better to link libnss3 once for Gecko/Firefox's use and then separately for use by tests and utilities like certutil. The tests and utilities use a lot of functionality that Gecko doesn't use, and you won't be able to minimize the size of Gecko if you only have one libnss that is the union of the functionality needed by everything. For example, and in particular, certutil calls CERT_VerifyCertificate, which depends on both libpkix and the classic NSS certificate verification. That means that mozilla::pkix replacing NSS for cert verification won't do any good without solving this problem. Finally, good work!

Or, one more thing: not only do the utilities like certutil require a lot more of NSS than Gecko does, but also desktop builds of Firefox use a lot more of NSS than mobile builds do, because the desktop builds have the certificate manager and they also have the legacy window.crypto functionality (MOZ_DISABLE_CRYPTOLEGACY). That means that ideally we'd be able to indicate which symbols to expose or exclude based on platform.

(In reply to Brian Smith (:briansmith, was :bsmith; NEEDINFO? for response) from comment #5) > I tried to build Firefox with these with tests enabled on Windows and > linking failed due to several missing symbols that are used in the test code. Yeah, my patch was not sufficiently complete. I've been updating the symbol lists, and should be landing a patch that builds everywhere this week. > IMO, it would be *much* better to just have 1 file that lists all the > exports from libnss3, instead of the multiple files that mirror the original > DEF files. When I tried to modify your patch to fix the above issue, it was > unnecessarily time-consuming to figure out which file to edit, even though I > have a lot of experience with NSS. That's two votes for that scheme; I'll switch things around, I suppose. > I also seemed to have trouble getting "mach build" to notice changes to the > "dummy" files. Please make sure the dependencies are set correctly so that a > change to any dummy file causes libnss to get relinked. Hm, that's worrisome; the exports file/linker script declares its dependency on the dummy files, and rules.mk should ensure that the NSS library depends on the export file/linker script. I'm not sure what else to fix here. > It would be better to link libnss3 once for Gecko/Firefox's use and then > separately for use by tests and utilities like certutil. The tests and > utilities use a lot of functionality that Gecko doesn't use, and you won't > be able to minimize the size of Gecko if you only have one libnss that is > the union of the functionality needed by everything. I have noticed this problem in updating the lists of symbols: all of the symbols I need to add are coming from the tests. Do you know of convenient ways to do this short of rewriting security/build/Makefile.in to build all of NSS by hand? It seems better to let NSS's build system do the necessary work there, rather than duplicating a raft of logic in Gecko... > Finally, good work! Thanks!

(In reply to Brian Smith (:briansmith, was :bsmith; NEEDINFO? for response) from comment #5) > It would be better to link libnss3 once for Gecko/Firefox's use and then > separately for use by tests and utilities like certutil. The tests and > utilities use a lot of functionality that Gecko doesn't use, and you won't > be able to minimize the size of Gecko if you only have one libnss that is > the union of the functionality needed by everything. For example, and in > particular, certutil calls CERT_VerifyCertificate, which depends on both > libpkix and the classic NSS certificate verification. That means that > mozilla::pkix replacing NSS for cert verification won't do any good without > solving this problem. We could plausibly work around this by using a prebuilt set of NSS utils for our test harnesses instead of building them alongside the browser. We already do this for mobile tests where we need host binaries.

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #8) > (In reply to Brian Smith (:briansmith, was :bsmith; NEEDINFO? for response) > from comment #5) > > It would be better to link libnss3 once for Gecko/Firefox's use and then > > separately for use by tests and utilities like certutil. The tests and > > utilities use a lot of functionality that Gecko doesn't use, and you won't > > be able to minimize the size of Gecko if you only have one libnss that is > > the union of the functionality needed by everything. For example, and in > > particular, certutil calls CERT_VerifyCertificate, which depends on both > > libpkix and the classic NSS certificate verification. That means that > > mozilla::pkix replacing NSS for cert verification won't do any good without > > solving this problem. > > We could plausibly work around this by using a prebuilt set of NSS utils for > our test harnesses instead of building them alongside the browser. We > already do this for mobile tests where we need host binaries. Do you mean statically-linked NSS utils? If so, then doesn't that somewhat defeat the purpose of testing them (since their test results ought to never change)? If not, how does that fix the utils requiring symbols from the dynamic libraries that most of the rest of the build doesn't care about?

No. The NSS utilities we use in tests (certutil, etc) are only used to build a certificate DB for test profiles (for adding our test certs to the Mochitest profile, mostly). We don't actually run tests against them. Because of this it would be fine to use the same tactic we use on mobile tests, which is that we simply have a prebuilt archive of them that we pull down to use during the test run.

Ah, thanks for the explanation. I thought we ran NSS's test suite at some point, but perhaps that was just a glimmer in somebody's eye.

That would be bug 799855. I have no idea how the NSS tests work, but if we could build separate test binaries or something that would probably still work since what you're doing is just link-time symbol removal. Anyway we can punt on that since it doesn't currently exist.

(In reply to Nathan Froyd (:froydnj) from comment #7) > (In reply to Brian Smith (:briansmith, was :bsmith; NEEDINFO? for response) > from comment #5) > > It would be better to link libnss3 once for Gecko/Firefox's use and then > > separately for use by tests and utilities like certutil. The tests and > > utilities use a lot of functionality that Gecko doesn't use, and you won't > > be able to minimize the size of Gecko if you only have one libnss that is > > the union of the functionality needed by everything. > > I have noticed this problem in updating the lists of symbols: all of the > symbols I need to add are coming from the tests. Do you know of convenient > ways to do this short of rewriting security/build/Makefile.in to build all > of NSS by hand? It seems better to let NSS's build system do the necessary > work there, rather than duplicating a raft of logic in Gecko... I would change the build logic in security/build so that it is possible to run the linking step twice, once outputting an optimized libnss3 to dist/bin (or dist/lib) and again a non-optimized libnss3 to dist/some-place-else. Then, when we run the GTests and when we run the utilities in security/manager/ssl/tests/unit/tlserver/cmd/, set the LD_LIBRARY_PATH so that the non-optimized variants are picked up. Using pre-built certutil and other NSS utilities would alleviate the problem to a certain extent. However, some of the tests are using otherwise-unused parts of NSS to generate test data, so of you really want to optimize away all the parts of NSS not used in Gecko, then the tests need to be linked to a version of libnss3 that contains more stuff than Gecko uses. Note that there is a difference between NSS's test suite, and Gecko's (PSM's) tests that use NSS. We don't run NSS's test suite, but we do have tests in security/pkix/test/gtest/* and security/manager/ssl/tests/gtest and security/manager/ssl/tests/unit/tlsserver/* that use NSS outside of Gecko.

Landed with a single nss.def: https://hg.mozilla.org/integration/mozilla-inbound/rev/8d6f286139ef https://hg.mozilla.org/integration/mozilla-inbound/rev/1ed1c6212f4c

Addons that use the NSS functions (through jsctypes or other means) that have been removed from Gecko with this change will probably stop working or may start working in a non-secure manner. The migration path depends on the individual NSS function. In the case of Thunderbird, Seamonkey, and other XULrunner-based-ish applications, I recommend building without MOZ_FOLD_LIBS=1 being set, to avoid the issue.

An attempt to fix sync digging around in NSS symbols: https://hg.mozilla.org/integration/mozilla-inbound/rev/bcd72aabb661 Backed out for xpcshell orange: https://hg.mozilla.org/integration/mozilla-inbound/rev/b124fea34d27

(In reply to Brian Smith (:briansmith, was :bsmith; NEEDINFO? for response) from comment #15) > In the case of Thunderbird, Seamonkey, and other XULrunner-based-ish > applications, I recommend building without MOZ_FOLD_LIBS=1 being set, to > avoid the issue. How about adding another knob for symbols reduction?

(In reply to Mike Hommey [:glandium] from comment #17) > (In reply to Brian Smith (:briansmith, was :bsmith; NEEDINFO? for response) > from comment #15) > > In the case of Thunderbird, Seamonkey, and other XULrunner-based-ish > > applications, I recommend building without MOZ_FOLD_LIBS=1 being set, to > > avoid the issue. > > How about adding another knob for symbols reduction? Either way is fine with me. Disabling MOZ_FOLD_LIBS for those things would reduce the variance across platforms and probably save the developers of those products more pain. But, folded libs probably improve the performance of those applications.

That could be seen the other way around: Keeping MOZ_FOLD_LIBS for those things would reduce the variance between versions, first, since they've been using MOZ_FOLD_LIBS for a while, and with Firefox.

Back in with a sufficient number of symbols: https://hg.mozilla.org/integration/mozilla-inbound/rev/486c857e934f https://hg.mozilla.org/integration/mozilla-inbound/rev/4b8680a3302d

https://hg.mozilla.org/mozilla-central/rev/486c857e934f https://hg.mozilla.org/mozilla-central/rev/4b8680a3302d

This is breaking TB builds, since NSS_CMSMessage_ContentLevelCount [at least, I haven't checked if there are more yet] isn't included in the smime export list.

Joshua, try building Thunderbird without MOZ_FOLD_LIBS=1. See comment #15 and the rest.

(In reply to Brian Smith (:briansmith, was :bsmith; NEEDINFO? for response) from comment #23) > Joshua, try building Thunderbird without MOZ_FOLD_LIBS=1. See comment #15 > and the rest. I'm trying to minimize divergence between Thunderbird and Firefox development, not increase it.

(In reply to Joshua Cranmer [:jcranmer] from comment #22) > This is breaking TB builds, since NSS_CMSMessage_ContentLevelCount [at > least, I haven't checked if there are more yet] isn't included in the smime > export list. Added to this list in this changeset: https://hg.mozilla.org/mozilla-central/rev/3018d1b623ab

Did this result in any size savings, or is it a precursor to other work?

(In reply to Nicholas Nethercote [:njn] from comment #26) > Did this result in any size savings, or is it a precursor to other work? This was a small size win (~20K uncompressed?) on Android. The real win from this will come if we can ever eliminate all the symbols that transitively call into NSS's certificate verification library (libpkix). The linker will then (theoretically, linker bugs notwithstanding) be able to eliminate all of libpkix from libnss3, which will save ~300K (uncompressed), or ~25% of libnss3 (again, measured on Android).

Hi all: Im not sure if this is related with this bug... Each release version of FF makes me to review all exported functions in NSS native library. I have an applet to use firefox certificates for cryto operations (sign, encrypt, ssl, etc...). For example, today I update my FF (windows 7) to latest version (33), and I found these functions are not exported (there are more), so my applet crash (java.lang.UnsatisfiedLinkError: Error looking up function 'SECMOD_UpdateSlotList'): SECMOD_UpdateSlotList PK11_FindPrivateKeyFromCert PK11_GetSlotFromPrivateKey In version 32 these 3 functions were exported. Where can I find the script witch create nss3.dll for firefox? My applet works on MacOSX and Linux, what about NSS on MacOSX and Linux? Apply the same export restrictions? Is there a way to getting notify about these changes? Thanks in advance. Bye.

Hi again: Firefox 32 uses NSS 3.16.5 and Firefox 33 uses NSS 3.17.1. Both versions export SECMOD_UpdateSlotList, PK11_FindPrivateKeyFromCert and PK11_GetSlotFromPrivateKey functions so Firefox build must exclude them. These functions are not in https://hg.mozilla.org/mozilla-central/rev/4b8680a3302d What is the reason? Under what circuntances? Who decides it? Any remark will be appreciate Looking in https://developer.mozilla.org/en-US/Firefox/Releases/33 under Security section: Security No change. Some explanation? Thanks again.

(In reply to Joaquin Sanchez Jimenez from comment #29) > What is the reason? Under what circuntances? Who decides it? Any remark will > be appreciate These symbols are not exported from the copy of NSS that is shipped with Firefox on some platforms because those symbols are not used by Firefox. Shipping libraries with code that isn't used by Firefox needlessly increases download sizes, size of the code on disk, and library load times.

Thanks Natham. Can you tell me where can I find nss.def for each release of FF (some URL pattern for example)? I see in MacOSX, FF 33 exports these 3 methods (and others), in Ubuntu for example, I load nss from system, so I think there is no problem in both platforms (I'll test later at home). I don't understand how the size of a native library could be a problem. Regarding the size of FF, nss library is very small. If someone decides to use low level PKCS11 instead PK11_*, SECMOD_*, PR_*, PL_*, nss will be dropped from some FF release... When I started with FF/Mozilla security (10 years ago), I used JSS. Then, some years later, JSS people decided to NOT provide native libraries for each platform (from 4.2 and later YOU must built them). Then, thow that I decide to implement my own native library, FF people has decided not to export NSS functions it uses. As you can see it is very frustrating... (In reply to Nathan Froyd (:froydnj) from comment #30) > (In reply to Joaquin Sanchez Jimenez from comment #29) > > What is the reason? Under what circuntances? Who decides it? Any remark will > > be appreciate > > These symbols are not exported from the copy of NSS that is shipped with > Firefox on some platforms because those symbols are not used by Firefox. > Shipping libraries with code that isn't used by Firefox needlessly increases > download sizes, size of the code on disk, and library load times.

(In reply to Joaquin Sanchez Jimenez from comment #31) > Can you tell me where can I find nss.def for each release of FF (some URL > pattern for example)? The current list is at http://hg.mozilla.org/mozilla-central/file/fe1513fc09f6/config/external/nss/nss.def The list used to reside under /security/build/nss.def before it was moved to its current location. > I don't understand how the size of a native library could be a problem. Just building NSS with properly exported symbols on Android saved ~120K off the packaged library size (bug 1018375). Building NSS with a smaller-than-normal list of exported symbols means that future work to reduce the number of NSS symbols that we use will result in even smaller libraries. Comment 27 suggests another win of 300K is reasonable. If you want an NSS library with the proper set of symbols, you need to distribute your own, or rely on the system version.

Thanks again Nathan I'm a developer and I understand your point of view, but: Why not build a nss version for android, like AIX, BSD, LINUX, MACOX, SOLARIS, etc... All NSS code has macros or defines to include or define conditionally. Why MacOSX distro is different from Windows distro? Android distro is built from Windows distro (from comment 27)? I decided to NOT provide my own nss native library due to the fact that I want to provide my users access to FF certificates (FF USES NSS), not another crypto container (NSSSharedDB, MSCAPI or Keychain for example). On Linux, when users open my applet in Chrome (chromium or similar), it access to NSSSharedDB (it is required to have NSS in system, chrome people has decide to do this form, like FF people decides to use NSS). On windows, chrome delegates to MSCAPI not to NSS and on MacOSX to Keychain so my applet delegates to Windows MSCAPI and Apple Keychain respectively. When I access to MSCAPI (via win32 api), I have not to rewrite my native code each time microsoft release a new version o patch for windows (this is not really true, but it is stable and well documented). My code works on windows 95, 98, ME, NT, Vista, 7 and 8 (these two last also in 64 bits). And of course I DON'T need to build my win32 API. Think about my code. I need to detect arch, so I need to include/define conditionally (uint32 for example). Then FF people drop some functions from NSS, so I need to conditionally detect if any function I use are present... For a developer, DON'T use FF in Windows: directly use chrome or IE. A simple remark: linux users which uses java with chrome 35 have a big problem with java support, just because chrome drops NPAPI plugins. If you search comments about java, chrome and linux (ubuntu forums fro example): MIGRATE to FF. (In reply to Nathan Froyd (:froydnj) from comment #32) > (In reply to Joaquin Sanchez Jimenez from comment #31) > > Can you tell me where can I find nss.def for each release of FF (some URL > > pattern for example)? > > The current list is at > http://hg.mozilla.org/mozilla-central/file/fe1513fc09f6/config/external/nss/ > nss.def > > The list used to reside under /security/build/nss.def before it was moved to > its current location. > > > I don't understand how the size of a native library could be a problem. > > Just building NSS with properly exported symbols on Android saved ~120K off > the packaged library size (bug 1018375). Building NSS with a > smaller-than-normal list of exported symbols means that future work to > reduce the number of NSS symbols that we use will result in even smaller > libraries. Comment 27 suggests another win of 300K is reasonable. > > If you want an NSS library with the proper set of symbols, you need to > distribute your own, or rely on the system version.

I'm sorry but your use case is not something we support. We are not ever going to do things to make this easier, and we are likely to do things to make it harder.

I agree, thanks (In reply to Ted Mielczarek [:ted.mielczarek] from comment #34) > I'm sorry but your use case is not something we support. We are not ever > going to do things to make this easier, and we are likely to do things to > make it harder.

You should investigate webcrypto. (also note that Chrome is due to stop using NSS at some point in the future)

Hi Mike: Here in Europe and specially in Spain (also in Germany and Belgium) there are some limitations about signature (TS 101 733 - V1.7.4 (CAdES) or ETSI 102 778 (PAdES) for example). For example, all PDF generated by the spanish public administration will be PDF-A, so we need PAdES_LongTerm (PDF-A v.2.b1 or something similar); in other scenarios we need CAdES, from BES to CAdES_A_v3, XAdES from XADES_v1_1_1 to XADES_A_v1_4_1 and eFactura (electronic invoice based on XMLDsig/XAdES). We need a way to manipulate CMS, XML or PDF structure not only digitally sign "simple" content. How can you sign a file of hundred Mb (that is, streaming mode: download a file and digitally sign it in any way)? You cannot from webcrypto. How about co-signature or counter-signature or tree-partial counter-signature? How about XMLDSig/XAdES with binary content? How about to digitally sign a PDF with PDF-Signature or PAdES? When Chrome change its crypto container, look into it and I'll change my code to access this container (if possible). If you read comment 31, I've been working with NSS and JSS 10 years, this is not a homework for me, so I do my best to integrate with all navigators in all platforms, including FF. Finally I decided to create my own distribution of nss. I do not want anyone from FF change the way they work, just expressing my problems and obstacles that certain changes in the distributions can cause developers. I understand that it is easier to simplify everything, I'm a developer, I do not want anyone to be offended, not my intention. Thanks

I won't claim to know anything about what's possible to do with webcrypto or not, but if it's not possible to sign documents without having them entirely in memory, that seems to me as something that should be fixed before it becomes a recommendation. But this is definitely off topic in this bug. I'm not sure who you should contact, though. I'm sure Sid can point to the right person.

Regarding the things with the certificate database, WebCrypto cannot do those things because WebCrypto cannot access the certificate database. Please see bug 1080046 where another user has requested that some PK11_* functions for accessing the certificate database be added back. I believe that David Keeler is considering adding back some of those functions, but I don't know what his final decision is.

Richard, David, can you two weigh in?

Hi again, a brief description of the problem: 1. I have an applet that delegates to FF NSS, because there is no NSS ecosystem on Windows (on Linux if there is no NSS ecosystem the applet delegates to FF linnss3.so and on MacOSX to FF libnss3.dylib or something similar). 2. the applet must be able to reload imported certificates once started (I submit a bug years ago about the hability to reload them using JSS). With FF 33 I can't use SECMOD_UpdateSlotList, it has been removed from the distro (FF don't use this function), so it is very hard to rewrite it, and maybe cannot be done (there are more functions that may be removed from the distro or for internal use). 3. some other functions can be rewriten, like PK11_FindPrivateKeyFromCert (use PK11_FindKeyByDERCert) and PK11_GetSlotFromPrivateKey (just use PK11_ReferenceSlot passing SECKEYPrivateKey->pkcs11Slot) and others not (bug 1080046). If FF people remove more and more functions from FF NSS, some functionality will be lost (for plugins, addons, etc...). So my question is: why in a desktop distro NSS is dropped? We are talking about 300K of the DLL/SO/DYLIB when the distro is about dozens of Mb, just keeping nss.def from NSS distro, there is nothing to do (no extra work). For example, all other dll are exactly the same in FF 33, 34.0a2 and 35.0a2 (if someone exclude some function from mozglue / msvcp100 / mcvcr100 / nssdbm3 / nssckbi / softokn3 maybe FF cannot access to its own certificates!!!). I undertand that for mobile or similar environments keep things as small as possible is very good.

I'm sorry this is a crappy situation, I agree that it sucks. Our position is that NSS in Firefox is not a public API and ideally we wouldn't export any of its symbols. Don't be surprised if we eventually fold it into libxul and expose none of the NSS API surface. We've had no end of problems with third-party components causing stability issues by poking into Firefox internals, so that's not an approach we support in any way. That your applet worked in the past was merely a technological accident. (Not to disparage your work, I'm sure it's a neat bit of engineering!)

I completely disagree with you, FF people can not avoid bad programming practices at any level. Due to the reason that you post, why not plugins, addons or XPCOM access are completely deleted?. For that matter, why not also deletes javascript?. If the removal of NSS were the case, what alternative would ?. You know the answer...

Joaquin, if we move further discussion of this issue to a new bug, I think we'll be able to get a clearer understanding of what it is you're trying to do and hopefully come up with a good solution.

bug 1080046 ?

It sounds like your situation is a little more involved than bug 1080046, so we should probably keep the two separate. Please file a new bug.

We have the exact same issues, The removal of the exports, is breaking not only add-on code, but application code. I am surprised you state NSS in Firefox is not a public API. Are we now required to ship our own copy of nss3 and install along side the nss3 already shipped with FF?

(In reply to Sid Stamm [:geekboy or :sstamm] from comment #40) > Richard, David, can you two weigh in? On the WebCrypto point: Brian is correct that WebCrypto cannot see the NSS cert db. It can't see any keys that weren't created using WebCrypto. Joaquin is correct that WebCrypto cannot operate in streaming mode. But honestly, 100MB isn't that much for the browser to handle in memory these days. I've run tests with 10MB files without a hiccup.