Sun BugTraq Bug Id: 4830310. Filed by pallab.bhattacharya@sun.com. Release summary: 3.3 Synopsis: NSS/HTTPS Performance is significantly poor due to excessive malloc() call Keywords: malloc, nss, performance Justification: NSS performance is very important to stay competitive in HTTPS arena use SunONE web-server Suggested fix: The SSL layer could accept a memory allocator hook from the webserver during NSS-Init -- The webserver would maintanin per-thread memory-pool, that can be used by NSS Using NSS 3.3 The number of malloc() call per HTTPS/ops is 298 -- This need to be reduced, I have analyzed each malloc call and the whole call stack A summary of the analysis is attached (malloc.stack) that can be used to aggresively work on reducing the number of malloc calls.

Note, this problem has been addressed since NSS 3.3, see: http://bugzilla.mozilla.org/show_bug.cgi?id=97485 malloc lock contention A zone allocator was invented to address this issue. It was incorporated into NSPR 4.2 in January, 2002. This solution was offered in part to satisfy the heap needs of NSS and NSPR without interfering with things like Java's run time (ala SmartHeap). To engage: setenv NSPR_USE_ZONE_ALLOCATOR 1 NSS 3.4 or later uses NSPR 4.2 or later. I do basically all selfserv stress with the zone allocator engaged because performance is dramatically improved, particularly on restarts. Added Pallab to CC list.

pallab.bhattacharya@sun.com 2003-03-11 Used NSS3.7 and NSPR4.3 -- the number of malloc is significantly down to 57 -- in these 2 stacks only -- malloc: size = 256, retval =a2ad08 ....../lib/libc.so.1:calloc+0x58 ....../lib/libfreebl_hybrid_3.so:mp_init_size+0x58 ....../lib/libfreebl_hybrid_3.so:s_mp_div+0x84 ....../lib/libfreebl_hybrid_3.so:mp_div+0x17c ....../lib/libfreebl_hybrid_3.so:s_mp_to_mont+0x2c ....../lib/libfreebl_hybrid_3.so:mp_exptmod_f+0x98 ....../lib/libfreebl_hybrid_3.so:mp_exptmod+0x1e0 ....../lib/libfreebl_hybrid_3.so:rsa_PrivateKeyOpCRTNoCheck+0x19c ....../lib/libfreebl_hybrid_3.so:rsa_PrivateKeyOp+0x218 ....../lib/libsoftokn3.so:RSA_DecryptBlock+0x54 ....../lib/libsoftokn3.so:NSC_Decrypt+0x104 malloc: size = 9760, retval =a2a9f0 ....../lib/libfreebl_hybrid_3.so:mp_exptmod_f+0xf4 ....../lib/libfreebl_hybrid_3.so:mp_exptmod+0x1e0 ....../lib/libfreebl_hybrid_3.so:rsa_PrivateKeyOpCRTNoCheck+0x19c ....../lib/libfreebl_hybrid_3.so:rsa_PrivateKeyOp+0x218 ....../lib/libsoftokn3.so:RSA_DecryptBlock+0x54 ....../lib/libsoftokn3.so:NSC_Decrypt+0x104 However, the Zone allocator has moved the problem from one location to another location (instead of fixing it) -- This zone allocator does not take into account the number of CPUs in the system -- and as such would make 'mz->lock' a hot lock -- a group of threads would contend on a given "[zone][pool]" on large systems, on small systems it may not exhibit any difference --

Slightly offtopic: Would it make sense to switch to an alternate memory allocator as workaround for this issue (see bug 193622 ("RFE: Use libmtmalloc() (or libbsdmalloc()) for memory allocations...")) ?

P2. Of business value to an active supporter of NSS. The present implementation of Zone Allocator has up to 11 different memory pools for each "zone" (size of allocation block). The CPUs/threads should be reasonably well distributed across those 11 pools. That suggests that there should be minimal contention until there are at least 11 CPUs, but there have been reports that contention becomes an issue with as few as 4 CPUs. If that is true, we should find out why. That may be easy to correct once we know why it is not happening.