The RSA algorithms has a hardcoded limit of 8192 bits in freebl/rsa.c . But that limit is only applied to public key ops. Key gen is not limited . The side effect of this is that C_GenerateKeyPair can generate any size key . However, you can't really generate >8192 keys with NSS because of pk11wrap . PK11_GenerateKeyPair calls C_GenerateKeyPair, which succeeds, but then it calls PK11_ExtractPublicKey, which fails . The fix is to apply the same limit to keygen and private key ops in freebl that we do to public key ops .

This patch should enforce the maximum modulus size (and public exponent size) when we generate a key pair. It does not check the modulus size during private key operations.

I want to broaden the scope of this bug to include the RSA key size limits in softoken also. While we're at it, we may also want to check out limits for DSA and DH keys and params, too. Softoken imposes lower bounds on RSA public key sizes with calls to sftk_ConstrainAttribute in sftk_handlePublicKeyObject (near http://lxr.mozilla.org/security/source/security/nss/lib/softoken/pkcs11.c#1046 ) but does not impose an upper bound. Since we now have an upper bound, I think it should be imposed at key import/derive/unwrap/gen time also, and imposing the limit in the calls to sftk_ConstrainAttribute would do that. sftk_handlePrivateKeyObject (near http://lxr.mozilla.org/security/source/security/nss/lib/softoken/pkcs11.c#1197 should impose similar limits on the corresponding private keys, IMO.

Comment on attachment 184426 [details] [diff] [review] Patch for key pair generation (checked in) This patch looks good as far as it goes. Do we also want to limit private key ops?

Comment on attachment 184426 [details] [diff] [review] Patch for key pair generation (checked in) Thanks for the code review, Nelson. Julien, do you have the setup to test this patch?

Comment on attachment 184426 [details] [diff] [review] Patch for key pair generation (checked in) I tested this patch and it works fine - it prevents keygen above 8192 bits from starting. I think we should limit private key ops as well.

Comment on attachment 184426 [details] [diff] [review] Patch for key pair generation (checked in) I checked in this patch on the NSS trunk for NSS 3.10.1.

Part of the fix was made on NSS_3_10_BRANCH for the keygen, but the private key size check remains to be done. For now, I'm tentatively setting the target to 3.10.1 .

I opened bug 308724 for the remaining work. I updated the bug's summary to reflect what was fixed in NSS 3.10.2.