This was discovered in PSM. It's possible to delete root certs, and the information is persisted into cert7.db somehow, so that the cert is no longer visible, even after closing the application and restarting it. This would all be good if we actually had implemented code to do this and if it worked for all certificates, but this behavior is erratic. I have only found it to occur with two of the built-in certificates, GTE ones. The bug exists both in NSS 3.4 and NSS 3.5 . I will try to create a simple test case outside of PSM for this.

This behavior is intentional. Since builtin certs can't _actually_ be deleted (the device is read-only), PSM deletes them by storing the cert in cert7.db with NULL trust. Later on, PSM should detect the NULL trust, and not display the cert (nor allow it to verify certs it has issued). The PSM code that does this is here: http://lxr.mozilla.org/security/source/security/manager/ssl/src/nsNSSCertificate.cpp#3478 For which certs does this not work?

Ian, I see. I started building a test case and found that PK11_FindCertFromNickname always found the cert regardless. However PK11_ListCert(PK11CertListCA, NULL) doesn't return the cert if I take the cert7.db database from the browser in which it was deleted. This all makes sense since the cert is not trusted. However, this doesn't work for most of the root certs in PSM. Just take the one from the top for example, ABA.ECOM Root CA , and delete it. Just close the PSM manage certs window, and reopen it, and that root CA immediately shows back again. Editing the trust shows that it is not trusted for either web sites, mail users, or software makers. Yet it still shows up in the list somehow. Is there some other bit that causes PK11_ListCerts to consider it a CA cert ?

Okay, I have a theory as to why this happens. Once the builtin's cert trust has changed, there are now two instances of it. In the function nssPKIObject_DeleteStoredObject, we iterate over all instances of a cert and attempt to delete them from their tokens. If any delete fails, the non-deleted instance is "recovered". I believe this function will delete the softoken instance with the null trust, while keeping the builtin instance (which it obviously can't delete). I guess it is open to interpretation as to how to fix this. 1. We could say, "NSS is doing the best it can, it deleted all the certs it could, but was left with the builtins." Since this isn't the behavior that PSM expects, PSM would have to *not* call SEC_DeletePermCertificate for root certs. I think this would be simple enough: in the code linked to above, simply do not mark builtin certs for permanent deletion (using the same branch where the null trust is set [incidentally, I think the test should be PK11_HasRootCerts(cert->slot)]). 2. Or, "NSS will either delete all instances or none." This is hard to recover from, and it may be outside of the scope of NSS. 3. A hack -- "NSS will not try to delete any instances of a builtin cert, EVER." Doable, but would produce hacky code.

Ian, I don't know about your theory on this, because I'm doing the delete with a brand-new profile and fresh database, without having modified the trust on any built-in cert. I just do a delete. It works for the first two GTE ones and deletes the trust on them, so that they don't show up anymore in the dialog, but it doesn't work for the ABAecom cert. For that one it appears to have no effect. I did a debug session and saw that CERT_ChangeCertTrust is getting called for each of the delete on all the root certs. It just doesn't have the same effect. I'm still trying to figure out really what we want the code to accomplish. I'm not clear why certs disappear from the CA cert list returned by PK11_ListCerts once they are "deleted", or more precisely, untrusted. If we do that, how would a mozilla user ever ever be able to display these CA certs again in order to re-enable trust on them if he so wanted ? Isn't there a property of the cert that says it's a CA, other than the trust bits, which would allow us to always list the certs ?

Assigned the bug to Julien.

This is a test program that looks for the GTE root cert with PK11_ListCerts , and then changes the trust bits to 0. It does this twice. In the second iteration, the certificate is no longer found.

FYI, here is the output of the program (starting with no databases, but libnssckbi.so needs to be in the current dir) : (strange)/u/jpierre/nss/35/mozilla/dist/SunOS5.8_DBG.OBJ/bin{121} ./delete Pass # 1 Pass # 2 Can't find GTE root cert (strange)/u/jpierre/nss/35/mozilla/dist/SunOS5.8_DBG.OBJ/bin{122} Apparently, in the second pass, PK11_ListCerts no longer recognizes the GTE root cert as a CA because its trust is zero. I thought what made a CA a CA was not only the trust but also other properties of the cert (extensions?).

As I recall, we have separate trust bits for "trusted CA" and "valid CA". Perhaps, instead of changing the trust bits to zero, you want to change them to show that the CA is valid, but not trusted.

The result from the test program is what I would expect. It is true that certs have properties that make them CA/not-CA certs. But trust is a higher-level decision, made by an application. Even if a cert has the required set of extensions to make it a CA cert, setting zero trust for it implies that the application no longer views it as a CA. In that sense, zeroing a cert's trust is analogous to deleting it. The PSM UI will not show certs with zero trust. This is the only way to "delete" a builtin cert -- while it is actually still present, none of the API calls made by PSM will display it. In short: zero trust : cert invisible (effectively deleted) valid trust : cert is still visible, but not trusted trusted : cert visible and trusted OTOH, certutil will still display certs with zero trust, because it traverses the slot and collects all certs (it doesn't filter by CA). Julien, in comment #4 you ask why deleted certs no longer show up, preventing a user from choosing to trust them in the future. There's a difference between choosing to not trust a cert, and choosing to delete it. A user shouldn't expect to ever see a cert they have deleted again (if the cert was in the softoken, it would have been physically removed).

Ian, The problem is that this program has different behavior on different root certs. It only has the behavior I reported for the GTE root certs. Other root certs will still be listed in the PK11_ListCerts output even after their trust is reset to zero. May I assume that it is because these other root certs contain a CA extension, and the GTE root cert does not ? Either way, I don't think the "delete" behavior in PSM should be different based on the extension. It is certainly very confusing to the user that when they delete the GTE root cert, it makes it disappear, and when they delete other root certs, it simply unsets the trust, but does not make it disappear.

Yup, the problem is that some of the certs aren't appearing to be deleted. I traced down into CERT_IsCACert(). If the cert has zero trust, that function will still return true as long as the cert has either an NSCertType or BasicConstraints extension. In PK11_ListCerts, we could change PK11CertListCA to not include certs with zero trust. But that wouldn't fix PSM, because it uses PK11_CertListUnique. PSM makes it's own decision about CA certs, and it uses the same logic: if the cert has zero trust, it calls CERT_IsCACert(). Is it safe to disregard certs that have zero trust from the CA list? IIRC, PKCS#12 was fixed to give valid CA trust to imported CA certs. Are there any other cases where the user would expect to see a CA cert with zero trust?

I'm not sure there's an NSS bug here. I think that WRT trust bits, NSS is behaving as designed. The "valid CA" trust bit is an _override_. It should only be set on certs that do not appear to be valid CA certs because of some lacking components that is required for CA certs, but which certs are known (somehow) to be valid CA certs. There are a number of malformed root CAs in existence, and some of them (IINM) are in the built-in list of root certs. The "valid CA" trust bit was created back before we had nssckbi and root certs were all stored in the perm cert DB. It was created explicitly so that malformed CA certs in the cert DB would still show up when Communicator attempted to list the known CAs, both trusted and untrusted. Communicator's UI had no way to set or remove the "valid CA" trust bit from a cert in the DB. Deleting a cert deleted it, and didn't instead mark it as simply having zero trust. The bit is also useful for malformed intermediate CAs, though those are less common (IINM). For a malformed intermediate CA cert, it simply allows the chain validation to continue past the intermediate CA, and for trust to be determined from the trust (if any) on the root. So, conclusion #1 is that if a malformed CA cert ceases to appear in the list of root CAs when the "valid CA" override trust is removed, that seems like the code is working as designed, not misbehaving. Now, in comment 9 above, Ian seems to have proposed that the "valid CA" bit's meaning be changed from being an override, to being required for all root CA certs. As I understand what he wrote, he proposes that any root CA cert with zero trust bits be treated as if it is not a root CA cert at all, and not appear in a list of root CAs. I have several comments about that proposal. Today, a root CA cert with zero trust bits can simply be an untrusted but valid CA. IINM, root CA certs are not typically marked with the valid CA trust bit unless they are malformed, and thus need it to appear to be CAs. Users who have previously marked root CA certs as untrusted in their DBs might find that all those CAs have simply disappeared if this new interpretation of zero trust were implemented. Prior to implementing this change, or in the act of installing code that implements this change, we'd have to search the user's cert DB and try to divine which certs are simply untrusted valid root CAs, and set the valid CA bit on them, so they don't disappear. Once a cert is marked with no trust, and disappears from the UI, how does the user ever get it back? It appears to have been deleted. If the user attempts to download it from somewhere, does that set the valid CA bit again? Does downloading a CA cert always set the valid CA bit? Does the user get an error that the CA cert is already present in the DB? When a user tries to delete a cert from softoken's perm cert DB, it seems that we want to actually delete it, unless it is a placeholder for a cert that's also in nssckbi. Perhaps the cert is only in the perm cert DB because the user has marked it as untrusted. If a user deletes a cert that is only in the perm cert DB, it is gone thereafter. But if a user deletes a cert that is also in nssckbi, thereafter the cert appears to be trusted, not deleted. So, do we want to start marking all CA certs with no trust rather than deleting them? Or do we delete them unless they're also in nssckbi, and mark them with zero trust oitherwise?

I think this is quite a mess to handle all the different cases with the existing set of bits. Would it be possible to add one more bit, ie. the "deleted" bit which would cause all identical certs in any token to be hidden, in case they are non-deletable like the root certs ? This would actually provide more logical behavior for PSM - clicking delete would delete and make the cert truly disappear from not only the UI but also from any use in NSS. Of course it wouldn't be a fully backwards compatible database format. But perhaps we have a free bit somewhere for this ?

If you invent a new deleted bit, how do you undelete that cert? What happens when you try to install it again?

You wouldn't be able to undelete it, it would just be gone. The only way to unset the bit would be if the cert was explicitly reinstalled by the user.

Okay, I guess I misunderstood the use of "valid CA". Perhaps we're missing the obvious: why not just disable the "Delete" button for builtin certs? I don't see why we need to hide from the user the fact that those certs are read-only. We could have a help message explaining, "Some root certs are shipped by default in a read-only device. These certs cannot be deleted. However, you may choose not to trust them, by clicking on the 'Edit' button." As far as the user is concerned, there is no functional difference between choosing not to trust a CA cert and deleting it, other than perhaps to remove a display line from the UI. I'm afraid adding another trust bit is piling another hack onto already existing hacks. We've already seen the confusion "valid CA" can cause, and we've seen the confusion caused by copying builtin certs to the softoken with zero trust. And there's additional confusion in what to do if the user attempts to reinstall a deleted builtin (I have no idea how the code would handle that right now). I guess we should ask, is the "feature" of tricking users into believing they can delete certs they really can't delete worth the cost? I tend to agree with Nelson that there is no NSS bug here, just a question of intrepretation.

I think that Ian suggestion is a good one. They can't be deleted. Can't NSS fail when asked to delete these certs. Then PSM would catch that error. NSS should maybe defined (if not existing) an error code for either read-only-cert (sort of general) or read-only-built-in-cert (more targetted).

In order to really make that interface work, you would need to expose to the user that the cert was in the databased and marked 'deleted'. bob

Stephane- PSM does not currently ask NSS to delete builtin certs, it detects them proactively and sets the trust to zero. I think it is better to disable the "Delete" button completely than to wait for a failure. But perhaps doing that is not as easy as I think. IIRC, there is a data structure for each certificate in the display (when I did it, it was nsIOutliner or something like that). That structure could have a boolean isDeletable, and when the user clicks on a display line for which isDeletable=false, the button is greyed.

You are right, the delete bit is a hack, I was just throwing that as a suggestion to check the potential problems with it. I already made the suggestion to disallow deletion of the root certs in the PSM bug 138818, comment #5. However, in comment #7, Kai said he implemented another fix which sets all trust to zero instead of disallowing deletion. The problem with Kai's fix is that it deletes the "VALID_CA" bit, which causes the improperly constructed root certs to disappear from the list. There are two possible fixes, both in PSM : 1) don't reset all trust bits to 0, always leave the valid CA bit if present. This would require checking trust, masking, and then setting 2) simply disallow root cert deletion PSM could also bring up a dialog when a user attempts to delete a root cert, telling him it can't be deleted, and offering him to either untrust it or abort deletion.

Per comment #20, this turned out to be a PSM bug, not NSS. Reassigning to Kai.

Ok, I agree to your proposal, that instead of deleting, we should show up a dialog and inform the user that the most we can do is to delete trust. I think this bug and bug 138818 are about the same thing. *** This bug has been marked as a duplicate of 138818 ***

V