+++ This bug was initially created as a clone of Bug #986019 +++ Our telemetry indicates turning off the SSL trust bit for the Equifax 1024-bit roots is causing considerable breakage. See http://mzl.la/1HbTGeF and http://mzl.la/1HbTGeF Please turn back on the WebSites trust bit for the following 1024-bit root certificates owned by Symantec. > Equifax > Equifax Secure Certificate Authority > Equifax Secure CA > 1998 Aug 22 > 2018 Aug 22 > SHA-1 > SHA1 Fingerprint: D2:32:09:AD:23:D3:14:23:21:74:E4:0D:7F:9D:62:13:97:86:63:3A > Equifax Secure Inc. > Equifax Secure Global eBusiness CA-1 > 1999 Jun 21 > 2020 Jun 21 > MD5 > SHA1 Fingerprint: 7E:78:4A:10:1C:82:65:CC:2D:E1:F1:6D:47:B4:40:CA:D9:0A:19:45

It was mentioned on the NSS call that it may be possible to fix this by adding a cross-signed intermediate instead of re-trusting the roots.

AFAIK, these certs are issued directly from the Equifax root, by now the remaining non-expired ones are mostly RapidSSL.

There was the idea to use scanning of certificate sets, in order to figure out the specific set of intermediate CA certificates that would be required to add as (neutral trust) helper certificates to our store, for finding all alternative trust chains, as an alternative solution to re-adding back to root. I know that Hubert has done a lot of scanning in the past. I don't know if he would be able to help with finding all necessary intermediates, adding him to CC.

If I understand correctly, we want to get this done prior to the Firefox 38 releases.

This is a list of sites affected by this that we found earlier: https://bug986019.bugzilla.mozilla.org/attachment.cgi?id=8571482

(In reply to Kai Engert (:kaie) from comment #3) > There was the idea to use scanning of certificate sets, in order to figure > out the specific set of intermediate CA certificates that would be required > to add as (neutral trust) helper certificates to our store, for finding all > alternative trust chains, as an alternative solution to re-adding back to > root. > > I know that Hubert has done a lot of scanning in the past. > I don't know if he would be able to help with finding all necessary > intermediates, adding him to CC. The intermediates I saw in scans that were signed by Equifax CAs are: Subject: C=US, O=GeoTrust Inc., CN=GeoTrust Global CA SHA1 Fingerprint=73:59:75:5C:6D:F9:A0:AB:C3:06:0B:CE:36:95:64:C8:EC:45:42:A3 Subject: C=US, O=Intel Corporation, CN=Intel External Basic Policy CA SHA1 Fingerprint=92:4B:35:7F:C7:B9:D8:C9:D2:6E:41:D4:AF:4D:C6:C4:BA:BE:90:E5 Subject: C=US, O=GeoTrust Inc., CN=GeoTrust Primary Certification Authority SHA1 Fingerprint=68:90:ED:2B:2C:11:10:72:91:2E:D6:25:54:59:AD:0D:B7:6F:3A:D1 Subject: C=US, O=GeoTrust Inc., CN=GeoTrust Primary Certification Authority SHA1 Fingerprint=15:B4:39:A0:9A:2B:84:6E:D2:1E:84:F4:42:24:CF:B8:7A:65:99:C9 All of those are 2048 bit RSA certs, so they could be re-signed by other CAs (though the GeoTrust ones may actually already be in the trust store, haven't checked) I'm rather busy now, so don't have the time to play around to figure out what certs are problematic and what can be done to fix it, but should be able to answer specific questions.

one more thing, I'll probably be starting a new scan today, so should have fresh results in ~2 weeks

The first entry in the list from comment 5 appears to have been signed directly by the root CA. In order to support them, we'd need to find an intermediate that use the same subject name as the removed Equifax root CA, right? The intermediates mentioned in comment 6 don't seem to help in this particular scenario.

(In reply to Kai Engert (:kaie) from comment #8) > The first entry in the list from comment 5 appears to have been signed > directly by the root CA. > > In order to support them, we'd need to find an intermediate that use the > same subject name as the removed Equifax root CA, right? No, it has to have the same subject name and a modulus that can be used to verify the EE certificate. If a certificate was signed by a 1024 bit key, it can only be verified using a 1024 bit cert. The EE certificates need to be resigned by a different CA to fix this problem.

(In reply to Hubert Kario from comment #9) > No, it has to have the same subject name and a modulus that can be used to > verify the EE certificate. If a certificate was signed by a 1024 bit key, it > can only be verified using a 1024 bit cert. Good point. So we can only expect fix sites that are already signed by a 2048 bit intermediate.

From bug 986019 comment 7: "We do not wish to include a cross-signed intermediate cert for either of these roots. The vast majority of certs were signed directly from these roots." If the majority was signed directly by the Equifax root, and if you wanted to avoid breaking them, then it seems adding the Equifax root back would be the only reliable approach. I manually looked at the cert for a few sites, and the one for access.endurance.bm is valid until 2016-08-08, so it would be required for at least one more year, which seems unfortunate. Do you think we could automatically lookup the domain registries for each affected domain, and send emails to all domain contacts about the need to replace their certs by a cutoff date, to ensure they will continue to be compatible with Firefox after that date?

Given how close we are to the release of Firefox 38, and given the need to prepare release packages etc. in Linux distributions, it would be appreciated if we could make a decision on this one quickly, sooner than next week. If there were agreement that re-adding the Equifax root CA as trusted were the only viable remedy, at least for the Firefox 38 short term, then I'd appreciate if we could get that done quicker, maybe by monday.

The Intel External Basic Policy CA intermediates expires in May 2015 BTW. I think going until the end of 2015 should cover the vast majority of the affected certificates.

Actually, it is the Intel External Basic Issuing CAs that expire in May 2015, but I don't see any other intermediates used under the Intel External Basic Policy CA.

Based on the discussion so far (and in particular comment 9), I think we should go ahead with re-enabling the Equifax root for 38. This will give us more time to come up with a more comprehensive solution for later versions.

That works for me, especially in the context of 38 being an ESR. It would be nice if we could have a patch today (Monday) for beta 6 or by Thrusday for beta 7.

This patch reverts the changes to both Equifax roots made in bug 1132496 for bug 986019.

Aloha! (I'm back from Maui...) (In reply to David Keeler [:keeler] (use needinfo?) from comment #0) > +++ This bug was initially created as a clone of Bug #986019 +++ > > Our telemetry indicates turning off the SSL trust bit for the Equifax > 1024-bit roots is causing considerable breakage. See http://mzl.la/1HbTGeF > and http://mzl.la/1HbTGeF > > Please turn back on the WebSites trust bit for the following 1024-bit root > certificates owned by Symantec. > > > Equifax > > Equifax Secure Certificate Authority > > Equifax Secure CA > > 1998 Aug 22 > > 2018 Aug 22 > > SHA-1 > > SHA1 Fingerprint: D2:32:09:AD:23:D3:14:23:21:74:E4:0D:7F:9D:62:13:97:86:63:3A > I'm fine with turning the websites trust bit back on for this "Equifax Secure Certificate Authority" root. It is the one we were concerned about based on the data here: https://people.mozilla.org/~dkeeler/ca-telemetry-dashboard/ Other telemetry showed that the removal was going to be OK. But apparently we are having some difficulty with our telemetry. > > Equifax Secure Inc. > > Equifax Secure Global eBusiness CA-1 > > 1999 Jun 21 > > 2020 Jun 21 > > MD5 > > SHA1 Fingerprint: 7E:78:4A:10:1C:82:65:CC:2D:E1:F1:6D:47:B4:40:CA:D9:0A:19:45 Is there *any* data indicating that the websites trust bit needs to be turned back on for this "Equifax Secure Global eBusiness CA-1" root? I haven't seen any data indicating wide usage of this root. If there is no such data, then please don't turn the websites trust bit back on for this root.

Here is data that Symantec gave me in early 2014 regarding these two roots, and Symantec has been working hard since then to transition their remaining customers off of these roots. So, this data backs up my understanding that we should be able to turn off the websites trust bit for the "Equifax Secure Global eBusiness CA-1" root. So, unless anyone has data showing otherwise, I think this change should only be to re-enabled the websites trust bit for the "Equifax Secure Certificate Authority" root. Equifax Secure Certificate Authority D23209AD23D314232174E40D7F9D62139786633A # Certs expiring after 2014: 7,845 # Certs expiring in 2014: 6,034 # Certs expiring in 2015: 7,564 # Certs expiring in 2016: 281 Last Cert expires: 29-Nov-2016 Last Cert issued: 23-Aug-2011 Equifax Secure Global eBusiness CA-1 7E784A101C8265CC2DE1F16D47B440CAD90A1945 # Certs expiring after 2014: 12 # Certs expiring in 2014: 1,713 # Certs expiring in 2015: 12 # Certs expiring in 2016: 0 Last Cert expires: 14-Apr-2015 Last Cert issued: 9-Jun-2009

Thanks Kathleen, your numbers seem pretty convincing. This patch only restores trust for the one more important root with Fingerprint (SHA1): D2:32:09:AD:23:D3:14:23:21:74:E4:0D:7F:9D:62:13:97:86:63:3A

Does anyone think it's necessary to test this patch on mozilla-central, prior to making a final release of the trust module? If we landed this patch on mozilla-central TODAY, how soon would your telemetry tell you that the spike of certificate errors is gone? Tuesday? Wednesday?

(In reply to Kai Engert (:kaie) from comment #20) > Created attachment 8594821 [details] [diff] [review] > 1155279-v2.patch > The patch looks good to me. Note that it turns both the websites and code signing trust bits back on for the "Equifax Secure Certificate Authority" root. I'm fine with that.

(In reply to Kai Engert (:kaie) from comment #21) > Does anyone think it's necessary to test this patch on mozilla-central, > prior to making a final release of the trust module? > > If we landed this patch on mozilla-central TODAY, > how soon would your telemetry tell you that the spike of certificate errors > is gone? > Tuesday? Wednesday? If we believe the numbers the CA gave us in comment 19 are correct, then we don't need to do test.

(In reply to Kai Engert (:kaie) from comment #21) > Does anyone think it's necessary to test this patch on mozilla-central, > prior to making a final release of the trust module? > > If we landed this patch on mozilla-central TODAY, > how soon would your telemetry tell you that the spike of certificate errors > is gone? > Tuesday? Wednesday? Telemetry on mozilla-central (i.e. Nightly) usually takes a few days at least. Unfortunately, in this case it wouldn't actually tell us much, because the relevant measurements for mozilla-central and mozilla-aurora already look fine. It's only mozilla-beta that's problematic (which is why we didn't even discover this issue until recently).

Thanks David. In that case, let's go ahead with releasing the roots module 2.4 with this change.

Landed into NSS_3_18_BRANCH https://hg.mozilla.org/projects/nss/rev/d034b5b1cba6

In order to fix this bug, we need a new PSM bug that upgrades Mozilla/PSM to NSS 3.18.1 I filed bug 1156428.

(In reply to Sylvestre Ledru [:sylvestre] from comment #16) > That works for me, especially in the context of 38 being an ESR. It would be > nice if we could have a patch today (Monday) for beta 6 or by Thrusday for > beta 7. Patch is ready and waiting for your approval in bug 1156428

"Last Cert expires: 29-Nov-2016" Actually, I have seen some eg Yahoo domains expire in Aug 2018. The more important date is ~5 years after RapidSSL switched to a 2048-bit root, which put it at around the end of 2015.

I believe I landed into mozilla-beta prior to creation of the separate 38 ESR branch.

(In reply to Kai Engert (:kaie) from comment #31) > I believe I landed into mozilla-beta prior to creation of the separate 38 > ESR branch. This is very likely as there is no ESR branch yet ;)

I filed Bug #1156844 to track creating a new plan for phasing out the "Equifax Secure Certificate Authority" 1024-bit root.

Can I get please some guiding on verifying this bug? I've tried reproducing the issue using a FF 37 release build and sites from the following list https://bug986019.bugzilla.mozilla.org/attachment.cgi?id=8571482 but the sites seem to work fine with FF 37.

To verify, you need to compare a build that doesn't contain this yet, with the latest build. If the older build fails with sites from the list (error message), but the latest build works (site loads without error message), then this bug is fixed.

The problem here was that the issue does not reproduce in Firefox 37. I was able to reproduce it though on Firefox 38 Beta 1 & 3, on Windows 7 x64, Ubuntu 12.04 x86, and Mac OS X 10.8.5, where I got an Untrusted Connection page (invalid security certificate) whenever I loaded a page from the attached list. The pages then loaded without error when opened in Firefox 38.0 ESR, on all 3 systems. Test sites used: https://atlasbridge.com/ https://actual-live.net/ https://aerobrowser.net/ https://bnc.mx/ https://client.lu/

(In reply to Florin Mezei, QA (:FlorinMezei) from comment #36) > The problem here was that the issue does not reproduce in Firefox 37. That's expected, because Firefox 37 had still contained the root. We had removed the Equifax root (or rather, we had disabled its trust), with a target milestone of Firefox 38, as part of bug 1132496, during the development of Firefox 38. That's why it fails in Firefox 37, but you see that the sites fail with the Firefox 38 beta versions. Very late in the development cycle of Firefox 38 it had been decided to undo the change for this Equifax root, as part of this bug 1155279. I think comment 36 is enough to set this bug to VERIFIED.

The Equifax certificate creates an attack vector for Advanced Persistent Threats. Check out the research at https://leeneubecker.com/equifax-equimelt-vulnerability/ #Equimelt Also, a recent white paper by Doctoral students at the University of Maryland details the potential for APT's by certificate injection. http://www.umiacs.umd.edu/~tdumitra/papers/CCS-2017.pdf These weak and untrusted CA's need to be removed. d23209ad23d314232174e40d7f9d62139786633a 323c118e1bf7b8b65254e2e2100dd6029037f096 627f8d7827656399d27d7f9044c9feb3f33efa9a 742c3192e607e424eb4549542be1bbc53e6174e2

The Equifax root ca has already been removed from Mozilla's trust list already, in september 2016, as part of bug 1296689 and NSS 3.27, I think was done in Firefox 51. https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/NSS_3.27_release_notes Please file a separate bug for further discussion of your new report. In that new bug, please provide more detailed identification of the 4 additional CAs you list as weak.

(In reply to Kai Engert (:kaie:) from comment #39) > Please file a separate bug for further discussion of your new report. I agree that Comment #38 should have been a separate bug. However, since it's already in this bug I suggest that we do not need another bug filed, because all of the roots listed in Comment #38 and in the article are or have already been dealt with... (In reply to me from comment #38) > These weak and untrusted CA's need to be removed. > d23209ad23d314232174e40d7f9d62139786633a Equifax Secure Certificate Authority -- Removed via Bug #1288250 > 323c118e1bf7b8b65254e2e2100dd6029037f096 GeoTrust Primary Certification Authority -- Part of Symantec distrust plan: https://wiki.mozilla.org/CA/Additional_Trust_Changes#Symantec > 627f8d7827656399d27d7f9044c9feb3f33efa9a Thawte Premium Server CA -- Removed via Bug #986014 > 742c3192e607e424eb4549542be1bbc53e6174e2 VeriSign Class 3 Public Primary Certification Authority -- Removed via Bug #1288250 Thanks!