User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.134 Safari/534.16 Build Identifier: I'm planning to send this email out to the EFF ssl observatory list but I wanted to give security@mozilla.com a heads up first. BEGIN EMAIL: "Detecting collusion between vendors and CAs: Revocation isn't enough and everyone knows it" This email is probably going to be somewhat controversial and so I'd like to caution people that I base my claims entirely on evidence linked withing this email. Please come to your own conclusions and consider that I mean no disrespect to any of the involved projects - this is a very serious issue that needs to be addressed. tl;dr: I think a CA was compromised and important high value certificates were issued to unknown attackers. Certificate revocation occurs for a number of reasons and the process of revocation is largely worthless. CRL and OCSP are both totally flawed processes for revocation. An attacker who is able to perform a SSL/TLS MITM is easily able to thwart the CRL and the OCSP process. It is total garbage in every browser except Chrome when interfacing with HSTS enabled sites. I watch a few software projects carefully and recently, I noticed a commit diff of extreme interest - revision 78478, Thu Mar 17 00:48:21 2011 UTC: http://src.chromium.org/viewvc/chrome/trunk/src/net/base/x509_certificate.cc?view=markup&pathrev=78478 In the function bool X509Certificate::IsBlacklisted() we see something rather interesting: {0x07,0x7a,0x59,0xbc,0xd5,0x34,0x59,0x60,0x1c,0xa6,0x90,0x72,0x67,0xa6,0xdd,0x1c}, {0x04,0x7e,0xcb,0xe9,0xfc,0xa5,0x5f,0x7b,0xd0,0x9e,0xae,0x36,0xe1,0x0c,0xae,0x1e}, {0xd8,0xf3,0x5f,0x4e,0xb7,0x87,0x2b,0x2d,0xab,0x06,0x92,0xe3,0x15,0x38,0x2f,0xb0}, {0xb0,0xb7,0x13,0x3e,0xd0,0x96,0xf9,0xb5,0x6f,0xae,0x91,0xc8,0x74,0xbd,0x3a,0xc0},{0x92,0x39,0xd5,0x34,0x8f,0x40,0xd1,0x69,0x5a,0x74,0x54,0x70,0xe1,0xf2,0x3f,0x43}, {0x92,0x39,0xd5,0x34,0x8f,0x40,0xd1,0x69,0x5a,0x74,0x54,0x70,0xe1,0xf2,0x3f,0x43}, {0xd7,0x55,0x8f,0xda,0xf5,0xf1,0x10,0x5b,0xb2,0x13,0x28,0x2b,0x70,0x77,0x29,0xa3}, {0xf5,0xc8,0x6a,0xf3,0x61,0x62,0xf1,0x3a,0x64,0xf5,0x4f,0x6d,0xc9,0x58,0x7c,0x06}, We can reduce those to the following easy to read serial numbers: 047ecbe9fca55f7bd09eae36e10cae1e d8f35f4eb7872b2dab0692e315382fb0 b0b7133ed096f9b56fae91c874bd3ac0 9239d5348f40d1695a745470e1f23f43 9239d5348f40d1695a745470e1f23f43 <-dupe d7558fdaf5f1105bb213282b707729a3 f5c86af36162f13a64f54f6dc9587c06 The first line is stated in a comment to be a test "Not a real certificate. For testing only" and this implies that the rest appear to be sixteen byte serial numbers that are now blocklisted. Lines five and six appear to be a mistake as they are identical. I suspect there will be an update to this function; it may be the case that a certificate in need of blocklisting was left out. This alone is not very compelling and without the leaf certificate, we're really not able to learn very much about the certificates that were blocked. The CRL used for revocation is generally present in the leaf certificate handed to a client - in theory, this means that an attacker is unable to tamper with the CRL distribution point without breaking the digital signature. While we lack the CRL distribution point in this function, the project that I announced yesterday, crlwatch, was specifically written to assist in finding who issued, and potentially revoked the serial numbers in question. About twelve hours (Thursday, March 17, 2011 | 13:00) after the above patch was pushed into source control - Google announced an important Chrome Update: http://googlechromereleases.blogspot.com/2011/03/stable-and-beta-channel-updates_17.html This also is mostly uninteresting until we notice that this is not isolated to Google. Mozilla pushed out two patches of interest: http://hg.mozilla.org/mozilla-central/rev/f6215eef2276 http://hg.mozilla.org/mozilla-central/rev/55f344578932 The first patch includes the following serial numbers in a struct called nsSerialBinaryBlacklistEntry: \x00\x92\x39\xd5\x34\x8f\x40\xd1\x69\x5a\x74\x54\x70\xe1\xf2\x3f\x43 \x00\xd8\xf3\x5f\x4e\xb7\x87\x2b\x2d\xab\x06\x92\xe3\x15\x38\x2f\xb0 \x72\x03\x21\x05\xc5\x0c\x08\x57\x3d\x8e\xa5\x30\x4e\xfe\xe8\xb0 \x00\xb0\xb7\x13\x3e\xd0\x96\xf9\xb5\x6f\xae\x91\xc8\x74\xbd\x3a\xc0 \x00\xe9\x02\x8b\x95\x78\xe4\x15\xdc\x1a\x71\x0a\x2b\x88\x15\x44\x47 \x00\xd7\x55\x8f\xda\xf5\xf1\x10\x5b\xb2\x13\x28\x2b\x70\x77\x29\xa3 \x04\x7e\xcb\xe9\xfc\xa5\x5f\x7b\xd0\x9e\xae\x36\xe1\x0c\xae\x1e \x00\xf5\xc8\x6a\xf3\x61\x62\xf1\x3a\x64\xf5\x4f\x6d\xc9\x58\x7c\x06 The second patch adds two more serial numbers to that same struct: \x39\x2a\x43\x4f\x0e\x07\xdf\x1f\x8a\xa3\x05\xde\x34\xe0\xc2\x29 \x3e\x75\xce\xd4\x6b\x69\x30\x21\x21\x88\x30\xae\x86\xa8\x2a\x71 Once again, we can reduce those to the following easy to read serial numbers: 009239d5348f40d1695a745470e1f23f43 00d8f35f4eb7872b2dab0692e315382fb0 72032105c50c08573d8ea5304efee8b0 00b0b7133ed096f9b56fae91c874bd3ac0 00e9028b9578e415dc1a710a2b88154447 00d7558fdaf5f1105bb213282b707729a3 047ecbe9fca55f7bd09eae36e10cae1e 00f5c86af36162f13a64f54f6dc9587c06 392a434f0e07df1f8aa305de34e0c229 3e75ced46b693021218830ae86a82a71 According to the commit, these serial numbers were added to address a secret Mozilla bug: https://bugzilla.mozilla.org/show_bug.cgi?id=642395 Attempts to access this bug results in the following error: "You are not authorized to access bug #642395. To see this bug, you must first log in to an account with the appropriate permissions." The full Mozilla change set list is here: http://hg.mozilla.org/mozilla-central/rev/f6215eef2276 It appears to have been made in this changeset: changeset: 63436:f6215eef2276 user: Kaie Engert <kaie@kuix.de> date: Thu Mar 17 14:40:13 2011 -0700 summary: Bug 642395 - change to handling of bad certificates. r=rrelyea, bsmith. a=sayrer This shows that Mozilla has blocklisted ten certificate serial numbers and Google has so far blocklisted eight certificates, one of which is a test and another that is a duplicate. Mozilla calls them bad, Google calls it a blacklist. Interesting timing too! Looks coordinated! The unified list of serials without uniqueness or sorting is: 047ecbe9fca55f7bd09eae36e10cae1e d8f35f4eb7872b2dab0692e315382fb0 b0b7133ed096f9b56fae91c874bd3ac0 9239d5348f40d1695a745470e1f23f43 9239d5348f40d1695a745470e1f23f43 <-dupe d7558fdaf5f1105bb213282b707729a3 f5c86af36162f13a64f54f6dc9587c06 009239d5348f40d1695a745470e1f23f43 00d8f35f4eb7872b2dab0692e315382fb0 72032105c50c08573d8ea5304efee8b0 00b0b7133ed096f9b56fae91c874bd3ac0 00e9028b9578e415dc1a710a2b88154447 00d7558fdaf5f1105bb213282b707729a3 047ecbe9fca55f7bd09eae36e10cae1e 00f5c86af36162f13a64f54f6dc9587c06 392a434f0e07df1f8aa305de34e0c229 3e75ced46b693021218830ae86a82a71 It's clear that there is some overlap here as we see both lists include identical entries. In total, when we make a sorted list with unique entries we find the following serial numbers: 009239d5348f40d1695a745470e1f23f43 00b0b7133ed096f9b56fae91c874bd3ac0 00d7558fdaf5f1105bb213282b707729a3 00d8f35f4eb7872b2dab0692e315382fb0 00e9028b9578e415dc1a710a2b88154447 00f5c86af36162f13a64f54f6dc9587c06 047ecbe9fca55f7bd09eae36e10cae1e 392a434f0e07df1f8aa305de34e0c229 3e75ced46b693021218830ae86a82a71 72032105c50c08573d8ea5304efee8b0 9239d5348f40d1695a745470e1f23f43 b0b7133ed096f9b56fae91c874bd3ac0 d7558fdaf5f1105bb213282b707729a3 d8f35f4eb7872b2dab0692e315382fb0 f5c86af36162f13a64f54f6dc9587c06 This gives us fifteen serial numbers worthy of blocking between both browsers. I suspect they should unify their list. Of course, this brings us to the next question - Why would Mozilla have one list and Google has another? As a slight conjecture tangent: I'm betting on a split disclosure - Mozilla was probably given a list that is important to them and Google was likely in the same position. This returns me to the reason for creating crlwatch yesterday - I want to find someone who knowingly revoked those special certificates. Because anyone looking lacks the leaf certificates in question, we have to look in a round about manner. Thanks to crlwatch, I now have a nearly canonical list of all CRLs. I parsed the CRLs into human readable text and searched for the above serial numbers. This was the result: Looking for 009239d5348f40d1695a745470e1f23f43 in parsed CRLs... Looking for 00b0b7133ed096f9b56fae91c874bd3ac0 in parsed CRLs... Looking for 00d7558fdaf5f1105bb213282b707729a3 in parsed CRLs... Looking for 00d8f35f4eb7872b2dab0692e315382fb0 in parsed CRLs... Looking for 00e9028b9578e415dc1a710a2b88154447 in parsed CRLs... Looking for 00f5c86af36162f13a64f54f6dc9587c06 in parsed CRLs... Looking for 047ecbe9fca55f7bd09eae36e10cae1e in parsed CRLs... 2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt: Serial Number: 047ECBE9FCA55F7BD09EAE36E10CAE1E 2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt: Serial Number: 047ECBE9FCA55F7BD09EAE36E10CAE1E 5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt: Serial Number: 047ECBE9FCA55F7BD09EAE36E10CAE1E Looking for 392a434f0e07df1f8aa305de34e0c229 in parsed CRLs... 2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt: Serial Number: 392A434F0E07DF1F8AA305DE34E0C229 2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt: Serial Number: 392A434F0E07DF1F8AA305DE34E0C229 5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt: Serial Number: 392A434F0E07DF1F8AA305DE34E0C229 Looking for 3e75ced46b693021218830ae86a82a71 in parsed CRLs... 2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt: Serial Number: 3E75CED46B693021218830AE86A82A71 2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt: Serial Number: 3E75CED46B693021218830AE86A82A71 5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt: Serial Number: 3E75CED46B693021218830AE86A82A71 Looking for 72032105c50c08573d8ea5304efee8b0 in parsed CRLs... Looking for 9239d5348f40d1695a745470e1f23f43 in parsed CRLs... 2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt: Serial Number: 9239D5348F40D1695A745470E1F23F43 2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt: Serial Number: 9239D5348F40D1695A745470E1F23F43 5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt: Serial Number: 9239D5348F40D1695A745470E1F23F43 Looking for b0b7133ed096f9b56fae91c874bd3ac0 in parsed CRLs... 2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt: Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0 2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt: Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0 5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt: Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0 Looking for d7558fdaf5f1105bb213282b707729a3 in parsed CRLs... 2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt: Serial Number: D7558FDAF5F1105BB213282B707729A3 2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt: Serial Number: D7558FDAF5F1105BB213282B707729A3 5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt: Serial Number: D7558FDAF5F1105BB213282B707729A3 Looking for d8f35f4eb7872b2dab0692e315382fb0 in parsed CRLs... 2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt: Serial Number: D8F35F4EB7872B2DAB0692E315382FB0 2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt: Serial Number: D8F35F4EB7872B2DAB0692E315382FB0 5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt: Serial Number: D8F35F4EB7872B2DAB0692E315382FB0 Looking for f5c86af36162f13a64f54f6dc9587c06 in parsed CRLs... 2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt: Serial Number: F5C86AF36162F13A64F54F6DC9587C06 2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt: Serial Number: F5C86AF36162F13A64F54F6DC9587C06 5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt: Serial Number: F5C86AF36162F13A64F54F6DC9587C06 Huzzah! It appears that we've found a few matches! Here are the three matching files: https://github.com/ioerror/crlwatch/blob/master/crl-parsed/2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt https://github.com/ioerror/crlwatch/blob/master/crl-parsed/2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt https://github.com/ioerror/crlwatch/blob/master/crl-parsed/5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt Matching entries in the list look like this: Serial Number: 392A434F0E07DF1F8AA305DE34E0C229 Revocation Date: Mar 15 20:15:38 2011 GMT An interesting note is that this date is a bit earlier than the above patches! The CA knew to revoke it and patches were worked into software a a few days later. I wonder how long the CA knew before they revoked it? All three of the CRLs in question belong to the same CA: issuer=/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN-USERFirst-Hardware This CA url is here: http://www.usertrust.com/about.html This appears to be a reseller or something similar for Comodo when you inspect the trust chain for their site: CN = COMODO High Assurance Secure Server CA We appear to have no initial matches for the following serials from the data that I gathered earlier this morning: 009239d5348f40d1695a745470e1f23f43 00b0b7133ed096f9b56fae91c874bd3ac0 00d7558fdaf5f1105bb213282b707729a3 00d8f35f4eb7872b2dab0692e315382fb0 00e9028b9578e415dc1a710a2b88154447 00f5c86af36162f13a64f54f6dc9587c06 Those serial numbers appear to not match, right? Nope. Mozilla appears to have a really serious bug in their secret bug patching process. If you remove the prefix of 00 from those serial numbers and search for the sixteen byte values, we find what we'd expect: Looking for 9239d5348f40d1695a745470e1f23f43 in parsed CRLs... 2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt: Serial Number: 9239D5348F40D1695A745470E1F23F43 2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt: Serial Number: 9239D5348F40D1695A745470E1F23F43 5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt: Serial Number: 9239D5348F40D1695A745470E1F23F43 Looking for b0b7133ed096f9b56fae91c874bd3ac0 in parsed CRLs... 2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt: Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0 2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt: Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0 5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt: Serial Number: B0B7133ED096F9B56FAE91C874BD3AC0 Looking for d7558fdaf5f1105bb213282b707729a3 in parsed CRLs... 2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt: Serial Number: D7558FDAF5F1105BB213282B707729A3 2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt: Serial Number: D7558FDAF5F1105BB213282B707729A3 5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt: Serial Number: D7558FDAF5F1105BB213282B707729A3 Looking for d8f35f4eb7872b2dab0692e315382fb0 in parsed CRLs... 2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt: Serial Number: D8F35F4EB7872B2DAB0692E315382FB0 2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt: Serial Number: D8F35F4EB7872B2DAB0692E315382FB0 5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt: Serial Number: D8F35F4EB7872B2DAB0692E315382FB0 Looking for e9028b9578e415dc1a710a2b88154447 in parsed CRLs... 2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt: Serial Number: E9028B9578E415DC1A710A2B88154447 2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt: Serial Number: E9028B9578E415DC1A710A2B88154447 5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt: Serial Number: E9028B9578E415DC1A710A2B88154447 Looking for f5c86af36162f13a64f54f6dc9587c06 in parsed CRLs... 2208d196b567691ff77bee6ed97ac6aea7825aa0.crl2txt: Serial Number: F5C86AF36162F13A64F54F6DC9587C06 2454652aa76e0dcf0c7531a2ed9dbeda5c0f5dd5.crl2txt: Serial Number: F5C86AF36162F13A64F54F6DC9587C06 5a65d440090271759080e6d0eaae0b6ccae2ff50.crl2txt: Serial Number: F5C86AF36162F13A64F54F6DC9587C06 Here's sample from one of those CRLs: Serial Number: D7558FDAF5F1105BB213282B707729A3 Revocation Date: Mar 15 20:15:26 2011 GMT It looks like Mozilla made a rather weird way of structuring their blocklisting serial struct! Ironically, the Mozilla patch also leaks the CA name: http://hg.mozilla.org/mozilla-central/rev/f6215eef2276#l1.57 I think that this is evidence of a rather serious event. If I had to make a bet, I'd wager that an attacker was able to issue high value certificates, probably by compromising USERTRUST in some manner, this was discovered sometime before the revocation date, the certificate was revoked, the vendors notified, the patches were shipped, and binary builds kicked off - people are probably still updating and thus many people are vulnerable to the failure that is the CRL and OCSP method for revocation. Even after they update, I'm guessing users are in trouble because the list isn't the same. Mozilla should force OCSP verification by default as part of their next release, I think. I've cc'ed Comodo, Google, and Mozilla for comment - I sure would love to get their side of the story here. It sounds like people are at risk and while everyone looks like they're trying their best, it looks like we REALLY need to find a better way to deal with issuing certificates, leaf certificate revocation, and provide a more transparent method for alerting people to these kinds of serious mishaps. All the best, Jacob Reproducible: Always I think this is eligible for a bug bounty.

I forgot to mention that if you remove the 00 prefix, sort and only list unique elements, the list is actually ten certificates long: 047ecbe9fca55f7bd09eae36e10cae1e 392a434f0e07df1f8aa305de34e0c229 3e75ced46b693021218830ae86a82a71 72032105c50c08573d8ea5304efee8b0 9239d5348f40d1695a745470e1f23f43 b0b7133ed096f9b56fae91c874bd3ac0 d7558fdaf5f1105bb213282b707729a3 d8f35f4eb7872b2dab0692e315382fb0 e9028b9578e415dc1a710a2b88154447 f5c86af36162f13a64f54f6dc9587c06

Jacob, what's the bug you're reporting on our end? We appreciate the heads-up on your planned mail, obviously!

I'll suggest that we change the title and make this bug become: > Mozilla should force OCSP verification by default as part of their next > release, I think. and start the investigation on how we might pull that off in firefox 5.

I'm on the phone with Dan at the moment, so perhaps he'll add more to this bug report. Basically the bug is that Comodo is doing split disclosures and is compromised. I guess all of their certs should be marked as invalid for the given three CRLs I listed. Also, that the lists of certs isn't the same between Mozilla and chrome.

> Basically the bug is that Comodo is doing split disclosures I don't think they are. For example, every single entry in Chrome's list is in our list. It does look like Chrome needs to update their list some more... Past that, I'll defer to Dan.

We also requested a test cert and received 72:03:21:05:C5:0C:08:57:3D:8E:A5:30:4E:FE:E8:B0 Ignoring the test certs it looks like Google and Mozilla were given the same initial list, at least if you assume the Chrome duplicate is a mistake and was intended to be e9:02:8b:95:78:e4:15:dc:1a:71:0a:2b:88:15:44:47 The two serial numbers from the second Mozilla patch were not sent to us until after the Chrome update was released. Presumably Google also have these SNs and will issue a future update. The strange initial null byte stripping in our patch is explained by Bob Relyea in bug 642395 comment 41, which I'll repeat here since that bug is inaccessible at the moment Serial numbers are unsigned integers. DER INTS are all signed, so when you encode an unsigned integer with DER you need to make sure the leading bit is not zero. This CA issues serial numbers which are 16 bytes long. If the lead byte starts with a zero (0x00-0x7f) then the serial number is 16 bytes long, If the lead starts with a 1 (0x80-0xff) then a zero is added as padding. Different uses of these values tend to decode the resulting bytes differently, so it's usually best to canonicalize the bytes before comparing them. the easiest way of doing that is to strip the leading bytes. You might ask why hardcode the leading 00 into our table of SNs if we're just going to strip it? The table was generated from the raw certs using a pre-existing tool that encodes with an initial null. Given the haste of the release it seemed best to leave the tool alone and not risk hand editing, and then re-hand-editing when we got the two new certs and regenerated the table. Although our update servers are not amongst the certs in question, anyone in a position to deploy those certs could easily block updates if they became aware we were addressing the issue. I would be grateful if you could wait to go public until users have a chance to get the update next week.

Ok. It's good to understand why the certs aren't in sync - I've been in touch with Google and they're appraised of this issue. It's bad to know that they're not in sync because the attack had some ability to generate new certs after the first set of certs was disclosed. I'm glad to hear that it may not be a split disclosure but it sure seems like Comodo should get an internet death sentence for this specific SSL reseller. This CA can't be trusted unless they can prove the key material wasn't compromised. That seems like a REALLY hard thing to prove given what I've been able to uncover. I won't go public until next week but I strongly urge you guys to enable mandatory OCSP *warning* at the very least before you ship the next Firefox. This is a security *disaster* if the certs issued are really what I believe them to be - it would allow for remote code execution on all Firefox clients when they query for an update. All an attacker has to do is reject OCSP/CRL queries with a HTTP 500 and MITM any attempts to upgrade any extensions. This will lead to native code execution, no? Read this blog post to understand the full ramifications: http://www.imperialviolet.org/2011/03/18/revocation.html I've said for *years* that relying on SSL alone for package and code verification in Firefox is not adequate. It seems like this event is a prime example of a near worst case scenario - if the attacker has specific people in mind, they're absolutely screwed. Mozilla needs to address this and not just by covering for a few certificates that Comodo has stated are compromised.

(In reply to comment #7) > This is a security *disaster* if the certs issued are really what I believe > them to be - it would allow for remote code execution on all Firefox clients > when they query for an update. The certs in question do not include the update servers for Firefox or extensions (aus2|aus3.mozilla.org and versioncheck.addons.mozilla.org) > This will lead to native code execution, no? If they had gotten the right certs; in this case we appear to have gotten lucky. Firefox 4 is safer: you'd have to obtain a spoofed intermediate cert in addition to the leaf cert. Not yet good enough.

(In reply to comment #8) > (In reply to comment #7) > > This is a security *disaster* if the certs issued are really what I believe > > them to be - it would allow for remote code execution on all Firefox clients > > when they query for an update. > > The certs in question do not include the update servers for Firefox or > extensions (aus2|aus3.mozilla.org and versioncheck.addons.mozilla.org) > Oh? I thought they did get AMO? > > This will lead to native code execution, no? > > If they had gotten the right certs; in this case we appear to have gotten > lucky. Firefox 4 is safer: you'd have to obtain a spoofed intermediate cert in > addition to the leaf cert. Not yet good enough. That's better, I agree. Still, there's a window of time now that these certs are useful. It seems like a seriously bad time for you guys to push out a new version of Firefox if you're not entirely sure of what they have, how they may use it, and what the impact will be. I'm guessing a lot of traffic will come your way and an attacker will be able to use those certs to their advantage in a big way. At least in the short term, I really think enabling OCSP to required is the minimum safe thing to do before downloading new code. It's a work around that will fail closed and while it's a bad thing, it seems like the best out of all of a bunch of bad choices. The only other option I really think might make a difference is to alert the world what certs are being looked for - not just the serial number but the entire cert chain, etc

(In reply to comment #5) > > Basically the bug is that Comodo is doing split disclosures > > I don't think they are. For example, every single entry in Chrome's list is in > our list. It does look like Chrome needs to update their list some more... > > Past that, I'll defer to Dan. I think I agree now. After talking with Google - it's clear that there was some lag. I think that there is something really seriously fishy going on with Comodo's CA though - they have many other revocations for the same day that are not in either list. I want to give Comodo credit - they could have swept this under the rug but I also think that the rest of the world needs to understand what happened. There are lots of other browsers out in the wild who won't have a blocklist of serials and they're all in trouble.

(BTW, congratulations on some good detective work :-) You assume that the attacker has a continued ability to issue certs; Comodo assure us that this is not the case. The reason two patches were required was that they forgot (!) to tell us about two of the certs until we had spun up builds once already. [You can draw your own conclusions as to how pleased we are about that.] I have been impressing upon Comodo the need to go public at the right moment, and explain what happened in detail - for all the reasons you give. Your work means I think you deserve to know more. One of their RAs was broken into (electronically, I understand) and their creds used to issue these certs. Because of the value of the targets, a post-issuance business intelligence ("we have a big new customer!") style check spotted them. That RAs account is now (of course) shut down, and all issuances have been scrutinized. As to your larger points, based on this event I want to start a discussion within Mozilla about where we see the future of internet identity verification going. But the time for that is not right now. (In reply to comment #9) > Oh? I thought they did get AMO? Your confusion is, fortunately, theirs :-) AMO != AUS. addons.mozilla.org (addons service) != aus.mozilla.org (update service). And there are extra protections around aus that we added in Firefox 4 where it checks the issuer, which means they could not compromise AUS even if they had a cert for it. > At least in the short term, I really think enabling OCSP to required is the > minimum safe thing to do before downloading new code. I strongly suspect we will look at that again but, given that the channel of bad certs is shut, the patch we have achieves the same effect with lesser fallout. Gerv

addons.mozilla.org is the website where you go to browse and download new extensions. versioncheck.addons.mozilla.org is where you check for addon updates, different cert. You can't even try a social engineering "if it's broken just add an exception" hack because the update checks do not allow you to override bad certs. Client updates ping aus2.mozilla.org in older versions, and aus3.mozilla.org in Firefox 4. Same deal as the extension update checks: bad cert overrides are not allowed. > there are extra protections around aus that we added in Firefox 4 where > it checks the issuer, which means they could not compromise AUS even if > they had a cert for it. True, but if you obtained a signing cert you could create both a fake issuer and fake leaf cert. We need to sign the contents of the update response and not just hack around it. Argh, we're not doing this for add-on update checks. That blows. You still need only one bogus cert to compromise anyone with an addon (luckily, a different cert than the ones we know were compromised). We estimate between 1/3 and 1/2 of our users have an addon installed so that's just about as good as getting the app-update cert. To repeat my initial point though, neither the application update server nor the addon update server were among the certs Comodo told us about.

(In reply to comment #11) > (BTW, congratulations on some good detective work :-) > > You assume that the attacker has a continued ability to issue certs; Comodo > assure us that this is not the case. The reason two patches were required was > that they forgot (!) to tell us about two of the certs until we had spun up > builds once already. [You can draw your own conclusions as to how pleased we > are about that.] Yes. I assumed that there was a reason that Comodo would send two batches was either that Comodo or UTN was further compromised, perhaps Comodo made a mistake, perhaps Comodo was incompetent, or worse. Your explanation seems to point to well meaning incompetence at the least. :-) > I have been impressing upon Comodo the need to go public at the right moment, > and explain what happened in detail - for all the reasons you give. I've informed the EFF about this issue, they've also agreed to keep quiet. It appear that from the observatory data we need to know _exactly_ what signed these keys as UTN is reasonably large: CA: C=US, ST=UT, L=Salt Lake City, O=The USERTRUST Network, OU=http://www.usertrust.com, CN=UTN-USERFirst-Hardware 50 total sub-CAs, 43 immediate 205794 total leaves, 85440 immediate Were these certs signed by a sub-CA as I assume? If so, what the key for signing stored in hardware? > Your work > means I think you deserve to know more. One of their RAs was broken into > (electronically, I understand) and their creds used to issue these certs. > Because of the value of the targets, a post-issuance business intelligence ("we > have a big new customer!") style check spotted them. That RAs account is now > (of course) shut down, and all issuances have been scrutinized. > All of them for all time? What prevents the other accounts on the same system from doing this same thing? > As to your larger points, based on this event I want to start a discussion > within Mozilla about where we see the future of internet identity verification > going. But the time for that is not right now. Yes, I agree. I think this will probably happen on the sslobservatory list run by the EFF very soon. I'm guessing Tuesday. > > (In reply to comment #9) > > Oh? I thought they did get AMO? > > Your confusion is, fortunately, theirs :-) AMO != AUS. addons.mozilla.org > (addons service) != aus.mozilla.org (update service). And there are extra > protections around aus that we added in Firefox 4 where it checks the issuer, > which means they could not compromise AUS even if they had a cert for it. > Theirs? The attacker? Oh the hilarity. > > At least in the short term, I really think enabling OCSP to required is the > > minimum safe thing to do before downloading new code. > > I strongly suspect we will look at that again but, given that the channel of > bad certs is shut, the patch we have achieves the same effect with lesser > fallout. One channel of bad certs that we know about is purportedly shut. The entire CA model is a disaster for privacy, security, confidentiality and so on. I'm glad that Comodo did the right thing but I don't think that it absolves their business model of being a losing direction for the internet in the long run.

(In reply to comment #12) > addons.mozilla.org is the website where you go to browse and download new > extensions. Right. Having control of this allows for code execution. > > versioncheck.addons.mozilla.org is where you check for addon updates, different > cert. You can't even try a social engineering "if it's broken just add an > exception" hack because the update checks do not allow you to override bad > certs. > OK. Unless there's a bug in that ssl code too. Here's hoping. > Client updates ping aus2.mozilla.org in older versions, and aus3.mozilla.org in > Firefox 4. Same deal as the extension update checks: bad cert overrides are not > allowed. OK. > > > there are extra protections around aus that we added in Firefox 4 where > > it checks the issuer, which means they could not compromise AUS even if > > they had a cert for it. > > True, but if you obtained a signing cert you could create both a fake issuer > and fake leaf cert. We need to sign the contents of the update response and not > just hack around it. > I agree. > Argh, we're not doing this for add-on update checks. That blows. You still need > only one bogus cert to compromise anyone with an addon (luckily, a different > cert than the ones we know were compromised). We estimate between 1/3 and 1/2 > of our users have an addon installed so that's just about as good as getting > the app-update cert. > Right. > To repeat my initial point though, neither the application update server nor > the addon update server were among the certs Comodo told us about. Ok - to be clearer: what was the FQDN list that you know of?

Some more stats on the following CA from the EFF SSL Observatory dataset: CA: C=US, ST=UT, L=Salt Lake City, O=The USERTRUST Network, OU=http://www.usertrust.com, CN=UTN-USERFirst-Hardware 50 total sub-CAs, 43 immediate 205794 total leaves, 85440 immediate 85,440 leaf certs were signed by that CA's key directly ~115,000 leaf certs were signed by intermediate CAs below it I'm not a fan of capital punishment in meatspace but I'd like to advocate for an internet death sentence for the specific signing CA. Unless we find extremely compelling evidence that demonstrates they could not have had a key compromise, we cannot assume that things are hunky dory.

(In reply to comment #11) > > At least in the short term, I really think enabling OCSP to required is the > > minimum safe thing to do before downloading new code. > > I strongly suspect we will look at that again but, given that the channel of > bad certs is shut, the patch we have achieves the same effect with lesser > fallout. > I wanted to really drive home a strong counter point. If we assume that Comodo is totally honest and has made no mistake (doubtful), I still think OCSP should be required (it's already enabled) because there are so many CAs that may also revoke silently. If revocation actually worked as it stands today, you guys wouldn't need to hard code serials into the binary.

(In reply to comment #16) > If revocation actually worked as it stands today, you guys > wouldn't need to hard code serials into the binary. I can only say that this is now the starting point for my own thinking on the matter. You are also right that Comodo could have just revoked silently, and we would have never known. So them coming forward does them credit. Gerv

I'm writing a guest blog post that will go live on the EFF website tomorrow - I'd like to get a few things together - one of which is the list of (CN) names for the certs - will you guys please disclose them?

We will if Comodo doesn't, but they've asked us not to for the time being because other vendors are affected who may not be able to update as quickly. There were a total of six domain names plus one that makes no sense. The two serial numbers added later were reissues of one of the original set of certs. I will confirm that 'addons.mozilla.org' was one of the certs. This is Not Good but could not be used to hijack add-on updates, only visits to the web site itself. The odd one no one can make any sense of was "global trustee" with a space in it. Don't know where such a cert would be valid. We did find a globaltrustee.com in Florida, but the cert issued would not work to MITM that site.

Jacob: we have just had a message from Comodo to say they have word from Microsoft that they intend to release a mitigation package tomorrow (Wednesday) at 10am PDT. They further say: "We are wondering if you [Mozilla] would consider delaying an announcement so we can all release together, tomorrow. This way, we can provide a coordinated response with as much mitigation for this issue as possible. They feel, and we agree, that this engenders the concept of community-based defence and best serves our respective customers. Regards Robin Alden M.Sc. MIET CTO -- Comodo" We are content to wait until then. Are you? Gerv

(In reply to comment #19) > We will if Comodo doesn't, but they've asked us not to for the time being > because other vendors are affected who may not be able to update as quickly. > There were a total of six domain names plus one that makes no sense. The two > serial numbers added later were reissues of one of the original set of certs. > > I will confirm that 'addons.mozilla.org' was one of the certs. This is Not Good > but could not be used to hijack add-on updates, only visits to the web site > itself. > Great. I'll include that information in the blog post. Thank you very much Dan. > The odd one no one can make any sense of was "global trustee" with a space in > it. Don't know where such a cert would be valid. We did find a > globaltrustee.com in Florida, but the cert issued would not work to MITM that > site. That's an odd one, I agree. Can the certs be used for code signing? I know that a friend disabled the CA and lost the ability to validate S/MIME - I suspect that the issues here run very deep.

(In reply to comment #20) > Jacob: we have just had a message from Comodo to say they have word from > Microsoft that they intend to release a mitigation package tomorrow (Wednesday) > at 10am PDT. > > They further say: > > "We are wondering if you [Mozilla] would consider delaying an announcement so > we can all release together, tomorrow. This way, we can provide a coordinated > response with as much mitigation for this issue as possible. This is not a normal attack. Disclosure does not allow anyone else to perform this attack - only the attacker with the certificate is able to take advantage of this situation. Only the attacker will benefit from a delay. > > They feel, and we agree, that this engenders the concept of community-based > defence and best serves our respective customers. End users are not customers. What is being done to protect them right now? Nothing unless they upgrade. This is really unacceptable as a solution. This kind of attack will happen again and again. Downloading a new binary isn't going to cut it. What happens when the attacker gets certificates for the Mozilla update process? The users will then be absolutely screwed without question. > > We are content to wait until then. Are you? > Yes, I will wait for a fixed, specific amount of time. 10AM PST tomorrow is when the blog post will go live. I will not agree to any further delays.

Sorry to be a pain in your collective asses but it looks like someone visiting https://addons.mozilla.org/en-US/firefox/addon/imtranslator/ will certainly install code (https://addons.mozilla.org/firefox/downloads/latest/2257/addon-2257-latest.xpi?src=addondetail) from the attacker who controls addons.mozilla.org. Is Mozilla really putting all of their users at risk to benefit Comodo and Microsoft? Why isn't Mozilla telling the entire world to enable OCSP and pushing for a security update? You guys can mitigate this without fully disclosing and the specific targeted users are totally in trouble because you guys have not done this.

Jacob, we have not determined if we are waiting for Microsoft and up until Gerv's comment above the potential to do so wasn't even mentioned...please don't jump to conclusions.

Have you guys released any information on the topic? Or pushed for mitigation solutions? If not, I'm correct until you guys do so. If so, you're right, I'm mistaken.

Hi Jacob, We are pushing fixes for this as part of the 4.0 launch and, now that we've confirmed our servers won't be DoS'd on launch day (a risk we've encountered during prior launches) we're pushing it live to 3.6 and 3.5. We will absolutely NOT put our users at risk to benefit anyone else's schedule. The planning around out-of-band security releases like this is often kept to a small group, and through no fault of his own, Gerv was accidentally left off that list on this issue, which meant his request about holding off was made without knowledge of the release plan. When we release an out of band fix like this that may have security impact for users, we typically accompany it, same day, with a post on blog.mozilla.org/security, and we have such a post queued for this release as well. It is currently slated to go live at 5pm PDT. This doesn't match Comodo/Microsoft's request, but much as that saddens me, our principle obligation is to our users, and that includes informing them about risks as soon as practical. As you anticipate, the post doesn't include specifics about the certificates involved since that needlessly jeopardizes users of unprotected software. NOW: - The software updates are already being put on the wire, no discussion of blog posts gets in their way. - The blog post itself was slated to go at 5pm PDT, but given the confusion in this bug and your restraint to date, I'm okay to delay it a small amount, if only to give you and others on this bug a chance to react to the comment I'm posting here. It will go live tonight, though. - The blog post is quite matter of fact and describes the issue and the status of supported Firefox versions. It explicitly does not preclude further action on our part, a question we are still very seriously discussing, and are likely to start public discussions on soon. I hope that you will be a part of those discussions. Sorry for the confusion here. Mozilla has to own the mixed messages we've sent and we'll do better in future. But I want to be absolutely clear that at no time was the topic of delaying a fix to our users for the sake of external stakeholders even a matter for discussion. If you want to talk more about this, I've sent you an email privately with my cell phone number.

I'm happy to post my blog post in a few hours rather than waiting until tomorrow morning. I'll give you a ring shortly.

FWIW the updates hit a bit of a snag, we're looking at having them available and announcing a couple hours or so from now (obviously later than 5pm PDT).

(In reply to comment #21) > That's an odd one, I agree. Can the certs be used for code signing? I know that > a friend disabled the CA and lost the ability to validate S/MIME - I suspect > that the issues here run very deep. The issued certs were strictly SSL Server certs. Not code signing, not S/MIME, not certificate signing certs. The CA root itself is trusted to issue S/MIME and object signing certs in addition to SSL certs. Disabling the root disables any leaf certs that chain to it for whatever purpose. (In reply to comment #22) > This is not a normal attack. Disclosure does not allow anyone else to perform > this attack - only the attacker with the certificate is able to take advantage > of this situation. Only the attacker will benefit from a delay. Not necessarily. If the attacker is content with the thought that blocking OCSP/CRL on the network is enough to keep the attack going they may not have thought to block browser updates. If they're smart they already thought about that. If they're breathing they'll catch on if there's a big todo about the issue and how people need to get updates. Fine for the people who got updated before the attacker caught on, sucks for the rest of them. It's anyone's guess whether this is one of the smart attackers or not, or how much attention they're paying, or whether the attack has been deployed somewhere or is just in the planning stages. > End users are not customers. What is being done to protect them right now? > Nothing unless they upgrade. This is really unacceptable as a solution. Right, and we're trying to get those updates out ASAP. Even Microsoft is pushing a _way_ out of cycle update tomorrow and you know how rare that is even when 0-day attacks are in the wild -- we're all taking this quite seriously. Blogging/tweeting about it isn't going to get the updates out any faster. Even if picked up by the tech press isn't going to reach that big an audience compared to the number of users on the internet. It's far more likely, in fact, to be noticed by an attacker paying attention than be noticed by most users. If that attacker is not already blocking updates that would be bad. If they are already blocking updates then it doesn't matter because most people still won't see our message to turn on OCSP. > This kind of attack will happen again and again. Downloading a new binary > isn't going to cut it. What happens when the attacker gets certificates for > the Mozilla update process? The users will then be absolutely screwed > without question. This is painfully obvious to us. We have several things we need to change in future releases. Not changes that can be rushed out in a security fix. (In reply to comment #23) > it looks like someone visiting [AMO] will certainly install code ... > from the attacker who controls addons.mozilla.org. Yes, said that in comment 12 > Is Mozilla really putting all of their users at risk to benefit Comodo and > Microsoft? No, a quick, quiet, update serves our users better than potentially alerting the attacker while only helping a tiny fraction that will hear the message. Or at least that's my belief, and it has no reference to other vendors. I don't believe a delay helps Comodo in any way: the thrashing will commence whether it starts now or tomorrow or next week.

(In reply to comment #29) > (In reply to comment #21) > > That's an odd one, I agree. Can the certs be used for code signing? I know that > > a friend disabled the CA and lost the ability to validate S/MIME - I suspect > > that the issues here run very deep. > > The issued certs were strictly SSL Server certs. Not code signing, not S/MIME, > not certificate signing certs. That's good to know - do we know if the private key was compromised? Or how the certs were issued with such certainty of compromise? > > The CA root itself is trusted to issue S/MIME and object signing certs in > addition to SSL certs. Disabling the root disables any leaf certs that chain > to it for whatever purpose. > Of course. > (In reply to comment #22) > > This is not a normal attack. Disclosure does not allow anyone else to perform > > this attack - only the attacker with the certificate is able to take advantage > > of this situation. Only the attacker will benefit from a delay. > > Not necessarily. If the attacker is content with the thought that blocking > OCSP/CRL on the network is enough to keep the attack going they may not have > thought to block browser updates. If they're smart they already thought about > that. If they're breathing they'll catch on if there's a big todo about the > issue and how people need to get updates. Fine for the people who got updated > before the attacker caught on, sucks for the rest of them. > I don't buy this argument at all. The attacker owned a real live CA and they're not smart enough to block updates? What do you know about this attacker that presents themselves as both a state level adversary and a total, absolutely untalented idiot? :-) > It's anyone's guess whether this is one of the smart attackers or not, or how > much attention they're paying, or whether the attack has been deployed > somewhere or is just in the planning stages. I think the evidence leans in the direction of a skilled and probably aware attacker. > > > End users are not customers. What is being done to protect them right now? > > Nothing unless they upgrade. This is really unacceptable as a solution. > > Right, and we're trying to get those updates out ASAP. Even Microsoft is > pushing a _way_ out of cycle update tomorrow and you know how rare that is even > when 0-day attacks are in the wild -- we're all taking this quite seriously. > I think that you are responding to this specific issue very reasonably. Chrome has already pushed their main update. There's a meta issue here though and I think actually Firefox has majorly dropped the ball here. After our rogue CA attack, Firefox should have taken steps towards fixing their CA policy. After Moxie's attack on OCSP, it should have been a priority for Firefox to create a way to ensure that users would be protected against his technique. After the entire CA model has shown time and time again to be security nightmare, Firefox should have led the way. That's what the internet expects from Mozilla and it's what I expect from Mozilla. You have some incredibly brilliant people and a metric ton of cash on hand. So enough with the half measures. It's time for everyone to work on new directions for trust metrics and we can stop this crazy cycle. We have to stop playing because everyone loses when we stay on this exploit-o-round. > Blogging/tweeting about it isn't going to get the updates out any faster. Even > if picked up by the tech press isn't going to reach that big an audience > compared to the number of users on the internet. It's far more likely, in fact, > to be noticed by an attacker paying attention than be noticed by most users. If > that attacker is not already blocking updates that would be bad. If they are > already blocking updates then it doesn't matter because most people still won't > see our message to turn on OCSP. > This is a seriously difficult issue and I agree that the incentives are all kinds of strange. So I propose a counter question to you - what have you done to ensure that all re-branders, all people who ship a Firefox, all other browsers, are notified of this compromise? The Tor Browser Bundle will be updated because I discovered it. Are other people who use entirely other web browsers so lucky? I think not. This is why telling the world is an important step. This is also the only way that a specifically targeted has any chance at all of hearing that something could be amiss at all. If you have the certificates, Mozilla should publish them. Send them to the EFF observatory mailing list - do something with them so that the internet can make the choices that you're currently not giving them. > > This kind of attack will happen again and again. Downloading a new binary > > isn't going to cut it. What happens when the attacker gets certificates for > > the Mozilla update process? The users will then be absolutely screwed > > without question. > > This is painfully obvious to us. We have several things we need to change in > future releases. Not changes that can be rushed out in a security fix. > So I'm sorry to harp about this as you and I have literally had this discussion for *years* but... I think it's worth clarifying: The future is now. It's time for Mozilla to make a public commitment to this seemingly difficult task. What is the plan for Mozilla? Tor's plan moving forward is with a peer reviewed update system we're calling Thandy: https://gitweb.torproject.org/thandy.git I really endorse that you guys dedicate some serious security engineering time to this, produce a road map and hopefully we can solve it together. Many free software projects are in a similar position and while Thandy solves most of the updating problems, it doesn't solve your specific deployment issues. I also really endorse you read the work by Justin Cappos (and many others) from the University of Washington: https://www.updateframework.com/browser/specs/tuf-spec.txt > (In reply to comment #23) > > it looks like someone visiting [AMO] will certainly install code ... > > from the attacker who controls addons.mozilla.org. > > Yes, said that in comment 12 Ok. I wanted to confirm that the updates server wasn't the only way to get code execution. There is no question now that the attacker has this ability and there is no way for the user to know that this is happening. > > > Is Mozilla really putting all of their users at risk to benefit Comodo and > > Microsoft? > > No, a quick, quiet, update serves our users better than potentially alerting > the attacker while only helping a tiny fraction that will hear the message. Or > at least that's my belief, and it has no reference to other vendors. I don't > believe a delay helps Comodo in any way: the thrashing will commence whether it > starts now or tomorrow or next week. I don't know how we measure this except by the data that I've seen so far. So I'll just have to voice my dissent based on the extremely limited ability of anyone other than the attacker to exploit this and the ability of every person to protect themselves.

(In reply to comment #31) > http://blog.mozilla.com/security/2011/03/22/firefox-blocking-fraudulent-certificates/ Thank you for keeping me in the loop. I'll publish something soon and update this bug with the blog url when I do.

I've posted my previously embargoed blog post here: https://blog.torproject.org/blog/detecting-certificate-authority-compromises-and-web-browser-collusion

Can you guys please make this a public bug?

The issue is now public and discussion should take place in more appropriate fora. We have product changes we'd like to make (OCSP improvements, for example) but those are tracked in separate bugs. I'm resolving this bug because it's hard to extract specific "next actions" that lead to a fix that aren't already covered in other bugs. From comment 0: > I think this is eligible for a bug bounty. Unfortunately it is not: we knew about (and were working to fix) the issue before you reported it.

Mozilla has made a more detailed statement about the Comodo misissuance incident: http://blog.mozilla.com/security/2011/03/25/comodo-certificate-issue-follow-up/ Please continue discussion in the mozilla.dev.security.policy discussion forum: https://www.mozilla.org/about/forums/#dev-security-policy Gerv