Now that https://bugzilla.mozilla.org/show_bug.cgi?id=787133 is complete, Firefox 35 and higher supports public key pinning via the HPKP header. Jake Maul expressed a strong preference for this approach rather than the static pinning that we've had since FF 32. This is a request to collect root CAs used by input.mozilla.org and any subdomains that it uses, and send the HPKP header as described here: http://tools.ietf.org/html/draft-ietf-websec-key-pinning-12 For reference, our static pinning implementation is described here: https://wiki.mozilla.org/SecurityEngineering/Public_Key_Pinning The difference between pinning statically and dynamically is that we (pinning team) would require a 14 week lead time before root CA changes for static pinning. For dynamic pinning, pinning information is sent as an HTTP header that includes things like TTL, and no changes are required to Firefox to deprecate pins.

David, can you help us find an owner for this?

I can add a header to Input HTTP responses, but someone has to tell me what it is we're adding more specifically than a link to a IETF spec. Clearing the needinfo from David figuring I'll be the owner.

(In reply to [:mmc] Monica Chew (please use needinfo) from comment #1) > David, can you help us find an owner for this? I am not sure what you mean by owner, but :willkg is the lead dev for input and will see this bug when he gets back from PTO on Monday.

Or today :)

Hi Will Headers look like so and can accommodate multiple root certs: Public-Key-Pins: max-age=604800; pin-sha256="<base64 encoded sha256 hash of SPKI of root cert 1>"; pin-sha256="<base64 encoded sha256 hash of SPKI of root cert 2>"; You may optionally include "includeSubDomains" if input.mozilla.org serves content from any subdomains. keeler, what's recommended max-age? I think we impose a minimum (like it has to be at least 1 day)

The spec more or less recommends 60 days (5184000 seconds). I think that's probably a reasonable value here.

Hmm, well if the point is to make the header more attractive than static pins (currently at 70 days or so) because it's more flexible, then it seems like they should be able to start with a shorter period. Also we don't have report only (right?) so they won't be able to test things out and recover from mistakes as easily.

I think the ZLB is the SSL endpoint for Input. So I think this header needs to be added at the ZLB. Jake: Does sound right?

(In reply to Will Kahn-Greene [:willkg] from comment #8) > I think the ZLB is the SSL endpoint for Input. So I think this header needs > to be added at the ZLB. > > Jake: Does sound right? Could be added anywhere really (ZLB, Apache, or in the app), as the ZLB will simply maintain any headers that the backend server sends it. But it's probably the most obvious if it's done in the ZLB so it's ephemerally "right next to" the cert being pinned, so I'll go that route. We could probably work this into a sort of "standard rule" that adds other related headers too, like HSTS. That'd make it easily apply-able to other sites. Nice added bonus. (In reply to [:mmc] Monica Chew (please use needinfo) from comment #5) > Hi Will > > Headers look like so and can accommodate multiple root certs: > > Public-Key-Pins: max-age=604800; > pin-sha256="<base64 encoded sha256 hash of SPKI of root cert 1>"; > pin-sha256="<base64 encoded sha256 hash of SPKI of root cert 2>"; > > You may optionally include "includeSubDomains" if input.mozilla.org serves > content from any subdomains. keeler, what's recommended max-age? I think we > impose a minimum (like it has to be at least 1 day) When you say "root cert 1" and "root cert 2", can that be anything in the cert chain? Most CA's sign certs with intermediates, so I'm wondering if I need to pin to that intermediate or if I can pin higher up. Also, with respect to the minimum max-age... I hope Firefox *does not* enforce a minimum. From the IETF spec, in the section on "Deployment Guidance": Operators should start small. Operators should first deploy public key pinning by using the report-only mode together with a report-uri directive that points to a reliable report collection endpoint. When moving out of report-only mode, operators should start by setting a max-age of minutes or a few hours, and gradually increase max-age as they gain confidence in their operational capability. I'm very interested in setting up a report-uri for this, so we can verify if or when things start to go south. If anyone knows of a reference implementation of such a tool, please let me know.

> When you say "root cert 1" and "root cert 2", can that be anything in the > cert chain? Most CA's sign certs with intermediates, so I'm wondering if I > need to pin to that intermediate or if I can pin higher up. We recommend pinning to root certs because they change less frequently, although it can be anything in the cert chain. > Also, with respect to the minimum max-age... I hope Firefox *does not* > enforce a minimum. Looking through https://mxr.mozilla.org/mozilla-central/source/security/manager/boot/src/nsSiteSecurityService.cpp#626 it doesn't seem we do. > I'm very interested in setting up a report-uri for this, so we can verify if > or when things start to go south. If anyone knows of a reference > implementation of such a tool, please let me know. I think this is currently unsupported. We are working on more generic SSL error reporting in https://bugzilla.mozilla.org/show_bug.cgi?id=846489.

I have set up a rule on input-dev.allizom.org, and I can see it in the headers: < Public-Key-Pins: max-age=30; pin-sha256="r/mIkG3eEpVdm+u/ko/cwxzOMo1bk4TyHIlByibiA5E="; pin-sha256="WoiWRyIOVNa9ihaBciRSC7XHjliYS9VwUGOIud4PB18="; Those fingerprints correspond to the DigiCert Global Root CA and DigiCert High Assurance EV Root CA (standard and EV certs, respectively), and the non-EV one should match already (for dev). What's the best way to verify that things are working properly? I can see the header with curl, but Firefox doesn't seem to generate any console output indicating that validation has happened. I could muck up the fingerprints and look for an error (which should be pretty safe given the 30s max-age)... is there a better way?

You could visit the site (thus observing the header) and then MITM yourself with ZAP or something ( https://www.owasp.org/index.php/OWASP_Zed_Attack_Proxy_Project ). The expected error in that case would be MOZILLA_PKIX_ERROR_KEY_PINNING_FAILURE.

Hmm... I munged the pin-sha256 fields in the headers, and the site still works. Running current Aurora (35.0a2). Is there a switch to enable (in about:config or elsewhere) to make it do something?

One of the features of HPKP is that if the given pinset wouldn't have succeeded for the given connection, it won't use that information. This prevents one way of shooting yourself in the foot. (So, you can't say pin-sha256="somebadpin" and then have future connections fail.) Also, you need to specify one hash that isn't in the verified certificate chain. This is supposed to be a backup key you can use in case all other certs are compromised or something.

I don't understand. If none of the pinsets succeed... it just displays the page normally? Then what's the point?

That's just on the initial visit. Once pinning information has been received and validated, on subsequent visits if none of the certificates presented match the stored pinning information, the load won't continue.

Ah, okay, that makes more sense. Sadly it still doesn't seem to work for me. I set input-dev.allizom.org to have a 300s max-age with the right pins, and visited it (in Aurora 35.0a2). Then I set it to have bad pins and revisited... still loads without issue. Is there any way to get Firefox to spit out information on pin validation? Console or log messages?

NSPR_LOG_MODULES=PublicKeyPinningService:5 might give some helpful console logging. Try also certverifier:5. Keep in mind what's being verified on a new connection is the stored pinning information vs. the certificate chain sent by the server.

(In reply to Jake Maul [:jakem] from comment #17) > Ah, okay, that makes more sense. > > Sadly it still doesn't seem to work for me. I set input-dev.allizom.org to > have a 300s max-age with the right pins, and visited it (in Aurora 35.0a2). > Then I set it to have bad pins and revisited... still loads without issue. > > Is there any way to get Firefox to spit out information on pin validation? > Console or log messages? Hi Jake, I don't think I understand your test. You can't ever trigger a pinning failure by setting HPKP headers. Those are only for advertising a pin, and won't be stored unless the connection they're received on passes the check. You have to MITM the connection with a bad cert. Thanks, Monica

I had a chat with Jake/WebOps about this during our weekly meeting. We need better tooling to continue testing and deployment of HPKP. It would be great to have a pining validation status in the developer tools. A command line tool that we can run as a nagios check would be useful as well. Anything that can facilitate deployment and compliance checking is most welcomed. Is this something :mmc or :keeler can provide?

Hi Julien and Jake, Thanks for chatting about this. The only user visible feedback for pinning is an ssl error, e.g. visiting https://pinningtest.appspot.com shows the warning. Otherwise, the pinning checks pass without error. Is that sufficient? If you mean that it's necessary to be able to inspect stored pins parsed from HPKP, I don't think there is currently a way but keeler may know different. Thanks, Monica

The visual test is good, but it only indicates failure. We need a way to indicate success such that operators can verify that HPKP is set correctly and validated. The large number of websites that we operate also makes manual validation inefficient. A command line way to validate a pining would greatly help adoption.

nsISiteSecurityService.isSecureHost (or isSecureURI) with HEADER_HPKP as the header type should tell you if the HPKP headers were successfully parsed/applied. You could write an xpcshell script to make an xhr to the host you want to test and then call isSecureHost.

I understand that it is possible to obtain the information through complex NSS engineering, but if our goal is broad adoption of HPKP, we really need better/easier operational tools, and clear deployment guidelines on the wiki: https://wiki.mozilla.org/Security/Server_Side_TLS#HPKP:_Public_Key_Pinning_Extension_for_HTTP Ideally, we could give ops teams a separate program (a python script? a standalone nss client?) that runs locally, outside of Firefox, and displays detailed information about pins. It should store the pining on disk, and verify known pins on subsequent runs. SecEng is the expert when it comes to HPKP. Is that something you can provide us with?

A compromise might be to expose the pinning status in the web console as Julien suggests in comment 20. Another possibility is modifying https://github.com/mozilla/pkix-compatibility-testing to verify HPKP. I don't think that seceng can own both the implementation in product and all of the toolchain for developers, too.

Exposing the pining status in the web console would be a great help! I am happy to help with the toolchain and documentation for dev & ops. In the future, we could capture and validate HPKP (and HSTS) in the observatory, because it will maintain state and be able to detect configuration changes over time.

This looks very much like something that would meet our needs: https://github.com/craigfrancis/dev-security

bug 932179 landed changes to the network console that now expose HSTS and PKP. (see screenshot) That's great news, and should provide the info we need to deploy these headers more easily.

(In reply to Jake Maul [:jakem] from comment #11) > I have set up a rule on input-dev.allizom.org, and I can see it in the > headers: > > < Public-Key-Pins: max-age=30; > pin-sha256="r/mIkG3eEpVdm+u/ko/cwxzOMo1bk4TyHIlByibiA5E="; > pin-sha256="WoiWRyIOVNa9ihaBciRSC7XHjliYS9VwUGOIud4PB18="; > > Those fingerprints correspond to the DigiCert Global Root CA and DigiCert > High Assurance EV Root CA (standard and EV certs, respectively), and the > non-EV one should match already (for dev). > > What's the best way to verify that things are working properly? I can see > the header with curl, but Firefox doesn't seem to generate any console > output indicating that validation has happened. > > I could muck up the fingerprints and look for an error (which should be > pretty safe given the 30s max-age)... is there a better way? :jakem - now that FF37 exposes the pining in the network->security dev console, would it be possible to retry setting the header on input.allizom.org? Or do you need more debug info to move forward with HPKP?

Great! this is working now on input.allizom.org. If I set an invalid pin, I get an error in the console. If I set a valid pin, I get nothing in the console... which kinda makes sense to me (no errors or warnings to report, which are the only two categories that exist in the "Security" type). Would anyone else like to eyeball this to see if things look reasonable? The two certificates pinned are: # Digicert Global Root CA # Digicert High Assurance EV Root CA The max-age is currently only 5 minutes, for testing purposes I didn't want to potentially break things for a long time. :) Things that are not included are: Digicert's SHA2 Root CA (I forget the exact CN right now) Mozilla's internal Root CA Any other public CA's I do have one question, and it's possible this has been handled already, but... what's the proper way to go about changing the CA used for a certificate, once this pinning is in place? For instance, if we wanted to change to another vendor for some reason, what's the right way to make the changeover?

Updating a pining is trivial: just update the HPKP header with the new value, serve it on a site that respects the old/current pining, and the browser will update the pining right away. Moving to a new vendor is not a concern as long as we have the key/cert from the old vendor to update the pining.

Welllll...... trivial feels like an understatement to me. It's trivial to *do*, but you can't "just do it". If I understand you correctly, this means we'd have to: 1) make the change to the HPKP header- adding the new pins (without removing the old ones, yet) 2) wait until the max-age of the pins expires before we can put the change in place 3) change the cert 4) remove the no-longer-useful original pins If we don't wait sufficiently long enough in step 2, infrequent visitors could still have a valid pin cached and would reject the new cert. So we need to wait long enough that either a) all such users have visited and gotten the new pins, or b) we're certain that browsers would ignore the cached pin anyway. To me this just means "wait out the TTL on the pin". Section 4.1 of the IETF spec seems relevant here: http://tools.ietf.org/html/draft-ietf-websec-key-pinning-12#section-4.1: There is probably no ideal upper limit to the max-age directive that would satisfy all use cases. However, a value on the order of 60 days (5184000 seconds) may be considered a balance between the two competing security concerns. Does this max-age (60 days) seem reasonable for us?

You are correct that infrequent users would be blocked if we replace the pining too fast. Maybe the webdev team can help us understand, via google analytics, how frequently people visit our sites and what a proper max-age is? I'd even start with a value lower than 60 days. 15 days is probably good enough as we test this. MITM attacks seem to be very targeted in time and space, so that should provide sufficient security for now.

I'm fine with 15 days as a starter. This is configured on input.mozilla.org. I'm going to mark this bug as resolved... changes to the max-age can be handled later on as we gain confidence and in-house knowledge with/about HPKP. :)