User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:71.0) Gecko/20100101 Firefox/71.0

Steps to reproduce:

FF 68.3.0 ESR returns a "MOZILLA_PKIX_ERROR_KEY_PINNING_FAILURE" for a website. However, the website turns out to be perfect. Other browsers (Safari and Chrome) load the website without a problem. All tests by https://observatory.mozilla.org, including all of its third party tests, return spectacularly good results.

Actual results:

MOZILLA_PKIX_ERROR_KEY_PINNING_FAILURE

Expected results:

If FF thinks there is a problem, it should return its supporting evidence. The error page should be humble enough to do this at the user's request.

what's the website in question?

Where is the supporting evidence I am asking for? The problem is not with the website. All relevant key pinning tests by third parties are positive in saying that the website has no problem whatsoever. Allow me to see why FF is returning an error, and I will send you the output.

These are the relevant settings on both FF ESR and FF-based tor browser: default

On tor-browser the website shows up.
On FF ESR it does not.

The website is public.

Other browsers tested and successful:
Safari
Chrome

All latest versions.

If I set security.cert_pinning.enforcement_level to 0, then FF ESR works.
I need to see the pinning mismatch that FF ESR believes to have found.

SiteSecurityServiceState.txt in your profile will contain stored HPKP records

(In reply to ruga from comment #3)

Other browsers tested and successful:
Safari
Chrome

All latest versions.

Because they don't support HPKP.

Re: on HPKP on Chrome

The claim that Chrome does not support HPKP is false.
Evidence: Open Chrome and edit the following in its search bar: "chrome://net-internals/#hsts".

I closed FF, re-set "security.cert_pinning.enforcement_level " to "2", and cleared the file SiteSecurityServiceState.txt from its contents, then re-opened FF and the problem did not manifest itself: FF renders the website without any error. The file SiteSecurityServiceState.txt was automatically re-populated, and the website appears in its listing. However, there seems to be something wrong. The file's columns, if I read them correctly, are as follows:

FQDN:[HSTS|HPKP] hits most_recent_access_day expiry_timestamp,HSTS_state,include_subdomains

where HSTS_state is
0: "unset"
1: "set"
2:: "knockout" / overrides HSTS preload information as "no HSTS information available"

and include_subdomains is either 0 or 1.

Ref: https://dxr.mozilla.org/mozilla-central/source/security/manager/ssl/nsSiteSecurityService.cpp#50

For the website in question, these are its entries:

example.com:HSTS 21 18250 1608370193362,1,1,2
example.com:HPKP 27 18250 1582018193368,1,1,ucF9xR6BJ/RH9t421PduWAIp7balG2u59OiQ9G9bJEw=L2aLiZHjPaHbCwwB414q/0NwfJLdi+95dm8ltY6uRWg=YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg=Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys=

This is the pinning policy, straight from the website:

add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
add_header Public-Key-Pins 'max-age=31536000; includeSubDomains; pin-sha256="ucF9xR6BJ/RH9t421PduWAIp7balG2u59OiQ9G9bJEw="; pin-sha256="L2aLiZHjPaHbCwwB414q/0NwfJLdi+95dm8ltY6uRWg="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="' always;

There is a matching.

Why did I have to clear the file SiteSecurityServiceState.txt to solve this problem?

Why nobody here mentioned this, and someone went as far as closing this bug by referencing it to another bug whose "solution" consists in the claim that FF should remove HSTS altogether?

HSTS is a good thing to have. It was born to solve a problem that still exists. So, if you remove HSTS, the old problem re-appears.

My standpoint here is that FF should keep HSTS/HPKP, but also investigate into why SiteSecurityServiceState.txt had to be cleared by hand in order to solve the problem I originally described in this bug report.

Problem solved.

calc 31536000/60/60/24
365

The website updated its certificates, changing CA. The file SiteSecurityServiceState.txt seems to act like a cache, had the old pins in place, did not update itself with the new ones given by the website's header.

Ref. https://www.fxsitecompat.dev/en-CA/docs/2019/http-public-key-pinning-is-no-longer-supported/

"Firefox 72 has dropped the support for HTTP Public Key Pinning (HPKP) because of the low adoption rate and interoperability risk."

Bad decision.

This is comparable to saying "The state has dropped the support for police because of the high crime rate."

Just educate webmasters and users, instead of taking away sound solutions to hard problems.

(In reply to ruga from comment #7)

Re: on HPKP on Chrome

The claim that Chrome does not support HPKP is false.
Evidence: Open Chrome and edit the following in its search bar: "chrome://net-internals/#hsts".

I saw "HSTS/PKP", not "HPKP". Chrome only supports static PKP that does not see HTTP headers at all. Google announced that they dropped support for HPKP since Chrome 72:
https://www.chromestatus.com/feature/5903385005916160

(In reply to ruga from comment #9)

Problem solved.

calc 31536000/60/60/24
365

The website updated its certificates, changing CA. The file SiteSecurityServiceState.txt seems to act like a cache, had the old pins in place, did not update itself with the new ones given by the website's header.

If Firefox updates the HPKP cache, attacker can take over the website, replace certificate, and overwrite the cache with the new header. It will defeat the sole purpose of HPKP.

When Chrome loads a broken site while Firefox does not, people tend to blame Firefox just like you are exactly doing. So Mozilla had to disable HPKP support.

Re: on HPKP on Chrome

https://www.chromestatus.com/feature/5903385005916160
One more reason to keep using FF ESR.

Re: on FF not updating its cache

If Firefox updates the HPKP cache, attacker can take over the website, replace certificate, and overwrite the cache with the new header. It will defeat the sole purpose of HPKP. --- When Chrome loads a broken site while Firefox does not, people tend to blame Firefox just like you are exactly doing. So Mozilla had to disable HPKP support.

Disabling HPKP support does not solve the problem, it just makes it easier to the attacker.

From my seat, a solution to all this in FF is to implement the verification of DANE/TLSA, where certificates are pinned into the DNS and signed by DNSSEC. I have it, but FF does not seem to know about its existence.