In bug 1138332 comment 10, Dick Riegner reported failure to override bad certificates with Firefox 36. Based on the screenshots that were provided, the error code reported is sec_error_unknown_issuer. See attachment 8573578 [details], attachment 8573580 [details], attachment 8573582 [details], and the comments from Dick in bug 1138332

Dick, your issue seems to be different in several ways. We must analyze what's happening for you. In bug 1138332, people weren't even able to add an override. The bug in that scenario was: The error page shown for some sites displayed an error message, but didn't over to override. Based on your screenshots, you were given the option to override. However, as you say, adding an override doesn't help with Firefox 36, you're still not able to access the host. The other unusual (different from the bugs I had seen) detail of your scenario is: In the past, you weren't able to add permanent overrides. Regarding the inability to override in the past: I wonder if your device generates a different certificate every time. The overrides in Firefox are bound to the combination of {hostname, port, certificiate, error produced}. If anything changed on the next execution, the remembered override is ignored. If you had connected with the same browser to the same host again, then the error would usually be the same (unless it had expired in the meantime). Thus, the only way to differ is if the server presents a different certificate. With Firefox 36, as you explained in the other bug, you are able to override, but then your connections are stuck, and Firefox doesn't show the web page. This means, Firefox is able to initiate the connection to the server, and is able to complete the initial part of the handshake, in which the certificate is sent by the server (which is deemed untrusted, und thus you are asked to override). Once you had added the override, Firefox will pass beyond that step, and attempt to establish a connection to your server (which is the Cisco 5505 VPN hardware box you mentioned). If Firefox "hangs", it seems that Firefox is expecting the server to send more data, but it isn't. If the server were simply disconnecting, you'd a different error page. So it seems, the server is unable to understand the communication with Firefox, and falsely assumes that it needs to wait for additional data, and with both sides waiting, you see that it hangs and never loads. (That's my guess.) If this problem, Firefox hangs on that site, was first seen with Firefox 36, then we should investigate which SSL/TLS options have changed between Firefox 35 and Firefox 36. I really wish I was able to connect to your system myself. If your device can be reached from the public internet, then it would be very helpful if you could enable me to connect to it myself, for purpose of diagnosing the server's beahviour.

Kai, Thanks for all your help with this problem. I cannot think of a way to give you access to my VPN box. Firefox is connecting to it through one of the VPN box's private network ports, not through the public network port. My top priority is getting Firefox back to its release 35 and previous behavior. It would then be nice to fix Firefox so that it could permanently store the VPN box's certificate. If storing the certificate is not feasible that is OK, I have lived with that limitation for years now. Let me know how you think we should proceed. I will let you know if I can think of a way to give you access to the private side of the VPN box. Dick

Has anyone taken a look at my problem?

Without more details, we can't.

What details do you need? I can gather whatever documentation you need.

I need to be able to connect to the service. If you cannot allow me to do that, someone with knowledge of how SSL/TLS works, and is able to use the tools, must run tests against your service, and tell us which versions work, and which version don't, and what the behaviour of those versions is. Preferably executed using command line tools like "ssltap -s", that can dump details of the handshake, which can give us a clue which detail is responsible for the failure.

The Cisco 5505 VPN hardware box is directly connected to my iMac over ethernet. I am running Mac OS X 10.8.5 and can run any tool that is available for Mac OS X. As an option, can you spin-up a debug version of Firefox that can dump the needed details? Since this is a regression from Firefox 35, any chance you can find the mod that caused this problem? Thanks.

A debug version of firefox isn't easy to distribute and wouldn't give us the necessary information in a logfile. That won't help. It would help if you can compile NSS and run the diagnostic utilities on the command line. Create a new local directory. Download https://ftp.mozilla.org/pub/mozilla.org/security/nss/releases/NSS_3_18_RTM/src/nss-3.18-with-nspr-4.10.8.tar.gz and place it into that directory. Open a terminal and use "cd" to go into that directory. Use the following commands to build: tar xzf nss-3.18-with-nspr-4.10.8.tar.gz cd nss-3.18 cd nss export USE_64=1 make nss_build_all cd .. Now execute the following commands to prepare the environment to run the NSS tools that you just built: OBJDIR=`cd nss/tests/common/ && make objdir_name` export PATH=$PWD/dist/$OBJDIR/bin:$PATH export DYLD_LIBRARY_PATH=$PWD/dist/$OBJDIR/lib:$DYLD_LIBRARY_PATH export ROOTS=$PWD/dist/$OBJDIR/lib/libnssckbi.dylib Now, if you type the name of a NSS tool, it should print it's help output, e.g. type tstclnt If that works, you can test connections to your 5505 box. Get the IP address and set it as a variable, so you can copy paste the following commands. export IP=1.2.3.4 (use the IP of your cicso 5505) I assume you connect to the regular https port, 443. Now run each of the following tests. Open a text editor where you can paste text into. For each of the tests you have executed, copy/paste the output that you see on the terminal, INCLUDING the respective command in front of the output, into your editor. tstclnt -D -p 443 -h $IP -R $ROOTS -V tls1.2:tls1.2 -C tstclnt -D -p 443 -h $IP -R $ROOTS -V tls1.1:tls1.1 -C tstclnt -D -p 443 -h $IP -R $ROOTS -V tls1.0:tls1.0 -C tstclnt -D -p 443 -h $IP -R $ROOTS -V ssl3:ssl3 -C tstclnt -D -p 443 -h $IP -R $ROOTS -V ssl2:ssl2 -C Please provide output of all 5 commands. At least one of the commands should print some diagnostic information. Then, run another command that will print much more information about the server's certificate, and after the -V use the respective parameter from above, which didn't result in an error, but printed more information. tstclnt -D -p 443 -h $IP -R $ROOTS -V replace-me -CCC Again, copy paste the results, including the command, into the editor. Send all that information by email to me. Leave the terminal window open, in case I need you to run additional commands later.

You should copy/paste the commands to the terminal, don't try to type them (except the very last command when you have to replace), because the commands contain some special characters, which are easy to get wrong, if you're aren't used to them.

After each time you run the tstclnt utility, after it printed some output, and if the connection worked, it will still run and wait for input. So, after each tstclnt command is done, you will have to press CTRL-C to terminate tstclnt.

Can I build NSS on a default Mac OS X 10.8.5 install or do I need to install some build tools? I have no special build tools installed. Is there an NSS binary already built that I can download? I won't be able to leave the window open. Once I am done with my VPN session into the private corporate network for the day, I revert my ethernet connection to the public network. My company's VPN blocks too many sites it deems unnecessary for business use. Also, once I have completed my login to the VPN box, I think that session is terminated. I usually then access our company's home page. For what it's worth, there may be ways to access the VPN box's certificate and security options through other pages that I currently don't access to. I should be able to get complete access to the VPN box administrator pages if needed.

Type gcc into a terminal. Is a program found? If not, you probably must install "Xcode", which is free. No, we don't usually provide binaries of NSS. Keeping the terminal was simply a convenience suggestion, no problem to open it later again, simply cd into the directory you made earlier, and again run these commands: OBJDIR=`cd nss/tests/common/ && make objdir_name` export PATH=$PWD/dist/$OBJDIR/bin:$PATH export DYLD_LIBRARY_PATH=$PWD/dist/$OBJDIR/lib:$DYLD_LIBRARY_PATH export ROOTS=$PWD/dist/$OBJDIR/lib/libnssckbi.dylib Then you can execute test tools again.

Any idea where I can get an NSS binary? I am willing to put some effort into helping debug this Firefox regression, but installing an entire devleopment environment and building NSS is getting a bit much. Migrating to Chrome to interface with my VPN box looks to be a very practical alternative. Thanks for your effort.

You earlier said you're willing to execute any tool on your machine to diagnose, and I've spent at least 30 minutes to write up and test the steps on a Mac for you, it's disappointing that you're no longer willing to analyze.

Firefox is a community project, and can only work if people contribute a little piece of work to make it better. This is your chance to help.

I only have access to a 32 bit Mac mini with OSX 10.6 I don't know if the binaries I built will work on your potentially newer 64 bit Mac with OSX 10.8

I am willing to help with the tools and environment I have at hand. At this time, I am unwilling to install a complete Mac OS X development environment just to build one tool. If I can find an NSS binary, I will gather the documentation you requested.

I suspect that a 32-bit binary will run fine on my 64-bit Mac OS X 10.8.5 machine. How can I get the binary?

I've uploaded binaries to https://kuix.de/mozilla/nss/nss-3.18-darwin-32bit.zip I cannot give any guarantees whatsoever, but I think running them should be safe. If you're worried these binaries might destroy any of your data, you could create a separate user account on your machine for running them. Save the file to your disk, e.g. to the "Downloads" folder. Open a terminal, "cd" into the directory where you stored the zip file, e.g. cd $HOME/Downloads and run unzip nss-3.18-darwin-32bit.zip cd nss-3.18-darwin-32bit The commands to prepare your terminal to run these NSS tools is different from what I said earlier. Execute these commands in the terminal: export PATH=$PWD/bin:$PATH export DYLD_LIBRARY_PATH=$PWD/lib:$DYLD_LIBRARY_PATH export ROOTS=$PWD/lib/libnssckbi.dylib Now, go to comment 8 above, please follow the instructions from "Now, if you type the name of a NSS tool" Let's cross fingers that the 32 bit binaries will work for you.

Ok, I downloaded and installed nss; it seems to run fine. Firefox attempts to connect to this IP and port on the VPN box, not 443: https://134.15.84.97:1443/netaccess/connstatus.html How should I set the IP variable?

export IP=134.15.84.97 and don't use -p 443, use: -p 1443

1) I ran these various tstclnt commands before attempting to connect to the VPN box: Riegner-iMac3:nss-3.18-darwin-32bit dick$ tstclnt -D -p 1443 -h $IP -R $ROOTS -V tls1.2:tls1.2 -C tstclnt: read from socket failed: SSL_ERROR_UNSUPPORTED_VERSION: Peer using unsupported version of security protocol. Riegner-iMac3:nss-3.18-darwin-32bit dick$ tstclnt -D -p 1443 -h $IP -R $ROOTS -V tls1.1:tls1.1 -C tstclnt: read from socket failed: SSL_ERROR_UNSUPPORTED_VERSION: Peer using unsupported version of security protocol. Riegner-iMac3:nss-3.18-darwin-32bit dick$ tstclnt -D -p 1443 -h $IP -R $ROOTS -V tls1.0:tls1.0 -C ==== certificate(s) sent by server: ==== Certificate: Data: Serial Number: -316472491 (0xed230355) Signature Algorithm: PKCS #1 SHA-1 With RSA Encryption Issuer: "CN=134.15.84.97" Validity: Not Before: Fri Mar 13 17:52:45 2015 Not After : Mon Mar 10 17:52:45 2025 Subject: "CN=134.15.84.97" Fingerprint (SHA-256): FC:B2:2B:B9:01:1B:B6:E1:65:DC:D0:B0:E7:72:11:6E:40:A8:B9:71:20:6A:37:8C:01:4E:CA:EC:4F:D0:41:D4 Fingerprint (SHA1): 66:88:7C:DB:C0:0D:B1:2F:BE:25:FB:F3:04:B9:2A:B7:63:C6:CE:DB ==== locally found issuer certificate(s): ==== ==== end of certificate chain information ==== tstclnt: authentication of server cert failed: SEC_ERROR_CA_CERT_INVALID: Issuer certificate is invalid. Riegner-iMac3:nss-3.18-darwin-32bit dick$ tstclnt -D -p 1443 -h $IP -R $ROOTS -V ssl3:ssl3 -C ==== certificate(s) sent by server: ==== Certificate: Data: Serial Number: -316472491 (0xed230355) Signature Algorithm: PKCS #1 SHA-1 With RSA Encryption Issuer: "CN=134.15.84.97" Validity: Not Before: Fri Mar 13 17:52:45 2015 Not After : Mon Mar 10 17:52:45 2025 Subject: "CN=134.15.84.97" Fingerprint (SHA-256): FC:B2:2B:B9:01:1B:B6:E1:65:DC:D0:B0:E7:72:11:6E:40:A8:B9:71:20:6A:37:8C:01:4E:CA:EC:4F:D0:41:D4 Fingerprint (SHA1): 66:88:7C:DB:C0:0D:B1:2F:BE:25:FB:F3:04:B9:2A:B7:63:C6:CE:DB ==== locally found issuer certificate(s): ==== ==== end of certificate chain information ==== tstclnt: authentication of server cert failed: SEC_ERROR_CA_CERT_INVALID: Issuer certificate is invalid. Riegner-iMac3:nss-3.18-darwin-32bit dick$ tstclnt -D -p 1443 -h $IP -R $ROOTS -V ssl2:ssl2 -C tstclnt: error setting SSL/TLS version range : SSL_ERROR_INVALID_VERSION_RANGE: SSL version range is not valid. 2) I then attempted to connect to the VPN box using Firefox 36 and got this: Riegner-iMac3:nss-3.18-darwin-32bit dick$ tstclnt -D -p 1443 -h $IP -R $ROOTS -V tls1.0:tls1.0 -C ==== certificate(s) sent by server: ==== Certificate: Data: Serial Number: -316472491 (0xed230355) Signature Algorithm: PKCS #1 SHA-1 With RSA Encryption Issuer: "CN=134.15.84.97" Validity: Not Before: Fri Mar 13 17:52:45 2015 Not After : Mon Mar 10 17:52:45 2025 Subject: "CN=134.15.84.97" Fingerprint (SHA-256): FC:B2:2B:B9:01:1B:B6:E1:65:DC:D0:B0:E7:72:11:6E:40:A8:B9:71:20:6A:37:8C:01:4E:CA:EC:4F:D0:41:D4 Fingerprint (SHA1): 66:88:7C:DB:C0:0D:B1:2F:BE:25:FB:F3:04:B9:2A:B7:63:C6:CE:DB ==== locally found issuer certificate(s): ==== ==== end of certificate chain information ==== tstclnt: authentication of server cert failed: SEC_ERROR_CA_CERT_INVALID: Issuer certificate is invalid. Riegner-iMac3:nss-3.18-darwin-32bit dick$ tstclnt -D -p 1443 -h $IP -R $ROOTS -V ssl3:ssl3 -C ==== certificate(s) sent by server: ==== Certificate: Data: Serial Number: -316472491 (0xed230355) Signature Algorithm: PKCS #1 SHA-1 With RSA Encryption Issuer: "CN=134.15.84.97" Validity: Not Before: Fri Mar 13 17:52:45 2015 Not After : Mon Mar 10 17:52:45 2025 Subject: "CN=134.15.84.97" Fingerprint (SHA-256): FC:B2:2B:B9:01:1B:B6:E1:65:DC:D0:B0:E7:72:11:6E:40:A8:B9:71:20:6A:37:8C:01:4E:CA:EC:4F:D0:41:D4 Fingerprint (SHA1): 66:88:7C:DB:C0:0D:B1:2F:BE:25:FB:F3:04:B9:2A:B7:63:C6:CE:DB ==== locally found issuer certificate(s): ==== ==== end of certificate chain information ==== tstclnt: authentication of server cert failed: SEC_ERROR_CA_CERT_INVALID: Issuer certificate is invalid. 3) I will now run the verbose version of the previous two commands.

4) I switched back to the VPN network, tried to connect to the VPN box using Firefox 36, and ran these commands: Riegner-iMac3:nss-3.18-darwin-32bit dick$ tstclnt -D -p 1443 -h $IP -R $ROOTS -V tls1.0:tls1.0 -CCC ==== certificate(s) sent by server: ==== Certificate: Data: Version: 3 (0x2) Serial Number: -316472491 (0xed230355) Signature Algorithm: PKCS #1 SHA-1 With RSA Encryption Issuer: "CN=134.15.84.97" Validity: Not Before: Fri Mar 13 17:52:45 2015 Not After : Mon Mar 10 17:52:45 2025 Subject: "CN=134.15.84.97" Subject Public Key Info: Public Key Algorithm: PKCS #1 RSA Encryption RSA Public Key: Modulus:c9:88:bf:42:de:f8:78:42:b8:e7:b7:d7:4f:1b:77:16:a0:c7:de:2b:29:e5:95:43:ff:df:4c:5a:23:76:ee:43:68:9b:20:4d:62:17:73:c7:63:fb:1c:a3:9d:88:07:29:24:c8:33:9a:f1:36:0e:7f:87:0d:de:b7:a5:b5:ea:61:8a:9f:69:68:14:01:78:56:9c:71:c4:af:5a:e6:44:23:1b:ef:5f:53:88:f9:8f:54:07:ee:42:42:4b:92:88:17:51:df:b5:37:bb:fb:0e:3a:51:ce:a8:b2:7e:c3:e8:7a:78:61:78:e3:29:1c:00:cb:a9:86:38:b4:7a:10:5a:23 Exponent: 65537 (0x10001) Fingerprint (SHA-256): FC:B2:2B:B9:01:1B:B6:E1:65:DC:D0:B0:E7:72:11:6E:40:A8:B9:71:20:6A:37:8C:01:4E:CA:EC:4F:D0:41:D4 Fingerprint (SHA1): 66:88:7C:DB:C0:0D:B1:2F:BE:25:FB:F3:04:B9:2A:B7:63:C6:CE:DB -----BEGIN CERTIFICATE----- MIIBpTCCAQ6gAwIBAgIE7SMDVTANBgkqhkiG9w0BAQUFADAXMRUwEwYDVQQDEwwx MzQuMTUuODQuOTcwHhcNMTUwMzEzMTc1MjQ1WhcNMjUwMzEwMTc1MjQ1WjAXMRUw EwYDVQQDEwwxMzQuMTUuODQuOTcwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGB AMmIv0Le+HhCuOe3108bdxagx94rKeWVQ//fTFojdu5DaJsgTWIXc8dj+xyjnYgH KSTIM5rxNg5/hw3et6W16mGKn2loFAF4VpxxxK9a5kQjG+9fU4j5j1QH7kJCS5KI F1HftTe7+w46Uc6osn7D6Hp4YXjjKRwAy6mGOLR6EFojAgMBAAEwDQYJKoZIhvcN AQEFBQADgYEAourbskHcBlTMFcNgzjS++So8qd4PFt4lidGfIj/HpT1x7IvG18Dp cE9kLn4zHkEk2cUcx7uhxlL0Vr55OnmxTLEeE8voPxbnYpXz7mcplUe+6Nc20Xnn 4iv6ikrQQtPQM/kyffckMVxtC5qlFFHSesOH+uRmpbn5HPNhsBgwI+Q= -----END CERTIFICATE----- ==== locally found issuer certificate(s): ==== ==== end of certificate chain information ==== tstclnt: authentication of server cert failed: SEC_ERROR_CA_CERT_INVALID: Issuer certificate is invalid. Riegner-iMac3:nss-3.18-darwin-32bit dick$ tstclnt -D -p 1443 -h $IP -R $ROOTS -V ssl3:ssl3 -CCC ==== certificate(s) sent by server: ==== Certificate: Data: Version: 3 (0x2) Serial Number: -316472491 (0xed230355) Signature Algorithm: PKCS #1 SHA-1 With RSA Encryption Issuer: "CN=134.15.84.97" Validity: Not Before: Fri Mar 13 17:52:45 2015 Not After : Mon Mar 10 17:52:45 2025 Subject: "CN=134.15.84.97" Subject Public Key Info: Public Key Algorithm: PKCS #1 RSA Encryption RSA Public Key: Modulus:c9:88:bf:42:de:f8:78:42:b8:e7:b7:d7:4f:1b:77:16:a0:c7:de:2b:29:e5:95:43:ff:df:4c:5a:23:76:ee:43:68:9b:20:4d:62:17:73:c7:63:fb:1c:a3:9d:88:07:29:24:c8:33:9a:f1:36:0e:7f:87:0d:de:b7:a5:b5:ea:61:8a:9f:69:68:14:01:78:56:9c:71:c4:af:5a:e6:44:23:1b:ef:5f:53:88:f9:8f:54:07:ee:42:42:4b:92:88:17:51:df:b5:37:bb:fb:0e:3a:51:ce:a8:b2:7e:c3:e8:7a:78:61:78:e3:29:1c:00:cb:a9:86:38:b4:7a:10:5a:23 Exponent: 65537 (0x10001) Fingerprint (SHA-256): FC:B2:2B:B9:01:1B:B6:E1:65:DC:D0:B0:E7:72:11:6E:40:A8:B9:71:20:6A:37:8C:01:4E:CA:EC:4F:D0:41:D4 Fingerprint (SHA1): 66:88:7C:DB:C0:0D:B1:2F:BE:25:FB:F3:04:B9:2A:B7:63:C6:CE:DB -----BEGIN CERTIFICATE----- MIIBpTCCAQ6gAwIBAgIE7SMDVTANBgkqhkiG9w0BAQUFADAXMRUwEwYDVQQDEwwx MzQuMTUuODQuOTcwHhcNMTUwMzEzMTc1MjQ1WhcNMjUwMzEwMTc1MjQ1WjAXMRUw EwYDVQQDEwwxMzQuMTUuODQuOTcwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGB AMmIv0Le+HhCuOe3108bdxagx94rKeWVQ//fTFojdu5DaJsgTWIXc8dj+xyjnYgH KSTIM5rxNg5/hw3et6W16mGKn2loFAF4VpxxxK9a5kQjG+9fU4j5j1QH7kJCS5KI F1HftTe7+w46Uc6osn7D6Hp4YXjjKRwAy6mGOLR6EFojAgMBAAEwDQYJKoZIhvcN AQEFBQADgYEAourbskHcBlTMFcNgzjS++So8qd4PFt4lidGfIj/HpT1x7IvG18Dp cE9kLn4zHkEk2cUcx7uhxlL0Vr55OnmxTLEeE8voPxbnYpXz7mcplUe+6Nc20Xnn 4iv6ikrQQtPQM/kyffckMVxtC5qlFFHSesOH+uRmpbn5HPNhsBgwI+Q= -----END CERTIFICATE----- ==== locally found issuer certificate(s): ==== ==== end of certificate chain information ==== tstclnt: authentication of server cert failed: SEC_ERROR_CA_CERT_INVALID: Issuer certificate is invalid.

Dick, thanks for the output. FYI, for the tools it shouldn't matter whether you have used Firefox previously or not. The output of (1) and (2) is identical. We have learned that your server doesn't support the most modern TLS 1.2 and TLS 1.1 protocols. Firefox 36 will by default try to run TLS 1.2, and might attempt to fall back to using older protocols. I wonder if something is wrong with that mechanism of falling back. You could try to *temporarily* disable TLS 1.2 and TLS 1.1 in Firefox 36. Go to about:config In the search box, type: tls Locate the line with security.tls.version.max This is equivalent to the highest version number enable. 3 == TLS 1.2 2 == TLS 1.1 1 == TLS 1.0 Double click the line, enter 1 and confirm. (I don't recommend to keep that setting, because you really should use those protocols with any site that supports it. Only use it for testing.) Restart Firefox. Try to connect with Firefox to your box. Does it work? (If it worked, it could mean bad implementation of TLS in your box, or a problem with the automatic fallback to older protocols in Firefox.)

Only after you have tried the instructions from comment 24, here is another test, which might produce more helpful information. - go to about:config again - enter tls - set security.tls.version.max back to 3 - restart firefox Open a terminal where you have prepared the NSS tools. Run the following command, which will execute a tunneling debug utility: cd /tmp/ ssltap -l -s 134.15.84.97:1443 2>&1 | tee /tmp/ssltap.log The tool will now listen on your local host port 1924 for connections. Go back to Firefox with tls max set to 3, and paste the following address into the location bar: https://127.0.0.1:1924 After it stops loading in Firefox, close Firefox. Go back to the terminal, and hit CTRL-C. Your terminal should have a long log of information. The same information will also have been written to the file /tmp/ssltap.log Please provide file ssltap.log The logfile will contain detailed SSL/TLS protocol information of the interaction between firefox and your server. If Firefox attempts to fall back or not, we'll learn that from the log.

(In reply to Kai Engert (:kaie) from comment #25) > Please provide file ssltap.log When you do, don't paste it into a bugzilla comment. Please either use the file attach mechanism in this bug, or please send the file to me by email. Thanks.

I set this variable: security.tls.version.max = 1 and restarted Firefox. But Firefox 36 still hangs when attempting to connect to the VPN box. I got the same prompts and warnings as shown in the attachments in bug 1138332. I will try the other test tonight.

I assume when you produced the ssltap.log file, Firefox showed an error page with a hostname mismatch. Please add an override. I need a logfile of the scenario where Firefox doesn't show an error page, but hangs. Thanks

Thanks. What I see is: - a first connection attempt, client disconnects after the handshake - now you've probably added the override - 8 seconds later, a second connection attempt - the handshake is completed - client sends a full request with encrypted data to the server, which most likely contains the http request - 15 seconds with no data exchanged - client disconnects I believe this means, client and server complete the handshake, in which they agree with each other on the parameters of the encryption. The client sends the http request to the server, to which the server is expected to reply. Instead of replying, the server doesn't send any response within 15 seconds. After 15 seconds, the client sends an alert message to the server, probably telling it that it gives up. Both client and server close the connection. It seems like the server doesn't understand the request that Firefox sends, and misbehaves by not sending any response. This is all I can guess without being able to connect to the server myself. If I were able to connect to the server myself, I could play with different requests.

Dick, would you be able to provide another log of a working connection with Firefox 35? One difference between FF 35 and FF 36: FF 36 no longer offers ciphers with RC4 to the server. It seems that it doesn't matter, because the server happily selects a non-RC4 cipher from the ones offered by Firefox 36. Maybe we can see another difference in the 35 vs. 36 logs.

Another suggestion: With FF 36, go to about:config, and in the search box, type: spdy Find the preference network.http.spdy.enabled.http2 I assume it's set to "true". Doubleclick the line, and it should switch to "false". Restart Firefox and try to connect to the 5505 again.

And if it still doesn't work, find network.http.spdy.enabled and switch it to false and restart and try again.

I will try the "spdy" tests this morning. I keep wondering how Firefox has worked for over 15 years with this Cisco hardware box, and the one before it, and now all of a sudden, all these problems?

I changed Firefox 36: network.http.spdy.enabled.http2 to false and re-started Firefox; still hangs trying to connect to the VPN box. Then changed: network.http.spdy.enabled to false and re-started Firefox; still hangs trying to connect to the VPN box. I changed both settings back to true. What do we try next?

I want to make clear that I appreciate all your help in resolving this regression. In that light, I make this suggestion: There can only be so many mods between Firefox 35 and Firefox 36 related to certificate handling. Why not pull all of those mods from Firefox 36, build a test Firefox, and verify that the regressions is indeed resolved? This conclusively indicates one of those mods introduced the regression. While a bit tedious, then start adding back mods one or two at a time until the problem returns. This gives us the mod responsible for introducing the regression. Of course, this is only practical when there is a limited number of mods. I will help with whatever debug course you take.

Dick, I have another theory, inspired by another bug I've investigated today. I wonder if Firefox is confused because you have added many overrides in the past. Could you please do the following: Prepare the terminal. Quit Firefox. Find the Firefox profile directory. Usually, if you have never used the "firefox profile manager" to create additional profiles, your profile should be named "whatever".default and the following command should work (please copy/paste): cd $HOME/Library/Application\ Support/Firefox/Profiles/*.default Confirm that you have the expected files: ls *.db You should have at least cert8.db, key3.db If you haven't found the right directory yet, use your OSX file software to find the correct name. Or use cd $HOME/Library/Application\ Support/Firefox/Profiles/ then ls which should print the name. Run cd the-name-that-you-saw-printed Now list the contents of your certificate database: certutil -d dbm:. -L This lists the names of intermediate CA certificates Firefox has cached, and personal certificates you might own, plus certificates that have been stored when you had added overrides in the past. Do you see entries that are named 134.15.84.97 ? How many of them? Now that firefox is still closed, I suggest you try the following to backup some files, and then clean up. mkdir backup mv -i cert8.db key3.db secmod.db cert_override.txt backup/ (Note this command will reset a lot of the SSL/TLS state your browser has remembered, and all your overrides, and if you own personal certificates, their keys will be moved, too. If you have such data, make sure you have a good backup of these files. If necessary, you can later restore them, by replacing any newer files with the ones from your backup.) Now start firefox and try again. Does it work now?

> Do you see entries that are named 134.15.84.97 ? How many of them? If you don't have such entries, maybe you could send me the output of the certutil -L command by email? Feel free to redact the output for privacy, if necessary.

I have two Firefox profiles; one for the public network the other for the company private network. 1) I tried running from the private profile directory and got: Riegner-iMac3:4qfu6u20.B_VPN dick$ /Users/dick/Download/nss/nss-3.18-darwin-32bit/bin/certutil -d dbm:. -L dyld: Library not loaded: @executable_path/libssl3.dylib Referenced from: /Users/dick/Download/nss/nss-3.18-darwin-32bit/bin/certutil Reason: image not found Trace/BPT trap: 5 2) And then tried running from the download bin directory and got: Riegner-iMac3:bin dick$ pwd /Users/dick/Download/nss/nss-3.18-darwin-32bit/bin Riegner-iMac3:bin dick$ certutil -d dbm:. -L certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. I moved the files to the backup directory and will restart Firefox on the VPN network.

Firefox 36 still hangs trying to access the VPN box after moving out those files. All four files have been recreated by Firefox in the VPN profile.

Do you need me to complete more tests or provide more documentation?

Are we still working this problem? There haven't been any updates from anyone but me for several weeks. I need to let the IT folks at work know if there will eventually be a Firefox fix for this VPN hardware problem.

Dick, I don't have further ideas what to do. Without direct access to the machine it's too complicated to analyze. I wish we'd see some support from Cisco, but I haven't gotten any response from them.

Try to run from within the download bin directory, because that seems to be required on your system to get the utlility I made to actually run. But then the parameters must be different and point to the firefox profile directory. cd /Users/dick/Download/nss/nss-3.18-darwin-32bit/bin certutil -d dbm:/full/path/to/your/firefox/profile/directory/ -L

Kai, I appreciate your help and frustration about not having access to my machine. I will gladly grant you access, if we can think of a way for you to login to my machine while it is connected to a private VPN network. I found that certutil was missing needed libraries, so I set this environment variable to point to them: cf-asa-hw-dickie-3:bin dick$ pwd /Users/dick/Download/nss/nss-3.18-darwin-32bit/bin export DYLD_LIBRARY_PATH=/Users/dick/Download/nss/nss-3.18-darwin-32bit/lib And now certutil seems to run, but says the database is old: cf-asa-hw-dickie-3:bin dick$ ./certutil -d dbm:/Users/dick/Library/Application Support/Firefox/Profiles/4qfu6u20.B_VPN/ -L certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. cf-asa-hw-dickie-3:bin dick$ Dick

Kai, Ok, dumb mistake of my part; I needed to protect the embedded space in the profile path name. Now we are getting some actual results: cf-asa-hw-dickie-3:bin dick$ ./certutil -d dbm:"/Users/dick/Library/Application Support/Firefox/Profiles/4qfu6u20.B_VPN/" -L Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI DigiCert Secure Server CA ,, DigiCert High Assurance CA-3 ,, EssentialSSL CA ,, DigiCert High Assurance EV CA-1 ,, 134.15.84.97 ,, DigiCert SHA2 Extended Validation Server CA ,, GeoTrust SSL CA - G2 ,, Verizon Akamai SureServer CA G14-SHA1 ,, GeoTrust SSL CA - G3 ,, Go Daddy Secure Certificate Authority - G2 ,, GeoTrust EV SSL CA - G4 ,, VeriSign Class 3 Secure Server CA - G3 ,, GeoTrust SSL CA ,, thawte SSL CA - G2 ,, DigiCert SHA2 High Assurance Server CA ,, Cybertrust Public SureServer SV CA ,, Entrust Certification Authority - L1C ,, COMODO High-Assurance Secure Server CA ,, COMODO RSA Organization Validation Secure Server CA ,, COMODO SSL CA ,, Verizon Akamai SureServer CA G14-SHA2 ,, PositiveSSL CA 2 ,, StartCom Class 2 Primary Intermediate Server CA ,, DigiCert SHA2 Secure Server CA ,, GeoTrust Global CA ,, Symantec Class 3 Secure Server CA - G4 ,, RapidSSL CA ,, GlobalSign Organization Validation CA - G2 ,, Google Internet Authority G2 ,, GeoTrust DV SSL CA ,, COMODO RSA Domain Validation Secure Server CA ,, GlobalSign Domain Validation CA - SHA256 - G2 ,, Starfield Secure Certificate Authority - G2 ,, cf-asa-hw-dickie-3:bin dick$

This is the IP address of my Cisco VPN box that shows up in the list: 134.15.84.97

I am guessing that you will want to see the details of the certificate from the VPN box at 134.15.84.97. So here it is: cf-asa-hw-dickie-3:bin dick$ ./certutil -d dbm:"/Users/dick/Library/Application Support/Firefox/Profiles/4qfu6u20.B_VPN/" -L -n 134.15.84.97 Certificate: Data: Version: 3 (0x2) Serial Number: 247927381 (0xec71255) Signature Algorithm: PKCS #1 SHA-1 With RSA Encryption Issuer: "CN=134.15.84.97" Validity: Not Before: Wed Mar 25 14:32:46 2015 Not After : Sat Mar 22 14:32:46 2025 Subject: "CN=134.15.84.97" Subject Public Key Info: Public Key Algorithm: PKCS #1 RSA Encryption RSA Public Key: Modulus: be:29:a8:fd:6d:44:8c:48:82:4b:5e:e6:83:b2:f5:63: 2c:ca:3c:2e:5d:01:11:f2:9e:30:60:cf:d9:97:d1:42: 47:b6:92:27:86:f5:6e:10:41:e6:0e:0f:41:cf:ca:03: 50:41:98:b6:53:bd:1a:74:33:dc:34:5c:3b:40:d7:a5: ba:cd:60:e6:78:8b:ef:4f:bc:0b:75:80:ed:ac:07:cc: b3:8e:f9:18:aa:88:ee:6b:f2:99:86:07:ee:39:d3:46: 50:b2:a8:d5:4c:5d:24:84:4f:96:69:ad:7d:e6:16:29: eb:ed:9f:49:dd:f3:48:82:67:b9:e1:36:a5:23:f4:eb Exponent: 65537 (0x10001) Signature Algorithm: PKCS #1 SHA-1 With RSA Encryption Signature: 73:18:0b:ce:92:f3:16:e6:9b:d8:3a:d7:33:ca:ad:df: fe:c0:fa:a5:11:5e:9e:4c:1d:5b:d5:50:80:09:14:d1: 5c:20:4b:c0:e0:dc:aa:3f:25:6e:a0:e7:4e:d8:cc:c7: ca:99:52:1c:da:0e:7f:77:59:3f:a2:b6:dc:13:e6:92: ae:f4:17:cc:d5:b1:8a:fa:e9:7c:b1:f2:8c:32:ad:1f: 05:0e:82:6b:ab:46:f8:aa:4f:ec:ac:cc:69:dc:f2:02: 0d:5d:38:df:a8:0b:b1:bc:9c:56:a8:9f:b7:4d:e8:57: 6b:dc:8e:5e:a4:e3:c2:03:8f:5b:73:6a:d4:a6:d9:69 Fingerprint (SHA-256): 3F:92:EE:87:A4:82:22:0B:B4:82:4B:6B:97:36:AA:01:1C:FD:E1:06:8D:4D:06:45:A6:F1:E3:12:65:2A:E0:F0 Fingerprint (SHA1): 77:13:C1:54:6D:DB:A1:72:FF:DD:8C:9C:B4:F2:82:B0:E8:BD:40:10 Certificate Trust Flags: SSL Flags: Email Flags: Object Signing Flags: cf-asa-hw-dickie-3:bin dick$

Thanks for that output. It doesn't show anything unusual. First, please make a backup of the cert8.db file in your directory, in case we might want to analyze it later. With Firefox closed, attempt to delete the cached copy of the router certificate: ./certutil -d dbm:"/Users/dick/Library/Application Support/Firefox/Profiles/4qfu6u20.B_VPN/" -D -n 134.15.84.97 Now list again ./certutil -d dbm:"/Users/dick/Library/Application Support/Firefox/Profiles/4qfu6u20.B_VPN/" -L -n 134.15.84.97 Does it still list the entry with the 134.... ? It yes, please continue to use the delete command, until the output of -L no longer shows the 134.... entry. With all entries gone, attempt to connect to the router again. Does it work? If it works now, then the reason might have been bug 1056341. It still doesn't work? Hmm.... Can you try to update to Firefox 37 ? Does it work or not? After all of the above fails: Did I already suggest that you try a fresh profile?

The serial number in comment 49 is different from the serial number in comment 22. Maybe it's really bug 1056341 and deleting all certs helps. I'm crossing fingers.

I backed-up the cert8.db file and then deleted the 134.15.84.97 entry with one run of certutil -D. I then tried to connect to the VPN box, got all the certificate warnings, but the connection still then hangs. :-( I then tried using my other profile and it too hangs trying to connect to the VPN box. :-( These tests were run with Firefox 37.0.2. I then went back to Firefox 35 to verify that the test environment had not changed, and I was able to connect with no problems. Thanks for the help.

Dick, I've created a self-signed certificate that's equivalent to the one you listed in comment 49, and I set up a local test server, and configured it to support SSL 3 and TLS 1.0, only. Then I used Firefox 37 to connect to it. After I added the override, the connection worked fine. You said, that you are successfully able to add an override for the self signed cert of that Cisco VPN box. I no longer believe the error is on the TLS protocol / security level. I suspect the error is on the http protocol level (which runs inside the TLS layer). There are too many variables, and without being able to connect to our hardware, I cannot provide any more help to analysis this issue for you. I'm resolving this bug as incomplete. In order to make progress on this bug, we'd need an unencrypted log of the communication between Firefox and the Cisco box. Maybe you could learn how to use a MITM proxy utility for the purposes of debugging your connection, but I cannot help you with learning how to use such a tool. Using an utility that shows all the http protocol connection going back and forth between firefox and the server, and by comparing those logs between working firefox 36 and nonworking firefox 37, we could understand, which side of the connection is guilty of not proceeding with the connection.

Our IT folks upgraded my Cisco ASA 5505 hardware VPN box to IOS version 8.4(6) and then to version 9.1(6). Running either version I was able to connect and login to my VPN box using Firefox 38.0.1. This problem has been resolved for me.

Thanks for the feedback, so let's conclude the implementation of TLS on older firmware versions was broken in one way or the other.