After bug 1547096, connections to SSL/TLS servers with "bad" certificates will fail in a different way, and might no longer offer an override.

As described in bug 1547096, the Mozilla core engine removes the callback interface nsIBadCertListener2.

A callback notification will no longer be offered. Instead, the code that handles the application protocol over a SSL/TLS must handle error results, and if necessary, query the socket/channel to find out if the error was related to a bad certificate, and act as desired.

This bug suggests to add a new implementation for that, to replace the previous use of nsIBadCertListener2 and notifyCertProblem in the following files:

• mail/base/content/mailWindow.js

Is my Bug 1605228 perhaps a duplicate or a consequence of this issue? (That bug is about Thunderbird failing to connect to an NNTP server with an expired certificate, and failing to explain this problem to the user.)

Beta users are now reporting problems related to this, as Magnus has seen in the recent emails to beta testers.

Here's one way to set up for proper reproduction, at least on ubuntu, but you can use the vm too.

sudo apt install dovecot-imapd dovecot-pop3d

# Make sure you have a user "guest" on your system with uid 1001
# Use `id -u guest` to find out the uid.

# Edit the configuration file and append the below (between ###s).
sudo nano /etc/dovecot/dovecot.conf

###
mail_home=/srv/mail/%Lu
mail_location=sdbox:~/Mail

## this is sometimes needed
first_valid_uid = 1001

# use system users
passdb {
  driver = pam
}

userdb {
  driver = passwd
  args = blocking=no
  override_fields = uid=guest gid=guest
}

ssl=yes
###

# Start the server
sudo systemctl start dovecot

Then get a Thunderbird account set up for guest@localhost. When connecting it should ask to override, because the included server certificate is self signed. Compare to 68. With the patch, it doesn't ask and seems to just end up in the connecting phase.

(In reply to Magnus Melin [:mkmelin] from comment #6)

Then get a Thunderbird account set up for guest@localhost. When connecting it should ask to override, because the included server certificate is self signed. Compare to 68. With the patch, it doesn't ask and seems to just end up in the connecting phase.

This patch is related to the mail window so when you try to send a mail or downloading the mail, it should show an error. The above error that you are talking about is related to account wizard part. Can you check with that patch or you had applied both the patches?

I had both patches applied. (It's unclear if the wizard part worked.) The situation is that the account is ready, but never fully connected because of the ssl problem. If I start the same profile with tb68 I get the dialog to override. I don't get that even with the patches applied on trunk, all it does is it stays in the "connecting to..... " mode.

Request to increase severity and priority. Anything I can do to help? Contact me via direct email. NOTE: evolution works with dovecot IMAPS selfsigned and letsencrypt where thunderbird fails.

Can I ask for this to be prioritized, now that 78 is being pushed through the update channel?

I have a setup similar to this and the update just broke mailing for me.

I can't add the missing cert exception through the certificate manager either, so the message "Checking mail server capabilities" is all TB is good for now.

We're also seeing reports of this in support, and clearly need this fixed (quickly) in 78.

khushil, can we move the patch forward?

Will it work this way? Where should we add the listener?

In c++, the notificationCallbacks that were set on the transport need to be changed to be called for protocol errors instead, or something like that. Would be my quick guess.

Hello,

Could you please fix the window to add self-signed certificates?

Problem also reported here:

https://bugzilla.mozilla.org/show_bug.cgi?id=1665340

Thank you.

It seems to me that this security failure has been talked about for 11 months in this thread, and was promoted to the release channel and automatic upgrade to v78.

The failure to work with self signed certificates (as an exception) raises incompatibility with Avast and possibly many other virus checkers that act as a 'friendly MITM' and use an self signed certificate between TB and themselves.

Reversion to an earlier version of TB carries warnings of the risk of corruption of the profile.

The simplistic work around seems to be to disable TLS and so compromise the security of one's email.

A better work around might be to discontinue TB and use another more secure email client in the interim (or permanently).

With no committed timeframe to fix an 11 month old problem that should never have been promoted to the release channel, the latter might seem the more secure option?

Have I got this wrong?

I've run into this bug once or twice over the past month or so, and the solution was to disable automatic updates for the user, then reinstall an OLD version of TB so the user could get back to work.

However, as the Profile's have been updated somehow, sure a reinstall of an old version is no longer a simple matter. For someone who has 7 email accounts, and multiple mail filters, it's torturous to deal with.

Right now I have a client who's machine conveniently updated TB and broke their main email account. The solution until this 11 month old bug is resolved is for her to use Outlook or Webmail to access that particular account. She's not a tech-savvy user and so I'll have to teach her how to use Outlook which is time I don't have right now.

This particular office has been using TB since the early days, it works for them, and I don't have to deal with any of the M$ Outlook issues that other clients I have experience. I use TB because it (used to) work.

Whatever got broken 11 months ago needs to be put back the way it was and a release pushed out. Requiring someone to use a "Let's Encrypt" or commercial certificate is not a solution. There's perfectly valid reasons to use self-signed certificates - they work quite well for a number of use cases, and Mozilla ought not to be breaking things because of their own philosophy that "this or that is good for you so we're going to force it on you."

When's it going to be fixed?

Ben, any thoughts on C++ part?

Yes! I can catch the bad cert in at least one case - checking for new mail (I was using pop3, but I'm sure it'll catch the issue on IMAP too):

Here's the code which asks the incomingserver to get new messages:
https://searchfox.org/comm-central/source/mail/base/content/mailWindowOverlay.js#2911

function GetNewMsgs(server, folder) {
  // Note that for Global Inbox folder.server != server when we want to get
  // messages for a specific account.

  const nsIMsgFolder = Ci.nsIMsgFolder;
  // Whenever we do get new messages, clear the old new messages.
  folder.biffState = nsIMsgFolder.nsMsgBiffState_NoMail;
  folder.clearNewMessages();
  server.getNewMessages(folder, msgWindow, null);
}

That last null param in server.getNewMessages call is the urlListener. If I hack up a listener to pass in, it does indeed get called (onStopRunningUrl()) with a bad-cert error code, and the url, which you can get the failedSecInfo from.

SO.

I'm now out of my depth - I don't know how the mailwindow and mailwindowoverlays interact, or where to put such a listener that could be shared over all these kinds of calls... but I think this will fix the problem.

The tedious aspect is that there are a lot of these operations which require a listener to be passed in.
I don't know how we track them all down, but this is probably a good starting point: https://searchfox.org/comm-central/search?q=nsIUrlListener&path=*.idl

There is a probably a small set of "critical" operations which give us enough coverage to get by. eg usually the first operation will be a "check for new messages". So if that runs, and the user adds an exception for the bad cert error, nobody will notice that subsequent operations don't have proper bad-cert handling!
Obviously though, it'd be much better if we did have proper bad-cert handling on everything :-)

I have added listener in server.getNewMessages.

Hello,

Thank you very much for your work in fixing this bug.

I watch this thread every day.

Cordially.

Updated the patch according to your suggestions.

Same thing with IMAP.

Sorry, I missed to do qrefresh. In the InformUserOfCertError function, we should have only two aruguements.

Also, we require beta and ESR uplift for this.

(In reply to Magnus Melin [:mkmelin] from comment #39)

Maybe we should name it TransportErrorUrlListener instead (after all) so
that we could e.g. pop up a message for bad TLS versions (in another bug).

The onStopRunningUrl() is the main error-handling point, so I think eventually we'd want to be handling all error types here, not just NSS ones.
At the moment it's not an issue because the various protocols often pop up prompts and error boxes before onStopRunningUrl() is called, so it's never been a problem that the listener was null. But it seems like it'd be good to decouple this the protocol away from the GUI more, and the obvious route would be to let onStopRunningUrl() handle the GUI side of things [1].

I wouldn't worry about that for naming it now (i.e. ignore this comment! :-)

[1] There could be really good reasons why I'm utterly wrong on this ;-)

Pushed by mkmelin@iki.fi:
https://hg.mozilla.org/comm-central/rev/c5f4ead2437f
Implement new handling for bad server certificates on SSL/TLS connections in Mail Window. r=mkmelin

Comment on attachment 9178689 [details] [diff] [review]
Bug-1590474_mail-window-bad-certificate-handling-6.patch

[Approval Request Comment]
Regression caused by (bug #): 1547096
User impact if declined: Bad certificate dialog will not be opened in the Mail Window.
Testing completed (on c-c, etc.): Yes, it's on the trunk for the last few days.
Risk to taking this patch (and alternatives if risky): Low

Comment on attachment 9178689 [details] [diff] [review]
Bug-1590474_mail-window-bad-certificate-handling-6.patch

[Triage Comment]
Approved for beta

Thunderbird 82.0b3:
https://hg.mozilla.org/releases/comm-beta/rev/801b0298e241

Comment on attachment 9178689 [details] [diff] [review]
Bug-1590474_mail-window-bad-certificate-handling-6.patch

[Triage Comment]
Approved for esr78

Thunderbird 78.4.0:
https://hg.mozilla.org/releases/comm-esr78/rev/87ed305fea4d

This morning got update 78.4.0.
After pressing "Get Messages" got prompt for certificate exception.
Also while sending got warning and then certificate exception prompt.

Adding exceptions worked, thank you guys!

Unfortunately no change in my environment with self-signed cert.

TB 68.12.1: When first connecting to the mail server a pop-up windows explains a problem with the certificate, because the idetity could not be verified. After adding an security exception (in German: Sicherheitsausnahmeregel hinzufügen), TB 68.12.1 establishes a TLS1.3 connection to the server. Access to all mails. No problems.
Analysis with openssl s_server shows communiction with certificates, ciphers, elliptical curves....

TB 78.4.0: When connection to the mail server (exactely same server, same certificate, same environment...) - no reaction. Window's status line says:

"MatthiasJ: Connected to imap1.<mydomain>.de..."

No popup to accept unverified certs, no error message...

Analysis with openssl shows the following output:

cobraimap1:/etc/ssl/certs # openssl s_server -key cobraimap.key -cert cobraimap.pem -accept 993
Using default temp DH parameters
ACCEPT
ERROR
140108269007296:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:ssl/record/rec_layer_s3.c:1544:SSL alert number 42
shutting down SSL
CONNECTION CLOSED

No change for me either with Thunderbird 78.4.0. When connecting to my IMAP server, which uses a self-signed certificate, Thunderbird (running in safe mode) doesn't get any further than displaying "Checking mail server capabilities…" in the status bar.

If I don't run Thunderbird in safe mode, but have all add-ons disabled, then attempting to connect to the IMAP server results in an "Add Security Exception" dialog that strangely lists the server's IP (followed by :143) rather than its hostname in the "Location" field. If I press the "Get Certificate" button, then I get the message "Unable to obtain identification status for this site." Ditto if I rewrite the Location field with the server's hostname, and also if I change the port to 993. At this point pressing "Cancel" is the only option. If I try checking mail again, the security exception dialog pops up again. If this time I try to confirm the security exception without getting the certificate, the dialog disappears, but it reappears as soon as I try to check mail again. Pressing the "View" button in the dialog does (eventually) bring up a new tab showing the correct certificate details.

To make sure the dialog to add exception pops up, make sure to click the "Get messages" toolbar button.

Once the dialog is up you should not be able to just accept the certificate that is there - and there should be one pre populated. "Get certificate" never worked for mail...

Note that in Thunderbird 78 TLS 1.2 or higher is required
https://support.mozilla.org/en-US/kb/thunderbird-78-faq#w_after-upgrading-to-thunderbird-78-i-cannot-download-email

(In reply to Magnus Melin [:mkmelin] from comment #57)

Once the dialog is up you should not be able to just accept the certificate that is there

No, that doesn't work.

Note that in Thunderbird 78 TLS 1.2 or higher is required
https://support.mozilla.org/en-US/kb/thunderbird-78-faq#w_after-upgrading-to-thunderbird-78-i-cannot-download-email

My IMAP server is offering TLS 1.2.

On further investigation, I've noticed that the IP address that the security dialog pre-fills in the Location field isn't actually the IP address of my IMAP server. It's actually the former IP address that I entered as the server's hostname, back when I originally set up the account in Thunderbird several years ago. (I later changed the configuration to use the actual hostname.) So I think what Thunderbird is populating in the field here isn't really a meaningful IP address but rather its own internal identifier for that server, per Bug 1464047. This is bad for any number of reasons.

I've also noticed that it's only the certificate for this particular account configuration that Thunderbird doesn't let me accept. If I set up a new IMAP account in Thunderbird, either in this profile or in a fresh one, using exactly the same server details, then the security exception dialog works properly. I'm able to accept the self-signed certificate and then go on to use the account normally.

So as a workaround, I deleted the old account configuration and am now using the new one that I've created. I backed up the old account configuration first in case anyone wants me to do further testing with it.

I would like to add that I am getting the following error on my Dovecot.

Oct 29 12:51:13 ns1 dovecot: imap-login: Disconnected (no auth attempts in 0 secs): user=<>, rip=192.168.3.254, lip=192.168.10.2, TLS handshaking: SSL_accept() failed: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate: SSL alert number 42, session=<WKfNg8WygI7AqAP+>

78.4 resolved the issue for me

May in 78.4 still be same thing with endless "Checking mail server capabilities..."?
On one Windows10 PC, with existing account. Solved by creating account from scratch.

Something still not right sometimes on some pc. Most likely in combination in Antivirus sertificate substitution MiM.
It feels like certificates are cached somewhere and deleting from "Servers" does not help.
When i somehow manage to use different working server name or port, only then i was able to get prompt for certificate exception.

Old server name was used as long as password was saved for old server names.
Only after old saved passwords was deleted from password manager, Thunderbird started to use new server address.

This bug seems to have come back. I had an SSL cert expire on my internal mail server. I generated a new, self-signed cert and restarted my IMAP server running on CentOS 7 current:
dovecot -n

2.2.36 (1f10bfa63): /etc/dovecot/dovecot.conf

OS: Linux 3.10.0-1160.45.1.el7.x86_64 x86_64 CentOS Linux release 7.9.2009 (Core)

I am getting the same problem described in the bug. That is, the new cert was not recognized and there doesn't appear to be anyway to tell TB to add the self-signed cert as an exception.

MOZ_LOG level 5 shows:
2022-02-16 04:54:28.087122 UTC - [Parent 983: Main Thread]: I/IMAP 7f87aff14000:mutilate:NA:SetupWithUrlCallback: clearing IMAP_CONNECTION_IS_OPEN
2022-02-16 04:54:28.087202 UTC - [Parent 983: IMAP]: I/IMAP 7f87aff14000:mutilate:NA:ProcessCurrentURL: entering
2022-02-16 04:54:28.087210 UTC - [Parent 983: IMAP]: I/IMAP 7f87aff14000:mutilate:NA:ProcessCurrentURL:imap://dave@mutilate:143/liteselect%3E/INBOX: = currentUrl
2022-02-16 04:54:28.137795 UTC - [Parent 983: IMAP]: I/IMAP 7f87aff14000:mutilate:NA:CreateNewLineFromSocket: * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS LOGINDISABLED] Dovecot ready.
2022-02-16 04:54:28.137982 UTC - [Parent 983: IMAP]: I/IMAP 7f87aff14000:mutilate:NA:SendData: 27 STARTTLS
2022-02-16 04:54:28.138489 UTC - [Parent 983: IMAP]: I/IMAP 7f87aff14000:mutilate:NA:CreateNewLineFromSocket: 27 OK Begin TLS negotiation now.
2022-02-16 04:54:28.138694 UTC - [Parent 983: IMAP]: I/IMAP 7f87aff14000:mutilate:NA:SendData: 28 capability
2022-02-16 04:54:28.157886 UTC - [Parent 983: IMAP]: I/IMAP 7f87aff14000:mutilate:NA:CreateNewLineFromSocket: clearing IMAP_CONNECTION_IS_OPEN - rv = 805a3ff2
2022-02-16 04:54:28.158121 UTC - [Parent 983: IMAP]: I/IMAP 7f87aff14000:mutilate:NA:TellThreadToDie: close socket connection
2022-02-16 04:54:28.158130 UTC - [Parent 983: IMAP]: I/IMAP 7f87aff14000:mutilate:NA:CreateNewLineFromSocket: (null)
2022-02-16 04:54:28.158323 UTC - [Parent 983: Main Thread]: I/IMAP offline imap url failed :imap://dave@mutilate:143/liteselect>/INBOX
2022-02-16 04:54:28.158351 UTC - [Parent 983: IMAP]: I/IMAP 7f87aff14000:mutilate:NA:ProcessCurrentURL: aborting queued urls
2022-02-16 04:54:31.243600 UTC - [Parent 983: Main Thread]: I/IMAP 7f87aff7c000:mutilate:NA:SetupWithUrlCallback: clearing IMAP_CONNECTION_IS_OPEN
2022-02-16 04:54:31.243924 UTC - [Parent 983: IMAP]: I/IMAP 7f87aff7c000:mutilate:NA:ProcessCurrentURL: entering
2022-02-16 04:54:31.243931 UTC - [Parent 983: IMAP]: I/IMAP 7f87aff7c000:mutilate:NA:ProcessCurrentURL:imap://dave@mutilate:143/discoverallboxes: = currentUrl
2022-02-16 04:54:31.243938 UTC - [Parent 983: Main Thread]: I/IMAP 7f87aff87000:mutilate:NA:SetupWithUrlCallback: clearing IMAP_CONNECTION_IS_OPEN
2022-02-16 04:54:31.244018 UTC - [Parent 983: IMAP]: I/IMAP 7f87aff87000:mutilate:NA:ProcessCurrentURL: entering
2022-02-16 04:54:31.244024 UTC - [Parent 983: IMAP]: I/IMAP 7f87aff87000:mutilate:NA:ProcessCurrentURL:imap://dave@mutilate:143/select%3E/Banking: = currentUrl
2022-02-16 04:54:31.244321 UTC - [Parent 983: Main Thread]: I/IMAP 7f87aff8c800:mutilate:NA:SetupWithUrlCallback: clearing IMAP_CONNECTION_IS_OPEN
2022-02-16 04:54:31.246870 UTC - [Parent 983: IMAP]: I/IMAP 7f87aff8c800:mutilate:NA:ProcessCurrentURL: entering
2022-02-16 04:54:31.246879 UTC - [Parent 983: IMAP]: I/IMAP 7f87aff8c800:mutilate:NA:ProcessCurrentURL:imap://dave@mutilate:143/fetch%3EUID%3E/Banking%3E33: = currentUrl
2022-02-16 04:54:31.304628 UTC - [Parent 983: IMAP]: I/IMAP 7f87aff7c000:mutilate:NA:CreateNewLineFromSocket: * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS LOGINDISABLED] Dovecot ready.
2022-02-16 04:54:31.304712 UTC - [Parent 983: IMAP]: I/IMAP 7f87aff7c000:mutilate:NA:SendData: 61 STARTTLS
2022-02-16 04:54:31.305027 UTC - [Parent 983: IMAP]: I/IMAP 7f87aff7c000:mutilate:NA:CreateNewLineFromSocket: 61 OK Begin TLS negotiation now.
2022-02-16 04:54:31.305157 UTC - [Parent 983: IMAP]: I/IMAP 7f87aff7c000:mutilate:NA:SendData: 62 capability
2022-02-16 04:54:31.310230 UTC - [Parent 983: IMAP]: I/IMAP 7f87aff87000:mutilate:NA:CreateNewLineFromSocket: * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS LOGINDISABLED] Dovecot ready.

TB Version is 91.6.0 which is current for CentOS 7.

Also, as per some of the earlier comments, I installed Evolution as a workaround and it works fine with the SSL certs on both the server and my client. Can either re-open this ug or start a new one.

New self-signed certificate in ProtonMail Bridge which serves IMAP and SMTP with STARTTLS on 127.0.0.1 caused Thunderbird 91.5.0 to claim that password authentication failed. Only examining the service logs revealed

ERRO[Feb 18 13:13:43.741] cannot upgrade connection: remote error: tls: bad certificate protocol=IMAP

Thunderbird doesn't prompt to accept the new certificate on IMAP request, even after restart, so went to Edit > Preferences > Privacy & Security > [Manage Certificates] > Servers, removed the two existing Permanent exceptions for the bridge, clicked '[Add Exception...]', entered imaps:127.0.0.1:1143, clicked [Get Certificate], resulting in

Checking Information
Attempting to identify this site...

and nothing happens in Thunderbird or the service log for at least a minute.

I resorted to the only workaround I know: deleting and re-creating the account in Thunderbird, which worked fine including prompting for the exception for the certificate.

When you hit "Get Messages" it will show the add exception dialog (if an exception is needed and possible).

Anyway, this bug is closed since long. Possible new issues should go into a new bugs.

(In reply to Magnus Melin [:mkmelin] from comment #69)

Anyway, this bug is closed since long. Possible new issues should go into a new bugs.

Thanks for the follow-up here. While I did try to load new messages by selecting them, I did not specifically try Get Messages. I'll update an existing bug or open a new one if and when I encounter the problem again.

(In reply to David G. Miller from comment #66)

This bug seems to have come back. I had an SSL cert expire on my internal
mail server. I generated a new, self-signed cert and restarted my IMAP
server running on CentOS 7 current:
dovecot -n

2.2.36 (1f10bfa63): /etc/dovecot/dovecot.conf

OS: Linux 3.10.0-1160.45.1.el7.x86_64 x86_64 CentOS Linux release 7.9.2009

(Core)
I am getting the same problem described in the bug. That is, the new cert
was not recognized and there doesn't appear to be anyway to tell TB to add
the self-signed cert as an exception.

TB logging shows:

Given that this is a regression, I wasn't sure whether to add to the existing bug or open a new one. Other organizations I've worked with prefer the "re-open existing but closed" approach. I will open a new bug but put a reference to this bug in the description so whoever works the bug will have easy access to the history.