User Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:99.0) Gecko/20100101 Firefox/99.0

Steps to reproduce:

Trying to retrieve email from IMAP server via STARTTLS, Normal password.

Actual results:

Since two days emails are not retrieved any more on accounts that previously worked fine. This issues happens on two different computers running Fedora 35.

Mail is not retrieved, and I see Checking mail server capabilities...

I think this is somehow related to certificates that are not trusted, because after removing a certificate of a still working email server, also this account started having this issue.

See also: https://support.mozilla.org/en-US/questions/1357919

I tried removing the certificates and the passwords for the accounts, but I am not asked for them when trying to connect to the mail server.

Expected results:

Mail should be retrieved.

As a test. I removed all certificates and passwords of the non-working email accounts. I do not get a popup or anything to ask me if I trust the certificate, or to enter the password.

Maybe bug 1737609 is related.

Maybe bug 1761946 is related.

Maybe bug 1743877 is the same issue as this one.

For info: I tried connecting to the email account with Evolution, and I get the attached popup about the certificate trust. Note that it is a wildcard certificate.

Here steps to reproduce this issue:

1. Install the current latest Thunderbird flatpak from https://flathub.org/apps/details/org.mozilla.Thunderbird
2. Go to Account Settings > Account Actions > Add Mail Account...
3. Create a new Mail Account with the following settings

• You full name: Thunderbird Test
• Email address: tb@janvlug.org
• Password: leave it empty for the test (but you can ask me if you need it)

4. Click: Configure manually
5. Click: Advanced config
6. Select OK in the Confirm Advanced Configuration pop-up window
7. Fill in the Server Settings:

• Server Type is already automatically: IMAP Mail Server
• Server Name: imap.janvlug.org
• Port: 143
• User Name: tb@janvlug.org
• Security Settings
  • Connection security: STARTTLS
  • Authentication method: Normal password

8. Leave the rest as the defaults.
9. Go to the Mail tab in Thunderbird
10. Close Thunderbird
11. Start Thunderbird
12. Click on the Inbox of the account with the name tb@janvlug.org
13. Notice the line at the bottom of the window: tb@janvlug.org: Checking mail server capabilities. (Note that you sometimes have to click a little around or wait for a few seconds for the message to appear). You get no other feedback.

I tried to give detailed steps, but if you cannot reproduce the issue, please let me know.

I checked the site of the provider. They state other settings (see attachment) that I applied, and now Thunderbird works again. This does probably not mean that this bug is invalid however, as the settings described in comment 6 can be used in Evolution to set up a working email account.

Confirming same issue. Thunderbird 91.3.0, Fedora 35.

Self-signed, date-valid certificate hosted on Dovecot server. Using IMAP w/STARTTLS and Normal Password authentication. Certificate Common Name matches server name used in Thunderbird Accounts wizard.

Setting up a new mail account, I'm unable to either "Confirm Security Exception", or import the certificate directly using Preferences -> Privacy & Security -> Manage Certificates -> "Servers" -> Add Exception. Dialog hangs at "Attempting to identify this site"....

If I click "View" on "Add Security Exception" dialog, the certificate details are correctly shown (so retrieval is not the issue).

The trick or work-around is to select Inbox of the problem account and click "get new mail" button. That will (usually) bring up the exception dialog box. I'm not sure why this is necessary but it always seems to work for me.
I have to do this with my local dovecot using STARTTLS (or TLS) having a self-signed cert.

How long have you been using beta?

I do not use a beta. I always use the version that is in the Fedora repositories. When reporting the bug, I used version 91.7.0, see: https://support.mozilla.org/en-US/questions/1357919#answer-1497432

Jan, did the suggestion in comment 12 help?

I did not try that. I worked around the issue this way: https://bugzilla.mozilla.org/show_bug.cgi?id=1764770#c7

(In reply to Jan Vlug from comment #16)

I did not try that. I worked around the issue this way: https://bugzilla.mozilla.org/show_bug.cgi?id=1764770#c7

Sorry, I missed comment 7 above when scanning the comments.
What was the change that your provider suggested? Looking at attachment in comment 8, it appears that a change was from STARTTLS to SSL/TLS. Was that what fixed it?

Set comment 6 for how I tied to give an option to reproduce this.

Note, I did not test the below again, because I'm a bit afraid to break my mail again, but it is what I got from my logbook regarding the issue.
Also note that the not in Thunderbird working config uses a "user friendly" host name (an alias?). The working configuration uses the direct host name of the server where the mail is.

This does not work in Thunderbird (but does work in Evolution):
imap.famvlug.nl:143
username: <redacted>
connection security: STARTTLS
authentication method: normal password

This works both for Thunderbird and Evolution:
Server: server101.hosting2go.nl
Port: 993
Connection: security: SSL/TLS
Authentication method: Normal password

I suspect it is because of the user friendly alias and not STARTTLS.
To verify this, you can set up a new "test" profile without disturbing your working default profile by starting tb from command line with -p option: $thunderbird -p and create a new account in the test profile using your user-friendly server name. Then when it's up and running, click on Inbox and if no subfolders appear or tb seems to be stuck checking the server, click get new mail button. Then you should see the exception to override.
If this works or not, you can start tb again with -p from command line and delete the test profile and go back to using your original default profile when you start tb normally.

I just ran into the same bug doing a server upgrade at my company. What I'm seeing is that if there is an existing account setup but the server presents a certificate that causes an exception (self signed or expired), there is NO pop up to ask for acceptance of the exception. TB just hangs. However, if I delete the account and set it up again, upon first loading of the mailbox it will pop up to ask to accept the exception. This is on newer versions. Some old versions seem to react properly

I have run into the same problem with version 102.0.2 on Windows 10. My server is Debian Bullseye running Dovecot for IMAP with a self-signed cert. Previously it worked fine. Now it hangs and is useless to me.

How can I troubleshoot this? How do I pull some kind of activity log?

hi Michael, have a look at:

https://wiki.mozilla.org/MailNews:Logging

(In reply to comment #21)

I'd bet you're seeing bug 1777717.

If you used an openssl command to generate your self-signed cert, please post the command there and I'll show you how to generate another that won't trigger the bug.

(In reply to Michael Soulier from comment #21)

I have run into the same problem with version 102.0.2 on Windows 10. My server is Debian Bullseye running Dovecot for IMAP with a self-signed cert. Previously it worked fine. Now it hangs and is useless to me.

How can I troubleshoot this? How do I pull some kind of activity log?

Let us know if this action from comment 12 above helps:

The trick or work-around is to select Inbox of the problem account and click "get new mail" button. That will (usually) bring up the exception dialog box. I'm not sure why this is necessary but it always seems to work for me.
I have to do this with my local dovecot using STARTTLS (or TLS) having a self-signed cert.

If that doesn't work to bring up the exception confirmation dialog to accept the self-signed cert, then an IMAP:5 log may be helpful as linked to in comment 22.

If that doesn't work to bring up the exception confirmation dialog to accept the self-signed cert, then an IMAP:5 log may be helpful as linked to in comment 22.

Probably better to check the info pointed to in comment 23 before recording an IMAP:5 log if the comment 12 thing doesn't help:
bug 1777717

hi,

I have now the same trouble on 2 differents windows 10. The only thing which is new is the windows 10 patch KB5015878.
I am using thunderbird 102.1.0 64 bits. I have an unsigned certificated on the email server. I am using the same certificat for smtp connexion and the certificat is working fine.
I have look in thunderbird log, it seems the trouble is went requesting the list of imap folder. We have a un null folder.
Regards
tg92

Sorry, don't know what you mean by "we have an un null folder". Also, if you can attach the log (I assume you mean the IMAP:5 log from https://wiki.mozilla.org/MailNews:Logging) it might help.
Thanks.

(In reply to gene smith from comment #27)

Sorry, don't know what you mean by "we have an un null folder". Also, if you can attach the log (I assume you mean the IMAP:5 log from https://wiki.mozilla.org/MailNews:Logging) it might help.
Thanks.

i have this log :
2022-07-31 06:34:12.777000 UTC - [Parent 15272: IMAP]: I/IMAP 1e527a75f00:mail.myServer.fr:NA:CreateNewLineFromSocket: clearing IMAP_CONNECTION_IS_OPEN - rv = 805a1ff3
2022-07-31 06:34:12.777000 UTC - [Parent 15272: IMAP]: I/IMAP 1e527a75f00:mail.myServer.fr:NA:TellThreadToDie: close socket connection
2022-07-31 06:34:12.777000 UTC - [Parent 15272: IMAP]: I/IMAP 1e527a75f00:mail.myServer.fr:NA:CreateNewLineFromSocket: (null)
2022-07-31 06:34:12.777000 UTC - [Parent 15272: IMAP]: D/IMAP SetConnectionStatus(0x805a1ff3)
2022-07-31 06:34:12.777000 UTC - [Parent 15272: IMAP]: D/IMAP URL failed with code 0x805a1ff3 (imap://myAccount%40myServer%2Efr@mail.myServer.fr:143/select%3E.INBOX)
2022-07-31 06:34:15.920000 UTC - [Parent 15272: IMAP]: I/IMAP 1e527a75f00:mail.myServer.fr:NA:ProcessCurrentURL: aborting queued urls
2022-07-31 06:34:16.098000 UTC - [Parent 15272: IMAP]: D/IMAP ImapThreadMainLoop leaving [this=1e527a75f00]

the trouble seems to be here CreateNewLineFromSocket: (null), it is looking like a null pointer

the thunderbird upgrade of this morning is fixing the trouble.

thanks

I have the same issue after the automatic update to Thunderbird 104 beta a few days ago. The log looks similar to comment 28:

2022-07-31 07:53:34.314000 UTC - [Parent 2596: IMAP]: D/IMAP ImapThreadMainLoop entering [this=1c2774b5100]
2022-07-31 07:53:34.315000 UTC - [Parent 2596: Main Thread]: I/IMAP queuing url:imap://xxxxx@tealdulcet.com@imap.tealdulcet.com:993/select>.INBOX
2022-07-31 07:53:34.428000 UTC - [Parent 2596: Main Thread]: I/IMAP 1c2774b5100:imap.tealdulcet.com:NA:SetupWithUrlCallback: clearing IMAP_CONNECTION_IS_OPEN
2022-07-31 07:53:34.428000 UTC - [Parent 2596: Main Thread]: I/IMAP 1c2774b5100:imap.tealdulcet.com:NA:SetupSinkProxy: got m_imapMailFolderSink
2022-07-31 07:53:34.428000 UTC - [Parent 2596: IMAP]: I/IMAP 1c2774b5100:imap.tealdulcet.com:NA:ProcessCurrentURL: entering
2022-07-31 07:53:34.429000 UTC - [Parent 2596: IMAP]: I/IMAP 1c2774b5100:imap.tealdulcet.com:NA:ProcessCurrentURL:imap://xxxxx%40tealdulcet%2Ecom@imap.tealdulcet.com:993/discoverallboxes:  = currentUrl
2022-07-31 07:53:34.527000 UTC - [Parent 2596: IMAP]: D/IMAP ReadNextLine [rv=0x804b0014 stream=1c275b9eb60 nb=0 needmore=1]
2022-07-31 07:53:34.527000 UTC - [Parent 2596: IMAP]: I/IMAP 1c2774b5100:imap.tealdulcet.com:NA:CreateNewLineFromSocket: clearing IMAP_CONNECTION_IS_OPEN - rv = 804b0014
2022-07-31 07:53:34.560000 UTC - [Parent 2596: IMAP]: I/IMAP 1c2774b5100:imap.tealdulcet.com:NA:TellThreadToDie: close socket connection
2022-07-31 07:53:34.560000 UTC - [Parent 2596: IMAP]: I/IMAP 1c2774b5100:imap.tealdulcet.com:NA:CreateNewLineFromSocket: (null)
2022-07-31 07:53:34.560000 UTC - [Parent 2596: IMAP]: D/IMAP SetConnectionStatus(0x804b0014)
2022-07-31 07:53:34.560000 UTC - [Parent 2596: IMAP]: D/IMAP URL failed with code 0x804b0014 (imap://xxxxx%40tealdulcet%2Ecom@imap.tealdulcet.com:993/discoverallboxes)
2022-07-31 07:53:34.579000 UTC - [Parent 2596: IMAP]: I/IMAP 1c2774b5100:imap.tealdulcet.com:NA:ProcessCurrentURL: aborting queued urls
2022-07-31 07:53:34.595000 UTC - [Parent 2596: IMAP]: D/IMAP ImapThreadMainLoop leaving [this=1c2774b5100]

(In reply to tg92 from comment #29)

the thunderbird upgrade of this morning is fixing the trouble.

thanks

Please report the version you upgraded to. You report above you are using release 102.1.0. I don't see a newer release in https://archive.mozilla.org/pub/thunderbird/releases/. Anyhow, it may also help the commenter Teal Dulcet and others to run the updated version too.

Thanks to both for the log snippets. It looks like the connection is being dropped probably due to cert problems. I don't think there's a null folder pointer.

with the same version 102.1.0 it is fixed on one of my 2 windows 10 but not on the second one. :(
quite strange

now it is working fine on my second windows 10
I have set all my account to none security
restart thunderbird
set all my account again to starttls
restart thunderbird
and everything is going well...
it is simply crazy and random. :(

set all my account again to starttls

Maybe STARTTLS is the problem. We had a user in Germany on netcologne server and TB refused to work with STARTTLS on imap, but only from some locations. This is bug 1747733. It worked for me using a test account in US. For that user in Germany it would occasionally work but not always.
When the user went to SSL/TLS security it worked reliably.
So you might try switching to SSL/TLS if your server supports it.

thanks i will see if i can setup ssl/tls on my server

(In reply to gene smith from comment #34)

So you might try switching to SSL/TLS if your server supports it.

I was using SSL/TLS in comment 30 and it does not work. I just tried switching to STARTTLS and unfortunately that does not work either...

(In reply to Teal Dulcet [:tdulcet] from comment #36)

(In reply to gene smith from comment #34)

So you might try switching to SSL/TLS if your server supports it.

I was using SSL/TLS in comment 30 and it does not work. I just tried switching to STARTTLS and unfortunately that does not work either...

So for you it has nothing to do with STARTTLS, obviously. I just tested my local dovecot server with self-signed cert on 104.0b1 and it worked as expected. What previous version were you using that worked OK?

The options I know of are:

1. Provide me a temporary test account on your server and I'll try to debug it from here, or,
2. You might try running wireshark and decode the TLS handshake to see where things are going wrong, or
3. You can run with added logging option nsSocketTransport:5 which may show the TLS error

I ran with 3 to see what it does and it doesn't look real useful for a successful TLS.

(In reply to gene smith from comment #37)

What previous version were you using that worked OK?

The affected TB installation has been using TB beta for years, so presumably it was the last build of 103 beta. Although based on the above comments, it looks like the bug might affect multiple TB versions.

3. You can run with added logging option nsSocketTransport:5 which may show the TLS error

I attached the log with this option. It does look like some kind of infinite loop issue, as this log started filling up very quickly after just a couple of seconds.

I just tested my local dovecot server with self-signed cert on 104.0b1

For me it is not self-signed, just an expired certificate (I do not have access to the server to renew it).

Can you provide me just the imap "server name" of the problem account with the expired cert (e.g., mail.myserver.com)? If you prefer, you can send it via my bugzilla profile email address. I assuming it's a public server and not on a private network. With just the address I should be able to at least get the imap CAPABILITY response from the server when the initial SSL handshake completes, and once that occurs you should be OK if I can find what's going wrong.

I couldn't tell anything from the attached log, as I expected.

Also, another possibility is that you may have changed the "server name" configured in TB for the account, maybe while debugging the problem. This may also cause a cert verification problem if the names don't match.

Well, actually I'm seeing the same thing if I delete the stored exception for my local dovecot. I see it with my trunk daily build and with the last beta 103. But if I go back to 91 release I can get the dialog to renew the exception.
With 91 there is no "spinner" when I select Inbox and I can click "get messages" button and the exception dialog appears. But with newer versions, the spinner keeps spinning and clicking "get messages" doesn't bring up the exception dialog. So something has definitely changed since 91. I'll look and see what I can find.

Teal, did you delete the exception at some point while debugging this? I didn't see a problem until I deleted it.

(In reply to gene smith from comment #39)

Can you provide me just the imap "server name" of the problem account with the expired cert (e.g., mail.myserver.com)?

It is imap.tealdulcet.com.

(In reply to gene smith from comment #40)

Teal, did you delete the exception at some point while debugging this?

Yes, well I never added the exception in the first place. The expired certificate is (hopefully) just a temporary issue, so for security reasons I have been unchecking that "Permanently store this exception" box when confirming the exception after starting TB. That worked fine until the automatic update to Thunderbird 104 beta. I just tried in the latest TB Daily 105 build and am also able to reproduce this bug.

Thanks. I should have seen your server name in your log snippet and didn't need to ask.

I have been unchecking that "Permanently store this exception" box when confirming the exception after starting TB.

I've found the problem as to why you are not seeing that exception prompt on startup but haven't arrived at a solution yet.
Anyhow, at this point the only work-around I know of is to install and run an older version, like https://archive.mozilla.org/pub/thunderbird/releases/102.0/ along side the beta. You will have to add the command line option --allow-downgrade when running 102.0 to avoid having to create a new profile (which would defeat the purpose). With 102.0 you should see the exception box. If you set the exception to "permanent" you should be able to go back to the beta version and use it with no problems. If you prefer to run with temporary exception and see the prompt on each startup you will have to stay at 102.0 until the issue is fixed.

This problem is a regression caused by bug 1768121 and the change was added to 102.0.1. I'm not sure when that change was added to beta but probably earlier.

I'm not sure this is the same problem reported in comment 0 since it was marked as version 97 and was reported well before the change in bug 1768121 was even in daily.

This bug is a bit overloaded with potential other issues, yes.
Thanks for investigating Gene! I think we might as well use this bug to fix the regression you tracked down.

(In reply to gene smith from comment #42)

Anyhow, at this point the only work-around I know of is to install and run an older version, like https://archive.mozilla.org/pub/thunderbird/releases/102.0/ along side the beta. You will have to add the command line option --allow-downgrade when running 102.0 to avoid having to create a new profile (which would defeat the purpose).

Thanks for investigating it and finding problem. I will look forward to your fix... For privacy/security reasons, I actually would prefer to use POP3 with the regular TB 91/102 release, but because of bug 1737609 I had to temporarily switch to IMAP last year.

I've found the problem as to why you are not seeing that exception prompt on startup but haven't arrived at a solution yet.

Note that the exception popup does not actually appear at startup as of TB 91, you have to manually click the "Get Messages" button due to bug 1743877/bug 1761946. It would of course be wonderful if you could fix all these related regressions in your fix for this.

(In reply to Steven Michaud [:smichaud] (Retired) from comment #23)

(In reply to comment #21)

I'd bet you're seeing bug 1777717.

If you used an openssl command to generate your self-signed cert, please post the command there and I'll show you how to generate another that won't trigger the bug.

Sorry, I've been away. I used the built-in Debian snakeoil cert. I can attach it if you like.

(In reply to gene smith from comment #24)

(In reply to Michael Soulier from comment #21)

I have run into the same problem with version 102.0.2 on Windows 10. My server is Debian Bullseye running Dovecot for IMAP with a self-signed cert. Previously it worked fine. Now it hangs and is useless to me.

How can I troubleshoot this? How do I pull some kind of activity log?

Let us know if this action from comment 12 above helps:

The trick or work-around is to select Inbox of the problem account and click "get new mail" button. That will (usually) bring up the exception dialog box. I'm not sure why this is necessary but it always seems to work for me.
I have to do this with my local dovecot using STARTTLS (or TLS) having a self-signed cert.

If that doesn't work to bring up the exception confirmation dialog to accept the self-signed cert, then an IMAP:5 log may be helpful as linked to in comment 22.

I'll give it a try. The affected machine was off for 2 weeks while I was away and it's not happening at the moment.

(In reply to Michael Soulier from comment #45)

(In reply to Steven Michaud [:smichaud] (Retired) from comment #23)

(In reply to comment #21)

I'd bet you're seeing bug 1777717.

If you used an openssl command to generate your self-signed cert, please post the command there and I'll show you how to generate another that won't trigger the bug.

Sorry, I've been away. I used the built-in Debian snakeoil cert. I can attach it if you like.

Please do attach it. Public key only, of course.

Edit: Never mind. I've already found out on my own that Debian snakeoil certs (at least current ones) contain subjectAltName fields. So you're probably not seeing bug 1777717.

Teal wrote in comment 44:

Note that the exception popup does not actually appear at startup as of TB 91, you have to manually click the "Get Messages" button due to bug 1743877/bug 1761946. It would of course be wonderful if you could fix all these related regressions in your fix for this.

I never understood exactly why that is necessary to click "Get Messages" to bring up the exception box, but it does seem to be. Also, usually you have to click it more than once before the box pops up.

On debugging the code and looking with wireshark, each time "Get Message" is clicked with a bad certificate, a TCP connection attempt is made and then the TLS handshake occurs. Tb then sends the server a "Bad Certificate" response and drops the TCP connection. However, it is random as to whether TB code sees a normal TCP level disconnect error like 0x804b0014 or if it sees a TLS level error that the certificate is bad and that it can be overridden, like 0x805a3ff2. So a problem is that half of the "Get Message" clicks produce a response that doesn't trigger the override box so more than one is often required to bring up the dialog to override.

On TB startup and when new messages are checked for on a timed basis (biff), with a bad certificate the response code is also random, just like when clicking "Get Messages". However, unlike for the "Get Message" button click, there is no "listener" code which calls OnStopRunningUrl() to run the override dialog (or at least I can't find it) . The exception dialog for "Get Messages" button is triggered here:
https://searchfox.org/comm-central/rev/5cf5c0334387b43482d4a5ff59e38b3a2671f776/mail/base/content/mailWindowOverlay.js#3151
So something similar would be needed for "non-Get Messages" triggered connections; however, it appears that there doesn't now exist a listener and I don't know how to create one. (Ben's the expert on creating url listeners: bug 1782374.)

A partial fix for this is to just add an alert message when the overridable error code is detected in the imap code so that this message appears in a pop-up notification and in Activity Manager:
Overridable TLS error is present. Select Inbox and click "Get Messages" until the "Add Security Exception" dialog appears.
There is already a non-overridable alert message for, e.g., TLS version on server is too old:
https://searchfox.org/comm-central/rev/c44ff13924b110e8c564074002851ad0eb61bcff/mail/locales/en-US/chrome/messenger/imapMsgs.properties#148

However, even with this alert added, it will only appear after a TLS overridable error is reported by low-level mozilla code. This may take several biff intervals to occur since randomly just a normal TCP disconnected error may be reported instead.

I also have a fix for regression cause by bug 1768121 that I'll submit.

This describes a work around for if you never see the exception dialog after clicking "Get Messages".

One other thing I noticed is that there appears to be another way to override a bad certificate but it doesn't work. Specifically I'm referring to:
Setting | Privacy & Security | Manage Certificates... | Servers | Add Exception...
You should be able to enter your configured imap server name and port and click "Get Certificate" and override it. However, when I entered mine, like this: wally.dbnet.lan:993 it just says no info and I see no network activity in wireshark.
I found that for this to work you have to add a new item in the TB Config Editor (Settings | General | Config Editor...). In the box at the top enter network.security.ports.banned.override, choose the "String" radio button and click +. Then enter the port number for imap TLS, 993 and click the blue "Save" button. Then probably restart TB.
Now when you enter your configured server name and port, e.g., wally.dbnet.lan:993 and click "Get Certificate", TB will successfully download the certificate and allow you to set the exception (permanent or temporary).

The only problem here is that this only works for security TLS (port 993) and not for STARTTLS (port 143). So you will need to switch your imap security from STARTTLS to SSL/TLS. (I think most servers that support STARTTLS also support TLS and is the preferred protocol.) If I include 143 in the comma separated list of network.security.ports.banned.override ports, e.g., 993,143 I see the connection made to the server port 143 but imap command STARTTLS is never sent to begin the TLS handshake so it fails.

Re: https://stackoverflow.com/questions/63947262/thunderbird-78-how-to-add-security-exception
Also after writing this I see that this issue has been mention in some other bug reports.

(In reply to gene smith from comment #48)

I never understood exactly why that is necessary to click "Get Messages" to bring up the exception box, but it does seem to be.

This was not always the case, as I said before it was a regression somewhere between TB 78 and 91 (maybe bug 1590474) that has still not yet been fixed... (see bug 1743877/bug 1761946)

Also, usually you have to click it more than once before the box pops up.

Thanks thoroughly debugging it. I have not seen this issue, a single click has always worked for me, but it could be a recent regression sometime after TB 103.

A partial fix for this is to just add an alert message when the overridable error code is detected in the imap code so that this message appears in a pop-up notification and in Activity Manager:

I would obviously prefer the real fix, but this partial fix would be a significant improvement. I have around a dozen e-mail accounts setup in TB and since it currently silently fails for any certificate issues, I did not notice for a couple of weeks that one of those accounts was not working. Any kind of error message would would have been extremely helpful (see bug 1681960).

However, even with this alert added, it will only appear after a TLS overridable error is reported by low-level mozilla code.

This is very concerning, could you please file a separate bug with Mozilla, so they can fix it.

Having a cert override dialog pop up in any occasion would be bad for security. Very easy to just click-through, and also disruptive. I think it would be ok to add for initial startup though. This is a dialog you should really ever see once if ever, and not again.

(In reply to Teal Dulcet [:tdulcet] from comment #50)

(In reply to gene smith from comment #48)

I never understood exactly why that is necessary to click "Get Messages" to bring up the exception box, but it does seem to be.

This was not always the case, as I said before it was a regression somewhere between TB 78 and 91 (maybe bug 1590474) that has still not yet been fixed... (see bug 1743877/bug 1761946)

At version 68 tb just popped up the exception dialog spontaneously on startup when it saw the appropriate class of error on the initial connection attempt that indicated a bad certificate. This was handled automatically by the mozilla level code and not by TB specific code. I just tested with v68 and I didn't have to click "Get Messages". On update to 78 this feature was removed from the mozilla code and had to be handled by TB specific code. It was decided to just add the functionality to the "Get Messages" button click. Re: bug 1590474. We also had to determine which errors indicated a bad and overridable certificate to cause "Get Messages" to call the override dialog only on the appropriate error.

Also, usually you have to click it more than once before the box pops up.

Thanks thoroughly debugging it. I have not seen this issue, a single click has always worked for me, but it could be a recent regression sometime after TB 103.

Not sure why that is true now. With 68 I only see the "bad certificate" error code each time I start up or if I click "Get Messages" multiple times and cancel the override dialog without overriding. I haven't seen a normal TCP level disconnect error with 68. This is with my local dovecot server with a self-signed cert.
Also, FWIW, in your log snippet at comment 30 I seen a normal disconnect error: clearing IMAP_CONNECTION_IS_OPEN - rv = 804b0014 and not a certificate error like 805a3ff2. So I think you would see this too, eventually.

A partial fix for this is to just add an alert message when the overridable error code is detected in the imap code so that this message appears in a pop-up notification and in Activity Manager:

I would obviously prefer the real fix, but this partial fix would be a significant improvement. I have around a dozen e-mail accounts setup in TB and since it currently silently fails for any certificate issues, I did not notice for a couple of weeks that one of those accounts was not working. Any kind of error message would would have been extremely helpful (see bug 1681960).

However, even with this alert added, it will only appear after a TLS overridable error is reported by low-level mozilla code.

This is very concerning, could you please file a separate bug with Mozilla, so they can fix it.

Yes, will open a bug report on this. However, I always see the alert and always after a few clicks of "Get Messages" the exception box comes up.

(In reply to Magnus Melin [:mkmelin] from comment #51)

Having a cert override dialog pop up in any occasion would be bad for security. Very easy to just click-through, and also disruptive.

Testing with version 68, that's how it worked. Whenever an overridable cert error occurs, not just at startup, the dialog comes up.

I think it would be ok to add for initial startup though. This is a dialog you should really ever see once if ever, and not again.

The way commenter Teal is using it, his ISP has an expired certificate so he wants to only override it temporarily on each startup. So he wants to see the override dialog on each startup until the ISP fixes the problem. So, for his use-case, it's not just a one-time dialog.

For security reasons, I think the override dialog should default to temporary. Currently it defaults to permanent (the checkbox comes up checked by default). So to permanently override, it should require user action and not just a click-through as it does now.
I tried to find where the default of "checked" is set but had no luck.

Here's a "try" build for my regression fix: try build 105.0a1

If there's an overridable certificate error (e.g., self-signed), when accessing folders in the account you should see a pop-up notification and/or entries in Activity Manage that you need to select Inbox and click "Get Messages", maybe more than once, until you see the override dialog pop-up.

I'm not sure why, but if your are not on Inbox when you click "Get Messages" the "onStopRunningUrl()" never runs and that's where the override dialog is invoked.

Also, when I run the try build I'm only seeing the error codes for bad certificate. So bug 1783538 is not happening. Don't know why. So a single click on "Get Messages" is probably all that's needed.

If anyone wants to try the "try" build here's the installation links:
linux64
win64
osx

Note: If the changes here are approved for uplift to 102, the added notification string will probably need to be removed since localization (translation) is needed. It can be added back in for daily and possibly beta.

Fixes the regression caused by changes in bug 1768121 so that the bad
certificate override dialog now appears when "Get Messages" is clicked.
Also adds a notification pop-up that the certificate error is present and the
user should click "Get Messages" to activate the override dialog.

I wrote in comment 54:

Also, when I run the try build I'm only seeing the error codes for bad certificate. So bug 1783538 is not happening. Don't know why. So a single click on "Get Messages" is probably all that's needed.

Now today I tested again with the "try" build and I'm seeing bug 1783538 again. On about 1/2 of the connection attempts I'm seeing a normal disconnect error instead of a bad certificate error, so often clicking "Get Message" doesn't invoke the override dialog.

Before I re-tested with try build, I thought maybe my mozilla code was out of date. So I updated both moz and comm to latest and on my local build I still see bug 1783538.

Fixes the regression caused by changes in bug 1768121 so that the bad
certificate override dialog now appears when "Get Messages" is clicked.
This changes the previous patch to remove the added alert messages as wanted by Magnus. So there will not yet be a notification that a overridable certificate error is present until the user clicks "Get Messages" and, due to bug 1783538, clicking it may not immediately bring up the override message so multiple clicks may be needed.

Pushed by geoff@darktrojan.net:
https://hg.mozilla.org/comm-central/rev/4548b508e3d1
Regression fix for bad TLS certificate override for imap. r=mkmelin

Comment on attachment 9289095 [details]
Bug 1764770 - Regression fix for bad TLS certificate override for imap. r=mkmelin

[Approval Request Comment]
Regression caused by (bug #): bug 1768121
User impact if declined: failure to add cert override dialog for self signed
Testing completed (on c-c, etc.): c-c
Risk to taking this patch (and alternatives if risky): not too risky as mostly just affect self signed scenarios

Comment on attachment 9289095 [details]
Bug 1764770 - Regression fix for bad TLS certificate override for imap. r=mkmelin

[Triage Comment]
Approved for beta

(In reply to gene smith from comment #54)

If there's an overridable certificate error (e.g., self-signed), when accessing folders in the account you should see a pop-up notification and/or entries in Activity Manage that you need to select Inbox and click "Get Messages", maybe more than once, until you see the override dialog pop-up.

Thanks again for working on this. Sorry for my belated reply, but over the last several days I have been able to confirm that issue is fixed in the latest TB Daily 105.

However, I have had to click "Get Messages" as much as a half dozen times before the popup appears and then even after confirming the exception I still have to click "Get Messages" one more time before the new messages actually download (yet another regression). Hopefully Mozilla will be able to fix Bug 1783538 soon... It looks like they want you to run mozregression.

(In reply to Teal Dulcet [:tdulcet] from comment #61)

I still have to click "Get Messages" one more time before the new messages actually download (yet another regression).

I'll have to look into what's happening with that.

Hopefully Mozilla will be able to fix Bug 1783538 soon... It looks like they want you to run mozregression.

Yes, I was hoping their staff would know how to fix it without me having to do more work. Anyhow, thanks for the reminder.

Thunderbird 104.0b4:
https://hg.mozilla.org/releases/comm-beta/rev/8235a3cab85e

Comment on attachment 9289095 [details]
Bug 1764770 - Regression fix for bad TLS certificate override for imap. r=mkmelin

[Triage Comment]
Approved for esr102

For unknown reasons it works OK again for now. I've tried versions from 68 to latest daily and clicking "Get Message" or pressing F5 causes the exception dialog to pop up and I'm not see a tcp disconnect error in the imap log. I saw this in comment 54 but then in comment 56 a few days later it started failing again, so I don't know what is going on.
Thought may be due to weak wifi but moved out away from router and still always see exception pop-up.
So bisecting this is not possible until, or if, I find a definite version that always works and one that always fails.

...after confirming the exception I still have to click "Get Messages" one more time before the new messages actually download (yet another regression).

This doesn't seem to be a change. I see this with 68 too. I'm not bringing in new messages but just looking at existing messages. I have to move to another message and then back to the one I want to open to see it. Clicking "get msgs" or opening another folder and then going back to Inbox should also bring in new mail or it will come in on the next timed "biff" interval if it's enabled.

(In reply to gene smith from comment #65)

For unknown reasons it works OK again for now. I've tried versions from 68 to latest daily and clicking "Get Message" or pressing F5 causes the exception dialog to pop up and I'm not see a tcp disconnect error in the imap log. I saw this in comment 54 but then in comment 56 a few days later it started failing again, so I don't know what is going on.

Hmm, I am been able to consistently reproduce it in TB Daily since your fix was pushed. You have mentioned a few times that you are using a "local dovecot server" (see comment 52), so I am presuming that means it is on your same local network. The issue seems to be a race condition in Mozilla's networking implementation, so maybe your local server does have enough of a network delay to consistently trigger it. That is the only major difference I can see between our servers.

This doesn't seem to be a change. I see this with 68 too.

I do not recall this always being the case, but maybe it is only a regression for POP3 and SMTP (you have to click "Send" twice).

Thunderbird 102.2.0:
https://hg.mozilla.org/releases/comm-esr102/rev/e1d578d67881

Don't know why but it works OK with all versions up to the latest beta with my testing profile. I only see the failure (no override dialog appears on clicking "get msgs") when I use my default profile.
A difference with my testing profile is that the account with the self-signed cert is listed first. In my default profile (used for everyday email) the self-signed server is down the list and not at the top. I don't know if this is significant.

Anyhow, using my default profile, and testing with TB nightly builds, I found that up to 2022-6-13 that clicking "get messages" always brings up the exception dialog since only "bad certificate" error (0x805a3ff2) is returned by the mozilla code.
But at TB daily build 2022-6-14, only the TCP disconnected error is returned (0x804b0014) so the exception dialog never appears.
Then after that and later (2022-6-15), there appears to be a "partial fix" in that both 0x805a3ff2 and 0x804b0014 occur randomly so sometimes the override dialog appears and sometimes it doesn't on click of get-msgs.

So in conclusion, the regression occurs with tb daily build 2022-6-14:
https://archive.mozilla.org/pub/thunderbird/nightly/2022/06/2022-06-14-10-56-18-comm-central/

See bug 1783538 comment 5 for a workaround for bug 1783538.
Specifically, changing security.tls.ech.grease_probability from default of 50 to 0 fixes the need to click "Get Messages" usually more than once.
Re: https://datatracker.ietf.org/doc/html/draft-ietf-tls-grease-01
Hopefully the network and security staff at Mozilla can find a true fix.

Info for possible release note:
The patch to fix this bug (click of "Get Message" never brings up the certificate exception dialog) is present in 104.0b4 and will also appear in the next ESR 102.2.0. However, due to bug 1783538, if the user has run a nightly build or a "try" build made with nightly code from 2022-06-14 or later and then run 104.0b4 or ESR 102.2.0 with the same profile, "Get Message" will often not invoke the certificate exception dialog. The workaround is to create a new profile or, with Config Editor, just set security.tls.ech.grease_probability to 0.

Even with this bug's fix, I still see a variant of it when my mail server (running Dovecot) uses a cert signed by a CA that's not in Thunderbird's list of authorities. This is with security.tls.ech.grease_probability set to 0. I tested with today's comm-central "daily".

In my case (with logging turned on as per https://wiki.mozilla.org/MailNews:Logging#Mac_OS_X), the error is 0x805a1ff3 == SEC_ERROR_UNKNOWN_ISSUER (https://james-ross.co.uk/mozilla/misc/nserror?0x805A1FF3). This is one of the errors considered overridable here.

When I right-click on my account name (in the Folders pane) and choose "Get Messages", nothing appears to happen at all (though I see error 0x805a1ff3 when logging is turned on). There's no dialog giving me a chance to override the error -- or any dialog at all.

Edit: For what it's worth, I never get asked for my password (when the bug happens). Gene, are you asked for your password in cases when you don't see the certificate exception dialog? Or do you have your password saved by Thunderbird's password manager?

Or do you have your password saved by Thunderbird's password manager?

In my case this makes no difference. I still see the bug even when I've saved my password.

(In reply to Steven Michaud [:smichaud] (Retired) from comment #71)

...
Edit: For what it's worth, I never get asked for my password (when the bug happens). Gene, are you asked for your password in cases when you don't see the certificate exception dialog? Or do you have your password saved by Thunderbird's password manager?

When I right-click on my account name (in the Folders pane) and choose "Get Messages", nothing appears to happen at all (though I see error 0x805a1ff3 when logging is turned on). There's no dialog giving me a chance to override the error -- or any dialog at all.

Sorry for the late reply. I forced/changed the error code I see, 0x805a3ff2, to the value 0x805a1ff3 (like you are seeing) and it brings up the exception dialog but it doesn't for you? There is also "security info" stored with the URL that the exception dialog code evaluates before bringing up the actual dialog. All I can figure is that it evaluates this info as "not overridable" and doesn't bring up the dialog. I need to look closer to see what is really going on.

I don't know how to create a self-signed certificated with an "unknown issuer".

Edit: For what it's worth, I never get asked for my password (when the bug happens). Gene, are you asked for your password in cases when you don't see the certificate exception dialog? Or do you have your password saved by Thunderbird's password manager?

No, I'm not asked for a password when the exception dialog fails to appear. There should be a alert pop up and/or something appear in Activity Mgr when there is a non-overridable error. However for 0x805a1ff3 that you see, the tb imap code thinks it overridable even though the dialog code seems to think it's not. Imap code: https://searchfox.org/comm-central/rev/1de28ff4bff691f753d7b30b2f7d7fcc9dc546a4/mailnews/imap/src/nsImapProtocol.cpp#5009. Dialog code: Need to find it.

(In reply to gene smith from comment #74)

I don't know how to create a self-signed certificated with an "unknown issuer".

You can't. I use openssl commands to create a CA certificate, and more openssl commands to sign server certificates with my CA certificate. Everything works fine if I import my CA certificate into Thunderbird's list of authorities. But if I don't do that (or if I remove it) I get error 0x805a1ff3 == SEC_ERROR_UNKNOWN_ISSUER.

I've documented how I do this at bug 1777717 comment #5 and bug 1777717 comment #15.

You can't. I use openssl commands to create a CA certificate, and more openssl commands to sign server certificates with my CA certificate.

Is what you are doing considered "self signed"?

Mine works with or without my personal CA under "Authorities", i.e., I only get error 805a3ff2 which triggers the override dialog when I override temporary or delete under "Servers".
I think I used dovecot's mkcert scripts to create my CA and self-signed certs but don't remember the details. (My dovecot is just for imap testing.)

(In reply to gene smith from comment #76)

Is what you are doing considered "self signed"?

No.

Mine works with or without my personal CA under "Authorities", i.e., I only get error 805a3ff2 which triggers the override dialog when I override temporary or delete under "Servers".
I think I used dovecot's mkcert scripts to create my CA and self-signed certs but don't remember the details. (My dovecot is just for imap testing.)

A self-signed cert doesn't use (or require) a CA. Interestingly, though, all top-level CA certs are themselves self-signed -- in effect the CA signs its own cert.

So are you saying I need to setup my certs and CA the way you describe in bug 1777717 comment #5 and bug 1777717 comment #15 to duplicate the issue with dovecot?

(In reply to gene smith from comment #78)

So are you saying I need to setup my certs and CA the way you describe in bug 1777717 comment #5 and bug 1777717 comment #15 to duplicate the issue with dovecot?

Probably. I'm not sure it's worth your trouble, though. Many more people use self-signed certs than use the kind I've been using.

At some point I may try to get to the bottom of this. That's not going to be soon, though.

I removed the need info flag for me, as I never used a beta version. This bug might still be relevant or maybe not, but I applied the workaround described here https://bugzilla.mozilla.org/show_bug.cgi?id=1764770#c7 (comment #7).