Open Bug 538809 Opened 15 years ago Updated 2 years ago

[autoconfig] When attempting to determine what kind of SMTP support is available, Thunderbird sends EHLO greeting too quickly

Categories

(Thunderbird :: Account Manager, defect)

defect

Tracking

(blocking-thunderbird3.1 -)

Tracking Status
blocking-thunderbird3.1 --- -

People

(Reporter: wildfire, Unassigned)

References

Details

Attachments

(1 file)

User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.7) Gecko/20100106 Ubuntu/10.04 (lucid) Firefox/3.5.7 Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.8pre) Gecko/20100109 Shredder/3.0.1pre Thunderbird when setting up a new mail account will attempt to integorgate the SMTP server. However it often, at least for me, seem to default to 587/None when it should have found 587/STARTTLS. Digging into the logs, I see that my SMTP server (exim) complains because Thunderbird has not waited until it has printed its banner -- a feature of exim is that ths banner is not generated straight away. In this manner it can detect clients which are not waiting for responses and drop the connection. This is because client which do not wait for the server to respond are typically malware. Server logs: 2010-01-10 13:04:27 no host name found for IP address 78.149.55.68 2010-01-10 13:04:28 SMTP protocol synchronization error (next input sent too soon: pipelining was not advertised): rejected "EHLO" H=[78.149.55.68] next input="QUIT\r\n" 2010-01-10 13:04:29 no host name found for IP address 78.149.55.68 2010-01-10 13:04:29 SMTP protocol synchronization error (next input sent too soon: pipelining was not advertised): rejected "EHLO" H=[78.149.55.68] next input="QUIT\r\n" 2010-01-10 13:04:31 no host name found for IP address 78.149.55.68 2010-01-10 13:04:31 SMTP protocol synchronization error (next input sent too soon: pipelining was not advertised): rejected "EHLO" H=[78.149.55.68] next input="QUIT\r\n" Reproducible: Always Steps to Reproduce: 1. Setup a new account 2. Have exim as the SMTP server 3. Thunderbird will send commands too quickly, and be considered spammy Expected Results: Thunderbird should wait until the banner appears, and then integorgate the server.
Related: bug 525643 on the EHLO not having an argument for the greeting, but apparently it's not even getting to that point in your case.
Summary: When attempting to determine what kind of SMTP support is available, thunderbird is spammy → [autoconfig] When attempting to determine what kind of SMTP support is available, thunderbird is spammy
This is probably causing Thunderbird to prefer port 465 to port 587. See bug 529763
Some more related issues: 1. Thunderbird doesn't give an argument to EHLO, which is illegal. 2. Thunderbird ignores a 5xx rejection in response to EHLO (2010-01-14 11:12:51 rejected EHLO from lewes.staff.uscs.susx.ac.uk [139.184.134.43]:65437 I=[139.184.32.17]:465: syntactically invalid argument(s): (no argument given)) 3. Thunderbird doesn't attempt to verify either the email address or the password during setup!
Actually, as I mentioned in my first comment, thunderbird is correctly preferring port 587 but without encryption. It appears the algorithm is: - attempt a connection on each of '587', '465' and '25'. - if the connection to '587' works, look for the string 'STARTTLS' if not found set encryption to 'None' and end - otherwise, if the connection to '465' work, use SSL - otherwise, if the connection to '25' works, look for the string 'STARTTLS', if not found set encryption to 'None' and end. The algorithm is correct - Thunderbird just needs to wait for the SMTP server to output its banner before continuing its investigation of the preferred port.
Ian, I think you are attempting to lump together a number of issues in this bug. This bug is solely related to the fact that Thunderbird is not waiting for the SMTP server to output its banner before continuing its investigation. Those other items you mention are bugs but they are not THIS bug.
(In reply to comment #3) > 3. Thunderbird doesn't attempt to verify either the email address or the > password during setup! It is doing that for the incoming server (e.g., bug 491202 for IMAP), but apparently no verification is performed during probing of SMTP connections. Given that authentication may be different (though unlikely) for incoming and outgoing servers, an oversight. And yes, this should be filed as a separate bug (your #1 is bug 525643).
Anand, fair enough, but if they're mentioned here then someone working on one of the bugs might fix them all. I guess they're all in the same part of the code, and all contribute to the general problem - that the autoconfiguration yields incorrect results. BTW, rsx11m, Thunderbird is quite happy to set up an IMAP account with incorrect credentials for me. So, I'm going to look at 491202, too!
It's sometimes hard to tell what the right "granularity" of a bug is; it should be specific enough, but you don't want to distribute an issue across too many reports. I had added Blake to the CC list as he had worked on the IMAP part (which definitely was checking the user's credentials when I debugged that, so your last point would be worrisome), maybe he can join the discussion how to proceed with this and the related bugs.
I can, and have been meaning to, but there's some large stuff I have to take care of first, so I'm going to be a while in responding... (I did check briefly into the code, and it looks like we're tying port 587 to STARTTLS, which concerned me a bit, and led me to think that this might be larger than I have time to tackle at the moment. But rest assured, it is listed in my Things, so I won't forget about it. But I will leave it unassigned, in case someone else wants to work on it while I'm otherwise occupied. :) Thanks, Blake.
Tying port 587 to STARTTLS isn't the problem that I have. We provide port 587 with STARTTLS but Thunderbird still seems to prefer port 465 with SSL. I think the reason is that our server rejects EHLO when it isn't given an argument. This bug is about a related issue - the client should let the server speak first. The bigger issue is that the SMTP protocols are violated in the autoconfiguration setup. However, if I manually edit the setup, and re-test, the test passes. In the retest, Thunderbird says "EHLO \rQUIT", without waiting for a reply, so STARTTLS isn't tested. This is also an RFC violation, but Thunderbird doesn't seem to care that it's given a rejection. Certainly Thunderbird needs to use STARTTLS if its available, in case it's on a mobile device. Our site allows some IP addresses to connect without authentication, but we're phasing out this facility.
Confirming based on comment #9, I've also confirmed bug 525643 and leave it up to Blake (or whoever takes those bugs) whether or not to merge these issues. Apparently connections to 587 check the EHLO response for STARTTLS (which is not present if a 550 error is returned), then try 465 with SSL as a secure fallback (where the EHLO response may be ignored), so comment #10 is plausible.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Setting SMTP_CMDS = ["EHLO example.com\r\n", "QUIT\r\n"]; in mailnews/base/util/autoconfigUtils.jsm would be a quick hack that would improve matters. It would work around servers that require correct syntax, but not those that check the accuracy of the EHLO argument. That would not get around the ballistic nature of the current bug, though. A five second pause before speaking might help - most servers should not wait too long on the submission port. A better approach might be to rewrite the whole thing in C using the SMTP routines that are actually used to deliver email. They look more robust. Of course, that's a lot more work.
To add to your last paragraph, rather than trying to reinvent the wheel for the autoconfiguration part, using the MailNews Core SMTP implementation may be more feasible to open an SMTP connection with a given set of parameters, see if it goes through, and then just close it again. I agree (you may still get bitten by bug 425652 or whatever else causes issues in the Core implementation), and it should also take care of the correct handling of server greeting/response.
Flags: blocking-thunderbird3.1?
(In reply to comment #6) > (In reply to comment #3) > > 3. Thunderbird doesn't attempt to verify either the email address or the > > password during setup! > > It is doing that for the incoming server (e.g., bug 491202 for IMAP), but > apparently no verification is performed during probing of SMTP connections. > Given that authentication may be different (though unlikely) for incoming and > outgoing servers, an oversight. Are you sure that it checks authentication for IMAP? I tried to set up a new account, as Thunderbird guesses the wrong username (my server requires full email adress, not just local part), the authentication should fail. However, there is neither a red nor a green light in front of the username field. After I correct the username to reflect the full email adress and have the correct password entered, an authentication test should succeed. However, there still is no red nor green light for the username. > And yes, this should be filed as a separate bug (your #1 is bug 525643). Has this already been done? I couldn't find anything...
(In reply to comment #14) > Are you sure that it checks authentication for IMAP? I tried to set up a new > account, as Thunderbird guesses the wrong username (my server requires full > email adress, not just local part), the authentication should fail. However, Dennis, this is a seperate bug too. Yes, thunderbird should use the full email address and *then* if authentication fails, fall back to just the username. i.e. if domain is example.com and discovered imap (or pop) is imap.example.com (or pop.example.com), try: => user@{imap|pop}.example.com => user@example.com => user Which should cover the majority of authentication cases. I actually suspect that 'user@example.com' would be more common than 'user@{imap|pop}.example.com' and it may warrant being tried first. But, again, that should be a seperate bug.
(In reply to comment #15) > => user@{imap|pop}.example.com > => user@example.com > => user > > But, again, that should be a seperate bug. I created a new bug for this: Bug 540415, "Username detection should be improved".
(In reply to comment #11) > Apparently connections to 587 check the EHLO response for STARTTLS (which is > not present if a 550 error is returned), then try 465 with SSL as a secure > fallback (where the EHLO response may be ignored), so comment #10 is plausible. And this case has been filed as bug 540319, thus handled there if not solved within a more general scope in the bug here.
This bug is talking about a bunch of different issues, so it's not at all clear to me what user-facing issue is nominated to block, if any. Comment 0 appears to talk about something which would effect the small number of sysadmins who happen to parse their logs looking for unusual strings. If that behavior, by itself, were the last bug standing, we would not hold a release for it. Ludo, if you had something else in mind when nominating, feel free to explain and renominate. It appears to me from comments 9 and 11 that we're mostly waiting on bwinton to find the time to nail down what he thinks this bug is truly about.
blocking-thunderbird3.1: --- → -
Flags: blocking-thunderbird3.1?
I think #525643 is related: Both bugs refer two the account manager beeing spammy and therefore causing the account manager to fail detecting SSL settings. I vote for flagging both bugs as blocking 3.1, because I think one goal of the account wizzard is that people use SSL connections. However, if detecting the possibility to use SSL fails due to a non-RFC implementation of SMTP in Thunderbird, then - so I think - the account wizzard is useless.
Yes, not recognizing that a server supports STARTTLS because the protocol isn't correctly followed is certainly the main issue, the log entries being a symptom of it. In the worst case, if TLS/SSL isn't supported on port 465, autoconfig may fall back to an unencrypted connection. To add yet another number to the mix, there is now also bug 542256, where the presence of AUTH isn't probed for either; while in theory completely unrelated, the need there to more properly adhere to the SMTP protocol may also solve the issue with prompting the EHLO too early. Meaning, this may be solved by a solution for either bug 525643 or bug 542256, or maybe yet another bug. The main question appears to be how to coordinate the various SMTP bugs best...
Blocks: 543957
(In reply to comment #18) > This bug is talking about a bunch of different issues, Indeed - the meta issue here is that the probing done by Thunderbird is prone to error. My bug is specifically about SMTP probing causing Thunderbird to assume no encryption capability. Additional comments relate to a similar problem with IMAP as well. > so it's not at all clear > to me what user-facing issue is nominated to block, if any. Account creation - particular via the Wizard component - > Comment 0 appears > to talk about something which would effect the small number of sysadmins who > happen to parse their logs looking for unusual strings. No, Comment 0, explains that the problem is Thunderbird violates the SMTP standards (in at least 2 ways) whilst probing for what the server is capable of. The logfiles is additional debug information which I had hoped would make it clearer to the next person who came along what the precise problem was. Obviously I missed the mark a bit. > Ludo, if you had something else in mind when nominating, feel free to explain > and renominate. > > It appears to me from comments 9 and 11 that we're mostly waiting on bwinton > to find the time to nail down what he thinks this bug is truly about. Perhaps bwinton could detail their current thinking; and if someone has the time to look into it they could. I downloaded the source myself, built it, but had not got around to determining where to look before I got ... oh! shiny.
Version: unspecified → 3.0
I think this bug should be about the problem mentioned in the first comment. It sounds like Thunderbird's auto-probe should pause a little before sending the EHLO, and it sounds like not pausing is causing some mail servers to return errors, thus throwing off our probe. On the plus side, it looks like a patch could be as easy as adding a "", in front of the "EHLO\r\n", on line 59 of mailnews/base/util/autoconfigUtils.jsm. (And, of course, add a test for this behaviour.) And if someone wants to create a meta-bug to track the other bugs people have mentioned and cc me on it, I would really appreciate it. :)
(In reply to comment #22) > On the plus side, it looks like a patch could be as easy as adding a "", in > front of the "EHLO\r\n", on line 59 of mailnews/base/util/autoconfigUtils.jsm. That's still hackish though if it just introduces an arbitrary pause before sending the EHLO greeting. Or would the "send nothing" addition actually wait for a server response before moving on to the next item to send?
From my reading of the code at http://mxr.mozilla.org/comm-central/source/mailnews/base/prefs/content/accountcreation/guessConfig.js#815 and #831, it would wait for the server response before moving on to the next item to send. The tests should confirm that, though. :)
(In reply to comment #22) > I think this bug should be about the problem mentioned in the first comment. > It sounds like Thunderbird's auto-probe should pause a little before sending > the EHLO, and it sounds like not pausing is causing some mail servers to return > errors, thus throwing off our probe. No, it's NOT a question of not pausing. It's a fundamental difference between HTTP and SMTP. In HTTP the client speaks first, but in SMTP the server speaks first. Because of the spam problem, many servers are very strict about this. Furthermore, they often deliberately wait an extended period - perhaps a few seconds - before speaking. That's designed to catch spammers out, and to limit the rate at which a given spammer can spam. > > On the plus side, it looks like a patch could be as easy as adding a "", in > front of the "EHLO\r\n", on line 59 of mailnews/base/util/autoconfigUtils.jsm. > (And, of course, add a test for this behaviour.) And EHLO requires an argument. It should be the FQDN of the host, if it has one. > And if someone wants to create a meta-bug to track the other bugs people have > mentioned and cc me on it, I would really appreciate it. :)
(In reply to comment #24) > From my reading of the code at > http://mxr.mozilla.org/comm-central/source/mailnews/base/prefs/content/accountcreation/guessConfig.js#815 > and #831, it would wait for the server response before moving on to the next > item to send. Never mind the code. The document to read is RFC 5321. > The tests should confirm that, though. :)
(In reply to comment #25) > > On the plus side, it looks like a patch could be as easy as adding a "", in > > front of the "EHLO\r\n", on line 59 of mailnews/base/util/autoconfigUtils.jsm. > > (And, of course, add a test for this behaviour.) > And EHLO requires an argument. It should be the FQDN of the host, if it has > one. I think you should probably add this information to bug 525643, if it's not already there. This bug is only about waiting for the server's banner before sending our EHLO. (In reply to comment #26) > > From my reading of the code at > > http://mxr.mozilla.org/comm-central/source/mailnews/base/prefs/content/accountcreation/guessConfig.js#815 > > and #831, it would wait for the server response before moving on to the next > > item to send. > Never mind the code. The document to read is RFC 5321. I don't see how the RFC will tell us what the code is currently doing, or what the proposed change will do, which is what my comment was addressing. ;) Admittedly, I could have made that clearer, and will try to do so in the future. Thanks, Blake.
Summary: [autoconfig] When attempting to determine what kind of SMTP support is available, thunderbird is spammy → [autoconfig] When attempting to determine what kind of SMTP support is available, thunderbird sends commands too quickly
Is this still applicable after the bug 549045 / bug 549040 redesign anyway? Can someone with a sufficiently picky server re-test with 5.0 beta 2?
Summary: [autoconfig] When attempting to determine what kind of SMTP support is available, thunderbird sends commands too quickly → [autoconfig] When attempting to determine what kind of SMTP support is available, Thunderbird sends EHLO greeting too quickly
Keywords: qawanted
No, the bug isn't fixed. 549045 is a gui change, and 549040 parallelizes the tests. Neither is likely to fix the problem. Furthermore, I've just tested account creation against my server, using Thunderbird 3.1.11 for OSX 10.6.8. these are the results: 1. Thunderbird still prefers port 465, even though this is NOT a standard email port, and even though port 587 is available. 2. Thunderbird *is* now using an argument in EHLO. It's using "we-guess.mozilla.org" even though a valid FQDN is available, but that's an improvement. 3. Thunderbird is NOT waiting for the banner before sending EHLO. My server says 'SMTP protocol synchronization error (next input sent too soon: pipelining was not advertised): rejected "EHLO we-guess.mozilla.org"'
Ian, did you try 3.1.11 only? The redesign won't be visible there, you'll have to check with, e.g., 5.0 beta 2 (5.0 final is supposed to be released tomorrow). http://www.mozillamessaging.com/en-US/thunderbird/early_releases/downloads/ It is my understanding that the EHLO greeting was changed by either bug 549045 or bug 549040, at least for the lack of an argument.
Using 5.0b2pre; Thunderbird (Miramar) still does not wait for the remote to be ready. This means that Thunderbird will mis-detect whether or not STARTTLS is available (it seems to correctly cycle through the ports 587, 465, 25 though) because it is not waiting to see that is can send the EHLO -- and it is disconnected without seeing the server response. 2011-06-28 20:14:39 SMTP protocol synchronization error (next input sent too soon: pipelining was not advertised): rejected "EHLO we-guess.mozilla.org" H=83-244-244-194.cust-83.exponential-e.net [83.244.244.194] next input="QUIT\r\n" 2011-06-28 20:14:39 SMTP protocol synchronization error (next input sent too soon: pipelining was not advertised): rejected "EHLO we-guess.mozilla.org" H=83-244-244-194.cust-83.exponential-e.net [83.244.244.194] next input="QUIT\r\n"
Thanks for re-testing, so this bug unfortunately still applies.
Keywords: qawanted
OS: Linux → All
Hardware: x86 → All
Version: 3.0 → Trunk
I can confirm that this bug still exists in Thunderbird 9.0.1. Exim is a popular SMTP server and its default configuration is to check for RFC compliance by connecting clients. bug 525643 suggests that Postfix (another popular server) is also quite picky.
I can assure you that at least Postfix is just fine. I used it for all my testing.
As of version 24.2.0 this behaviour still occurs on both smtp & submission tests: postfix/postscreen[5354]: PREGREET 33 after 0.03 from [127.0.0.1]:43525: EHLO we-guess.mozilla.org\r\nQUIT\r\n postfix/submission/smtpd[5353]: improper command pipelining after EHLO from example.com[127.0.0.1]: QUIT\r\n
Thunderbird 31 still fails to detect STARTTLS! From the server log (Exim): 2014-08-07 23:16:24 SMTP protocol synchronization error (next input sent too soon: pipelining was not advertised): rejected "EHLO we-guess.mozilla.org" H=xxx [yyy] next input="QUIT\r\n"
> Thunderbird 31 still fails to detect STARTTLS Don't over-generalize. This statement is false. It works with most servers I've tested. It just doesn't work with *some* picky servers. And yes, it should be fixed. But don't say nothing works, because that's wrong.
Attached patch autoconfig-wait-for-ehlo.patch (deleted) — Splinter Review
Bug 538809 - Wait for connection greeting before sending EHLO RFC 5321 strongly recommends to wait for the connection greeting before sending the EHLO command: The communication between the sender and receiver is an alternating dialogue, controlled by the sender. As such, the sender issues a command and the receiver responds with a reply. Unless other arrangements are negotiated through service extensions, the sender MUST wait for this response before sending further commands. One important reply is the connection greeting. Normally, a receiver will send a 220 "Service ready" reply when the connection is completed. The sender SHOULD wait for this greeting message before sending any commands. Exim closes the connection if the EHLO command is sent before the connection greeting and thus auto-configuration fails. SocketUtil is simplistic and does not properly parse the protocols. SocketUtil first sends the first command, then waits for a reply, then sends the next command and so on. Adding the empty string as the first command causes SocketUtil to wait for the first reply and to send the first real command afterwards. This is certainly a hack but minimally invasively solves the problem. Ideally HostTry should properly parse the resepective protocols. This patch picks up the idea of comment #22.
Comment on attachment 8542447 [details] [diff] [review] autoconfig-wait-for-ehlo.patch Ben, could you comment on this?
Attachment #8542447 - Flags: feedback?(ben.bucksch)
Comment on attachment 8542447 [details] [diff] [review] autoconfig-wait-for-ehlo.patch Review of attachment 8542447 [details] [diff] [review]: ----------------------------------------------------------------- It's not obvious to me that this will fix the problem. It may help for some server configurations, but if you say "" to Exim, it interprets that as an unrecognised command, and returns a 550 error. If Exim -or any SMTP server- is configured to reject a connection that speaks first, then it will reject this. In conclusion, changing the configuration in this way will help, but it is probably not sufficient to close the bug.
Dear developers, It's sad to report that this 8 years old bug still exists, and it's too bad that it impacts our users. I got an issue with this TB bug: - I have Postfix postscreen service enabled, it has the Pregreet test enabled. FYI: http://www.postfix.org/POSTSCREEN_README.html#pregreet - We have Fail2ban filter to block client IP which cannot pass pregreet test. - While setting new account, TB doesn't wait for the connection greeting/banner sent by Postfix, this causes TB caught by Fail2ban and blocked. Postfix log looks like below: Jan 25 18:04:32 mail postfix/postscreen[13246]: PREGREET 33 after 0.11 from [219.80.xx.yy]:15135: EHLO we-guess.mozilla.org\r\nQUIT\r\n Please help fix it. It's been 8 years, let's make TB more RFC-compatible.

Comment on attachment 8542447 [details] [diff] [review]
autoconfig-wait-for-ehlo.patch

If this fixes the bug, I'm all for it.

Attachment #8542447 - Flags: feedback?(ben.bucksch) → feedback+

Since I've been sent here from bug#1681946 I'll repeat here: in ESMTP lingo PIPELINING feature means that IF the feature is active (offered) THEN
the client is allowed to send command one after another without waiting for the server response for each and every command.

If the PIPELINING extension is not active (or not offered) then the client MUST wait for the HELO/EHLO and MUST wait for every response after a command, or else it is an SMTP protocol violation.

And as comment#38 mentioned the client is strongly requested to wait for "service ready" instead of sending anything. It is not a protocol violation to send EHLO early but really advised against.

It is not a protocol violation to send EHLO early

So, what is the accused protocol violation here, then? Because we only send EHLO and QUIT. And you say that we're not strictly required to wait with EHLO. But we don't send anything else than that. So what is the protocol violation, then? Is it that we sent QUIT too early, or that we send EHLO too early?

In layment terms, we just say "Hello" to the server and then just leave, and the server is offended that we didn't wait for his Hello ;-P ... But: We're just saying Hello to everybody. We're not actually talking to the server. The server is offended that we greet everybody quickly.

(In reply to Ben Bucksch (:BenB) from comment #45)

It is not a protocol violation to send EHLO early

So, what is the accused protocol violation here, then?

You cannot send EHLO and not wait for the reply and send immediately any other command, neither QUIT nor anything else.

Because we only send EHLO and QUIT. And you say that we're not strictly required to wait with EHLO.

You are not required (but STRONGLY advised) to wait before sending EHLO. This is not a protocol violation, just bad practice.

But we don't send anything else than that.

False, since you send QUIT without waiting for the response to EHLO.
(As a helpful sidenote PIPELINING extension is defined in RFC2920, and the SMTP extension mode in RFC5321 section 2.2.)

(Also worths a mention that Exim is strict on purpose [and it could be also deactivated, which is definitely not advised]: most spambots spew a lot of commands onto SMTP servers to speed up spam delivery and also do not have full SMTP protocol stack implemented. This also stops some horrible implementations [some camera servers as an example] to keep desyncing connections over and over again and use up server resources.)

So what is the protocol violation, then? Is it that we sent QUIT too early,

Yes, QUIT is the violation.

or that we send EHLO too early?

That is not a violation, but bad practice.

In layment terms, we just say "Hello" to the server and then just leave, and the server is offended that we didn't wait for his Hello ;-P ... But: We're just saying Hello to everybody. We're not actually talking to the server. The server is offended that we greet everybody quickly.

I am no sure whether this was an attempt at humour or a serious question; you connect TO one server and start talk TO one server, not everybody; HELO/EHLO is not a "greeting" but a protocol synchronisation device, and the server does not get offended but rejects invalid protocol if detected.

Your connection is used to detect a server speaking a specific protocol and it's a bad idea to try to do that by violating that specific protocol, since you may interpret the error your misbehaving results from the server as a server problem, which it is not. (Also detection SHOULD :-) follow advised best practices since you intend to detect well-behaving servers in the wizard, and not just anything which barely look like an smtp server.)

Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: