User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0 Build ID: 20170306192303 Steps to reproduce: The server (IMAP) has the email file (please see attached...) and yes it is SPAM. I think it has bad malformed header.... Actual results: 1. Thunderbird can NOT get this file. 2. Thunderbird does NOT mark this email as read. 3. In Thunderbird, the account name is always highlighted in blue (new email). When I log in to the same IMAP account via android gmail app, I can see the email (or part of it). Therefore I can delete it. AFTER that, the Thunderbird does NOT indicate(not blue) that there is a new email in the box. Expected results: The bad/malformed email should at least be available for deletion....

Sorry, forgot to attach the raw email file....

I can confirm that this malformed message is not well tolerated by TB. You can import it into a local folder, but in an IMAP folder it's not visible at all. I can't confirm point 3 since I manually moved the message, but I have no reason to doubt the report.

Gene, any spare cycles for this one?

Sorry, I remember seeing this but then forgot about it. I took the attached file and "dragged" it from my file manager to a local folder on tb. I see the email on the folder's list in tb but only a date shows (no subject etc). I can then copy it to an imap folder and see the same thing. I *can* delete it at local and imap. However, when I open it all I see is the raw text of the message. So not sure I am doing this right. Not sure how this email appeared in tb to the reporter when it was actually received. Is he saying he can't see it at all except there is an indication of an unread phantom email? Jorg, maybe I don't understand how to "import" this file into thunderbird so it appears as a newly received.

The easiest way to reproduce this is via an SMTP session. Please see examples below... Notice the EMPTY LINE(<CR><LF>) right after the “354 End data with <CR><LF>.<CR><LF>”, or before the “Subject: ” line in example 2. It looks like if the “Subject: “ header field is NOT IN the message HEADER or MISSING, Thunderbird can not “access/pull” it from an IMAP server. However according to RFC 2822 the ‘Subject: ’ header field is “syntactically optional.” EXAMPLE 1: telnet yourserver.tld 25 Trying 123.456.78.90... Connected to yourserver.tld. Escape character is '^]'. 220 mail.yourserver.tld ESMTP EHLO someserver.tld 250-mail.yourserver.tld 250-PIPELINING 250-SIZE 10240000 250-ETRN 250-STARTTLS 250-AUTH PLAIN LOGIN 250-AUTH=PLAIN LOGIN 250-ENHANCEDSTATUSCODES 250-8BITMIME 250 DSN MAIL FROM: <someuser@someserver.tld> 250 2.1.0 Ok RCPT TO: someuser@yourserver.tld 250 2.1.5 Ok DATA 354 End data with <CR><LF>.<CR><LF> Subject: kfjgkfdjkjgfdg dfgfdgfdgfdgfdg dfgfdgfdgfd fdgdfgffd . 250 2.0.0 Ok: queued as E9BEF15A67D QUIT 221 2.0.0 Bye Connection closed by foreign host. ----- END EXAMPLE 1 ----- EXAMPLE 2: (DATA ONLY) … 250 2.1.5 Ok DATA 354 End data with <CR><LF>.<CR><LF> X-MAILER: test test Subject: test test test test . 250 2.0.0 Ok: queued as 47D2915A67D quit 221 2.0.0 Bye Connection closed by foreign host. ----- END EXAMPLE 2 ----- Thanks for looking in to this...

Correction: This behavior is not triggered if the "Date: " header field is present.

I left off the Subject: and I could still see the received message. What is the significance of the blank lines before the subject that you mention? Also, I didn't put in a Date: but apparently the smtp server added it since I see it in the received email source. I shutdown tb and removed the date: line from the ....ImapMail/<account>/INBOX file and restarted tb but still see email but with no time received now, even when offline. What am I missing?

(In reply to gene smith from comment #8) > I left off the Subject: and I could still see the received message. What is > the significance of the blank lines before the subject that you mention? > Also, I didn't put in a Date: but apparently the smtp server added it since > I see it in the received email source. I shutdown tb and removed the date: > line from the ....ImapMail/<account>/INBOX file and restarted tb but still > see email but with no time received now, even when offline. What am I > missing? Since your SMTP server adds the “Date: ” field automatically you will not be able to reproduce it via an SMTP session. Please see the new attachment. When I drag and drop it into my IMAP inbox, I can NOT see it in Thunderbird, but I can see it via an other email client or when I ssh into the server. Thanks

I don't think it has anything to do with the date or subject lines. It seems to be the blank line between the top header items and the rest. Here is the top of your original bad message: Return-Path: <2979-102-16829025-159-serge=theblacklistnyc.com@mail.planwtrs.xyz> X-Original-To: serge@theblacklistnyc.com Delivered-To: serge@theblacklistnyc.com Received: from app.planwtrs.xyz (unknown [23.95.29.137]) by mail.theblacklistnyc.com (Postfix) with ESMTP id 55C1215A67D for <serge@theblacklistnyc.com>; Tue, 7 Mar 2017 06:15:57 -0500 (EST) Authentication-Results: mail.theblacklistnyc.com; domainkeys=permerror (bad format) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; s=k1; d=planwtrs.xyz; h=Content-Type:Date:From:Reply-To:Subject:To:Message-ID; i=IncreaseYourPenileSensitivity@planwtrs.xyz; bh=Ks9hIEKuNjAorHBlW0iQmAY65Ow=; b=yBRECiI9nkV/jAP6RtN6lAVNHisNSg5gteUbtsIcaQGjMg6qqpowTbmg5n7OFXd51vu4ZmVJbl1x /+XdHNHo54VniBNUI+yybHz0kjMk94E1IUa9UWBPABrU2ImEvJYe1wAzNA0YglCGE8z477j54hSG MX1V/6GBWrCRi+LNHkw= DomainKey-Signature: a=rsa-sha1; c=nofws; q=dns; s=k1; d=planwtrs.xyz; b=hOzv0dEyLP+Qr+3KNy2VMxvj8EuU9MJQx3+GK1xEaEnGOP6+8XSgH83uxqdUmqOQ1KUxi1K9pnJQ +gpoX1RFUYmYitaWRmKQjMHCbq5H6TtG2UMdS6YWqx/5AYBpZhwGPMY5Zbr8LjA/keYFrDNchX6j Y7ncICKv5Raq38ygfQk=; <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<-- tb dislikes this blank line! Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="deafcd07a32153ee8c3c82ccde7a3009" Date: Tue, 7 Mar 2017 06:06:06 -0500 : : If I take out the blank line the email displays fine and I can open it with File/Open saved message and see the spam email, not just the raw text. If I rename the file to a .eml I can drag it to an online imap folder and it shows up fine there too.

I made the same correction to the attached file in bug: 1388518 (removed blank line after the top header on each email) and they all display the expected (spam) content in an online imap folder. So is this blank line actually legal in emails? Even if not, it seem tb should list something for the email.

Oops, the link is: bug 1388518. (Wish you could edit these!)

How did you save the original attached file raw_email.txt? Did you save it from a file thunderbird creates (like .../ImapMail/<account>/INBOX) or did you save it with another app? At this time I am don't know if tb actually keeps the entire received email or just the top header part up to the blank line and throws away the rest.

(In reply to Serioga from comment #9) > Please see the new attachment. When I drag and drop it into my IMAP inbox, I > can NOT see it in Thunderbird, but I can see it via an other email client or > when I ssh into the server. > Somethings changed or I just missed this earlier. Now I can drag and drop both of your bad emails to a tb imap mbox and they do appear. However, only the current time (no date) appears on the summary line. When I click on one that is real spam, I see the raw text starting right after the 1st blank line: Mime-Version: 1.0 Content-Type ... : : I also see the messages on another tb instance on another device so it is on the imap server too. Of course, if I remove the 1st blank line, the spam message appears correctly with the proper subject date, from and email body. But don't know what happens when the email is actually sent via smtp and tb fetches the bad email from the imap server. I wouldn't think it would matter. Can you try to send me a malformed email so I can see what happens? I've seen a similar thing when downloading (sync'ing) a large mbox via a somewhat slow/error prone wireless link. Sometimes only part of a message was received and only the current time showed for the partial email that was actually from years ago.

Just to summarize Both of the test messages have badly malformed header. The “Date: “ and “Subject: “ header fields are missing. Servers set-up: IMAP (maildir) + SMTP (configured NOT add missing headers automatically). Under these conditions TB can not fetch the malformed message from the server. Upon further investigation, per RFC2822 Sec 3.6 https://tools.ietf.org/html/rfc2822#section-3.6 : The only required header fields are the origination date field and the originator address field(s). All other header fields are syntactically optional. Therefore it is really the job for an MTA(SMTP server) to either add the missing headers or to drop the malformed messages. TB is probably doing the right thing and not fetching the bad messages. However, it would be nice to be able to delete these emails, [but TB would have to fetch them first :) ] Considering all of the above, feel free to close this bug. Thanks. P.S. Gene, if I send you a malformed email, your SMTP server will add the missing headers as per Comment 8 P.P.S. Re: Comment 12, the first blank line separates headers from the body https://tools.ietf.org/html/rfc2822#section-2.1

(In reply to Serioga from comment #16) > Just to summarize > > Both of the test messages have badly malformed header. The “Date: “ and > “Subject: “ header fields are missing. That is definitely true for you short email attachment 8938550 [details] and technically true for the spam email attachment 8844674 [details] since it does contain what looks like a Data and Subject header but they are after the 1st blank line so they are not seen as headers but as part of the email body. > > Servers set-up: > IMAP (maildir) + SMTP (configured NOT add missing headers automatically). Is your thunderbird configured to user maildir format and not the default mbox format? You are connecting to a standard IMAP remote server or possibly your IMAP server is running locally, e.g., dovecot? > > Under these conditions TB can not fetch the malformed message from the > server. Why can I fetch them? On machine A I can drag the malformed messages to a tb IMAP folder. The messages appear on machine A. Then I go to machine B that accesses the same IMAP account with tb and see the malformed messages on it too. On both machines only the current time appears in the summary but I can look at and even delete the malformed messages. I haven't checked using tb imap logging or wireshark, but I expect the act of dragging to the imap folder causes tb to do an imap APPEND operation. An SMTP server is not involved so not missing headers are being added back in. > > Upon further investigation, per RFC2822 Sec 3.6 > https://tools.ietf.org/html/rfc2822#section-3.6 : > > The only required header fields are the origination date field and > the originator address field(s). All other header fields are > syntactically optional. > > Therefore it is really the job for an MTA(SMTP server) to either add the > missing headers or to drop the malformed messages. TB is probably doing the > right thing and not fetching the bad messages. However, it would be nice to > be able to delete these emails, [but TB would have to fetch them first :) ] > > Considering all of the above, feel free to close this bug. Was it also observed that the invisible messages(s) caused the new message indication (blue highlight) and the unread count to increase on the folder? > > Thanks. > > P.S. Gene, if I send you a malformed email, your SMTP server will add the > missing headers as per Comment 8 If you send me a message I don't think my SMTP server will be involved at all, just my IMAP server. ?? > P.P.S. Re: Comment 12, the first blank line separates headers from the body Yes, definitely true. When tb parses the email and sees the 1st blank line it assumes everything past it is *not* headers. > https://tools.ietf.org/html/rfc2822#section-2.1

(In reply to gene smith from comment #17) > Is your thunderbird configured to user maildir format and not the default > mbox format? You are connecting to a standard IMAP remote server or possibly > your IMAP server is running locally, e.g., dovecot? Dovecot IMAP (maildir) on a remote host. > Why can I fetch them? Not sure. Maybe because your IMAP server is using mbox > Was it also observed that the invisible messages(s) caused the new message > indication (blue highlight) and the unread count to increase on the folder? YES. > If you send me a message I don't think my SMTP server will be involved at > all, just my IMAP server. ?? I just tried, check your mail. If the message source has a "Date:" header than your SMPT server added it automatically: ~# host -t MX chartertn.net chartertn.net mail is handled by 10 mx1.charter.net. ~# telnet mx1.charter.net 25 Trying 68.114.188.69... Connected to mx1.charter.net. Escape character is '^]'. 220 impin001 charter.net ?? ESMTP server ready 20171222 203312 EHLO sergem.org 250-impin001 hello [66.228.40.106], ?? pleased to meet you 250-HELP 250-SIZE 50000000 250-ENHANCEDSTATUSCODES 250-8BITMIME 250 OK MAIL FROM: <serge@sergem.org> 250 2.1.0 pEZC1w00t2HRgCq01EZqKg <serge@sergem.org> sender ok RCPT TO: <gds@chartertn.net> 250 2.1.5 pEZC1w00t2HRgCq01EZqKg <gds@chartertn.net> recipient ok DATA 354 enter mail, end with "." on a line by itself This message has no headers. Re: BUG 1345266 . 250 2.0.0 pEZC1w00t2HRgCq01EZqKg Message received: 20171223023603.JQHI3837.mtain005.msg.strl.va.charter.net@impin001 E0000 QUIT 221 2.0.0 impin001 charter.net closing connection Connection closed by foreign host.

Yes, it does. Right after the message-id and before the blank line. By "my SMTP server" I assume you mean the charter server that is receiving your email?

(In reply to gene smith from comment #19) > Yes, it does. Right after the message-id and before the blank line. By "my > SMTP server" I assume you mean the charter server that is receiving your > email? Yes, the charter server added the "Date:" and "Message-ID:" header fields automatically, therefor the message is visible to your Thunderbird.

So is there an "popular" imap/smtp server that doesn't add the missing date? I have several other accounts. I can use (gmail, yahoo, aol, outlook, dovecot)

Steps to replicate using a gmail account: 1. Sign into your gmail account using Thunderbird. 2. Drag and drop the second test email (raw_txt.eml) into the "Inbox" folder IN Thunderbird 2a. The email is not going to be visible in Thunderbird. 3. Log into the gmail account in a browser 3a. You should see the email

You're right. I noticed this earlier where I dragged raw_txt.eml into tb/gmail and it didn't show. But if I drag to tb/charter first and then drag from tb/charter to tb/gmail it does show in tb/gmail. But in both cases it is visible in webmail/gmail. Also, in webmail/gmail, doing "show original" you can see the raw email. In both cases there is no date header and is unchanged from raw_txt.eml contents.

I see what's happening. TB wants to see at least one of a set of headers: From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type Reply-To If none are present it doesn't even try to fetch the whole email via imap. When I drag the bad email to tb/charter, charter adds a Message-ID so the email is visible in tb. If I drag it to tb/yahoo, nothing is added so tb gets an empty response and doesn't fetch the email body at all and acts like it doesn't exist (yet I see it in android-Blue Mail app). I suspect a similar thing is happening with gmail (when I drag the email from tb/charter to tb/gmail the message ID is present so it shows up in tb/gmail). I haven't checked the tb code to see what it is actually trying to do.

This is just a proposed patch since additional comments and testing are needed. Both yahoo and gmail, and probably others (*), don't add one of the expected header fields when the email is appended to a mailbox (via O/S drag/drop or tb copy/move), so the email never appears in the summary for tb, but does show up in some other email apps and on webmail for gmail and yahoo. Probably the same thing happens when yahoo, gmail and other SMTP servers receive emails when all of the header fields that tb expects are missing. Note: With this change, if at least one of the expected fields is present, the email is accessible in tb. This fixes the problem by not considering a zero length header a bad header and not printing the warning message "Header text should always end in a newline" and not returning an error from ParseHeaders() when just the header length is 0. (*) Some do add an expected field. E.g., charters add a Message_ID so the email appears with this proposed change.

Correction: with->without, charters->Charter (*) Some do add an expected field. E.g., Charter add a Message_ID so the email appears without this proposed change.

Comment on attachment 8938779 [details] [diff] [review] nsParseMailbox.cpp.diff This works well. I'd give r+ if there weren't the "This is just a proposed patch since additional comments and testing are needed". Please let me know what else you want to do with this apart from removing the debug.

I have removed the printf's and made an official patch. Should be OK.

Comment on attachment 8939349 [details] [diff] [review] 1345266-review-changes0.patch Thanks again for the investigation leading to this small change!

Pushed by mozilla@jorgk.com: https://hg.mozilla.org/comm-central/rev/cecc2f11dcca Allow access to messages with no expected header. r=jorgk

Beta (TB 58): https://hg.mozilla.org/releases/comm-beta/rev/ebe246a8d5f2

TB 52.6 ESR: https://hg.mozilla.org/releases/comm-esr52/rev/db2e4ca7c51302f77b0d2fc3cd69ac1855c5042b