User Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:53.0) Gecko/20100101 Firefox/53.0 Build ID: 20170413192749 Steps to reproduce: On a new installation, old account. first time download from an IMAP folder Actual results: some of the downloades mails get timestamped with the download time Expected results: if first time header is not found. it should look at a secondary time/date header

I am running Daily 55.0a1 (2017-04-27) (32-bit) on Windows and I can confirm that when an email lacks "Date:" in its header, it is timestamped with current time and date. Though it simply may be sender's client fault for not including this parameter.

(In reply to Kire Dyfvelsten from comment #0) > if first time header is not found. it should look at a secondary time/date > header What's a "secondary" time/date header? For IMAP mail, the Date and Received are taken from the Date: header of the message. It's different for POP.

i am not a programmer. so i dont know anything about the delivery methods but inside the mail there is a date (when viewing the source) i do not have these mail (deleted) so i cant send an example :( most of these mails are from mailrobots

We are still migrating to Thunderbird in our organisations so i can get an example later..

Hmm, no Date: header, and we're not digging out a date from the Received: headers.

this is the header of an mail that is not sent from a robot. and it still get timestaped with the download date/time

a screenshoot showing the result on an normal mail that get download time timestamped

Gene, do you have time to look into the IMAP time-stamping. I sadly don't have any spare cycles for this.

I think this might be what I saw here: https://bugzilla.mozilla.org/show_bug.cgi?id=1348762#c54 see item (1) in the list. What I saw again with item 1 was one or sometimes two random emails from the past would be listed with the current time (but no date) but when you look at the email, it seems to have the correct time and date shown. I think that is what attachment "Till Kire_2.png" is showing too. If you go into where tb stores the email folder (usually 2 files) and delete them and let tb re-download from the server it may work OK this time, or other random emails might have the problem. I only saw this on fairly large folders, inbox and send having about 4000 and 2000 messages respectively. I only saw this with a flaky wifi usb dongle so sometimes tb would (maybe) lose then restore the connection to the imap server when downloading. When I used a reliable ethernet connection to my router I never saw this problem.

Going back to my usb wifi, I again see the same problems downloading from the imap server. And it often leaves a random email from the past with current time and no date at the top of the list. Once I even saw one with no subject, no sender and no date at the top! When I open these out of order emails, all content is there and is correct (time, date, subject, body etc all OK). Kire, are you seeing any communication interruptions or delays when you download? I only see these when I use a defective wifi adapter. What I am seeing with wireshark is tb receives and acks imap-on-tls packets from the remote imap servers. Then after a random number of headers are downloaded (e.g., 700), tb sends an ack and there is no response from imap server. Tb gets tired of waiting for more data and attempts to reset the connect but still gets no response from imap server and closes imap and attempts a tcp FIN to disconnect (several times) with no response from imap. While this is going on I can ping the imap server so communication is still possible. Tb will eventually sync back up with imap server and continue the header download. I'm not sure, but I think the corrupted emails occur during this communication glitch.

We have two very different issues here: 1st attachment 8864559 [details]: It has no Date header. In this case, the date of the download is displayed as the date. The Reporter suggests (bug description) to use the 'internal date' of the IMAP server instead. This is rather a design decision. Because the RFCs do not specify this case. (In reply to Jorg K (GMT+2) from comment #6) > Hmm, no Date: header, and we're not digging out a date from the Received: > headers. That Date is already displayed in the received-column. 2nd attachment 8864749 [details]: It has a Date header. When I import the mail, the date is displayed without problems. (In reply to gene smith from comment #10) > I think this might be what I saw here: > https://bugzilla.mozilla.org/show_bug.cgi?id=1348762#c54 see item (1) in the > list. > What I saw again with item 1 was one or sometimes two random emails from the > past would be listed with the current time (but no date) but when you look > at the email, it seems to have the correct time and date shown. I think that This could also explain the case of the reporter. This is some sort of timing problem. In this case, the correct date, and not the IMAP date, should be displayed. So we should create a new bug for this.

Took a while to see a badly listed email today so I could look at headers. Today I have an email from 2009 with "Date" and "Subject" headers OK in "view source"; but, in the list, I see empty/blank subject and only current time. (I didn't see the case where Date header is not present at all in source like reporter's 1st attachment.) Anyhow, I did find a way to "fix" my badly displayed email: delete it to Trash and then move or copy from trash back to original location (Inbox, Sent etc.). When the email is moved to Trash (or probably any other folder), for me the Subject and Date in the list are restored and they still are when moved from Trash back to the original location (so the "bad" email from 2009 is now at the correct place when sorted by date). >In this case, the correct date, and not the IMAP date, should be displayed. So we should create a new bug for this. Not sure what this means. What is "correct date" and "IMAP date"? I think you are saying for the 1st attachment from the reporter that there is no "Date" header in the envelope. In this case another date from the header should be used. This is caused by the sending client not putting in the Date header and is not a random decoding error (repeats on re-download from imap server). In the 2nd and 3rd attachment (which are more like what I am seeing) something else is going wrong since the envelope has all the expected headers but they don't display in the email list correctly. For me this is completely random since re-downloading all folders from the imap server either shows no problem or a different email from the past now has the problem. Is this true for the reporter? (In reply to Alfred Peters from comment #12) > > (In reply to Jorg K (GMT+2) from comment #6) > > Hmm, no Date: header, and we're not digging out a date from the Received: > > headers. > > That Date is already displayed in the received-column. Does this mean this header? Received: from jEDMailer (unknown [192.168.0.20]) by mail-13.relationbrand.com (Postfix) with ESMTP id 432E140E3A for <christer.jonsson@thages.se>; Fri, 10 Feb 2017 07:04:44 +0100 (CET) If so, wouldn't the 10 Feb 2017 put the email at a more "correct" point in the list and not at the top? The reporter also says it's only shows the "download time" and no date displayed in the list (meaning when he downloaded from imap server, I think).

(In reply to gene smith from comment #13) > Anyhow, I did find a way to "fix" my badly displayed email: delete it to > Trash and then move or copy from trash back to original location (Inbox, Right Mouse Click on the Folder -> 'Properties' -> /General Information\ -> [Repair Folder] ...should also work. > (so the "bad" email from 2009 > is now at the correct place when sorted by date). These mails are sorted incorrectly, and have only a time display because TB uses the current date for them. The short date format is used for recent dates. > >In this case, the correct date, and not the IMAP date, should be displayed. So we should create a new bug for this. > Not sure what this means. What is "correct date" and "IMAP date"? "correct date": If the Mail has a date header that header should be used. "IMAP date": See "RFC 3501 - 2.3.3. Internal Date Message Attribute" | The internal date and time of the message on the server. This | is not the date and time in the [RFC-2822] header, but rather a | date and time which reflects when the message was received. It is possible to request this date from the server. I just do not know if we already make use of it. > I think you are saying for the 1st attachment from the reporter that there > is no "Date" header in the envelope. For clarity: do you mean 'envelope' as described by RFC 3501? We don't have access to it at display time. > In this case another date from the > header should be used. If so, we cold use the Date from the received-column. (If you eventually don't know what I mead, configure your 'Thread Pane' -> 'Select columns to display' -> 'Received') > This is caused by the sending client not putting in > the Date header and is not a random decoding error (repeats on re-download > from imap server). Yes of cause. As I said. this are two different Bugs. We should split them to not complicate the matters. > (In reply to Alfred Peters from comment #12) > > That Date is already displayed in the received-column. > > Does this mean this header? Not exactly. See above. > Received: from jEDMailer (unknown [192.168.0.20]) > by mail-13.relationbrand.com (Postfix) with ESMTP id 432E140E3A > for <christer.jonsson@thages.se>; Fri, 10 Feb 2017 07:04:44 +0100 (CET) But in deed this Time is already used in the received-column of the Thread Pane.

(In reply to gene smith from comment #11) > Kire, are you seeing any communication interruptions or delays when you > download? most connections is via 100mbit ipsec lan to lan tunnels i dont notice any communication interrupptions

Another date / time related note... one user got an mail from estonia (we are located in sweden) when he replied.. he had replayed 1h earlier than the mail was sent (timestamped in TB)

(In reply to Alfred Peters from comment #14) > (In reply to gene smith from comment #13) > > > Anyhow, I did find a way to "fix" my badly displayed email: delete it to > > Trash and then move or copy from trash back to original location (Inbox, > > Right Mouse Click on the Folder -> 'Properties' -> /General Information\ > -> [Repair Folder] > > ...should also work. Well, that sort of works. If you are only downloading headers (not full messages) this "repairs" the folder by downloading the headers again and recreating the *.msf file which seems to be about the same as deleting the *.msf file and letting tb re-download the folder. A similar thing happens when the "bad" email is moved to another folder and then back, except only the message header or the message UID is fetched and not the whole folder. Haven't tried it when full messages are downloaded, but I think it also "repairs" the *.msf by downloading all headers again. > > > (so the "bad" email from 2009 > > is now at the correct place when sorted by date). > > These mails are sorted incorrectly, and have only a time display because TB > uses the current date for them. > The short date format is used for recent dates. That's true. If downloaded at 23:55, it shows just 23:55. But after midnight it then shows yesterday's date and 23:55 too. (I assume that's what you mean by "short date" format.) > > > >In this case, the correct date, and not the IMAP date, should be displayed. So we should create a new bug for this. > > Not sure what this means. What is "correct date" and "IMAP date"? > > "correct date": If the Mail has a date header that header should be used. > > "IMAP date": See "RFC 3501 - 2.3.3. Internal Date Message Attribute" > | The internal date and time of the message on the server. This > | is not the date and time in the [RFC-2822] header, but rather a > | date and time which reflects when the message was received. > > It is possible to request this date from the server. I just do not know if > we already make use of it. Currently, the "received" headers are only obtained if "full messages" are downloaded and not just headers. They are not parsed but are only displayed when the user does "show source". > > > I think you are saying for the 1st attachment from the reporter that there > > is no "Date" header in the envelope. > > For clarity: do you mean 'envelope' as described by RFC 3501? > We don't have access to it at display time. I probably used the wrong term "envelope". What I mean is the header date that typically looks like this: Date: Tue, 03 Nov 2009 12:45:24 -0500: Tue, 03 Nov 2009 12:45:24 -0500 When headers are downloaded, this time/date (adjusted for local timezone) seems to get displayed in the "Date" column. And the same values are displayed in the "Received" column if enabled. > > > In this case another date from the > > header should be used. Currently Date: is the only date header downloaded. Maybe with imap you could also download the "received" header that would only be used in the rare case that the Date: header was missing. That would add quite a bit of redundant downloading and storage (I assume in the *.msf files), plus decoding/parsing. > > If so, we cold use the Date from the received-column. (If you eventually > don't know what I mead, configure your 'Thread Pane' > -> 'Select columns to display' -> 'Received') I tried that. It shows exactly the same date/time as the "Date" column. > > > This is caused by the sending client not putting in > > the Date header and is not a random decoding error (repeats on re-download > > from imap server). > > Yes of cause. As I said. this are two different Bugs. > We should split them to not complicate the matters. > > > (In reply to Alfred Peters from comment #12) > > > > That Date is already displayed in the received-column. > > > > Does this mean this header? > > Not exactly. See above. > > > Received: from jEDMailer (unknown [192.168.0.20]) > > by mail-13.relationbrand.com (Postfix) with ESMTP id 432E140E3A > > for <christer.jonsson@thages.se>; Fri, 10 Feb 2017 07:04:44 +0100 (CET) > > But in deed this Time is already used in the received-column of the Thread > Pane. No, the time/date from this Received: item/header never appears and is not even downloaded when only headers are stored (there can be more than one or some emails don't have these at all). It is only downloaded when full messages are stored but never used (except when "show source" is displayed). Anyhow, I would say that if the sender intentionally leaves off the Date: header the email is essentially invalid and is probably spam, junk or bulk. I think the reporter indicated these were sent by "robots" and he deleted them. So I don't see a reason to fix this by searching the email for an alternate date and displaying that. The *real* bug here is that sometimes incomplete headers are received due to network errors which I can duplicate easily with my setup. I will provide more information in a subsequent comment.

(In reply to Kire Dyfvelsten from comment #15) > (In reply to gene smith from comment #11) > > > Kire, are you seeing any communication interruptions or delays when you > > download? > > most connections is via 100mbit ipsec lan to lan tunnels > i dont notice any communication interrupptions Ok, do I understand correctly that your original report was that some emails had no Date: header? I assume that if you repeat the download by deleting the *.msf file(s) (or "repair" the folder as described above) that the email would still have no Date: header after this? I suspect Tb is not just dropping or losing the header since probably the sender didn't put it in. Also, the 2nd problem you describe (your 2nd and 3rd attachments) seem somewhat different. In this case there is a Date: header but it still didn't display at the correct place. This is what I see when I have a communication glitch while downloading. I don't think it really matters but are you downloading just headers or full messages? Also, if you retry the download (stop tb, delete the *.msf file(s), start tb, select folders) you shouldn't see the the same emails with the problem (if my theory is right). When I download by wifi, I see the the download "hang" after a random number of messages for INBOX or Sent folders (where most emails are). I can switch to another folder and that usually resume the download. If it hangs again I can switch to another folder and it again resumes. Eventually all headers in all folders download but it needs some help like this. During the hangs, if I look at my imap.log I can predict if there will be a problem. If the hang occurs with a partial header download, then a email will display in the list with a problem. E.g, if the Date: is not downloaded, an email from the past will appear at the top of the list with the Date or Received column showing the time of the hang (approximately). It is also possible, as the following attachment shows, to only have the message ID and the date. In this case the message is listed with blank subject and no recipient or sender listed, but has the correct date and shows up at the right place in the list, date wise. Sometime, the hang occurs with only messageID header downloaded. In this case only the hang date/time is shows for the message (no subject, sender, or recipient. In any case, if you show the email or list its source, it shows all fields since the whole email is downloaded on the fly at that time (but this doesn't fix the listing). I have no idea why I have problems downloading over wifi with tb. I tried another email client (evolution) on the same machine using the same wifi network interface and downloading simultaneously the same account (6000+ emails) with tb and evolution has absolutely no problem while tb hangs several times during the header download. The evolution is also set to download only headers but it seems to also download entire messages to INBOX (unless you cancel the download, then you only have mostly headers). One other observation: If I set tb to download entire messages and not just headers, I don't see hangs when it is downloading messages via wifi. The hangs only occur during header downloads (server stops responding to acks from tb).

This is my imap log where the hang using wifi occurred. This resulted in a message the the right place, date wise, but with no subject or other information listed.

(In reply to Alfred Peters from comment #12) > This could also explain the case of the reporter. > This is some sort of timing problem. > In this case, the correct date, and not the IMAP date, should be displayed. > So we should create a new bug for this. How are we going with this? A new bug has just arrived where the dates are not shown correctly in IMAP, bug 1498002.

(In reply to Jorg K (GMT+2) from comment #20) > How are we going with this? A new bug has just arrived where the dates are > not shown correctly in IMAP, bug 1498002. Been about a year since I looked at this. I will re-read (and re-learn) this. I did try the attachment with bug 1498002 and when imported to local folders it is OK (different and correct Received and Date). Import to imap folder, both the same as was reported.

In tb code I only see Received being parsed for local folders, not for imap folders. I found a workaround that fixes the problem, where imap Date and Received are the same, by setting a pref: In Config Editor Find mailnews.customDBHeaders that has a default blank value (unless maybe an extension has tweaked it). If it is blank, set it to |Received| -- without the |'s. Restart tb. Any newly received imap mail will now display the Received column correctly. For older mails to display the Received column correctly you must "repair" the imap folder so that everything is re-downloaded. (Apparently, this causes the Received: headers to now be stored in the database.) There is also an extension that supposedly works up to tb v60 that does the same thing. See http://blog.dmitryleskov.com/small-hacks/putting-the-received-column-in-thunderbird-to-work/ The extension is here: https://addons.thunderbird.net/en-US/thunderbird/addon/imap-received-date/ Not sure why tb works like this and requires this. It is more of a workaround for Bug 1498002 than this bug. Most of what I discussed above over a year ago is a different problem (corrupted headers due to network errors). I think this may also help the reporter of this bug since when he had missing Data: header the Date column defaulted to the current date/time. With this workaround it should now show the Received date/time instead.

(In reply to gene smith from comment #22) > I think this may also help the reporter of this bug since when he > had missing Date: header the Date column defaulted to the current date/time. > With this workaround it should now show the Received date/time instead. This is not quite right. If Date header is missing this work around still shows the current date/time in the Date column. But the work around will now show the actual Received date/time from the Received header in the Received column.

(In reply to gene smith from comment #22) > > Not sure why tb works like this ... With IMAP, tb can selectively download certain parts of the message and not the whole message. So the Received: headers are not downloaded for db storage by default. There can be quite a few of them (one for each hop) and they probably would occupy a lot more database storage (.msf file) for a minor benefit (probably only really need the top one?). The Date: header is downloaded by default and stored in database and is displayed as a default column and is OK for most purposes. It could be argued that if you really want to know the Received times you can look at the message source and see it. The complete message source (body), including all the Received: headers, is stored in the mbox or maildir files or kept in memory cache if no offline storage is used. So even though it is all downloaded it is not all stored in the database and not parsed much at all. So if the user attempts to show the Received column for an imap folder, tb might say "Are you sure? This will require that Received: headers be downloaded and stored in your database. Please restart and repair the folder to enable Received column". Tb could then automatically enable the pref to fetch Received: headers and maybe automatically repair the folder. Or maybe it would just be better to disable the Received column for imap folders? Or document this behavior and do nothing (my vote!)?

Not an exact duplicate but has much in common with this old bug: Bug 402594