It seems the Received date does not work on IMAP. I have yet to see any of my IMAP mails differ in time for Date and Received date - even junk. Don't know if it's the last or first received date we use, but for my test mail both were other than the Date, and date was still used.

According to IMAP log attached to Bug 402426 Comment #15, all header data looks to be obtained when initial header FETCH, if ENVELOPE is specified in FETCH with BODY.PEEK by setting mail.imap.use_envelope_cmd=true. (Q1) Will mail.imap.use_envelope_cmd=true be a workaround? According to Bug 390795, whole data including all mail headers seems to be obtained on initial fetch, if mail.server.serverXX.fetch_by_chunks=false is set. (Q2) Will mail.server.serverXX.fetch_by_chunks=false be a workaround? (XX is the server number for the IMAP account) Note: Above won't work on already downloaded mails, even if they are workarounds.

Correction of (Q1). Sorry for spam. mail.imap.use_envelope_cmd=true/false didn't have relation to "download Received: headers or not". So setting mail.imap.use_envelope_cmd=true can do nothing for this bug. > http://lxr.mozilla.org/seamonkey/source/mailnews/imap/src/nsImapProtocol.cpp#3001 gUseEnvelopeCmd=true/false relates to existence of ENVELOPE only. true : UID RFC822.SIZE ENVELOPE BODY.PEEK[HEADER.FIELDS (...)] FLAGS false : UID RFC822.SIZE BODY.PEEK[HEADER.FIELDS (...)] FLAGS Inclusion of additional headers in "BODY.PEEK[HEADER.FIELDS (...)" depends on other settings or conditions. Use of "UID RFC822.SIZE BODY.PEEK[HEADER] FLAGS" or "UID RFC822.SIZE RFC822.HEADER FLAGS" depends on other settings or conditions.

Sure would be nice to get this working...

Shouldn't this be marked as dupe of Bug 392774? Or the other way around?

This problem persists for this build nightly build of 3.0b3 Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b5pre) Gecko/20090504 Lightning/1.0pre Shredder/3.0b3pre The date field is incorrectly used for Received sort - and of course Date is frequently wrong.

This bug has also been reported as 514328 and 496042. In bug report 514328 I provided a mail header that produced the problem. I am using the default Gmail (IMAP) setup.

(In addition to comment #2) > Inclusion of additional headers in "BODY.PEEK[HEADER.FIELDS (...)" depends on > other settings or conditions. Message Filter's customized header is an "other settings or conditions". When a message filter rule for IMAP account has condition of "If Received contains aaa, Add Star"(enabled or disabled was irrelevant), Tb trunk displayed "2009/08/pp qq:21" at Received column for next mail, instead of "2009/08/pp qq:20" after rebuild-index of IMAP folder. (pp, qq depends on your time-zone) > Received: by 10.204.123.20 with SMTP id n20cs143091bkr; > Sat, 29 Aug 2009 09:21:20 -0700 (PDT) > Date: Sat, 29 Aug 2009 19:20:08 +0300 (EEST) It looks that adding filter rule of "If Received ..." can be a workaround.

(In reply to comment #8) > In bug report 514328 I provided a mail header that produced the problem. > I am using the default Gmail (IMAP) setup. There is no need to look into IMAP log when this bug occurs. (Why this bug occurs itself is apparent - Received: header is not obtained upon header fetch.) Torrance, can you check workaround I wrote in comment #9, with getting IMAP log? Are Received: headers obtained upon fetching of mail headers of new mail?

Maybe rkent know off hand where to make that happen ;)

@WADA I think I understood what you wanted me to do. I added a custom header category in the message filter entitled "Received", and set up the filter to search this header for "(PDT)" (which was appearing in my received headers) and to star these emails. I then sent myself a future dated email, and sure enough when it arrived the message filter ran, starred it, and it was dated correctly in the Received field. Oddly, if I tried to run the message filter on a message *already* received, it refused to recognise that the rule matched and that it should have starred the email. It would appear, then, that the failure to fetch all the header information upon fetching the email has a number of flow on effects.

(In reply to comment #12) > Oddly, if I tried to run the message filter on a message *already* received, > it refused to recognise that the rule matched and that it should have starred the email. It's Bug 184490.

I think there is a case to be made that this is really quite important functonality. Coming from Apple's Mail.app, I've come to expect mail to be ordered by the date and time my mail server receives it, not the date that the sender appends to the mail. Without this, I have mail ordered all over the place: new mail appearing way down in my inbox unnoticed, future-dated mail sitting perpetually at the top, and so on. But given that this bug was opened in 2007 (and many duplicates have been opened since), is it unlikely it will be resolved before Thunderbird 3 final?

This bug still exists in TB3 Beta 4, even after reindexing folders.

This bug exists in TB3 production. There only appears to be one "Date" column to sort on, and the date present in that column seems to be date sent rather than date received even if the hidden pref "mailnews.use_received_date" is set to true. If the workaround described in comment 9 is something that can be done as an end-user rather than by changing code and recompiling, I'd appreciate someone explaining it in a bit more detail. I couldn't quite follow it.

The column is called "Received". The workaround described in comment #9 is as follows: Tools > Message Filters > New Create a filter on the "Received" header, which you can add to the list using the "Customize" option in the list of headers. The rest of the filter doesn't really matter, and once it is created, the filter can even be disabled; it just has to exist. Do that once per IMAP account. Then you need to re-index folders. this can be done in the General Information tab of a folder's Properties dialog. I don't know if there's a way to re-index all folders in the UI.

That work around doesn't work if you view your emails in threaded view. The received field displays the correct date, yet the sorting order ignores it and appears to order by the date field anyway.

(In reply to comment #18) > yet the sorting order ignores it and appears to order by the date field anyway. "Received: header value" is currently irrelevant to threading and display order of main-thread/sub-thread/mails-in-sub-thread etc. Torrance, read bugs such as bug 383985, bug 462637, bug 533409 well, and open separate bug for your request after search B.M.O well for bugs relate to threading, please.

I opened bug 543956 to 'Always download All headers, even for folders that are *not* set to offline mode', which would apparently (if I read the comments right) fix this, if anyone is interested.

As an alternative to the workaround from Comment #17, Brian's post from 2008 suggested adding the "Received" header to the string preference "mailnews.customDBHeaders" (which is a space delimited list). http://forums.mozillazine.org/viewtopic.php?f=39&t=729645#p3788505 This way I won't have to create a fake filter per IMAP account. I have this pref in the user.js file that I use in multiple installations. But hopefully Thunderbird source will eventually get patched.

Why go through all of this hassle? Just implement bug 543956 and be done with it. It would probably silently solve a lot of other minor issues along the way...

I'm running minefield against a dovecot IMAP server and I implemented suggestions from comment #17 and comment #22, and neither has had any effect, even after rebuilding indexes (recently renamed "Repair Folder"). In my message list, the Received column and Date column are the same for every message, and I got two bad "Date:"d messages just today. The mozillazine link in comment #22 was able to get the full "Received:" text into my message pane's header display though, which does make it easier to verify that it's not working. I think that rules out dovecot as the cause of the problem, too. Just to be sure, I did an IMAP log and I see that the received date is being downloaded along with other headers (not later with the body), but it is still the Date: (or worse, the current time, for rare messages with no Date: field) that is appearing in the message list's "Received" column. I don't always use the release version of thunderbird but I just reinstalled 3.0.4 and tried to rebuild index with the headers settings and a "received:" message filter, and it's the same thing. Do you think this is still the same bug?

Charles, FWIW even with IMAP folders selected for offline use that have been fully synchronized/downloaded (and re-indexed), I still see the Received column mirroring the sent Date, unless I apply the workaround from Comment #22 or Comment #17. I have no idea why, perhaps TB (currently 3.0.4) simply does not parse extraneous headers (such as Received) into its index even if the whole message is already in the offline store? And who knows, in the case of IMAP folders *not* selected for offline use, perhaps all the headers (including Received) are in fact retrieved on the initial fetch, but again only some of the headers are processed into the index? Sorry if I'm completely off-base here and had misunderstood the earlier discussions. Regardless, a proper fix in Thunderbird source would be ideal as Torrance mentioned that the workarounds don't work for sorting in threaded view, and Jack mentioned that they don't work at all for him... P.S. I'm curious about the exact nature of the preference "mailnews.customDBHeaders" (note that there's also a "mailnews.customHeaders"). MozillaZine KB doesn't seem to have any (correct) info: http://kb.mozillazine.org/Mail_and_news_settings#Mailnews. I'm guessing that "mailnews.customDBHeaders" forces the specified headers to be included in the index, independent of the fact that they had already been downloaded/fetched from the IMAP server? P.P.S. There have been discussions elsewhere about sorting on the "Order Received" column, which AFAIK does *not* pertain to the Received header at all (and which isn't useful with moved or imported messages). For IMAP accounts, is "Order Received" based on the internal indexing property of the IMAP server, or Thunderbird's own local index?

"Date:" header field should never be displayed by any mail client other than calling it something like 'Date based on Senders Computer Clock' or 'Senders Date' The ONLY relevant date is that from the receiving mail server - period. Its odd that this bug is still being diddled on after 3 years ... its not just about imap its TB using the wrong header for the message date. If you really want to get a date/time closer to the senders time (which I believe is wrong) - the the first Received Header should be used.

(In reply to comment #25) > Charles, FWIW even with IMAP folders selected for offline use that have been > fully synchronized/downloaded (and re-indexed), I still see the Received column > mirroring the sent Date, unless I apply the workaround from Comment #22 or > Comment #17. Ok, looks like this is an actual bug then...

I have a more basic question. To populate the Received column for IMAP accounts, why does Thunderbird 3 not use the IMAP server's INTERNALDATE message attribute (instead of parsing the message's Received: header)? There's an old MozillaZine KB from Jan 2008 describing this IMAP quirk, which apparently has not been remedied: http://kb.mozillazine.org/Invalid_date_in_IMAP_messages I believe that most other clients take the "Received date" value from the server's INTERNALDATE attribute (for IMAP), or from the client's date of download (for POP). But Thunderbird takes the "Received date" value from the message's Received: header (for both IMAP and POP), in much the same way that the "Sent date" value is traditionally taken from the message's Date: header. TB's IMAP behavior is the one I'm questioning, as querying the INTERNALDATE attribute would seem to be more straightforward and less expensive? Should this be filed as a separate report? Searching Bugzilla for "INTERNALDATE" right now only turned up two non-relevant IMAP bugs: https://bugzilla.mozilla.org/buglist.cgi?query_format=specific&order=relevance+desc&bug_status=__all__&product=&content=internaldate For example, one of them mentions that detaching attachments from an IMAP message also changes that message's internal date to the current date -- based on the fact that modifying a message in IMAP has to be done by creating a new, modified copy of the original message. With the bug fix (see Bug 521867 Comment #8), the IMAP APPEND command used to create the message copy now sets the INTERNALDATE parameter (to the same date as the original message).

(In reply to comment #28) st63z, can you open separate bug(enhancement) for next? Use INTERNALDATE attr for Received column value of IMAP mail

and file that bug on me, thx. I agree that INTERNALDATE is a lot cheaper.

Whoops I didn't see the already-resolved Bug 166254 (with 278 comments!), which is listed as being blocked by this bug. It dealt with adding the Received column and the concept of "Received date" in general, but some of the earlier comments from 2004 also discussed the use of IMAP INTERNALDATE and POP download date (see Bug 166254 Comment #11 - #17). (On a side note, Bug 166254 Comment #158 also relates to Comment #18 on this page...) Anyways there's also Bug 272065 (marked as a dupe of 166254) that's more specific to INTERNALDATE. It may be a good bug to re-open, except that it was filed before the Received column was added, and thus the part of the bug summary requesting the addition of that column is no longer relevant. So as requested, I've opened a new enhancement request for IMAP INTERNALDATE: Bug 570355 (although I need help with assigning it to David).

I just discovered that when I reset/unset this: mail.imap.use_envelope_cmd = true Received date started working for me again after a restart and repair folder. Back at the time of comment #2, perhaps, it "does nothing" but as of now, it has the opposite effect at least for me on my dovecot server. I don't even know what use_envelope_cmd does or how/why I set it, so I don't know whether it should be a second bug.

use_envelope_cmd makes us use the IMAP envelope command to fetch the envelope headers (From To Cc Bcc Subject Date Message-ID). We stopped using it by default because servers often had issues with it so it was more trouble than it was worth.

(In reply to comment #33) > use_envelope_cmd makes us use the IMAP envelope command to fetch the envelope > headers (From To Cc Bcc Subject Date Message-ID). We stopped using it by > default because servers often had issues with it so it was more trouble than it > was worth. With TB 3.1.7 (Windows & Linux versions), use_envelope_cmd : - is not present in the config editor ; - defaults to TRUE ; - needs to be explicitly set to FALSE in order to report the correct Received date.

For the record, the last/bottommost Received: header is most likely to be faked, since MTAs will prefix Received: headers, not append. Best to use the topmost/first Received: header, since that would be affixed by the MTA that is also the mailbox host. Followed the instructions in Comment #17 (create new filter on custom header 'Received', 'contains', empty field, action is to stop the filter, and disable the filter after saving it). Didn't see a "reindex" on the properties of my IMAP folder, so used the "Repair Folder" button instead. Spam now correctly listed as received "last night at 10 minutes after midnight," instead of "3 in the afternoon four months from now."

(In reply to a_geek from comment #40) > Problem still exists in TB 6.0.1 (Linux/i386). Please don't post unless specifically asked to re-confirm a bug or if the steps to repeat have changed, or the bug has fixed itself. It doesn't help readability of the bug for people who may come to help fix it.

FWIW, this is _still_ a problem in TB 8.0. But my point here is really, that people should not have to go through these kinds of gyrations to get something as basic as sorting by Received date working. And they only know about these gyrations if they start hunting around for why this does not work as anyone and everyone would expect it should. Quite simply, if I add the Received column to my message summary pane, the change to actually fetch the received headers (i.e. the comment #22 workaround) should just be automatic.

Perhaps a more accurate re-stating of the problem is that it is difficult to tell whether "Received" column is going to work or not, or why. I second the notion that when "Received" is displayed, tbird should set the appropriate preferences so that the header can actually be retrieved. Failing that, part of the problem here is that "Date" is substituted for "Received" at all. It's just lying to the user. When thunderbird can't read a date from a "Received" line, it should leave the column blank.

Just out of curiosity... Is there any [good] reason that TB doesn't simply fetch *all* headers? I really don't see any advantage to not doing so, and there are a lot of disadvantages (see this bug)...

(In reply to Charles from comment #44) > Just out of curiosity... > > Is there any [good] reason that TB doesn't simply fetch *all* headers? I > really don't see any advantage to not doing so, and there are a lot of > disadvantages (see this bug)... By "fetching headers" I assume what you mean is to pull them into the message summary database for easy access. The issue with fetching all headers is that the current mork database essentially brings in the entire database for a folder as a memory-resident object, so adding non-essential items to that database would have a large impact on memory usage.

Hmmm... can you elaborate on what 'large' means using real numbers?

Is this a 20-years problem to workaround this by implementing a setting bool 'Use first fetch from server as Received column value' in case of IMAP acount?

(In reply to Andrey Semyonov from comment #47) > Is this a 20-years problem to workaround this by implementing a setting bool > 'Use first fetch from server as Received column value' in case of IMAP > acount? Um.. that would be worse than just copying the sent date. Also, 20 years? Don't be ridiculous! It will take at least 50 years to add this basic core feature that should have been there from day 1. (And yes, I'm using the customDBHeaders workaround)

what is happening here? getting this corrected is not a nicety that should be spanning five (5) years ... or even a few minutes. the 'received date' is just that and not something that was generated by the sender. i just received a msg delivered 13 hours after the 'date:'. it is unacceptable for email to be marked received with anything other than the date it was delivered to the MTA. -- thank you,

(In reply to Jonathan Kamens from comment #50) > https://addons.mozilla.org/en-US/thunderbird/addon/imap-received-date/ Does this do anything to fix the fact that even when the Received column contains valid Received dates, one cannot sort using them and that choose a sort on that column actually still sorts by Date:?

(In reply to Brian Murrell from comment #51) > Does this do anything to fix the fact that even when the Received column > contains valid Received dates, one cannot sort using them and that choose a > sort on that column actually still sorts by Date:? I noticed that issue last night after posting my comment. Oddly, it only occurs for me on Windows, not on Linux. I'm not sure whether that's because there's a platform difference or whether it's been fixed between TB 13, which I'm using on Windows, and the current trunk, which I'm using on Linux. I'm going to investigate that today. I'll let you know what I find out.

Oh, wait, after reviewing the comments here and testing it out a little bit, I think I understand better. When you group the thread pane by thread, it gets sorted by Date even when you've told it to sort by the Received header, but when you _don't_ group by thread, it gets sorted by Received header properly. That's bizarre.

OK, so first of all, the fact that threads are sorted by date even when Received is your primary sort column is bug 383985. Second, I can't fix that issue in an add-on because it's buried deep in the innards of the core C++ code, which I can't override in my add-on. So my add-on is a partial fix -- to get the correct value into the Received column, and to allow sorting by it when the thread pane is not threaded -- but not a complete one.

thank you, kamens, for attempting to rectify a problem that goes back seven (7) years now (see bug 166254). it is hard to believe that getting the actual received date (per MTA) into the record is such an impossible feat. i use seamonkey and must do with the TB hand-me-down. i look back over the 'debate' re this date business with incredulity. ***** the 'received' date has to be, legally, a constructive date and that is the date it is received by the recipient's mail server. ***** this date is *not* the download or display date (due to recipient vagaries) or "sent" header (due to potential sender manipulation). this also eliminates TB's "received order" threading until the recipient's MTA's actual date of receipt is used (more bugs/debates). a serious user (one concerned re message integrity, server-maintained data, and history backup away from the end-user's vulnerable terminal/pc) will utilize IMAP and this is my only concern. if mozilla would just make IMAP work the way it should, then POP3 can be what ever anyone else wants it to be. some 30 years ago, RFC 822 set out the 'received' header format and such headers [one (1) for each relaying MTA] are served sorted in a newest-to-oldest relay order (thus, by date/time). fortunately, this forum is not 30 years old or we would probably have this debate extending back that far! but, there cannot be a debate when we understand the 'constructive receipt' issue mandates that the newest MTA receipt date from the 'received' headers IS the only received date that can be used. if mozilla must refer this issue to an outside attorney to confirm what should be common understanding, sobeit. just do it or get on with modifying the code to reflect what the received date should be. -- thank you,

Johann, no one disagrees that the Received date should show when the message was actually received. There's really no need to invoke lawyers or legal jargon to convince anyone of something they already agree with. It's just a bug. There are lots of bugs in Thunderbird. I wish it were fixed as much as you do, but ranting and raving about lawyers, message integrity, constructive receipt, etc. isn't going to get it fixed any faster.

perhaps, kamens, the coders are not so all-aware as yourself and it was in this spirit that i -- as you put it -- went to 'ranting and raving' to point out what should have been self-evident years ago and why. now, as far as who might be agreeing with myself; i feel somewhat lonely given the going-on decade-long lack of a fix from the developers' side and now it seems that even some end-users are of the opinion that this 'issue' should be soft-pedaled ... or something. my pointing out that there are real legal/business reasons why the MTA-receipt date should always have been used is because such citing has never been entered into the 'debate' thus far and it should be the prime (if not only) reason why we need a date of certainty. and, we are not talking re man-months of effort to get the job done. one must locate the code and find where it says something like ReceivedDate=HeaderArray[15] and it gets changed to ReceivedDate=HeaderArray[2]. hopefully, the developer(s) have read my comments, supra, and have a better understanding of the underlying issues re the received date and are not thrown tangentially by specificity. certainly, i cannot see how derision re legitimate concerns and the presentation of valid points will constructively contribute to the furtherance of all of our goals to get this matter properly resolved. -- thank you

(In reply to aditsu from comment #57) > Just wondering, WHAT is going to get this bug fixed any faster? > Would it help if I buy some life insurance then proceed to dive into the TB > code and attempt to write a patch? Patches are always appreciated, both because they are a real, substantive contribution to fixing the issue, and because they show that the person submitting them is actually interested in helping, as opposed to just criticizing others for not doing what s/he wants (participants in this disucssion would be well advised to read https://bugzilla.mozilla.org/page.cgi?id=etiquette.html). I will warn you, however, that (a) I spent some time looking at the code yesterday, and it became clear quite quickly that it is _not_ very straightforward to fix any of the issues that are involved here; (b) just replacing Date with Received in calculations everywhere (e.g., thread sorting) is not the right answer, and a patch that does that is unlikely to be accepted -- users need to be given the choice of which to use, and in particular the right solution with thread sorting, as detailed in bug 383985, is to check whether the user has clicked on the Date or Received column for sorting, and use the selected column when grouping by threads; and (c) patches without automated test cases are almost never accepted. You may be able to sweet-talk someone on the TB dev team to write a test case for you once you've met them halfway by implementing a patch, but that's by no means a sure thing, and they're certainly under no obligation to do so.

three (3) months ago (#49) i succinctly stated that this fix was long over-due. since the mozilla point-person has not been heard from for almost exactly two (2) years (now), i realized that my chiming in was unlikely to elicit much of a developer response. but, what i said needed to be said. my history-to-date with antecedent software has spanned the 'net' from text==>mosaic==>netscape communicator/navigator==>mozilla suite. mozilla suite eventually spawned (phoenix==>firebird==>) FF, TB, and SM. i have not investigated mozilla's funding, but i truly doubt that the $4+ billion AOL plopped down to netscape's founders filtered down to the mozilla non-profit in a huge way. thus, one's patience has been rewarded with products that have far out-paced MS in standards compliance and user utility while operating on what to MS is a less-than shoe-string budget. so please understand that my recent commenting is not borne of impatience (oh well, maybe a little); but of wonderment that this mis-application of the many email standards has not been brought into line. then, as though to restore my faith, a few days ago this thread was awakened from the dead by a user who took the initiative to develop a work-around for this problem. i complimented this person (#55) for the yeoman effort, but this turned out to prove the rule that no good deed goes unpunished. i had to note (as did another user in #52) that the threading still had problems from the way mozilla sources the IMAP 'INTERNALDATE'. i tried to illustrate that the recipient MTA's 'Received:' date is the only one that matters and that my experience (from forensics and auditing of financial systems) proves that there is only this very date that is important. most -- if not all -- SMB's do not backup their entire email traffic. one has to make do with only the server logs: if you are lucky. of course, the way things are now, the email 'Date:' field can bear no resemblance to the mailserver's time-stamp in its logs. please, let us forget the "to-ing" and fro-ing" and suggested reading and get to the heart of the problem. RFC (822==>2822==>) 3051 says the following re IMAP (v4) 'trace' info: > 45) Clarify that the ENVELOPE, INTERNALDATE, RFC822*, BODY*, and UID > for a message is static. (i.e., immutable) AND RFC 2060 re IMAP (v4rev1) 'INTERNALDATE' states: > The internal date and time of the message on the server. This is not > the date and time in the [RFC-822] header, but rather a date and time > which reflects when the message was received. In the case of > messages delivered via [SMTP], this SHOULD be the date and time of > final delivery of the message as defined by [SMTP]. RFC (821==>2821==>) 5321 for SMTP states re what happens when the last MTA takes the message off the net (there can be several 'relay' servers between origination and end points) and internalizes it: > When an SMTP server receives a message for delivery or further > processing, it MUST insert trace ("time stamp" or "Received") > information at the beginning of the message content, as discussed in > Section 4.1.1.4. ... ('Section 4.1.1.4' is just a formatting spec) AND > An Internet mail program MUST NOT change or delete a Received: line > that was previously added to the message header section. SMTP > servers MUST prepend Received lines to messages; they MUST NOT change > the order of existing lines or insert Received lines in any other > location. ... AND > When the delivery SMTP server makes the "final delivery" of a > message, it inserts a return-path line at the beginning of the mail > data. now, what do we end up with at this point. the first line (record, if you will) of the email is the 'return-path' and the second line of the email is the last "Received:" line. this second line is broken into two (2) fields: the 'from'/'by', FS=';', and date/time. the date/time is in two (2) sub-fields: alpha day-of-the-week, FS=',', and an RFC-compliant date and time (i.e., Thu, 19 May 2011 14:51:08 +0200). if the time zone is -0000 it indicates UTC and if it is +0000 it indicates the london, UK, time zone. you may have noticed that a lot of email clients are set to '+0000 UTC'. this is wrong and the RFC's state using "UTC" instead of a real time zone is a 'SHOULD NOT'. after all of this, then, what is the real problem. here is a real-world example: > From: Kleinhans <@gmx.de> > Date: 2012 Apr 17 11:25 > To: "Johann v. Preußen" <@2secure.us> > Return-Path: <@gmx.de> > Delivered-To: @2secure.us > Received: from (mailout-de.gmx.net [213.165.64.23]) by ("mail.4ssl.us MTA") with > SMTP id 5086880A0190 for <@2secure.us>; Tue, 17 Apr 2012 11:32:37 -0700 (PDT) > Received: from mobile-198-228-222-049.mycingular.net (EHLO [10.8.99.246]) by > mail.gmx.net (mp031) with SMTP; 17 Apr 2012 20:25:54 +0200 TB mixes up the email records a bit, but you will notice that the 'Date:' reflects a time of '11:25' which is localized for me to PDT from my mailserver in germany (+0200 which is 20:25 in germany and 11:25 in PDT). however, the mail was actually "received" by the mail.4ssl.us MTA at 11:32 which is seven (7) minutes later than the 'Date:' field's time. this time differential can be quite large and can be significant in some business situations. i may be unusual, but when i see "received" i want to know when it got here not when it left somewhere else. RECAP (did you think you would ever get here?!): if you open up your 'all-headers' view you will see near the top of the "header" listing the first 'Received:' line which has the time that your server actually received the email. this may not be the date/time you see in the 'Date:' record. this is what all the hoopla is all about. the date we need is right there in the second field of the second raw email record. it just needs to be used for the 'Date:' ('INTERNALDATE' in IMAP) and everything would be fine. i cannot really imagine a user who is more interested in the time that something was supposedly sent from the point of origin when that can be spoofed; much earlier than the received time for a large, metered mailing list; and/or just plain delayed somewhere in the path to the user's server. the way things are now, the threading can be all jumbled as well and this is certainly not optimal ... either. i know this post is far, far more than most readers want to wade through; but it is -- at least -- a definitive explanation of the scope of the problem and a roadmap to the solution. unfortunately, i am not much of a programmer (maintenance and scripts at the best); but i hope that somehow this matter can be straightened out by somebody with the talent for this task. -- thank you

Some additional comments, since I've looked at this bug in connection with the review of Jonathan Kamens's addon (https://addons.mozilla.org/en-US/thunderbird/addon/imap-received-date/) That addon simply does what is suggested in comment 22. Jonathan, you might consider creating a bug that adds the Received header to customDBHeaders in core, and ask bienvenu to review it. Only he would know the possible impacts on IMAP performance to know the possible regressions from that. I also don't know if there is a specific IMAP command that can get the information, rather than have to download the entire Received header all of the time.

Adding Received to customDBHeaders is a hack, not the correct fix for this bug. The correct fix for this bug is to retrieve the IMAP INTERNALDATE message attributes and display them as the Received column.

> Adding Received to customDBHeaders is a hack, not the correct fix for this bug. Are you just trying to remind us that additional work would still be required, or would you even go so far as to say that it is NOT better than nothing? The default behavior, until one thing or the other happens, is to fail and lie to the user about it.

(In reply to Jack Eidsness from comment #63) > Are you just trying to remind us that additional work would still be > required, or would you even go so far as to say that it is NOT better than > nothing? The default behavior, until one thing or the other happens, is to > fail and lie to the user about it. I am saying that given that (a) there is an easy workaround for users who care about this, i.e., edit customDBHeaders or let my add-on do it for you; (b) TB and SM developers have limited time to work on bug fixes; (c) checking a change into the source tree requires test cases and overhead that an add-on does not; and (d) changes like this often have unanticipated side effects; I think the developers' time would be better served by working on fixing the bug properly than by shepherding a workaround through the release process. If the developers see things differently, they are of course free to check in the workaround proposed by Kent in comment 61.

The workaround is only easy for people who understand what is happening, and know how to find this bug and read past the outdated, ineffective fixes (which are also out in various forums). The "Received" field is not an advanced feature for geeks. There is no way of knowing how many people see a false "Received" date and assume it is accurate, and perhaps even apologize for something that is not their fault (ignoring or overlooking an email). The default behavior on IMAP is far worse than some imperfect code.

(In reply to Jonathan Kamens from comment #62) > The correct fix for this bug is to retrieve the IMAP INTERNALDATE > message attributes and display them as the Received column. As seen in phenomenon of Bug 693353 Comment #15, Tb requests "INTERNALDATE obtained from Date: header" at least upon "append by Detatch/Delete operation". In this case, if INTERNALDATE is used as Received column value, the column value becomes wrong. And, if malformed Date: header, result may be unpredictable. Such case is edge case, but should be cared. As far as definition of "Received column value in Tb" is "timestamp in last added Received: header", it should always be obtained from Received: header which will never be altered. To use INTERNALDATE as Received column value, definition of "Received column value" should be carefully changed.

(In reply to WADA from comment #66) > (In reply to Jonathan Kamens from comment #62) > > The correct fix for this bug is to retrieve the IMAP INTERNALDATE > > message attributes and display them as the Received column. > > As seen in phenomenon of Bug 693353 Comment #15, Tb requests "INTERNALDATE > obtained from Date: header" at least upon "append by Detatch/Delete > operation". > In this case, if INTERNALDATE is used as Received column value, the column > value becomes wrong. And, if malformed Date: header, result may be > unpredictable. > Such case is edge case, but should be cared. This is an interesting edge case, but it is not clear to me that your solution to it, i.e., using the Received header rather than the internal date, is correct. It is also not clear to me if TB is to blame for an IMAP server refusing to accept an internal date setting legitimately specified by a client when saving a message into a mailbox. > As far as definition of "Received column value in Tb" is "timestamp in last > added Received: header", it should always be obtained from Received: header > which will never be altered. I beg to differ. I just deleted an attachment from an IMAP message in Outlook, and Outlook preserved the internal date on the redeposited message, but stripped all the Received headers. Perhaps Thunderbird should fetch both the most recent Received header and the internal date and use whichever one is older. (1/2 kidding, 1/2 serious)

(In reply to Jonathan Kamens from comment #67) > I beg to differ. I just deleted an attachment from an IMAP message in > Outlook, and Outlook preserved the internal date on the redeposited message, > but stripped all the Received headers. Outlook's Detach/Delete sounds "Edit As New in Tb with removal of attachment" + "append with original INTERNALDATE". If so, I think following can be an acceptable solution. (1) Definition change of "Received column value". Received: header if local mail folder(POP3), INTERNALDATE if IMAP. (2) If IMAP, always use INTERNALDATE only. (3) Column header like "Received(InernalDate)" for IMAP folder, to avoid unwanted user's complaints on different value from his expectation. Note: When upload of .eml file or mail copy to IMAP folder, Tb currently doesn't request specific INTERNALDATE value in append command, then INTERNALDATE is set to "upload time" by server. So, I think this kind of "protection from unwanted bug open" is needed. (4) Upon mail copy/move from IMAP to local, don't copy "Received value". Set it from last added Received: header according to definition.

I installed Thunderbird 13.0.1 today and immediately ran into an issue where a number of my messages failed to display the correct received date/time, despite it being in the header (both Date and Received were just a time stamp, no date). I followed the instructions in Comment 22 to create a functionless filter on the Received header, and this does seem to have worked around the issue. Rebuilding now shows the full date as expected. I understand there is some debate here about the proper way to handle imap dates in the core, and doing a proper fix that accomodates threading et al is probably very involved from a coding perspective, but I'm curious why the Received header workaround can't be implemented in the short term in a line or two? Without looking at the code, I assume that creating the filter is somehow telling Thunderbird to fetch or store this header when otherwise it wouldn't. The GUI and message view clearly know what to do with it once it's been processed. Doesn't that then imply that somewhere in the code, someone just has to add "Received" to the list of headers to always be fetched or stored? Even if it doesn't fix the entire problem, it is apparently sufficient to at least partially resolve a glaring bug in the product, so why not start there?

My bug 647689 seems to be a newer duplicate of this.

I ran into the same issue, and found that the mail server even didn't return *Received* field. Could Thunderbird handle this? Or, does anyone have a workaround for this kind of mail server? Thanks. -511707392[7f16d91542c0]: d7759800:imap.xxxxxxxxxx.com:S-INBOX:SendData: 19 UID fetch 1427856759:1427856761 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type Reply-To Received)])^M -511707392[7f16d91542c0]: ReadNextLine [stream=d7780c10 nb=210 needmore=0] -511707392[7f16d91542c0]: d7759800:imap.xxxxxxxxxx.com:S-INBOX:CreateNewLineFromSocket: * 1 FETCH (UID 1427856759 FLAGS (\Seen) RFC822.SIZE 2234 BODY[HEADER.FIELDS (FROM TO CC BCC SUBJECT DATE MESSAGE-ID PRIORITY X-PRIORITY REFERENCES NEWSGROUPS IN-REPLY-TO CONTENT-TYPE REPLY-TO RECEIVED)] {233}^M -511707392[7f16d91542c0]: d7759800:imap.xxxxxxxxxx.com:S-INBOX:STREAM:OPEN Size: 2234: Begin Message Download Stream -511707392[7f16d91542c0]: ReadNextLine [stream=d7780c10 nb=58 needmore=0] -511707392[7f16d91542c0]: d7759800:imap.xxxxxxxxxx.com:S-INBOX:CreateNewLineFromSocket: From: thunderbird-test <thunderbird-test@xxxxxxxxxx.com>^M -511707392[7f16d91542c0]: ReadNextLine [stream=d7780c10 nb=56 needmore=0] -511707392[7f16d91542c0]: d7759800:imap.xxxxxxxxxx.com:S-INBOX:CreateNewLineFromSocket: To: thunderbird-test <thunderbird-test@xxxxxxxxxx.com>^M -511707392[7f16d91542c0]: ReadNextLine [stream=d7780c10 nb=94 needmore=0] -511707392[7f16d91542c0]: d7759800:imap.xxxxxxxxxx.com:S-INBOX:CreateNewLineFromSocket: Message-Id: <AIwAmADRAEWpDq7GHv0Y8aqI.1.1427856876678.Hmail.thunderbird-test@xxxxxxxxxx.com>^M -511707392[7f16d91542c0]: ReadNextLine [stream=d7780c10 nb=25 needmore=0] -511707392[7f16d91542c0]: d7759800:imap.xxxxxxxxxx.com:S-INBOX:CreateNewLineFromSocket: Subject: test1 at 10:54^M -511707392[7f16d91542c0]: ReadNextLine [stream=d7780c10 nb=3 needmore=0] -511707392[7f16d91542c0]: d7759800:imap.xxxxxxxxxx.com:S-INBOX:CreateNewLineFromSocket: )^M -511707392[7f16d91542c0]: d7759800:imap.xxxxxxxxxx.com:S-INBOX:STREAM:CLOSE: Normal Message End Download Stream -511707392[7f16d91542c0]: ReadNextLine [stream=d7780c10 nb=210 needmore=0]

i do not recognize what server you have here, but it appears to be doing what is required. please see your posting indicating 'Date' and 'Received' as being processed: 'BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type Reply-To Received)]' notice within the parens the sixth field is 'Date' and the last field is 'Received'. in the raw email the first line is the 'Return-Path' and the second is the 'Received' added by your MTA. all of this discussion is re whether the MTA 'Received' date/time is correctly reflected in 'Date'. if you expand the headers for this email, i am certain you will see your server's 'Received' line right under 'Return-Path'. these headers will be listed several lines down in TB's header display. there will likely be several additional 'Received' lines beneath the first that indicate when prior relays processed the email and the first (furthest down) is from the originating server. you might be interested in using the the "Order Received' column sorting rather than the 'Date'. if you sort down you will see the newest at the top. if you use threading, the order reverses within the thread from the oldest to newest. of course, the first of the thread (the oldest in the thread) is in the full listing in its proper order. -- thank you

Hi, I was referred to this bug 402594 thread by a comment in the bug 1183490 thread. Bug 1183490 is a bug in the Order Received field that makes sorting a folder by Order Received unreliable. Someone there suggested sorting by Received instead of by Order Received. Someone else advised reading about the 402594 bug before deciding to sort by Received. I've read or skimmed the comments in the 402594 thread, plus the description in Jonathan Kamen's add-on that partially mitigates 402594. If I've understood correctly, there are actually two 402594 bugs: (1) Thunderbird doesn't fetch messages' Received header field from IMAP servers, and (2) when a user sorts by Received, Thunderbird actually sorts the folder by Date (date sent). JK's add-on mitigates the former (by somehow getting Thunderbird to pay attention to the values in the Received fields) but doesn't solve Thunderbird's inability to sort by Received. It's troubling that the two bugs in 402594 persist; it was reported in 2007. According to comments posted here over the years, other email clients don't have the issue, which strongly suggests it shouldn't be that hard to fix. Furthermore, it's a major bug; it's very important for users to be able to reliably sort by date, and at present Thunderbird has no way to do so. Sorting by Date is unreliable because it can be spoofed by the sender. Sorting by Received fails utterly due to the 402584 bugs. Sorting by Order Received is unreliable because of bug 1183490. (And even if 1183490 is fixed, Order Received would still be quirky and misleading because moving a message from one folder to another causes Thunderbird to treat it as if it were newly received. In other words, it doesn't really mean "order received"; it means "order added to this folder.") In addition to pleading for the bugs to be fixed, I also propose revising the misleading or ambiguous or vague labels in Thunderbird's field picker (and in the column headers when the fields are selected). "Date" should say "Date Sent" to be less ambiguous. "Received" should say "Date Received" to be less vague. "Order Received" should say "Order Added to This Folder" to be less misleading.

(In reply to seppley from comment #76) > If I've > understood correctly, there are actually two 402594 bugs: (1) Thunderbird > doesn't fetch messages' Received header field from IMAP servers, and (2) > when a user sorts by Received, Thunderbird actually sorts the folder by Date > (date sent). In my understanding, by default TB doesn't fetch the Received header, but still gives you the option to show the Received date and sort by it. When you try to use it, TB misleadingly gives you the data from the (sent) Date header instead, without any warning. > JK's add-on mitigates the former (by somehow getting Thunderbird to pay > attention to the values in the Received fields) but doesn't solve > Thunderbird's inability to sort by Received. There is a workaround that solves all the problems, without any add-on, already mentioned in other comments: go to the config editor, and add/modify a preference called "mailnews.customDBHeaders" with the value "Received". You will likely need to restart TB, and also "repair folder" if you already tried to use the Received header and got the sent date instead. That will give you the real received date in the Received column, and you can sort by it correctly. > it doesn't really mean "order received"; it means "order added to this folder." Yes, that seems to be the case (regarding "Order Received"). It's the order in which it was last added.

(In reply to aditsu from comment #77) > In my understanding, by default TB doesn't fetch the Received header, Correct, hence my 5 1/2 year old bug 543956 to just always download all of the headers all of the time. Downloading partial headers in my opinion is just plain silly, and is the cause of numerous bugs like this...

Replying to Charles comment 78: -snip- > Downloading partial headers in my opinion is just plain silly, and is the > cause of numerous bugs like this... It isn't necessary to download all headers to solve bugs like this. They can presumably be fixed by downloading & storing only the fields available in the column field picker menu that Thunderbird isn't already downloading & storing. In particular, downloading & storing the Received field should allow 402594 to be easily fixed. Downloading & storing all header fields seems like a waste of storage space. At minimum, Thunderbird should download (or construct if necessary, such as Order Received) and store the data that users can pick with the column field picker (and thus can sort on). Storing less than the fields in the picker must surely be buggy.

Replying to aditsu comment 77) > (In reply to seppley from comment #76) >> If I've >> understood correctly, there are actually two 402594 bugs: (1) Thunderbird >> doesn't fetch messages' Received header field from IMAP servers, and (2) >> when a user sorts by Received, Thunderbird actually sorts the folder by Date >> (date sent). > > In my understanding, by default TB doesn't fetch the Received header, but > still gives you the option to show the Received date and sort by it. When > you try to use it, TB misleadingly gives you the data from the (sent) Date > header instead, without any warning. That description appears equivalent to what I wrote about 402594 being two bugs: (1) fails to fetch the Received datum, and (2) sorts by Date (Sent) when user selects sort by Received. -snip- > There is a workaround that solves all the problems, without any add-on, > already mentioned in other comments: go to the config editor, and add/modify > a preference called "mailnews.customDBHeaders" with the value "Received". > You will likely need to restart TB, and also "repair folder" if you already > tried to use the Received header and got the sent date instead. That will > give you the real received date in the Received column, and you can sort by > it correctly. That workaround (first suggested in comment 22) was reported not to work in comment 24, without an additional step described later in comments 32 & 38. However, comment 33 said that additional step could cause "more problems than it's worth." Is there a consensus now (mid-2015) about whether the additional step should be performed too? Also, if it can still cause trouble, how can a user determine whether it will cause too much trouble in his/her case?

I've an additional thought about Charles' suggestion to download all headers... Comment 45 said the reason not to download all headersis that it would greatly increase the amount of memory (RAM, presumably) used by Thunderbird because the entire database is--or was, back in 2011--stored in memory. However, Charles didn't explicitly propose *storing* all the header fields. So, why not download all headers as Charles suggests, then store only the data needed to fix bugs, so that the database won't be significantly larger? I'm assuming the only way to fetch at least one of the fields needed to fix at least one bug is to download all headers. In other words, I'm assuming there is no alternative that would both provide the missing needed field(s) and be more efficient.

(In reply to seppley from comment #81) > That description appears equivalent to what I wrote about 402594 being > two bugs Yeah I guess you can say so. > That workaround (first suggested in comment 22) was reported not to work in > comment 24, without an additional step described later in comments 32 & 38. It works for me, without any additional step. I don't know why it didn't work in some cases, could be a different issue in old TB, user error, imap server issue... who knows. Does it work for you?

(In reply to seppley from comment #80) > It isn't necessary to download all headers to solve bugs like this. I didn't say it was - but it certainly would fix it. Now, maybe address my main point... wht NOT download all headers, all the time? > Downloading & storing all header fields seems like a waste of storage space. Ridiculous notion in the modern age. Yes, maybe way back in the day of 20GB hard drives... but today such an argument just doesn't hold water. > At minimum, Thunderbird should download (or construct if necessary, such as > Order Received) and store the data that users can pick with the column field > picker (and thus can sort on). Storing less than the fields in the picker > must surely be buggy. And what if I want to use a custom header for a filter action that isn't displayed/picked via the column picker? Not to mention such code is open to even more bugs... Again, just download ALL the header sand be done with it.

Replying to aditsu comment 83: -snip- >> That workaround (first suggested in comment 22) was reported not to work in >> comment 24, without an additional step described later in comments 32 & 38. > > It works for me, without any additional step. > I don't know why it didn't work in some cases, could be a different issue in > old TB, user error, imap server issue... who knows. Does it work for you? I don't know yet whether the comment 22 workaround is working for me. I did the workaround after aditsu asked, but not the "additional step" of comments 32 & 38, nor the "repair folder" step. I'm reluctant to repair a folder unnecessarily because I read a few weeks ago that repair is buggy and can lose messages. Is there a quick way to test whether the workaround is causing Thunderbird to sort on Received correctly? I don't know how to send (to myself) an email with a spoofed Date Sent.

(In reply to seppley from comment #86) > I'm reluctant to repair a folder > unnecessarily because I read a few weeks ago that repair is buggy and can > lose messages. Aaaaaargh!?!? Can you elaborate on this? Maybe it has to do with possibly losing power during the repair? Please tell me it isn't about just random bugginess where repairing a folder could silently lose data?

Replying to Charles comment 87: > (In reply to seppley from comment #86) >> I'm reluctant to repair a folder unnecessarily because I read >> a few weeks ago that repair is buggy and can lose messages. > > Aaaaaargh!?!? > > Can you elaborate on this? Maybe it has to do with possibly losing power > during the repair? Please tell me it isn't about just random bugginess where > repairing a folder could silently lose data? I recall it being described as a bug, and that power loss wasn't the cause. To try to find it again, I googled: thunderbird repair folder bug server delete OR lose Firefox shows the top google result is one I've already visited: https://support.mozilla.org/questions/987918 Looking at that webpage again, it looks familiar. I think that's where I read about the possible bug in repair. Some of the other Google results look scary too, though. For example: http://codeverge.com/mozilla.support.thunderbird/repair-folder-deleted-my-inbox-ma/1961101

BASIC Q&A: what does a USER expect: that the 'Date' is the "received" date free of any timing ambiguities created at the source of the email what do some USER's incrementally want: to be able to see when it is claimed that the email was sent what does the TB system require to answer those requests: two (2) DB/UI columns: 'Date Received' and 'Date Sent' how does the system get these dates: both dates are provided by POP3/IMAP headers IAW numerous RFC's i applaud @aditsu for proposing the use of the hidden 'prefs.js' solution by setting the 'mailnews.customDBHeaders' to the string 'Received'. hopefully, this route is a persistent answer for power users, developers, and IT administrators in lieu of a coding solution that seems to have escaped so many developers. of course, that proc leaves the average TB USER out-in-the-cold. thus, what the vast majority of TB/SM clients deserves is not the covert use of hidden preferences with skimpy or non-existent doc's. the USER deserves a completely documented and transparent patch bringing the DB/UI into RFC compliance and normal expectations. also, with many estimates of email volume putting the spam/scam/intrusion segment in the very high double-digits, there is absolutely no reason that the sent date and everything that could be wrong with it is given preference over a known commodity: the received date. in an informal survey i did last year of almost 50 users and two (2) each POP3/IMAP administrators, the farthest out they thought the date might go was to the interface of the receiving MTA. obviously, USER ignorance has prevented this bug from being clogged with comments. and, if the truth would be known, i would not even mess with the sent date. tell the USER to open up the headers and take a look if they feel the need. at the same time, educate them as to why the sent date is unreliable and may be indicative of security issues with the sender.

You're all anxious to have this bug fixed - that's understood. But this isn't a forum, so please do not comment here unless you are adding technical information or actively working on fixing the bug. If you just have an strong interest please make it known with vote or cc - but anything else is spam. TIA.

(In reply to seppley from comment #88) > Replying to Charles comment 85: > -snip- > > Now, I'm wondering just how much more RAM it would consume? If you're > > talking the difference between 10MB and 20MB, then I don't see that as a > > legitimate argument in the modern age either. > > Someone else would need to answer that; I'm just a naive user who doesn't > normally need to pay attention to obscure details in message headers. It's > a shame that Kent James, who authored comment 45 claiming RAM would be an > issue, didn't answer your question when you asked it in comment 46. Those concerns are serious, in part beause memory usage is only one factor. There is also time to read the file from disk and CPU load for processing the contents. And the size in memory is larger than the size on disk. These are all major factors in Thunderbird overall performance - affecting more than just folder operations. This can be mitigated by not putting ALL downloaded header information into the folder database, as I believe someone else has written > I'll take a wild guess. Suppose the header data that Thunderbird doesn't > already store averages 256 bytes per message, and suppose Thunderbird needs > to be able to store the data for 100,000 messages. (Mine is already storing > about 25,000 messages, and I doubt I'd be considered a high volume user.) > Given these assumptions, it would cost approximately 25 MBytes. correct, that's not an especially high volume folder > Another relevant question is whether Thunderbird still keeps the entire > database in RAM. yes it does, the data coming from the msf file that is. Something that might be improved in the future, but not something we can resolve in this bug :)

Replying to seppley comment 86: > Replying to aditsu comment 83: -snip- >> Does it work for you? > > I don't know yet whether the comment 22 workaround is working for me. I did > the workaround after aditsu asked, but not the "additional step" of comments > 32 & 38, nor the "repair folder" step. -snip- It's at least partially working. An email received after applying the workaround has a Received date that displays as a minute later than Date sent. (Thunderbird truncates the seconds in the columns, so for most messages the two dates display identical values.) I don't know yet whether it's sorting properly by Received; the order is the same as it would be if sorted by Date. To test this, I need to wait until I receive an email with a Date Sent that's significantly wrong.