'Sent date' is not a good field for ordering mail by. The problem is, if the sender has the incorrect date set, or is in a different timezone, finding the unread mail in a folder requires scrolling up and down to find it. Having a 'Received date' field as well as the 'Sent' date field would alleviate this problem entirely.

Use: View->Sort by->Order received duping to bug 72463 for the Received date field *** This bug has been marked as a duplicate of 72463 ***

verified as duplicate

reopening - bug 72463 has now been fixed, so "order received" is possible, but this bug is requesting a field for "date received", which isn't the same. (I guess this would need a back end change as well, but I'm not sure...)

-> NEW for consideration

Dupe of 123786?

possibly - depends what happens to that bug. based on the original report, bug 123786 is just a dupe of bug 72463. but further comments imply an actual received date, in which case this a dupe of that bug. either that bug should be duped to the original 72463, or that bug should be clearly changed to be a request for an actual received date field and this bug should be duped to it.

*** Bug 200955 has been marked as a duplicate of this bug. ***

IMO the "Order received" field should be replaced by a "Date received" field since the number in the order received field is completely irrelevant for most end users. If the "Date received" field gets added, the current "Date" field should be renamed to "Date sent" or at least have its tooltip changed accordingly. In order to make this work right even after moving between folders, it should probably be implemented using what is suggested in bug 190337, therefore making dependent on that bug.

FYI, the From_ header in mozilla's mbox-format mailbox files appears to already contain the date received, as mentioned in bug 190337.

(In reply to comment #8) > IMO the "Order received" field should be replaced by a "Date received" field > since the number in the order received field is completely irrelevant for most > end users. ... Completely agree to this as moving a message from folder to folder changes order (at least here using an IMAP mailbox with Cyrus mail server) which is not kept any more in the real order of incoming so this is not a workaround I could use.

I just had a look at some mail messages' sources. It is a good idea to store he date mozilla received a message. But it would be better to use the date one's _mail server_ received the mail message: Received: (qmail 25558 invoked by uid 65534); 26 Apr 2004 14:13:26 -0000 That would be a _very_ meaningful timestamp. Maybe we should have three dates stored and displayed: 1.) Sent 2.) Received (by my mail server) 3.) Downloaded

(In reply to comment #11) Completely agree. ... > > Maybe we should have three dates stored and displayed: > > 1.) Sent > 2.) Received (by my mail server) > 3.) Downloaded

IMAP specifies an "internal date" field which is the date received at your server. This seems to be what Outlook Express uses for both sorting and display, and it certainly makes the most sense...

I have done a little bit of research into how to fix this bug. While there is an IMAP to supply the date an email is received there is not one for POP. However, according to the mail RFCs the 'Received' fields must be top appended, not interfered with, and the receiving server must append a received field. This means that the date received can be reliably extracted from the topmost 'Received:' field in the email. Currently the complete 'Received:' fields string is stored in the mail database under the token kReferencesColumnName = "references". So a proposal of how to fix the bug is to add a case statement to nsMsgDBView::GetLongField for the date-received sort type where the function returns the computed date received from the references string (easy if you know how to work strings and time in mozilla, unfortunately I do not) [? using functions nsMsgHdr::ParseReferences or nsMsgHdr::GetStringReference?]. All that is then required in a few back end tweaks to open up this sort option to javascript (easy) and the associated front end tweaks (easy).

The IMAP date is not ideal because shares one of the problems with "order received" - when you move a message from one folder to another, the IMAP date of the message in the destination will be the date you moved it, not the date of the message. I think the date in the top-most Received: header is the one to go for.

(In reply to comment #15) > The IMAP date is not ideal because shares one of the problems with "order > received" - when you move a message from one folder to another, the IMAP date of > the message in the destination will be the date you moved it, not the date of > the message. RFC 2060 says: "In the case of messages delivered by the IMAP4rev1 COPY command, this SHOULD be the internal date and time of the source message." I believe Cyrus IMAP does this correctly; I'll confirm this on request if anyone has any doubt. From my experiences of the chaos caused by not restoring the internal date correctly I'm confident that it is reliable for a "Date Received" field, though as mentioned this would only work for IMAP and for POP3 the topmost Received: header would have to be parsed.

*** Bug 272065 has been marked as a duplicate of this bug. ***

I've noticed that email by some providers is missing a Date: field, in which case Thunderbird dates the email at 12/31/1969 despite the presence of several other keyable fields it could use for a more accurate date. This results in email being shoved in strange order when it's viewed. I like the idea to switch to received date, but also ensure that the date has a fallback to sort to.

this patch makes us parse the received: headers for a date, preferring the last header, which will usually be the server closest to the sender. If we fail to parse a Received hdr, we will use the date: header, as before.

(In reply to comment #19) > this patch makes us parse the received: headers for a date, preferring the last > header, which will usually be the server closest to the sender. If we fail to > parse a Received hdr, we will use the date: header, as before. I think that using the first date, which will be the server closest to the _receiver_, will be more likely to be reliable as an indicator of date received. The server closest to the receiver is likely to be a constant factor for all emails received, whereas the server closest to the sender will depend on the sender and may even be spoofed.

I can do that. That was the wrong patch, anyway.

OK, use the first received: header. And if there's no Date: or Received: hdr, use PR_Now()

(In reply to comment #22) > Created an attachment (id=199777) [edit] > proposed fix > > OK, use the first received: header. And if there's no Date: or Received: hdr, > use PR_Now() Sorry to bother you, I wish to know if this patch is going to be included in the 1.5 release of Tb, because it is likely to resolve another RFE (number 216033), which has been issued over two years ago. Thanks in advance.

If I understand correctly, this will change the "date" field shown to be the received date. From me, that would be great (this issue is pretty much the only thing stopping us from switching at work), but I suspect there will be people who will prefer the current behaviour of displaying and sorting by the sender's date and will consider this fix a backwards step. For example, a previous work email system I used collected email infrequently and added a Received header at that point, so the received date would be one of a couple of times each day, and you wouldn't be able to tell if a message had been sent in the morning of the current day or the evening of the previous day. Or I may be misunderstanding...

The original bug report asked for an _additional_ column for "Received date". Presumably the existing header should be renamed to "Sent date" to avoid confusion. This is what I'd like, anyway. IMO, "Sent date" should be taken from the RFC 2822 "Date" header, and if not present then perhaps the first (bottom-most) "Received" header. This can, of course, be spoofed. The "Received date" should be taken from the IMAP internal date if possible, and where not available should come from the last (topmost) "Received" header. Outlook Express on OS 9 does this perfectly.

(In reply to comment #24) > If I understand correctly, this will change the "date" field shown to be the > received date. From me, that would be great (this issue is pretty much the only > thing stopping us from switching at work), but I suspect there will be people > who will prefer the current behaviour of displaying and sorting by the sender's > date and will consider this fix a backwards step. > > For example, a previous work email system I used collected email infrequently > and added a Received header at that point, so the received date would be one of > a couple of times each day, and you wouldn't be able to tell if a message had > been sent in the morning of the current day or the evening of the previous day. > > Or I may be misunderstanding... IMHO I think that the current behaviour presents at least one serious drawback: if the sender has incorrect date / time set (as for a system with bad CMOS battery), that email message, for Mozilla (and also Thunderbird) is displayed as sent at the wrong date/time. This is ALWAYS reproducible.

(In reply to comment #26) > IMHO I think that the current behaviour presents at least one serious drawback: > if the sender has incorrect date / time set (as for a system with bad CMOS > battery), that email message, for Mozilla (and also Thunderbird) is displayed as > sent at the wrong date/time. This is ALWAYS reproducible. The "Date sent" is always going to actually be "The date the sender claims that the email was sent" and this is unavoidable. Therefore, if the sender's date is wrong, then this field is going to be wrong. If this is a problem, then we should have a "Date received" column, which is likely to be correct, but has shortcomings as discussed. This is exactly what this bug is about. I think both columns should be present. I don't care which the default is; that's a separate argument. Users can pick and choose which they want to use.

(In reply to comment #26) > if the sender has incorrect date / time set (as for a system with bad CMOS > battery), that email message, for Mozilla (and also Thunderbird) is displayed as > sent at the wrong date/time. This is ALWAYS reproducible. Yes, and this can be easily the case. I had the problem, that for example spammers used sent-dates in the future in their mails to be on top in my inbox. Some people might want to see the date sent also. IMHO the best is to have an additional field for that. So you either can have both visible in the list.

Ok - my fault for starting it, but please don't add any further comments to this discussion. The issues have been raised; there's no need for us all to discuss it in the bug as more comments just makes it harder for the developers to find the important information in the bug later.

David, I have some questions on your implementation/improvements. (Q1)Which has higher priority in displaying date as "Mail date" when both exist? Date: header? or Received: header? (Q2)What will be done when malformed Date: header or malformed Received: header? Same as No Date: header/No Received: header case? (Q3)Is your "new mail date" displayed in current Date field in thread pane(mail list pane)?

this only affects newly downloaded headers, or newly parsed folders (e.g., after deleting .msf file)

In what I checked in, the first Received: hdr that has a parseable date has highest priority; failing that, the date: header, and failing that, we use the download time as the date to display. We're only displaying and storing one date. And that's pretty much all I intend to do with this. If anyone else wants to do more, they're more than welcome to submit a patch.

> the first Received: hdr that has a parseable date has highest priority Urgh, I'm appalled. I didn't expect such a "solution" which also isn't what the reporter wanted. Ok, in best case date/time from Received and Date headers are only seconds appart. But I guess most users trust that the time in the date column is the one the user composed the mail. And not few users compose the mail offline to send them in a batch. In that case Received and date can differ hours and days. Also now it's no more possible to see the senders time zone without looking at the source. I promise not few people will ask about wrong dates in support forums.

Again, and what if the system time is wrong (knowingly or not)? I think that the solution proposed is equal to the "normal" behaviour of many email clients (of which Outlook Express is not the last) and I don't think so many people are complaining 'bout that. Sure, if both fields were present, it would be better.

I'm happy to back it out, or go back to my original idea of using the last received: header, which is going to be the one closest to the time the user sent the e-mail. But that was shot down...

(In reply to comment #35) > I'm happy to back it out, or go back to my original idea of using the last > received: header, which is going to be the one closest to the time the user sent > the e-mail. But that was shot down... Excuse me, then maybe I haven't understood WHICH was the final patch you proposed. I thought it was the idea of using the last received: header. Am I wrong?

what's checked in now uses the first Received: header it sees, and ignores the others. The first Received: header seen happens to be the last header added, since each server adds its own Received: header at the top. So it's the server closest to the recipient whose date gets used. Presumably, that's the most trusted server. The tradeoff is between using the most likely to be accurate time, vs using the least likely to be spoofed time.

(In reply to comment #37) > what's checked in now uses the first Received: header it sees, and ignores the > others. The first Received: header seen happens to be the last header added, > since each server adds its own Received: header at the top. So it's the server > closest to the recipient whose date gets used. Presumably, that's the most > trusted server. The tradeoff is between using the most likely to be accurate > time, vs using the least likely to be spoofed time. And so I don't understand who your comment (#35) was referred to... Can't you include all two headers in the sorting columns list, i.e. the last AND the first?

I don't want to make the UI and code more complicated than it already is by adding a second date column.

> I don't want to make the UI and code more complicated than it already is by > adding a second date column. Well, I do understand that and despite what I wrote in comment #33 I'm thankful someone works on this and provided some functionality for the way to fix this bug. But IMHO letting this code be the whole fix is wrong. I'd expected at least a pref (yes I know, another one) and a separate field (where the priorities are first Received header, now and Date header) at best. And for a Received date field using the first (last added) Received header is ok. But my position is, that everything else then the Date header's value ist wrong for a Date field.

If the Received field is used, how should the messages without that field be dealt with? (You can have those if you place a copy in the mailbox as you send a message.) Use the Date field instead?

David, thanks for taking this bug forward. I for one argue for the date from the topmost received header (as explained in comment 14, the RFC analysis being better than my then understanding of the mozilla codebase). This date is most likely to allow new mail to arrive in order when sorting by date. However, I do see a good argument for a ‘date sent’ (declared by mail client) and a ‘date received’ (topmost received header). Often large institutions have mail server issues which lead to email to be delayed before leaving the institution network. This means it is desirable to 1) use ‘date received’ to sort email, 2) use ‘date sent’ while interpreting the content of the email. Unfortunately (from what I, subsequently, inferred from the code) this entails adding an additional mail database field.

Hi all, Could we wait a couple of days, to see how the second patch looks? It just seems like the patch is being shot down by everyone. Thanks a lot for the patch David - it's sad, that it doesn't make the 1.5, but I do understand why.

(In reply to comment #39) > I don't want to make the UI and code more complicated than it already is by > adding a second date column. I can understand it, and I can explain to users about design/spec after your latest patch. But using first Received: header (last added Received: header) changes meaning of Date column from "date in Date: header" to "arrival date at POP3 server of mail receiver". This will produce new confusion on term "date", in message filtering, in junk purging, etc. (and many unwanted INVALID bugs... ;-) This should be user choosable, should be intentional change by user, I believe. David, what do you think? Can you add new pref.js entry and use it?

(In reply to comment #31) > this only affects newly downloaded headers, or newly parsed folders > (e.g., after deleting .msf file) If folder name contains special character(e.g. "/", "#"), file name for the folder becomes hexa-string(converted name). In this case, deletion of ".msf" causes loss of original folder name because the original folder name is saved in ".msf" file. David, is there any easy way in this case? User have to keep folder name and file name pair in a text file before deletion of ".msf"? User have to rename hexa-string folder name to original folder name after restart?

I agree that displaying the date from the received headers instead of the Date: header by default was a bad idea. So I'm going to do the following: 1. Use the Received: date if there's no Date: header (and only use the download time if neither header exists) 2. Possibly add a hidden pref, default to false, to use the received by header instead of the Date: header for the Date field.

(In reply to comment #46) > I agree that displaying the date from the received headers instead of the Date: > header by default was a bad idea. So I'm going to do the following: I dunno - a whole heck of a lot better than nothing! Consistently altering the date column to mean "best guess of time recieved" ought to be fine, IMHO. If people really want, adding back a "best guess of time sent" column. Frankly, I don't want my mail sorted by sent time - recieved time is far more useful, as long as the mail headers are left alone. > 1. Use the Received: date if there's no Date: header (and only use the download > time if neither header exists) If you are going to do this, then I'd use the oldest (last) Received: date, otherwise you are swapping the start and end of transit time, even though the later headers are more likely to be untrustworthy and/or broken. > 2. Possibly add a hidden pref, default to false, to use the received by header > instead of the Date: header for the Date field. OK, but a separate (default disabled) displayable/sortable column would be better, IMHO.

> > 2. Possibly add a hidden pref, default to false, to use the received by header > > instead of the Date: header for the Date field. > OK, but a separate (default disabled) displayable/sortable column would be > better, IMHO. I agree, they are two separate concepts and they should have two independent columns.

> I don't want to make the UI and code more complicated than it already is by > adding a second date column. I suggest that the "Order received" column is removed from the UI, which would be completely obsolete now. I still don't get the notion that people want the date received as close to the sender as possible - this bug was opened to resolve that issue once and for all. There are many legit reasons for keeping the date received as close to the receiver as possible (people working with deadlines, across timezones etc). If this is too much of a change then the user interface should clearly indicate which date is being displayed and sorted on - thus both date fields should be selectable from the interface and this should not be hidden at all. Again, the "order received" field can be removed now and thus this would not "net" add options, only add a clearly understandable one in place for a non-intuitive field.

If I understood well it is difficult to add one more column to UI. So as the order received field is not really needed this field could be used to store the most recent (first) received date. So we could have the original date unchanged (if used by junk mail controls etc. and the order received contains the received date which in descending order is quite the same as the order received was - and I guess nobody interested interested in that counter, but the date is more information. Is this possible? - Or are there problems because of probably different variable types?

(In reply to comment #49) > I suggest that the "Order received" column is removed from the UI, which would > be completely obsolete now. No, it is not. Order Received has its uses, obscure as they may be -- for instance, that shows the order that messages were copied into a (local) folder, regardless of the date information on the messages.

But, if we must choose only one of the two columns (I think that THIS is the main develop problem, am I wrong?), with little or no doubt the majority of people would prefer to have date received as close to the receiver as possible, again IMHO.

Yes, indeed. And I really hope that there will be a solution in the near future. It's been (at least) since 2002 that this issue is talked about. I really would like to propose Thunderbird and OpenSource to other people but such circumstances do not make it easy in a lot of cases. In my case it was completely annoying and finally I talked our network admin into finding a solution and he patched the mail server in the way that the original date sent header is overwritten for each incoming mail. But folks, that's really not the way to do it! If Outlook would behave that way, thousands of people would argue at Microsoft... - SCNR. If it is really so difficult to fix this and/or either to add a new column then there is something going rather wrong. It probably is time to rework some core things completely.

it's easy to add another column. The problem is that it's easy to add many columns, until you have a bloated UI and a bloated DB. It's a slippery slope. We could do like OE does and change the name of the column and displayed header to Received, but I'm sure there would be a percentage of people unhappy with that too.

There was already the proposal of adding an option for this. So everybody could be happy. So in my opinion - and I guess I am not alone - there is no more doubt that this change would be are really needed thing.

(In reply to comment #54) > it's easy to add another column. I really don't want to add to all the bugspam (why doesn't bugzilla have a seperate area for discussion of a bug?) but I have an idea. I don't now if this is workable, but I hope so. How about a hidden pref, say DateIsDateReceived (true/false, default false). This pref, if true, changes the Date column header to "Date Rec'd" and from then on puts the current time and date into the date field in the database, disreguarding any dates in the email headers, and doesn't change any dates already stored in the database. If the pref is false, things are basically the same as they are now, except use David's code to help with munged dates. How does that sound? Is it possible?

Code bloat is always going to be a problem with any sophisticated tool The more things we want a tool to do the more code is required, don't get me wrong, code bloat is something to be avoided. but most of what I consider code bloat is not added features but its poor programming practises. TB NEEDS a date recieved column and sort function, maybe not for every one but certainly for many of us, enough to make it an issue. a serious one! those people who don't see the need can ignore it, but for those of use who do it is a ***glaring*** ommission from TB. for my vote it should be as close to the reciever as possible. if there is no way to seperate it from the senders date then TB should add it. Is it so wrong that a client adds a header? perhaps X-DATE-RECIEVED-TB. headers are added all the time. I have no problems with TB adding a few. The sent date is also important, I often use it to determine the date on the senders computer. I have often solved a users problem (wrong date) using this field but it is easy to look at the headers and get it it does not really need to be a ui matter, unless lots of people need it. I agree with the sentiment that the order recieved column should go, not the field just the UI column. retain the field for whatever use it is currently put to.

(In reply to comment #54) > it's easy to add another column. The problem is that it's easy to add many > columns, until you have a bloated UI and a bloated DB. It's a slippery slope. We > could do like OE does and change the name of the column and displayed header to > Received, but I'm sure there would be a percentage of people unhappy with that too. I'm one in the "percentage of people unhappy" :-) How about changing "Order received" column to "Date received by your server"? As Comment #49 says, and already opened bug for removing "Order recieved" says, I think "Order recieved" is not so usefull, especially after "Global Inbox". Meaning of "Order recieved" column is very near to server arrival date/time sequence, and I think "Order" is better to be sequence of server arrival date/time, when Global Inbox, and when moved/copied mail. (I think this is also applicable to IMAP.) And changing existing colomn satisfys your comment #39, except making code complicated a little. > I don't want to make the UI and code more complicated than it already is > by adding a second date column. My prefer. (1) Change current "Order received" column to "Date received by your server". (2) Add an option to display date in last added Recieved: Header in date column when no/corrupted Date: header. (3) If option of (2) is on, display date colmn as Outlook Express does, as you say in your Comment #46. Almost all logic for (3) is already implemented, or is ready to implement, by you. An if it's optional by (2), we, who don't want MS's behaviour, will not be unhappy. And if (1) is implemented, this bug's original request is satisfied, without making people, who don't want MS's behaviour, unhappy. - "Having a 'Received date'" is bug opener's request. See comment #0. Since last added Received: header is added by POP3 sever which mail receiber is using, clock time of the server is usually proper, and I think user can afford incorrect date display due to his server misconfiguration, his server operator's error etc. If the mail server is maintained by himself, incorrect date display is caused by himself, then he can afford to it. David, what do you think?

------- Additional Comment #51 From Mike Cowperthwaite 2005-10-19 11:43 PDT (In addition to comment #58) As Mike says in comment #49, "Order Received" has its uses and is currently used, then removing "Order received" may be impossibble. But many users want Received: header column, even though developers never want increasing number of columns... How about user choosable "Order received" column content? - Add an option for data in "Order received" column : (a) Download sequence number by my PC(current one) (b) Server arrival sequence(Date/time in GMT in last added Received: header) If this option, option for GMT display or Local time display. - Save both in ".msf", for both compatibility with older Thunderbird and future enhancement. I think this is not an excellent solution, but I feel practical solution for us and practical solution for developers. David, what do you think?

Corrrection. Sorry for spam, and my misoperation in copy&paste. "As Mike says in comment #51" <== "As Mike says in comment #49"

The term "date" is underspecified - why not just make it mean "date received"? Yes, I agree that "date sent" is occasionally useful, even perhaps to sort by, but what matters most for general mail is when *I* saw it, not when the sender claimed to send it, not the order that it got filed into my various folders. Proposal: this bug: make the "date" column mean "date received". Include a tooltip defining the term "date" and make sure it gets into the FAQ. If you want to see date sent, look at the header(s). This immediately corrects the very serious TB deficiency without any visible UI impact. new bug1: add a "sent" column to UI, hidden by default. new bug2: retitle the "date" column as "received" In any case, this bug should probably be reopened...

Accordingly to what Michael Thome said, who I agree with, I think that this "alternate format" could be managed (is not importantwhich is the default) with an entry in the prefs.js, and a description in the right places (in MozillaZine, just to make an example). How does it sound?

(In reply to comment #61) > The term "date" is underspecified - why not just make it mean "date received"? It's a really bad idea to change the meaning of a part of a UI that is already well established - Thunderbird's Date column means "Date Sent" and that's the way it should stay. There is a group of users who have a strong preference for the Date column to show the date sent. I'm one of that group. I'd object to a change in the Date column display. David's understandable reluctance to add complexity to the UI notwithstanding, I'd rather see the implementation of a separate "Date Received" column. Failing that, a pref for which of date sent or date received is shown in the Date column could also be a satisfactory solution. But, please, let's not fight over which date is the "right" or "best" date to show in the Date column because it shouldn't have to be a choice between one or the other - we need both.

David, could you re-open this bug, please. Since your current patch changed meaning of "date column" from "Date sent(Date: header) to "Date received by server(Received: header)", this bug's request, "Received date field", was implemeted, then this bug was changed to FIXED. But, as you say in your comment #46, you are going to change your patch. The new patach will not be for this bug. The new patch will be for Bug 73565(and may be for Bug 32216). Because you changed this bug FIXED, and because you are going to change your ptach, I asked you some questions about how to display "Received date" in cuurently existing column with minimum changes/workload. But, as jwq says in comment #63, both "Sent date" and "Received date" are needed. I absolutely agree with jwq. I believe request of this bug, Bug 166254, should be kept.

David, thanks for quick re-open of this bug.

only used Received hdr for date column if there's no Date: header, or user sets a hidden pref.

(In reply to comment #66) > Created an attachment ...(snip) > only used received: hdr if no Date: header or user sets hidden pref The patch aparently is not solution of this bug(request of Received column, see comment #0). If the patch will be checked-in as fix for this bug, it will produce confusion in bug tracking/fix verification testing. I believe the patch should be proposed as the solution of Bug 73565.

pinging Scott for SR...

David, thanks for referencing this bug. Bug #209501 also is about reading dates. From, Received and Date all may start lines that have date info shown in mail source. But I didn't see anything in my mail source that looked like it could be converted into a date about 3 days in the future. Some msgs I get are typically only one day in the the future, and are from particular other mailing lists. Bug #209501 noted that there could be a problem in headers with punctuation in dates, the presence or absence of a comma between a dayname (Thur, for example) and the actual date. I noted that the format of dates isn't always the same. Sometimes day of the month is before month, sometimes after. Some header times say -0000, others say GMT. Parsing all that seems tricky. As noted by IRC, some spams actually are future-dated. But the mailing list msgs with future dates show up with reasonable dates and times when opened. I don't have any suggestions as to which date info should be used to form msg lists and write to the msf file, except that it seems better to make that info consistent with when the msg was sent.

Hi, I'm somewhat confused about the resolution of this bug. I belong to the group for whom "Date Received" is indispensible, and this issue is pretty much the single reason why I've been Evolution rather than Thunderbird. The last I understood (message #66), the current solution is to use "Date Sent" if the Date: field is defined, and "Date Received" otherwise unless some mysterious secret handshake is performed instead. This is horribly mixed up! I feel a bit disappointed to read that the developers oppose the identification of the two concepts. Email is an intrinsically asynchronous process. It's not reasonable in principle to assume "Date Sent" and "Date Received" are the same thing. I routinely use "Date Received" for my normal email management, but I may want to use "Date Sent" to check for a missed email which a correspondent says she sent on such-and-such a date. They can easily differ by days. Why not just have the two columns available separately and let users rearrange their own UI the way they want it?

The other big issue is that any variety of date-sent is inherently untrustworthy. It would be far better, IMHO, to use NOW (local download time) to fill-in a missing received-date than depend on any possibly bogus time - at least if your localtime is hosed, it is your own fault. To summarize: - time sent is always undependable, both in terms of existance and accuracy (intentional or not). - trustable time received should always be gettable from the first received header if you've got something close to a reasonable server. - if your server is unreasonable, then you can construct a trustable received time from download time or imap's date (if available) - order received is (a) less informative than date received and (b) wrong because imap changes it when you move folders. The "proposed fix" patch ( https://bugzilla.mozilla.org/attachment.cgi?id=199777 ) comes pretty close, though I worry that if the first received header is malformed, you might end up trusting the next (possibly spoofed) received header. Also, it would be nice if you saved it back to storage (e.g. an added/modified header) if you needed to use imap. Also also, it looks like this patch (still) just changes the semantics of the Date column - while I'd prefer this, clearly others do not. The next patch makes it possible to set a pref to determine if Date is received or a guess at sent, but leaves the default behavior of the Date column even more confused than before. In any case, no combination of these patches quite addresses the basic request to add received field parsing and column. Bottom line (for me): time received is absolutely crucial, time sent is sometimes useful. Reasonable default behavior of the mail reader must not depend on the sender's good will and correctness.

Please note bug #209501 and comment #69 in this bug. Improper punctuation in a header messes up the correct reading of a date and/or time. Also, any additional headers should not be specific to Thunderbird, Mozilla 1.7x or Seamonkey. They should be generic headers that future versions of Mozilla-family software will be able to read without kludges.

It isn't clear to me what the status is of this bug. The bug was reopened but nothing has been posted here since March. In the hope that this bug has been fixed and is in the pipeline, I just downloaded and installed the most recent nightly (well, yesterday's, anyway - Thunderbird build 2006060705). That version, which is labeled version 3.0a1 does not seem to have this fixed. There is still just a single "Date" column. It isn't clear if it is "Date Sent" or "Date Received". Some of the comments here indicate that the solution might be to change the was date is determined rather than adding this new column. If that's the accepted solution, please, please, change the column heading so it's clear what date is being used. Can someone clarify the situation? Is there going to be a "Date Received" column in Thunderbird?

Comment on attachment 205084 [details] [diff] [review] only used received: hdr if no Date: header or user sets hidden pref ping - we're getting complaints about using the received date header instead of the date header, so I'd like to get this, or something similar (perhaps not using the pref) checked in.

(In reply to comment #74) > (From update of attachment 205084 [details] [diff] [review] [edit]) > ping - we're getting complaints about using the received date header instead of > the date header, so I'd like to get this, or something similar (perhaps not > using the pref) checked in. > please note the global effect this might have on RSS feeds' date. while email sent/received may (in the non spoofed aggregate) be on the order of seconds apart, RSS feeds are fundamentally different. blogs and forum posts are only useful for the sent date, usually verified by phpbb (for example in case of forums), or other. and RSS feeds are by nature batched with half hour or more TTL; not sure how Tb would handle. identical received times would render threaded discussion reading useless. imo, two columns are required due to the importance of being clear with the meaning of Date.

Repeating David's ping from two months ago; Scott, sr is being sought here. I'm not opposed to having the pref, but I want the displayed Date header to match the Date column. and the Date header *must* be from the RFC822 Date header if that's available.

I never understood why there can't be an additional column for that, so everyone has the choice to display one or the other - or even display both...

Would any of the suggested changes affect how dates of newsgroup posts are displayed?

(In reply to comment #76) > ... > I'm not opposed to having the pref, but I want the displayed Date header to > match the Date column. and the Date header *must* be from the RFC822 Date > header if that's available. I'm curious - Why is the RFC822 date (if present) useful? Or is it merely a concern that column heading should match header lines? Given how large a percentage of the time it is wrong/misformatted/spoofed/missing, I personally would prefer to ignore it completely, as it simply not trustable.

(In reply to comment #79) > Given how large a > percentage of the time it is wrong/misformatted/spoofed/missing, I personally > would prefer to ignore it completely, as it simply not trustable. I would agree with you - but as there was discussion about it I do not see any reason why not to store all three, see: (In reply to comment #11) > [...] > Maybe we should have three dates stored and displayed: > > 1.) Sent > 2.) Received (by my mail server) > 3.) Downloaded

(In reply to comment #79) > I'm curious - Why is the RFC822 date (if present) useful? Or is it merely a > concern that column heading should match header lines? Given how large a > percentage of the time it is wrong/misformatted/spoofed/missing, I personally > would prefer to ignore it completely, as it simply not trustable. It is generally not spoofed or missing except on spam, which you wouldn't want to read anyway. If the sender is legitimate and using a normal mail client, it is typically correct. It is useful for the same reason that is customary to write a date on a paper letter. The recipient knows the point when the email was written which is very helpful to interpret the content of most business emails, especially if many correspondents and/or emails are involved. I think the RFC2822 date MUST be displayed. If you choose to ignore it, Thunderbird already has an option to turn off columns. No other MUA I know of excludes the "Sent" (RFC2822) date. Microsoft Outlook has the concept of both a "Sent" and a "Received" (presumably top Received: line or IMAP internal date) date, both of which are available to a user. Given that no other MUA does this I think you need to provide a rationale for excluding this from Thunderbird, not the other way round. I've just read through the comments, and I think the following solution will cause no problems for anyone: Define "Received" as the first of these that is available: 1. IMAP internal date 2. POP3 fetch date 3. Date listed in the top "Received:" RFC2822 header line Define "Sent" as the RFC2822 Date: field. For either "Sent" or "Received", if the date is unavailable using the above logic, then we do not know the corresponding date. Ideally this would be represented by an "unknown" value like null, and the GUI would work accordingly (just list "unknown" or leave a space or whatever). I don't think this is essential, as "Received" would normally be available in all cases as there will be at least one "Received" line. Both Sent and Received columns should be available in the message index view. I suggest that the default is Received, as the user would expect emails to arrive at one end or the other, but not in the middle. The GUI already allows the user to add and remove columns to suit, and both would be available, so what the default is doesn't affect functionality. I'm happy for whoever submits the patch to choose. When viewing a message, the Date shown in the message (rather than the index) should be the date Sent, and the text should be changed from "Date:" to "Sent:" to make this clear. This is analogous to how someone would read snail mail.

(In reply to comment #81) > If the sender is legitimate and using a normal mail client, it is typically > correct. Unless the clock on the sender's computer is wrong - at work we have a couple of contacts who send emails with a selection of dates and times in January of 2000. > Given that no other MUA does this I think you need to provide a rationale for > excluding this from Thunderbird, not the other way round. The rationale for swapping one for the other is based on the decision that there can be only one. > I've just read through the comments, and I think the following solution will > cause no problems for anyone... You either missed or ignored the comments which said that having 2 date fields to choose from (which I'd actually be in favour of) adds complexity (because the user when picking fields has more things to choose from), and they don't want to do that.

(In reply to comment #81) > (In reply to comment #79) > > I'm curious - Why is the RFC822 date (if present) useful? > > It is generally not spoofed or missing except on spam, which you wouldn't want > to read anyway. You are correct that spam often spoofs or drops it (though not so much any more), but several major help-desk systems drop it, including those used by major ISPs. Furthermore, even in these days of ntp, it is extremely common for me to get email from people with significantly mis-set system clocks, especially on windows machines pre-NT. I'm also not sure I understand your argument about disregarding the RCF822 date for spam - IMHO, that is precisely the point: you don't want to read spam, but mis-dated spam/etc will get inappropriately sorted into your inbox (for instance), leaving sorting by sent date less useful. > If the sender is legitimate and using a normal mail client, it is typically > correct. but see above. > It is useful for the same reason that is customary to write a date on a paper > letter. The recipient knows the point when the email was written which is very > helpful to interpret the content of most business emails, especially if many > correspondents and/or emails are involved. displaying this date in a message != providing a column and expecting to be able to use it to sort messages. > I think the RFC2822 date MUST be displayed. If you choose to ignore it, > Thunderbird already has an option to turn off columns. > > No other MUA I know of excludes the "Sent" (RFC2822) date. Microsoft Outlook > has the concept of both a "Sent" and a "Received" (presumably top Received: > line or IMAP internal date) date, both of which are available to a user. > > Given that no other MUA does this I think you need to provide a rationale for > excluding this from Thunderbird, not the other way round. I'm not arguing directly for dropping sent date: I am arguing that received date must be a first-class message parameter (with column) because it is both critical and dependable (unlike Sent), and that the mozilla family of MUAs are at a major disadvantage to their competitors because of this omission. > I've just read through the comments, and I think the following solution will > cause no problems for anyone: Except that the GUI folks seem to be balking at adding a Received Date column, so we're still stuck here. The excuse was to avoid "code bloat" but this seems more like a boogeyman than anything else, given how many bugs have been opened and generated so much discussion for so long: check out the discussion on all of the blocklist. > Define "Received" as the first of these that is available: > 1. IMAP internal date > 2. POP3 fetch date > 3. Date listed in the top "Received:" RFC2822 header line I was under the impression that IMAP and POP3 dates changed on folder transfers... personally, I'd recommend reversing the order.

Any chance we could at least get the two active patches applied in time for thunderbird 2? While they fall short of addressing either the summary or the original request, they do represent significant progress toward fixing this problem and we could then open a new RFE on the other side to get the fields included in the GUI. As an aside, this open bug is #35 on the Core/MailNews* most votes list, and nearly identical bug 216033 is #12 on the Thunderbird/* list.

(In reply to comment #83) > Except that the GUI folks seem to be balking at adding a Received > Date column, so we're still stuck here. The excuse was to avoid > "code bloat" but this seems more like a boogeyman than anything > else, given how many bugs have been opened and generated so much > discussion for so long: check out the discussion on all of the > blocklist. It seems to me that the argument that adding an additional date column would result in too many options is a bit silly. There are currently (v1.5.0.5) 16 column options to choose from. One more isn't going to push this over some limit into unusability. In any case, even if it did, I would gladly give up Unread, Total, and Account columns in a trade for a Received Date field and possibly a future draft choice. > > Define "Received" as the first of these that is available: > > 1. IMAP internal date > > 2. POP3 fetch date > > 3. Date listed in the top "Received:" RFC2822 header line > I was under the impression that IMAP and POP3 dates changed on folder > transfers... personally, I'd recommend reversing the order. That's my understanding, as well, and it's the reason the value of the Order Received field is somewhat limited. While we're talking about traiding fields, a Received Date could easily replace the Order Received field, as far as I'm concerned.

(In reply to comment #85) > While we're talking about traiding fields, > a Received Date could easily replace the Order Received field, as far as I'm > concerned. That would be logical. If the UI argument is that adding a new field is cumbersome, then the argument could be made that we are not adding a new field, we are making the existing 'Order Received' field more usable. To the end user, the 'Order Received' field is a meaningless number (they don't even appear consecutive), and does not always sort by received date either.

> If the UI argument is that adding a new field is > cumbersome, then the argument could be made that we are not adding a new field, > we are making the existing 'Order Received' field more usable. To the end > user, the 'Order Received' field is a meaningless number (they don't even > appear consecutive), and does not always sort by received date either. I will respectfully disagree with your comment that 'Order Received' is a meaningless number, as has already been pointed out in comment #51. I would not like to see it replaced, but I do find the lack of 'Date Received' distressing. The "Date Received" column should really not be competing with other columns, it should just be added as a new column, separate from 'Date' and 'Order Received', and as a matter of personal opinion I think it should reflect the date in the final (top-most, last in chronological order) received header.

we won't stop ship for this but that doesn't mean we won't end up approving the patch for the branch.

(In reply to comment #88) > we won't stop ship for this but that doesn't mean we won't end up approving the > patch for the branch. Scott, Any reason not to SR the patch for the trunk? The patch fixes a regression on the trunk (bug 341548).

(Modification of my previous comment #67) > If the patch will be checked-in as fix for this bug, it will produce confusion > in bug tracking/fix verification testing. > I believe the patch should be proposed as the solution of Bug 73565. It seems to have been wrong. After I reconsidered about David's second patch by Comment #66 and I wrote Bug 341548 Comment #10, I was aware that my understanding about "Date column" was wrong. (Wrong) "Date column" means "Date: header" display. (Correct) "Date column" means "mail date" display which held in ".msf". The "mail date" is date used for mail age calculation etc. (I don't know correct term for "mail date", so call it "mail date") Current : "mail date" is always obtained from "Date: header" First patch : "mail date" is always obtained from "Received: header" Second patch : A) When user doesn't set mailnews.use_received_date=true "mail date" is obtained from "Date: header". Only if no "Date: header", from "Received header". => User can continue to see value in "Date: header" in "Date column" as current. But user can't taste fruit of "mail date from Received:". Bug 73565 is resolved by this enhancement. B) When user sets mailnews.use_received_date=true "mail date" is always obtained from "Received: header". => User looses "Date: header" value in "Date column", and user gets "Received: header" value in "Date column". Column name is unchanged(still "Date"). Bug 73565, Bug 32216 and some other bugs which complaint about old date or future date set by spammer, will be automatically resolved by this enhancement. David, is it right? As far as user sets mailnews.use_received_date=true intentionally, and as far as user can accept difference from current(value in "Date column" is changed from value in "Date: header" to value in "Received: header"), second patch can be a solution of this bug, although it satisfy this bug's request partially. And user can taste fruit by second patch ; (1) User needn't see future date in "Date column" of mail list pane. (2) spam of future Date: header is deleted normally/shortly by usual Junk Purge setting such as "Delete after 14 days". I believe many users can accept difference from current like me. However, it is true that many users need both two columns - column for "Date: header" value and column for "Received: header" value, as seen in many comments in this bug. I think next enhancement will be sufficient for this requirement, once your second patch will be released. - Implement "Date: header value" column and "Received: header value" column. (No change is required for "mail date"="Date column") Although "Date column" value becomes same as value of one of two new columns, it is not a big issue, I believe. David, what do you think? (still hate new column? :-)) Bug opener and who want both "Received: header value column" and "Date: header value column", what do you think?

(In reply to comment #90) > and who want both "Received: header value column" and "Date: header > value column", what do you think? For me this solution is sufficient.

(In reply to comment #90) > However, it is true that many users need both two columns - column for "Date: > header" value and column for "Received: header" value, as seen in many comments > in this bug. > I think next enhancement will be sufficient for this requirement, once your > second patch will be released. > - Implement "Date: header value" column and "Received: header value" column. > (No change is required for "mail date"="Date column") This change would be fine - indeed, it is no more or less than what the original RFE requested 4 years ago.

we could potentially do this with an extension, if we added a hidden pref to collect the Received date header and store it in the msf file. bug 348504 adds the ability for extensions to add custom columns, so an extension could add the received: date.

(In reply to comment #93) > we could potentially do this with an extension, if we added a hidden pref to > collect the Received date header and store it in the msf file. > bug 348504 adds the ability for extensions to add custom columns, so an > extension could add the received: date. Oh good news. I wrongly guessed the reasons why "developers don't want to add new column" were "don't want to add new fields in msf" and "msf compatibility with older versions". Is enhancement request bug for hidden prefs/storing in msf already opened? I don't like extension for enhancement which I like, because of compatibility issues when upgrade of Mozilla family, which always caused/will certainly cause many bugs or many forum threads of "not working well after software update!". If there are several predefined customizable columns, I think new columns for sorting requirement can be satisfied in many cases although not in all cases. - Several customizable columns, and a special column handler for it - User can change column name text for a customizable column - User can choose mail header from predefined mail header names - Predefined mail header names can be limited because mail header (which is very convenient if sorting at mail list pane can be done) is not so many. - If Recieved: header and Date: header are included in predefined headers, this bug's remaining issue will automatically resolved. (If unix mail box separator of "From - ..." is included,) (user can sort mails by "Download local time".) One of biggest issues is when/how invoke msf re-creation for all mail folders if option is changed... David, is there any possibility that such request for enhancement will be accepted by Mozilla mail&news/Thunderbird developers? Or this kind of enhancement should be done by extension? By the way, David, when will your second patch proposed in 2005/12 be applied to trunk? I'm enjoying "mail date" from "Received:" every day and is very satisfied. But I don't want to read duplicate bugs of bug 341548, don't want to do DUP closing work.

*** Bug 354813 has been marked as a duplicate of this bug. ***

Here's a data point: I just received two pieces of bugmail that for some reason were delayed. They have the following timestamps: Date: 11:27am Received: 2:55:55pm Date: 11:15am Received: 2:56:45pm In this case, at least, the correct order is by the Date field, not the Received field. (I have no idea why they arrived out of order.) I wouldn't have seen this if I hadn't been running a trunk build, with this bug's checked-in patch, when they arrived.

*** Bug 363971 has been marked as a duplicate of this bug. ***

Having just investigated bug 355113, I'm more in favor of this patch, as the date stored in the index file (the received date, with this patch) is the date used when folder retention is being used. Given the possibility of automatic deletion of a desired message because of a wrong date -- especially since auto deletion can't be undone, to my knowledge -- it seems more important to use the received date than if it were simply a matter of not being able to find a message in the expected position in the sort order. This has been the case with junk mail purging all along. While I'm sympathetic to the concerns of bug 341548, it seems that the big fix requires adding columns for each (probably storing both dates in the index). Assuming that's not feasible in the near future, I'd say this patch should get into the 1.8.x branch.

I really disagree with comment 99. If I may restate it, it says that: (a) because the sender's date may be wrong, and because some folders automatically purge messages older than a certain number of dayes, a message sent by a system with the wrong date may get purged a day or two before it should have been, because of the bad date. (b) the risk of purging a message a day or two early (due to wrong date) is SO great and consequences SO terrible that we should continue to force ALL users to put up with the also-HUGE usability problem of showing the wrong dates in the message list pane. I totally disagree with that last conclusion. Yes, I use folders that expunge messages based on a retention limit. Obviously, I don't lose any sleep about those messages not being retained. It they get purged a day or two early, so what? I surely do not want to continue to put up with the problem of wrong dates being displayed forever because of this.

(In reply to comment #1) > Use: View->Sort by->Order received > Problem with using Order Received is if you move mail from one folder to another it changes the Order Received number. Meaning if you are going though your inbox moving messages and don't start from oldest to newest then in the sub folder they are not in true order received. One of the bigger problems with the date field that is being used is spammers seem to be taking advantage of this and setting there dates to a date years in the future (like 2083). It would be nice to be able to choose either the date and time the e-mail server sent it or received it.

Can someone give some indication of where this is in the process? This thread is four and a half years old. There are posts from six month ago about patches. I'm running version 2 beta 2 and have this line: user_pref("mailnews.use_received_date", true); in prefs.js. Nevertheless, my Date column is still giving me the "Date: header" rather than the last "Received: header". So, two possibilities: 1) my prefs.js entry is wrong 2) This patch is not in the beta Which is it? If my entry is wrong, can someone please give me the correct entry? If this patch is not in the beta can someone please comment on when we're likely to see any of this?

The patch is not in the beta. You can get a nightly trunk build, which does have the patch, at: ftp.mozilla.org/pub/thunderbird/nightly/latest-trunk

Comment on attachment 205084 [details] [diff] [review] only used received: hdr if no Date: header or user sets hidden pref Scott MacGregor, Is there anything I can do to get you to review & approve this patch? Is there any dollar figure that will do the job? Please look at bug 341548 comment 10 to see why this is important.

Comment on attachment 205084 [details] [diff] [review] only used received: hdr if no Date: header or user sets hidden pref looks good David.

i just installed thunderbird-3.0a1.en-US.win32 created 4/12/07 and the date column is not using the received date and time. Unless the patch just does not update already received e-mails. Below is source from an e-mail I received. From - Thu Apr 12 13:12:14 2007 X-Account-Key: account2 X-UIDL: 00017238c2441865 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 X-Mozilla-Keys: Return-Path: <XXXX@XXX.XX> Received: from yokoyama1 (000.000.000.00) by smtp.mailserver.com with ESMTP (Eudora Internet Mail Server X 3.2.6); Thu, 12 Apr 2007 13:09:45 -0400 Return-Path: <XXXX@XXX.XX> Received: from 00.000.00.00 (HELO mail.XXX.XX) by mailserver.com with esmtp (LF-:7V'6,2E= *N4'8) id G.=VK;-T00AU1-JU for XXXXX@XXXXXX.XXX; Thu, 12 Apr 2007 17:10:20 -0900 Date: Thu, 12 Apr 2007 17:10:20 -0900 If you sort by date it is showing this e-mail was sent 4 hours in the future.

Actually this is exactly what is the problem - spammers send e-mails in the future as some sort of "keep on top"-"feature". On the last e-mail server I was using they had implemented a server side exim script that did overwrite the Date header with the received date. Currently on a new mail server unfortunately I get future-e-mails again. And what is more annoying: Thunderbird is showing even other e-mails not necessarily in their order of receiving which can be very confusing. Shortly: the Date stamp is not reliable. The only one reliable date is the last - or spoken from the order of the headers the first "Received" stamp. So please: Take the date from the first "Received" header instead for that column in Thunderbird. For the people who would like to say to use the order column instead: Whenever I move the e-mail to another folder I get new order numbers so this is not an option.

This bug is specifically a request that "received date" be added as a first-class concept to thunderbird and other mozilla mail components. Any other solution would not address the problem and so would be non-responsive to the bug. The reason that this is an issue is precisely that there is no current way to reliably order email temporally that is not either (a) under control of the sender or (b) destroyed by inbox/folder maintenance. The traditional "Date" field while "RFC required" is absolutely undependable in many, many instances - yes, spammers often spoof them, but it is also clear that many people are incapable of keeping their system clocks in sync with reality. Even worse, several major billing, support, and helpdesk systems don't bother to put any Date field at all on their messages, leading to thunderbird hiding such generally critical messages at the beginning of time. All this conspires to make sorting by Date worse then useless: it is asking for trouble. Date *might* be genuinely useful as a idiot detector. The "Received Order" field is interesting, but is reset when messages are moved between folders and boxes (depending on mailserver behavior). Obviously, while it is possibly useful *within* a single folder, it is only a partial ordering when (for instance) merging views from multiple boxes. Furthermore, it is only useful for sorting purposes, not for viewing, making it a waste of screen space. IMHO a much more useful value here would be define "order received" as "filed date": there would be absolutely no difference in sort order and the associated information would be independently useful. I can understand the reluctance to change the semantics of the Date column (even if I disagree - I seriously don't see any redeeming value in display or order-by Date header...), but I don't understand why adding received date as a column is in any way a less attractive option than a "fix" that allows only expert users to change Date=Date to Date=Received... Better than nothing, I suppose - at least it will finally become possible for expert users to order their email in a sane way.

I could hardly disagree more. While I agree that other means must be found if the Date: field is not present, or perhaps if it's utterly outside sane bounds, anything other than the Date: header (or perhaps the first Received header date) means that messages close in time will commonly be mis-ordered due to the vagaries of the Internet. If people you care about are sending emails with bogus Date fields, tell them so. I'm sure they will be happy to fix their clocks, and have their reminders go off at the right time and so on. And if spam is a problem, use a spam filter. It can note the bogus date to do its job better. This proposed patch seems to me to be exactly the right thing, and it should be checked in ASAP. Gerv

I agree completely with Michael Thome and completely disagree with Gervase Markham . Like Michael, I consider the bug to be the reason that it was raised for - that a "Date Received" column (regardless of what it is called) is required. As Michael said, the patch is useful as an obtuse workaround but doesn't not resolve the problem, and therefore would not resolve this bug. My vote for this bug is for a "Date Received" column to exist. This is what the bug was raised for and this is what I suspect many votes represent. Obviously there is a division on this matter. Can I propose that this be split into two bugs. Then it will be clear who is voting for what.

I also vote for a new column but it seems to me from the discussions (not only in this issue) that it is too difficult to add a further column. In that case I can perfectly live also with an option somewhere (hidden or not) where I can decide to use the received date in the date column. Telling everybody to fix their clock is something I could do if I am very out of work (if I know the name where to drop that notice) otherwise I assume that they will find out that their clock is wrong themselves. However I cannot assume that others will solve my problems. I have integrated spam filters on nearly all accounts I am using but as all of you might experience - those do not always find out every spam. And actually I cannot understand how it can be more work to add that option rather than discussing it over years...

Yes, I too agree with Michael Tome. The (obvious for me) reason for the existence of this bug was lack of Date Received column. This can (and probably should) be an additional column - most end users are have no interest in original (as per rfc) message date, they want to know when a message was received into their own, private mailbox!

> they want to know when a message was > received into their own, private mailbox! And presumably they love having their discussions out of order if one message gets delayed and a subsequent one does not? This behaviour happens to me regularly under the current regime - far more regularly than receiving non-spam messages with bogus Date: headers. The only way to guarantee that a conversation is threaded in the order that the people involved wrote the messages is for it to be threaded by Date: (i.e. when it was written) and for the Date headers to be correct. The latter is, in my experience, almost always the case for non-spam mail. Therefore, we should do the former so that the two combine together to achieve the correct result. But it is correct that there is no point in arguing about "what this bug is for". The fact remains that there's a patch here that solves the problem for some people without making life more difficult for anyone else. So let's get it checked in. If you want to argue for something else, you can then do so. Gerv

(In reply to comment #109) > I could hardly disagree more. While I agree that other means must be found if > the Date: field is not present, or perhaps if it's utterly outside sane bounds, > anything other than the Date: header (or perhaps the first Received header > date) means that messages close in time will commonly be mis-ordered due to the > vagaries of the Internet. I'm not trying to be argumentative here - I'm genuinely baffled as to why one would care (let alone trust) that the date that someone *claims* to have sent a message is anywhere near as important as when you got it - when one gets physical mail, what matters is when it arrived in your mailbox. Sure, I *occasionally* check the postmark on an envelope when it arrives in poor condition or later than I expect, but having someone tell me "the check is in the mail" isn't something that I'm likely to ever trust. The bottom line for me is that the Date header is, in my experience the Date header is often wrong and surprisingly often significantly so, even for legitimate mail. Received is always trustworthy. I would vastly prefer a trustworthy measure of absolute ordering than a known untrustworthy, often a bit off, and sometimes totally bogus measure... when you are dealing with a good citizen with decent connectivity, then the measures are going to be nearly identical, but when senders are badly behaved, the "Date solution" rewards the bad behavior, where the "Received solution" is neutral to it. > If people you care about are sending emails with bogus Date fields, tell them > so. I'm sure they will be happy to fix their clocks, and have their reminders > go off at the right time and so on. And if spam is a problem, use a spam > filter. It can note the bogus date to do its job better. Ah - Well, so far, Verizon Wireless has ignored my complaints, and my ISP uses a commercial help desk system that doesn't send Date at all. It took a remarkably long time for people to fix their machines w/the recent Daylight Savings Time change. And ntp is nowhere near ubiquitous, though it is slowly getting better since windows added it in. What *really* grates on me is how often when I complain, the response is "Well, maybe you ought to use a decent email program". This *ought* to be truly horrifying to anyone maintaining Thunderbird - it is an extremely serious competitive disadvantage that impacts users far more than most of the new features and bugfixes that have been addressed in the 4.5 years since this bug was opened. The implicit assumptions behind your response are that (1) I know the people sending me legitimate email (often not true), and that (2) entities with broken clocks/MTAs care that they are annoying me (often not true - see above). I get legitimate email with bad Date fields that are broken enough to result in visibly misordered email almost every day - thunderbird's baffling lack of the trivial capability to properly order my messages is not something that is possible for me to ignore - the defect is in my face *constantly*. Threading email discussions cannot reliably be done with the Date header, not only because it is often wrong, but if Date is not approximately the same as Received already, then you cannot know when the author of a message saw the message that they are responding to. On the other hand, real message threading is perfectly and properly accomplished using the already fully implemented In-Reply-To headers. FWIW, this bug is the 7th most voted for open MailNews backend bug (out of 570). The closely related bug #216033 is the 8th most voted for and the 38th oldest open bug in Thunderbird (out of 5051).

(In reply to comment #114) > I'm not trying to be argumentative here I'm sure :-) > - I'm genuinely baffled as to why one > would care (let alone trust) that the date that someone *claims* to have sent a > message is anywhere near as important as when you got it Because the people I correspond with don't lie about it, and because there is no other way of accurately putting messages in the order which they were written. Because email is not an instant-delivery medium. > - when one gets > physical mail, what matters is when it arrived in your mailbox. I don't tend to have physical mail conversations with multiple people participating in the same conversation. And if I were involved in some sort of round-robin multiple-recipient letter scheme, I would doubtless look at the dates written at the top of the letters to ascertain in what order they were written, so I could see who was responding to what. > Sure, I > *occasionally* check the postmark on an envelope when it arrives in poor > condition or later than I expect, but having someone tell me "the check is in > the mail" isn't something that I'm likely to ever trust. Utterly irrelevant example, surely? > The bottom line for > me is that the Date header is, in my experience the Date header is often wrong > and surprisingly often significantly so, even for legitimate mail. Received is > always trustworthy. It's an absolutely trustworthy measure of when a particular message was received by a particular machine. That doesn't mean it's always *what you want to know*. For example, I go offline and write some messages, and press Send Later. Meantime, other people are writing messages in the same email conversation. When I go back online, the Received header would erroneously imply I wrote those messages after theirs, when in fact that was not the case. My MUA has carefully noted when I wrote them, and stored the value in the Date: header so that other participants in the thread can see when I wrote it. Gerv

>For example, I go offline and write some messages, and press Send Later. >Meantime, other people are writing messages in the same email conversation. >When I go back online, the Received header would erroneously imply I wrote >those messages after theirs, when in fact that was not the case. My MUA has >carefully noted when I wrote them, and stored the value in the Date: header so >that other participants in the thread can see when I wrote it. Errr, sorry, but it would not. It is *received* date, not a *sent* date. BTW - have anyone made any assesment of message threading popularity (ie. how many people use it)? Issues with threading seem to be an important argument in this discussion, but out of my ~10 friends using Thunderbird, no one uses threads (and all complain about lack of Received Date column/field...).

(In reply to comment #115) > For example, I go offline and write some messages, and press Send Later. > Meantime, other people are writing messages in the same email conversation. > When I go back online, the Received header would erroneously imply I wrote > those messages after theirs, when in fact that was not the case. My MUA has > carefully noted when I wrote them, and stored the value in the Date: header so > that other participants in the thread can see when I wrote it. And the date field would erroneously imply that the others who had sent their emails when they finished writing them had read yours. That's what conversation threading is for. Your message should indicate what it's replying to and fall quite nicely into the threaded conversation. (More generally) I personally care more about Date Received than I do about Date, though I believe that all three of Date, Date Received, and Order Received have their place as first class columns, and there are times I'd want to be able to see all three of them. My vote for this bug was for a fix to that effect. All three columns, no hacks, no extras. This is a basic MUA feature.

Seems to me that this discussion has the obvious conclusion that tb/mailnews needs to have both these columns and let us pick and choose which we want to use and when. Sometime's I'll sort some messages by Date Received, sometimes by Date Sent, sometimes by something else. In that context I don't like the proposed patch. Too often such temporary fixes serve as an excuse for allowing serious issues to remain unresolved (thing of bug 2920 for example). > BTW - have anyone made any assesment of message threading popularity (ie. how many people use it)? This is an interesting question but this bug page is not the place to discuss it.

Seems to me that this discussion has the obvious conclusion that tb/mailnews needs to have both these columns and let us pick and choose which we want to use and when. Sometime's I'll sort some messages by Date Received, sometimes by Date Sent, sometimes by something else. In that context I don't like the proposed patch. Too often such temporary fixes serve as an excuse for allowing serious issues to remain unresolved (thing of bug 2920 for example). > BTW - have anyone made any assesment of message threading popularity (ie. how many people use it)? This is an interesting question but this bug page is not the place to discuss it.

Re comments #114 and #115, some people use email like a chatroom, sending one-liners back and forth with time differences of seconds or minutes, not hours or days. I just read a pile of msgs from 10K to 80K passed back and forth with one-line additions, and deleted a bunch of them. I would suggest using an irc or local chat service, not email, for such light banter. I too must say that the received-by date is the only reliable one for the end recipient of an email. If minute-by-minute messages get out of order, that's tough bippies. Use a more suitable medium for chat.

Re comments #114 and #115, some people use email like a chatroom, sending one-liners back and forth with time differences of seconds or minutes, not hours or days. I just read a pile of msgs from 10K to 80K passed back and forth with one-line additions, and deleted a bunch of them. I would suggest using an irc or local chat service, not email, for such light banter. I too must say that the received-by date is the only reliable one for the end recipient of an email. If minute-by-minute messages get out of order, that's tough bippies. Use a more suitable medium for chat.

This patch gives users a choice. Today, users have NO choice. Clearly, no one solution is preferred by everybody. The people who do the work on this code say that adding another column is too much work. I believe them. No one who wants another column is volunteering to do the work. That means: it won't happen soon. In the meantime, we have a patch that gives us all the choice. It should go in. For myself, I absolutely don't care about dates in spam, because they all get filtered/deleted, so they never show up in my folders. I care MOST that successive emails sent by the same individual show up in the order SENT by that individual, even if that is not the order received. Today, without this patch, I simply don't have that option.

Anyone else seeing "midair collision" msgs resulting from Different senders (not the same sender) sending messages at the same time? It seems to me that if the senders are different, it shouldn't be considered a midair collision by bugzilla.

As the person who originally reported this bug, let me just chime in again.. This seems like a pretty cut and dry issue to me. Some people want date sent, some people want date received. Regardless of your opinion of which is more useful, several, if not all of the most popular MUA's out there today have this ability, making this a serious competitive failure. It's clear that vast numbers of people would appreciate this functionality whether everyone agrees with it or not. So, can we all agree on the intention and move onto implementation details? In my opinion, making this a "hidden" setting is a stop-gap solution which doesn't really address the problem properly. Use cases have been proposed by people who want both a date sent field and a date received field, so the ability to switch it via a prefs file from one to the other is lacking, at best. Adding a new column may be difficult, but I don't for a second believe that's a good enough reason to implement a half hearted hack which barely solves the problem. Surely there's a developer out there willing and able to do the work to add it properly? I was gearing up to send this bug a "Happy 5th birthday" card in a few months. Perhaps we can have a goal of getting a resolution before that happens.

I agree that having separate columns for Date Sent (or date ostensibly sent, if you prefer), Date Received, and Order Received is the ideal situation. I've found myself wanting all three at various times and the fact that I have two of them is a good start. Having a Date Received column would be a great addition. This is, by the way, how many other systems report date. At work, where I use MS Outlook for mail, I can have either or both date fields. I switch back and forth regularly. I also appreciate that the date field defaults to newest at the top since that's what I want nine times out of ten. That's a subject for a different bug thread, I suppose. It's clear that the folks able to do this are not willing. I don't hold that against them. I'm quite grateful for all they have done and continue to do. Others of us, myself included, are quite willing but not particularly able. Frankly, I wouldn't know where to start. If someone wanted to look at the code involved, where would he start?

(In reply to comment #124) Congrats on (so far) having outlived your bug! I agree, the bug is very straightforward. The caption says: "Received date field" When I switched from Outlook a few years back, this was one of the more noticeable differences; the order of emails is subtly incorrect when I sort by Date. No, I cannot ask people to fix their computer time; it would be unprofessional to ask this in most cases. Order Received is (with all due respect) useless since it is reset too easily by moving a file, or even just deleting an attachment. How about, for now, implement the bug fix for "Date Sent" and RENAME the UI column from "Date" to "Date Sent" so it is more clear what it is. However it does not fix the problem that the bug is for SO PLEASE DON'T CLOSE THE BUG until we have DATE RECEIVED. Sorry about the bug spam but this bug is important.

Mattias wrote: > Order Received is (with all due respect) useless since it is reset > too easily by moving a file, or even just deleting an attachment. Maybe that's a bug and should be fixed.

Order received has it's use but become inacturate if you move an e-mail to another folder. That is not what we are reporting here. The date collum is not a bug as it works the way it was written. it sorts the date the e-mail client said it was written. (if it's faked it's faked) I think we are asking that a feature be put in to sort by date received like most of the major e-mail clients on the market do. granted 99% of the e-mails I get that have wrong dates are spam, there are a few that i get that are from customers that have to change there computer dates. then you get the ppl who can't set there time zone correctly so that messes up there sent date. as for the person that likes to have thread style e-mail conversations. might i suggest a message board.

> The date collum is not a bug as it works the way it was written. > it sorts the date the e-mail client said it was written. Actually it does NOT do that, not in trunk builds. I would be happy if it did, but it does not. See comment 90. The trunk now behaves as described there for "first patch". It's horrible (IMO :).

Why is it most everyone talks about IN THEIR EXPERIENCE and then ignore other peoples experience like it doesnt matter. We all have different experiences If you have good spam filters thats nice, but thats you. I get hundreds of items of spam a day, and in my role as a webmaster I can't automatically filter all of it. I have to err on the side of caution so a lot of bulk email ends up in my inbox. The same role prevents me from sending back a "fix your date you idiot" message even if I could be bothered in the first place. We are not our brothers keepers. This bug as pointed out by micheal in #114 is one of the "most popular" which REALLY means most annoying, it is also one of the longest running IMHO the rfc that is used as a basis for many of the arguments IS DATED it doesnt account for the changing netscape (lol) But the RFC allows for MTA's to add headers so adding a received header is not going to break it. The solution that seems to resolve all wants is an additional column "Received Date". taken from the added received header. Oh yes the MUA is also an MTA so it has the legal right to add the header :) The user should be able to pick this column out of the column picker NOT via some internal switch, How many users are able to do that? The bulk of people don't stick with it and submit bug reports and argue the point, they simply up and leave. Why would you want to alienate more people? As a programmer I struggle to see it as being too hard to add a column. but then I haven't investigated the source code so perhaps some bad design decisions have left Thunderbird in this (appalling) state. Those that want threading perhaps you should start a request for a threading mechanism 'Sent' date was not designed for that process and will never reliably work for that process. But the main point is this bug report is not about threading. Why are you even mentioning it? and no one is saying get rid of the sent header anyway so you can continue to use it. Hmmm actually ignore my posting. I give up I am going back to outlook

(In reply to comment #127) > Mattias wrote: > > Order Received is (with all due respect) useless since it is reset > > too easily by moving a file, or even just deleting an attachment. > > Maybe that's a bug and should be fixed. > I was just pointing out why the Order Received column is irrelevant to the bug at hand. And it would not take the place of Date Received even if it was fixed. Developers who can fix this are obviously unwilling. You must be reading these comments too. Considering all the amazing features of TB; to a user it seems like a small deal to add a Date Received column, and that it would make a lot of people happy to do so. Perhaps one of you could explain what the actual problem is that prevents this bug from being fixed so that us laypeople can appreciate it?

Nelson, I've landed the patch on the trunk; sorry for the delay. re adding a Date Received column, I'd be happy to help someone who wanted to do a patch for that. It's not hard, just tedious.

Comment on attachment 205084 [details] [diff] [review] only used received: hdr if no Date: header or user sets hidden pref There's a certain amount of clamoring for this

Comment on attachment 205084 [details] [diff] [review] only used received: hdr if no Date: header or user sets hidden pref a=mscott for mailnews only change to 1.8.1.5.

support for hidden pref landed in 1.8.1.5 - I'm going to add the fixed1.8.1.5 keyword, even though the main bug itself (a whole new received by date column) is not fixed on the trunk or branch.

Again, as the original requester of this feature, I'd like to re-iterate that I don't consider a hidden preference a great fix. To my mind, there are acceptable use cases for both received and sent date fields, and practically every other MTA available has received as not only an option, but the default. I appreciate the hidden preference as a stop-gap solution, but I don't believe this bug can ever be truly resolved until the option is more obvious to your every-day user. I can't recommend Thunderbird to my parents if I have to tell them to open a user prefs js file, or go edit config variables, in order to get their mail in the same way they've received it for years with other clients.

The instructions for changing this setting using the config editor are only a little bit more involved than they would be for finding an option in the prefs UI: Tools | options | advanced (general) - config editor type received right click on the line that says "mailnews.use_received_date" and pick toggle That sets it to false. Setting it to true by default provoked a lot of negative reaction so we're not going there again. If someone wants to write a patch to add a received column, again, I'm happy to help.

(In reply to comment #137) > If someone wants to write a patch to add a received column, again, I'm happy to > help. David, first of all thanks for your efforts about Thunderbird code. I'm not a C++ coder, so I can't help with the patch, but I'd like to ask you if it would be possible to have the received date as separate properity of nsIMsgDBHdr. In this way it would be very easy to create a custom column to display it.

CCing self. (In reply to comment #137) > The instructions for changing this setting using the config editor are only a > little bit more involved than they would be for finding an option in the prefs > UI: > > Tools | options | advanced (general) - config editor > type received > right click on the line that says "mailnews.use_received_date" and pick toggle > That sets it to false. I don't see that option. *shrug* > > Setting it to true by default provoked a lot of negative reaction so we're not > going there again. You're saying that ordering e-mails by the Delivery-date instead of Date, a vastly more sensible method and the same as OE and Outlook do it, provoked negative reaction? By who? Idiots? It's just way better and more sensible. > > If someone wants to write a patch to add a received column, again, I'm happy to > help. I think I might try to hack this. I doubt the columns usable for sorting use much C++, it's more likely XUL and JavaScript.

> I don't see that option. *shrug* You're probably not running the latest trunk or 2.0.0.5-pre build >You're saying that ordering e-mails by the Delivery-date instead of Date... I'm saying have a column that says "Date" and then putting the data from the Received header instead of the Date header bothered people. >I think I might try to hack this. I doubt the columns usable for sorting use >much C++, it's more likely XUL and JavaScript. Right - the only C++ required is to set a dateReceived property on the msg hdr when we're parsing the incoming message. We have code to do something very similar now, obviously, but it would need to set a dateReceived property, not the date property. I could help you with that part, if needs be. I wouldn't add it to nsIMsgHdr.idl, but rather, just do something like msgHdr->SetUint32Property("dateReceived", intDate); and pull that date out for the Received column with GetUint32Property. Actually, unless you use a custom column, you need a little bit of code in nsMsgDBView.cpp to add a case for the received date column.

David: Why does your code try to parse the Received: header first? Shouldn't it look at the Delivery-date header, then fall thru to Received?

Jeremy, I think you're right - we should be looking for the Delivery Date header first. Do you want me to make a patch to do that? It would be a little while before I can get to it.

Sure. I'm currently trying to figure out the nightmarish dark arts of hacking Mozilla, and being reminded of why I try to avoid it whenever possible, so I might come out with something before. Have to see.

Jeremy: You've come a little late to the party, you're speaking a little too loudly, and you keep referring to the caviar as "bean dip". It's awkward. I completely agree with you that the current implementation is incorrect, but I think that "idiots" is a strong word for those who feel it's better than the alternatives. We've done our best to keep this thread civil; please don't spoil it now. If you refer to the past five years of comments, you'll see we've all been discussing what the best behavior is and what the limitations of the current architecture are. You may wish to review these at your leisure to save some time. ObActualContribution: Delivery-date: is a non-standard header added, I think, only by exim and mh. On other mailers, it won't be created or modified and is therefore easily forgeable, leading right back to the Date: problem.

Delivery-date is described here: http://www.faqs.org/rfcs/rfc2156.html - but if it's easily forgeable, then that's a potential problem (not sure if it's a real world problem, though). Jeremy, what's the advantage of using the Delivery Date over the topmost received header? I wonder if one way of dealing with forged dates is to trust the date closest to the current download time, or conversely, trust dates past the current download time less, since they're most likely forged.

(In reply to comment #144) ... > ObActualContribution: Delivery-date: is a non-standard header added, I > think, only by exim and mh. On other mailers, it won't be created or modified > and is therefore easily forgeable, leading right back to the Date: problem. > ... and ... (In reply to comment #145) > Delivery-date is described here: http://www.faqs.org/rfcs/rfc2156.html - but if > it's easily forgeable, then that's a potential problem (not sure if it's a real > world problem, though). Jeremy, what's the advantage of using the Delivery Date > over the topmost received header? Correct me if I'm wrong, but surely everything in a POP-format e-mail is easily forgeable... by the MTA. However, we trust the MTA *waaaay* more than anything that passed the e-mail to the MTA. Otherwise we'd have a pretty useless e-mail service. So, I'm taking the approach that if we can say the MTA added it, we should assume that's the most trustworthy info we have. You could just look at the topmost received header, I guess. I was just thinking that it might possibly not represent the time when the MTA received the e-mail, or there might be a few seconds' difference. Delivery-date is specifically designed for the MTA to tell you when it received the mail, so I'd give it top priority; obviously fall through to Received for MTAs that don't add that header. In other words, I'd put the order of priority as: 1. Delivery-date 2. Received 3. Local system time (PR_Now()) 4. Date ... where 4 should probably not be used because some idiot could (and many spammers do) set it to 2011 or 1995 and make it appear in a silly place in the list. You start looking at 1., and fall through to 3. > > I wonder if one way of dealing with forged dates is to trust the date closest > to the current download time, or conversely, trust dates past the current > download time less, since they're most likely forged. > That assumes that the user's local date/time are correctly configured, which is very dangerous. At the end of the day, I'd give 1. and 2. in the above list priority over the local system's date/time. In other words, if I set my system to June 6th, 1997 and get an email in with a Delivery-date header of July 3rd, 2007, I want it to display the latter. It's more likely I misconfigured my system than that the MTA lied, and if the MTA lied, I should find a better MTA. :-P By the way, I've compiled Thunderbird version 3.0a1pre (2007070321), toggled the mailnews.use_received_date value to true, and yet an e-mail with a bogus Date header (but valid Received date) is still having its date listed as the bogus date. Perhaps you could give me some idea of why this might not be working, David.

@David: Oops, I should have been more precise about "non-standard". Delivery-Date: isn't in RFC2822. RFC2156 describes only how X.400-to-MIME (MIXER) gateways should work. RFC4021, the standards-track registration of MIME header fields, lists it as MIXER-only and "not for general use" - and RFC4021 itself is only a proposed standard, step 1 of 3 in becoming a standard. So it's not standard. But more to the point, it's not particularly common, let alone ubiquitous, and so I don't think it can really be trusted the way Received: can. There's an option to create Delivery-Date: in exim, and there's another option to remove any existing header; I don't know if those options are defaults. Other MTAs, say sendmail or postfix, would simply pass it through if it already existed. @Jeremy: You're exactly right - we trust our local MTA. But we can't assume that the local MTA is the one that added Delivery-Date: unless (a) we are running exim and (b) we've configured exim to add that header (or at least remove such an existing header). Otherwise, it could have been added anywhere along the chain. Received:, at least, is nearly-ubiquitously added - and is additive, not, er, replace-a-tive - and so thus more trustworthy when looked at in reverse order.

(In reply to comment #140) > Right - the only C++ required is to set a dateReceived property on the msg hdr > when we're parsing the incoming message. We have code to do something very > similar now, obviously, but it would need to set a dateReceived property, not > the date property. I could help you with that part, if needs be. I wouldn't add > it to nsIMsgHdr.idl, but rather, just do something like > > msgHdr->SetUint32Property("dateReceived", intDate); > > and pull that date out for the Received column with GetUint32Property. > Actually, unless you use a custom column, you need a little bit of code in > nsMsgDBView.cpp to add a case for the received date column. > David, how is this safe? It looks like a PRTime type, which your patch is using to store the parsed Received: date, is defined as a 64 bit unsigned int. Wouldn't setting it as a uint32 property lose data?

at some point maybe you have to convert from milliseconds to seconds - you want to be storing seconds, in any case.

Added a pre-alpha patch. This sort of starts to work - it adds a 'Received' column but all the received dates are off. I'm guessing this is because it's trying to cram a PRTime (64 bits) into a 32 bit attribute, David. Please advise on how to fix.

Look at http://lxr.mozilla.org/mozilla/source/mailnews/db/msgdb/src/nsMsgHdr.cpp#527 and when you get ready to ask for a review, please remove all those JEZ comments :-)

OK, added an updated version.

David, I just realised what you meant by those 64-32-64 bit helper functions. New patch actually seems to display the Received dates correctly! It displays a space " " if there's no, or an invalid, Received date. I've changed a bit of your date generation code. It used to fall back to the Date header, and then PR_Now(), if it couldn't find a Received header. I've just made it display a space " " character. My rationale behind this is that the Received header should be a very trustworthy indicator of when the MTA received the e-mail. It's nearly always there, and when it isn't, I want to know about it and I don't want the sender to have the opportunity of 'fooling' the user into thinking the Received date is what they specified as Date. If you want to know the Date header, you can look at the Date column. The Date column is much more of a 'probably received at this date' than Received, which is why it just falls back to PR_Now() if the Date header isn't there - it's much more prone to spoofing... but that won't matter if we deprecate its use, and display & sort by Received by default, instead of Date.

looks like it's coming along, good job! + <treecol id="receivedCol" persist="hidden ordinal width" flex="2" + label="Received" tooltiptext="Received tooltip"/> + <splitter class="tree-splitter"/> the tooltip text wants to be in a dtd file, like the other tooltips (this needs to be fixed in several places) + if (!LL_IS_ZERO(m_receivedTime)) { + PRTime2Seconds(m_receivedTime, &rcvTimeSecs); + m_newMsgHdr->SetUint32Property("dateReceived", rcvTimeSecs); + } + else { this file uses this braces style: if (a) { } else { } and in general, you should follow the prevailing braces style why are these lines changed? Is your editor messing with the line endings, or were they broken in this file? +// set to 0 if you don't want to ignore timestamp differences between +// local mail folders and the value stored in the corresponding .msf file. +// 0 was the default up to and including 1.5. I've made the default +// be greater than one hour so daylight savings time changes don't affect us. +// We will still always regenerate .msf files if the file size changes. +pref("mail.db_timestamp_leeway", 4000); + -pref("mailnews.use_received_date", false); - If you get rid of this pref, you basically break users who want to use Received Date for things like sorting within threads (we use the date value for things other than the date column). I'm not sure what we want to do about that, unless it's to switch to using the received date for that kind of sorting... Is your plan to break this out into a shared routine so that both date columns can share this code? + nsDateFormatSelector dateFormat = m_dateFormatDefault; +// if (explodedCurrentTime.tm_year == explodedMsgTime.tm_year && +// explodedCurrentTime.tm_month == explodedMsgTime.tm_month && +// explodedCurrentTime.tm_mday == explodedMsgTime.tm_mday) +// { +// // same day... +// dateFormat = m_dateFormatToday; +// } +// // the following chunk of code causes us to show a day instead of a number if the message was received +// // within the last 7 days. i.e. Mon 5:10pm. +// // The concrete format used is dependent on a preference setting (see InitDisplayFormats), but the default +// // is the format described above. +// else if (LL_CMP(currentTime, >, dateOfMsg)) +// { +// // some constants for calculation +// static PRInt64 microSecondsPerSecond; +// static PRInt64 secondsPerDay; +// static PRInt64 microSecondsPerDay; +// static PRInt64 microSecondsPer6Days; +// +// static PRBool bGotConstants = PR_FALSE; +// if ( !bGotConstants ) +// { +// // seeds +// LL_I2L ( microSecondsPerSecond, PR_USEC_PER_SEC ); +// LL_UI2L ( secondsPerDay, 60 * 60 * 24 ); +// +// // derivees +// LL_MUL( microSecondsPerDay, secondsPerDay, microSecondsPerSecond ); +// LL_MUL( microSecondsPer6Days, microSecondsPerDay, 6 ); +// +// bGotConstants = PR_TRUE; +// } +// +// // setting the time variables to local time +// PRInt64 GMTLocalTimeShift; +// LL_ADD( GMTLocalTimeShift, explodedCurrentTime.tm_params.tp_gmt_offset, explodedCurrentTime.tm_params.tp_dst_offset ); +// LL_MUL( GMTLocalTimeShift, GMTLocalTimeShift, microSecondsPerSecond ); +// LL_ADD( currentTime, currentTime, GMTLocalTimeShift ); +// LL_ADD( dateOfMsgLocal, dateOfMsg, GMTLocalTimeShift ); +// +// // the most recent midnight, counting from current time +// PRInt64 todaysMicroSeconds, mostRecentMidnight; +// LL_MOD( todaysMicroSeconds, currentTime, microSecondsPerDay ); +// LL_SUB( mostRecentMidnight, currentTime, todaysMicroSeconds ); +// +// // most recent midnight minus 6 days +// PRInt64 mostRecentWeek; +// LL_SUB( mostRecentWeek, mostRecentMidnight, microSecondsPer6Days ); +// +// // was the message sent during the last week? +// if ( LL_CMP( dateOfMsgLocal, >=, mostRecentWeek ) ) +// { // yes .... +// dateFormat = m_dateFormatThisWeek; +// } +// }

David: Talking to you on IRC about the FetchDate function would be useful. Unless I'm mistaken, it isn't working properly.

A bit of research for the record: Outlook Express uses the 'Received' date first, then drops to 'Date', then uses the local system time, to mark the received date. I think we should do the same. I initially said that you should totally trust the MTA to put a Received header in, but on the rare occasions it doesn't, Date should be used (or the local system time) so that the Received column can be more comprehensive for sorting e-mails. IOW, you're OK with the Received date being potentially wrong because it only applies to very few e-mails that for some reason don't have a Received header.

(In reply to comment #155) > David: Talking to you on IRC about the FetchDate function would be useful. > Unless I'm mistaken, it isn't working properly. Seems I was mistaken; it does work properly. Ignore that. :-P (In reply to comment #154) > looks like it's coming along, good job! > [...] > > the tooltip text wants to be in a dtd file, like the other tooltips (this needs > to be fixed in several places) I've done this with the new patch. > > this file uses this braces style: > > if (a) > { > } > else > { > } > > and in general, you should follow the prevailing braces style Damn, I hate that braces style. :-) But it's fixed in the new patch. > why are these lines changed? Is your editor messing with the line endings, or > were they broken in this file? Good question. I've re-checked out the file from CVS and re-made the changes, and it doesn't seem to be diff'ing these lines now. > -pref("mailnews.use_received_date", false); > - > > If you get rid of this pref, you basically break users who want to use Received > Date for things like sorting within threads (we use the date value for things > other than the date column). I'm not sure what we want to do about that, unless > it's to switch to using the received date for that kind of sorting... That seems like a good idea. The Received date column is intended to be more reliable than or as reliable as the Date column, in all cases, so this sorting should be switched to use Received by default. > Is your plan to break this out into a shared routine so that both date columns > can share this code? It wasn't, but that seems like a good idea now. :-) I've done this in the new patch with the help of an optional bool argument, set to true if you want the Received date. I've also made the FinalizeHeaders() function always set 'Received' to something, with the order of priority going Received -> Delivery-date -> Date -> PR_Now(). The assumption is that an MTA might somehow not set Received, but might set Delivery-Date, which is slightly more reliable than Date (almost always set, but often spoofed to be wrong). The problem with PR_Now(), I guess, is that it will calculate the date wrongly if you 'rebuild index' for the folder. It will set the date to the date when you rebuilt the index, instead of the date it was tagged with when the e-mail was actually received. I'm not sure if there's an easy way to get around this, and it probably isn't important because so few e-mails will actually need to resort to PR_Now() anyway. A review of this new patch would be appreciated. AFAICT, the only thing needed now is to allow sorting by Received column, and (I think) set it to be displayed and used to sort messages by default upon Thunderbird install, instead of Date.

(In reply to comment #157) > > -pref("mailnews.use_received_date", false); > > - > > > > If you get rid of this pref, you basically break users who want to use Received > > Date for things like sorting within threads (we use the date value for things > > other than the date column). I'm not sure what we want to do about that, unless > > it's to switch to using the received date for that kind of sorting... > > That seems like a good idea. The Received date column is intended to be more > reliable than or as reliable as the Date column, in all cases, so this sorting > should be switched to use Received by default. Using Date for threads is really the preferred thing IMO. Elsewhere I wouldn't care so much which is used, but in threads I definitely want Date to read it the order it's written, not the order it happens to be received. Date is now used on trunk, but on 2.0 it's still threaded by Order Received which probably corresponds quite well with Received - and that causes problems surprisingly often.

> I've changed a bit of your date generation code. It used to fall back to the > Date header, and then PR_Now(), if it couldn't find a Received header. I've > just made it display a space " " character. I think it needs some kind of fallback. While you might trust the MTA to insert a Receceived header, you cannot trust all mails to be handled by an MTA. Locally generated mail could well be sent directly to the MDA or even appended to an MBOX file (or Maildir) without passing the systems normal MDA.

(In reply to comment #159) > > I've changed a bit of your date generation code. It used to fall back to the > > Date header, and then PR_Now(), if it couldn't find a Received header. I've > > just made it display a space " " character. > > I think it needs some kind of fallback. Yep, I changed my mind on this one. The latest patch does fall back.

Patch v1.0 submitted. This one does quite a bit more; adds the received column, and the ability to sort by it, as well as group messages by it. It hides itself for folders other than Inbox, Trash, and other POP folders (except unsent, sent, and drafts).

Iif you change the sort type enums, you break all the .msf files with saved sort orders, because we write out the ordinal value of the current sort in the .msf file. So I don't think you can do that. Why did you want to that?

I thought I saw somwhere in the codebase that 'byCustom' had to be last?

I don't know why that would be - mxr link?

OK, I tried adding the const after 'custom' and recompiled - seems to work. Here's the new patch.

Comment on attachment 273854 [details] [diff] [review] Patch v1.1 Looking pretty good. Some things I noticed: // sort_type is an int value reflecting nsMsgViewSortType values // as defined in nsIMsgDBView.idl (byDate = 18, byId = 21 etc.) pref("mailnews.default_sort_order", 1); // for Mail/RSS/... (nsMsgDatabase) -pref("mailnews.default_sort_type", 18); // +pref("mailnews.default_sort_type", 19); // pref("mailnews.default_news_sort_order", 1); // for News (nsNewsDatabase) -pref("mailnews.default_news_sort_type", 21); // +pref("mailnews.default_news_sort_type", 22); // are these values right? Since you're not changing the enum anymore, it doesn't seem like you should be changing the news default, since it can't be received... This code looks like it has a tab in it, and the else if brace should be on a line by itself. same with this line: + if (!LL_IS_ZERO(m_receivedTime)) { // Upgrade 'Received' to Received: ? + PRTime2Seconds(m_receivedTime, &rcvTimeSecs); + else if (deliveryDate) { // Upgrade 'Received' to Delivery-date: ? + PRTime resultTime; + PRStatus timeStatus = PR_ParseTimeString (deliveryDate->value, PR_FALSE, &resultTime); + if (PR_SUCCESS == timeStatus) + PRTime2Seconds(resultTime, &rcvTimeSecs); + } + m_newMsgHdr->SetUint32Property("dateReceived", rcvTimeSecs); + and here: if (date) { // Date: I still haven't had a chance to apply this patch because my tree has some changes to the view code to implement stable secondary sorts, which I'm still finishing up, but I should have that done in the next few days.

(In reply to comment #166) [...] > // sort_type is an int value reflecting nsMsgViewSortType values > // as defined in nsIMsgDBView.idl (byDate = 18, byId = 21 etc.) > pref("mailnews.default_sort_order", 1); // for Mail/RSS/... (nsMsgDatabase) > -pref("mailnews.default_sort_type", 18); // > +pref("mailnews.default_sort_type", 19); // > pref("mailnews.default_news_sort_order", 1); // for News (nsNewsDatabase) > -pref("mailnews.default_news_sort_type", 21); // > +pref("mailnews.default_news_sort_type", 22); // > > are these values right? Since you're not changing the enum anymore, it doesn't > seem like you should be changing the news default, since it can't be > received... Well spotted. I've fixed this now. > This code looks like it has a tab in it, and the else if brace should be on a > line by itself. Looks like you missed out the code you meant to reference here? > same with this line: I've been through with a hex editor :-) and fixed the two lines I could see that I added that had tabs in, as well as the brackets. By the way, there's a bunch of existing code with inconsistent bracketing and tabs spattered all over the place. Maybe you should give up on the tab consistency thing. :-P > I still haven't had a chance to apply this patch because my tree has some > changes to the view code to implement stable secondary sorts, which I'm still > finishing up, but I should have that done in the next few days. I also fixed some code that prevented the 'sort by' menu from having the Received option selected or the 'group by sort' option enabled, when sorting by Received. The one remaining thing I can see that might be wrong is the behaviour when sorting e-mails in threaded mode. However, I really can't understand how TB is meant to do this and I'll need to look into it more. It seems to sort the threads by the Received date of the root thread e-mail, rather than the Received date of the most recent thread entry. Looking at the official TB release I have, it looks like it should do the latter. Not sure what code is messed up there.

(In reply to comment #167) [...] > The one remaining thing I can see that might be wrong is the behaviour when > sorting e-mails in threaded mode. However, I really can't understand how TB is > meant to do this and I'll need to look into it more. It seems to sort the > threads by the Received date of the root thread e-mail, rather than the > Received date of the most recent thread entry. Looking at the official TB > release I have, it looks like it should do the latter. Not sure what code is > messed up there. This error happens when sorting by Received and turning threading on. In addition, when you sort by Date and then turn on threading, the threads get sorted by the Received date of the thread's most recently Received message, when it should be using the Date of the thread's most recently Dated message. The behaviour is what I want when sorting by Received and turning threading on, but not when sorting by Date and turning threading on. So both seem to be behaving incorrectly at this point. The threading code is complex.

FYI. There are at least 5 bugs which can be observed while thread display. (some are fixed on trunk) 1. Bug 254159 : Order of main thread. 2. Bug 262316 : Order in a thread. 3. Bug 181446 : Thread will be broken when child message is followed by parent message if subjects are not the same 4. Bug 264941(Thunderbird) / Bug 388205(Seamonkey) View->Threads->"Threads with Unread" changes sorting to "Order Received" 5. Bug 369620(Thunderbird) / Bug 389935(Seamonkey) Threading via column header forces Sort by "Order Received", when previous sort order is chosen by column header click (not via View/Sort By menu) Bug 181446.

(In reply to comment #169) > 1. Bug 254159 : Order of main thread. I've advised that this should be WONTFIXed, following the fix to this bug being checked in. > 3. Bug 181446 : Thread will be broken when child message is followed by parent message if subjects are not the same It looks like this bug is on the verge of being fixed, with a patch just waiting for a superreview. > 5. Bug 369620(Thunderbird) / Bug 389935(Seamonkey) > Threading via column header forces Sort by "Order Received", > when previous sort order is chosen by column header click > (not via View/Sort By menu) I have a working patch for that bug, that I'll submit once the fix to this bug is checked in.

Fixed 1 or 2 things so small that I can't remember what...

OK, sorry for the delay. I've been running with the patch, and I think it does what it intends. I have a couple issues with the experience, however. I'm not sure what the solutions are, but here are the issues: The Date Received column is blank for existing headers. I understand why, but it's not a very nice out of the box experience. I wonder if the view code could display the Date: value in the Received column if the Received value is 0. Just a thought, some people will barf, I'm sure. The alternative of rebuilding all the summary files isn't very attractive. Out of the box, we show both Date: and Received columns. That's a waste of space, IMO (though mail.app does it, I believe); I think most users are only going to want one. For new profiles, I'd be happy just showing the Received column; for existing profiles, you can't really switch to showing the Received column instead of the Date column, because they very likely are sorted by the Date: column. And I don't think we can just not show the Received column by default - for one thing, Jeremy's head would explode, and for an other, as the patch stands now, the default sort for new folders is by Received, and having a default sort by a column you don't display is not nice. Thoughts? The patch looks fine, code-wise, but I'd like to deal with these usability issues.

(In reply to comment #172) > a thought, some people will barf, I'm sure. The alternative of rebuilding all > the summary files isn't very attractive. It does seem rather unpleasant, but would it be unacceptably bad as a kind of once-off "Updating your mailboxes for Thunderbird 3" thing? The only other options I can think of are (a) if the received column is displayed, then initially display nothing if the received value is 0, but immediately calculate and add the received values for any messages that are visible in the pane, so they'd appear soon after, or (b) substitute "date:" for "received" to start with, but start rebuilding the summary files in the background whenever cycles are spare.

(In reply to comment #173) > It does seem rather unpleasant, but would it be unacceptably bad as a kind of > once-off "Updating your mailboxes for Thunderbird 3" thing? But this might well be implemented before TB3... > The only other options I can think of are (a) if the received column is > displayed, then initially display nothing if the received value is 0, but > immediately calculate and add the received values for any messages that are > visible in the pane I'm not sure this is possible with the current code, it's generally all-or-nothing when it comes to calculating message index data. > (b) substitute "date:" for > "received" to start with, but start rebuilding the summary files in the > background whenever cycles are spare. Again, not sure this is possible with the current code. I think I prefer David's idea of displaying the Date: value instead for any received fields that contain 0.

(In reply to comment #174) > Again, not sure this is possible with the current code. I think I prefer > David's idea of displaying the Date: value instead for any received fields that > contain 0. I don't like this solution at all. There's no way to tell that the value you're seeing in the field is date instead of received. What's wrong with recalculating the index? I have folders with >~10,000 mails in them on an imap server, and I've had a number of times when I've had to re-download the headers and recalculate. It's never taken longer than I would consider reasonable..

(In reply to comment #175) > What's wrong with recalculating the index? I have folders with >~10,000 mails > in them on an imap server, and I've had a number of times when I've had to > re-download the headers and recalculate. It's never taken longer than I would > consider reasonable.. By now my profile is 6.7 GB. It does take quite some time to recalculate the indexes and the application isn't very informative during that time. More so on machines with slow disks and not much RAM. I know what's going on in the background and what has triggered it. Others don't. So at least an assessment of the impact on the average user should be made.

(In reply to comment #176) > By now my profile is 6.7 GB. It does take quite some time to recalculate the [snip] > of the impact on the average user should be made. I don't think an average user would have even a 1GB profile, so I suspect >6GB is an exception. I agree, some performance analysis would be nice, but at the end of the day, no matter how slow it is, I think it's preferable to having inconsistent values in the field.

Even if it's a bit slow to recalculate because a user has an enormous mailbox, I think it'll be fairly acceptable if there's an alert message that this is a one-off necessary convert / rebuild.

We can't just indiscriminately blow away .msf files. For example, if you have an imap server that doesn't support keywords on the server, blowing away the .msf file will remove all your tags. In my experience, you mostly care about the difference between received and date values for new incoming mail, e.g., when some spammer sets the mail to be from the future so that it will show up at the top (or bottom) of your mail.

Updated patch. Now received column echoes date column if it has nothing to display. Received column is inserted in the columns list just before Date. Received column will appear when TB is upgraded, and Date is hidden by default on new installs. Threading behaviour is totally correct, AFAICT. Unfortunately, whilst grouping works for today's messages and earlier, it is broken for messages whoes dates are in the future - they each get grouped in their own bucket. I can't for the life of my figure out why, so help! Apart from that, I think the patch is there, though, feature-wise.

Comment on attachment 275585 [details] [diff] [review] Patch v1.4 thx, Jeremy. Finally got a chance to run with this. It looks good - I tried a new profile and verified that it just shows the Received column, not the Date column.

Am I to understand that the patch changes the default behavior for new profiles to show only the Received column and not the Date column? If so, I object to that change. On a technical note, new code should not use the LL macros, which are deprecated. All supported compilers on all supported platforms now support 64-bit types (even if the generated code emulates the behavior). In new code, when using the PRInt64 and PRUint64 types, ordinary arithmetic statements and ordinary comparisons should be used rather than the LL macros. The compilers are likely to generate more efficient code using their own 64-bit implementations than when using NSPR's macros.

(In reply to comment #182) > Am I to understand that the patch changes the default behavior for new > profiles to show only the Received column and not the Date column? > If so, I object to that change. Using Received is superior to using Date because it's generally more reliable, and it's also what most other mail clients do.

This patch fixes the problem where, when you tried to sort by Received in a folder whose index hadn't been rebuilt, it fell back on ID for messages that didn't have the Received attribute. Now it uses Date instead.

Apologies, the 1.5 patch still had a debug printf() in that I left. This doesn't.

Received is UTTERLY USELESS in environments where servers randomly delay some emails, by up to 24 hours. When the messages finally arrive, I want to see them in the order they were sent. I don't have a problem with senders having bad clocks. I have a BIG problem with screwey mail relays. I'm not alone. All the people served by my various mail service providers have the same problems. This is a change that affects how the product works for ALL the users who have used it for years, to benefit a few newcomers who expect it to be different. Do we want to alienate our established user base to entice some Outlook Express users?

No offence, but why not get your ISP(s) to fix their MTAs? MTAs should NOT be doing that on a regular basis. In my experience, it's a far bigger problem to have misconfigured local system clocks or just plain spoofed dates (usually by spammers) than for MTAs to be broken. I run my own mail server, I'm not even that much of an expert, and it doesn't hold mail up randomly.

Jeremy: No offence, but why not get your correspondent(s) to fix their machine's local system clocks? If you can say "just fix the MTAs", why can't Nelson say "just fix the local clocks"? After all, having your local system clock wrong can lead to all sorts of other problems too. BTW, spammers spoofing dates in the future is a feature, not a bug. It causes all the spam (which the filter doesn't catch) to collect together in one place, ripe for deletion. It's the nicest thing they do. Gerv

I've been thinking for a long time since mailnews.use_received_date=false/true was landed by David that; - Nelson can go same way as current by mailnews.use_received_date=false and by display of Mail_date(=Date:) column, and can display Received: column if needed. - Jeremy can change to new way by mailnews.use_received_date=true and by display of Mail_date(=Receieved:) column, and can display Date: column if needed, although "rebuild index" is required to change to new way. - Mail_date is always used for sorting, mail aging and so on. Are these already incorrect? (Sorry but I don't understand new design well yet.)

The obvious answer to this dilemma is to have an "OE compatibility mode". Lots of products have these modes of compatibility with their competitors' products (word processors, especially!). Word has a WordPerfect compatibility mode. Excell has (or had) a 1-2-3 compatibilty mode. If we're trying to attract OE users, let's make it clear that we're doing that, and make it be an option chosen at profile setup time (default: OFF). BTW, you can construct a filter to junk or delete all email from the future. It shouldn't be necessary to reorder the way you display ALL emails, just to make it more convenient to delete a few spams that can be filtered out.

(In reply to comment #186) > This is a change that affects how the product works for ALL the users who > have used it for years, to benefit a few newcomers who expect it to be > different. Do we want to alienate our established user base to entice > some Outlook Express users? I'm a user who has used TB for years. I logged this very feature request almost 5 years ago, so it's been at least that long. Prior to that, I used Netscape Mail & News, which also used sent date. Personally, I've always found it to be a major annoyance, and I can't believe I'm the only one. So right off the bat, I'd guess your concerns are probably only shared by a small percentage of current users. I really don't believe changing this to a default will "alienate [the] established user base". Those who rely on sent date will simply switch it back. They're more likely to realise that sent date is even available as an option than a new user who probably hasn't even considered the possibility that a client might use something other than received. Usage stats for MUAs is hard to come by, but I can't believe TB has more than maybe 20%. That means there's an 80% market share to be gained, most of whom will be currently using clients which use received date. Is the goal to entice new users? I assume so.. in which case, why not make it easy for them? People who are trialling software are going to be far less forgiving of perceived inadequacies than long term users seeing a change (which can be simply reverted). The idea of an "OE compatibility mode" is, to my mind, completely over the top. We're talking about one simple column and whether it's on by default. Why not make both on by default and let people remove the one they don't want? Doesn't that seem a lot simpler?

(In reply to comment #191) > (In reply to comment #186) > > This is a change that affects how the product works for ALL the users who > > have used it for years, to benefit a few newcomers who expect it to be > > different. Do we want to alienate our established user base to entice > > some Outlook Express users? > I'm a user who has used TB for years. I logged this very feature request > almost 5 years ago, so it's been at least that long. The same applies for me - and I NEVER used Outlook Express and also desired that since a lot of years. But that were days with less spam... (In reply to comment #188) > BTW, spammers spoofing dates in the future is a feature, not a bug. It causes > all the spam (which the filter doesn't catch) to collect together in one > place, ripe for deletion. It's the nicest thing they do. I did not think about this. "Just fix the local times" is something I experienced in the past that people simply did not care that much. But with the rising amount of spam we have no THE reason for everybody to take care of there local times. And actually since aggressive anti-spam-software do filter often more than it should people now have a real interest in having their clocks set correctly and so I recognize that actually the original problem does occur less and less. So in the meantime I think I can live without that additional column - so for me this issue is obsolete.

(In reply to comment #191) [...] > The idea of an "OE compatibility mode" is, to my mind, completely over the top. Indeed. Whatsmore, calling it a 'compatibility mode' implies that it's only being used for compatibility with something, perhaps to be deprecated or phased out later. That is not the case. Received is just a more reliable alternative to Date, in most cases. I have to say that this is not the kind of thing I was expecting a Holy War to break out over, and i'm quite disappointed that it has. > We're talking about one simple column and whether it's on by default. Why not > make both on by default and let people remove the one they don't want? Doesn't > that seem a lot simpler? It's not the column that's in contention, it's the default mode for sorting threads. Say you sort by Subject, then go into threaded mode. When you expand a thread, what do you sort the child threads by? Not subject, because the subjects are probably all the same. Previously it was by Date, which made sense. Now some want it to be by Received (me included), because we think it's more reliable, others want Date still. I say make Received the default and have a pref to switch it back. Nelson, what is your problem with that? It could use the existing mailnews.use_received_date pref, which if set to true (by default) will cause threads to be sorted by received.

(In reply to comment #193) > it's the default mode for sorting threads. I've understood what is the current issue now. Thanks Jeremy. Question: What about mail aging, filtering based on date related data? (i.e. what is used as "mail_date" after your enhancement?)

I don't know what mail_date is. There's no such token in the sourcecode.

Sorry about the bug spam. Judging from programming comments, it seems like you're really close to a resolution of this bug. Let me just say I very much appreciate this! (In reply to comment #186) > This is a change that affects how the product works for ALL the users who > have used it for years, to benefit a few newcomers who expect it to be > different. Do we want to alienate our established user base to entice > some Outlook Express users? As a long time user, I feel compelled to speak up and say I will not be alienated. For me, sorting by and displaying Received date is the right thing to do. Displaying "Date" is too ambiguous. You might say "this is a change that affects how the product works for ALL the users who have used it for years, finally fixing a long time bug" I don't really care what the default is, or if it's changed, but it sure makes sense to me to make it something that is used by most other email clients (afaik). For instance, when I moved from Outlook to TB, this bug was obvious, and a major annoyance to me. I never even considered the "Date" column in Outlook because it was always correct. This was one of the items on the "minus" side when I considered whether to stay with TB, or return to Outlook. (In reply to comment #188) > Jeremy: No offence, but why not get your correspondent(s) to fix their > machine's local system clocks? If you can say "just fix the MTAs", why can't > Nelson say "just fix the local clocks"? After all, having your local system > clock wrong can lead to all sorts of other problems too. It's a matter of the relationship with the sender. You might receive email from clients. It might not be appropriate to ask your client to fix their system clock. To paraphrase, please consider that many who use Thunderbird might not be in position to ask the sender to correct their system clock. On the other hand, if your mail server is giving you an incorrect "received date" then you only have one entity to deal with, and (without any exception I can think of) you are dealing with someone who views you as a customer. Personally I would not even ask friends to fix their system clock. Why? Because they would think I'm a moron if I tried to explain that for my email client to work correctly, I need their system clock to be correct.

(In reply to comment #193) > (In reply to comment #191) > > The idea of an "OE compatibility mode" is, to my mind, completely over the top. > Indeed. Whatsmore, calling it a 'compatibility mode' implies that it's only > being used for compatibility with something, perhaps to be deprecated or Plus "Date received" is also the default date column for Apple's Mail app. So the use of it can't be reduced to some Microsoft "deviation" from an accepted standard. Although, interestingly, Mail.app does appear to display "date sent" in the mail header itself.... (In reply to comment #188) > BTW, spammers spoofing dates in the future is a feature, not a bug. It causes > all the spam (which the filter doesn't catch) to collect together in one place, For the few spam mails that spoof future dates, anyway. And I receive a number of emails from friends that appear to have been sent before they're received. But I presume this wasn't asserted too seriously. Otherwise it'd be like saying "some weeds have purple flowers, so if I segregate all the purple flowers I can fix my weed problem" :)

good people. just an observation that as of january this bug already had 100 comments and it will soon hit 200. It's hard to imagine there is *anything* that hasn't already been said. Please let the devs decide what will be implemented based on the patch as supplied. Besides, for now, it's only a patch on trunk.

This patch adds the mailnews.use_received_date pref that controls whether grouped/threaded messages are sorted by Received or Date.

Comment on attachment 277116 [details] [diff] [review] Patch v1.6 m_useReceiveDate should be cached in a static global, instead of once per thread. If you have 1000 threads, you'll have 4000 bytes of data with the exact same value in it. You should be able to find lots of places in the code that cache prefs in a global var. for example, http://mxr.mozilla.org/mailnews/source/mailnews/db/msgdb/src/nsMsgDatabase.cpp#228

the example you give of a static global creates inflexibility - see https://bugzilla.mozilla.org/show_bug.cgi?id=377573 while received date may work well for pop/imap, have we considered that batch received rss feeds would all have the same received date - set at time of poll interval - and it really only makes sense to display sent (posted) date? date spoofing is not an issue here. a one time all-or-none private var set at load is not good. the pref should be an attribute of a folder, a g var, or at least should be reread/updated on folder/account change.

true, you shouldn't be using received date for news or rss feeds. But, the way I'd wrote the code, anyway, if there's no received header, we use the date: value anyway.

This patch's functionality is identical to 1.6, but it uses static globals to hold the use received data instead of class members.

(In reply to comment #195) > I don't know what mail_date is. There's no such token in the sourcecode. Sorry for my incorrect wording. My question is: When mailnews.use_received_date=true, and when "sort order" is based on Date: header value, which timestamp value is used for filter of "Age in Days is ..."? What for filter of "Date is ..."? (predefined Date, not user added Date) Note: User added item name Date in filter means Date: header(string), but predefined item name of Date is date value obtained from Date: header or Received: header(date type data). To distinguish these two different "Date", I called the latter "mail_date" in previous question.

(In reply to comment #204) OK, I think I understand now. > When mailnews.use_received_date=true, > and when "sort order" is based on Date: header value, > which timestamp value is used for filter of "Age in Days is ..."? The Date: header. > What for filter of "Date is ..."? (predefined Date, not user added Date) The Date: header.

To Nelson Bolyard: What is your expectation for "Age In Days is ..." and "Date is ..."(Not for Date: header user added) of filter in next four cases? What is "should be" do you think? 1. mailnews.use_received_date=false, Thread sorting based on Date: header 2. mailnews.use_received_date=false, Thread sorting based on Received: header 3. mailnews.use_received_date=true, Thread sorting based on Date: header 4. mailnews.use_received_date=true, Thread sorting based on Received: header

Comment on attachment 277170 [details] [diff] [review] Patch v1.7 if you'll look at the example I pointed you at before, you'll see that there are two globals - one that says whether we've read the global pref, and one for the global value. This saves us the trouble of reading the pref for every thread that's created. I'm pretty sure we're going to want to default this behavior to use the date header, not the received header, from what Scott says. What I'd really like to do is get a patch that does the Received column, off by default, and get that checked in, and do all the other stuff in a subsequent patch(like your recent changes for this global pref). That way, something gets landed, and can get used and tested, and that way, the subsequent patches are smaller and easier to review.

(In reply to comment #207) > I'm pretty sure we're going to want to default this behavior to use the date > header, not the received header, from what Scott says. Well I don't know how to have this discussion with you anywhere else, because you don't seem to respond outside Bugzilla which is unfortunate, so I'll have to do it here. Please explain the rationale behind this decision. A few people want the current behaviour to remain and stuff to get sorted by Date by deault, but it seems to me from reading this bug that for every one of them, there are 2 people who think the current behaviour is bad and want it replaced with sorting by Received. Why not listen to the majority, especially when they're logically in the right? > What I'd really like to do is get a patch that does the Received column, off by > default, and get that checked in, and do all the other stuff in a subsequent > patch(like your recent changes for this global pref). As you say, stuff that gets landed can get used and tested. Whatsmore, it would be landed in the trunk, which gets 'experimental' stuff landed in it all the time, and is largely for testing purposes. Why, then, would you want me to take out the thread and group sorting functionality for this patch? Are you seriously telling me it's too 'big' to review? Why isn't it better to get this functionality in ASAP so it too can be tested? Allowing to sort by Received column but not sort threads/groups by received seems inconsistent to me.

I'd like to get the non-controversial parts landed. Is that hard to understand?

Well, ok. Patch v1.8 doesn't include the "controversial" stuff.

Comment on attachment 277591 [details] [diff] [review] Patch v1.8 seems like if we're not showing the received column by default in this phase, we wouldn't need any changes to the UpgradeThreadPaneUI method. I'll try the patch w/o those changes

What is the purpose of the trunk build? Seriously? Is it for something other than testing, because I didn't realise. If you hide the column, nobody will use it. Why are you even bothering to check it in? And please, satisfy my curiosity, are there *REALLY* significant numbers of people (please give me a rough number) who are actually going to the effort of contacting you to complain about a patch that hasn't been checked in yet? Who are these people? You seem to have done a virtual U-turn in opinion; you wanted to *replace* the date column with received, and the thread ordering too. I can understand wanting the pref and everything, I can even understand making it off by default (although I disagree), but my understanding has completely ceased at 'put in the functionality but make sure nobody notices it'.

(In reply to comment #206) > To Nelson Bolyard: > What is your expectation for "Age In Days is ..." and "Date is ..."(Not for > Date: header user added) of filter in next four cases? > What is "should be" do you think? > 1. mailnews.use_received_date=false, Thread sorting based on Date: header > 2. mailnews.use_received_date=false, Thread sorting based on Received: header > 3. mailnews.use_received_date=true, Thread sorting based on Date: header > 4. mailnews.use_received_date=true, Thread sorting based on Received: header Wada's first question seems to be whether to use Date or Received header in FILTERS for filtering of incoming mail. Do I understand that right? The next 4 questions seem to be about "Thread sorting", which clearly happens after filtering (messages don't appear in the message list pane until they have been filtered). I think they are actually asking two questions, which are: a) when mailnews.use_received_date=false, should threads be sorted by Date: header or by Received: header? b) when mailnews.use_received_date=true, should threads be sorted by Date: header or by Received: header? My answer to those last questions is the subject of bug 383985, which states that "Order of *threads* in msg list pane should not be controlled by Date Sent vs Date Received". But maybe Wada's didn't mean to ask about the order of threads, but rather the older of messages (in threads? when display is threaded? or not?). Maybe the questions can be restated?

Nelson: I think you're failing to realise that Received and Order Received are two very different things. Received isn't currently supported in TB; this patch aims to make it so. Order Received is something that is pretty much internal to TB; it's TB's internal track of the order messages were received and has *nothing* to do with the Received header - the name just happens to be similar. What I think WADA was trying to suggest in his/her question was that 'Age in Days' and 'Date', in the filters, are currently based solely on the Date header, and perhaps some people would want them to be based on the Received header instead. This is true, but the way I would suggest implementing that is: 'Date' always uses the Date header. 'Age in Days' perhaps uses the mailnews.use_received_date pref to determine which header to use. To filter by Received, add a 'Received date' filter option. However, that stuff will definitely not be added in the patch to this bug, and will need to be noted as a new RFE bug.

In answer to comment 212, the trunk is the place where the next MAJOR revision of the product is supposed to come from. It's not a playground. Changes made there are changed intended to appear in the next major release.

In reply to comment 214, Jeremy, thanks for your reply. I'm OK with mailnews.use_received_date= consistently controlling whether date is taken from Date: or Received: headers. I'd certainly want WADA's choices 1 & 2, not 3 & 4! IOW, false == date from Date:, true== date from Received, except for places where the user has an explicit choice between the two.

(In reply to comment #215) > the trunk is the place where the next MAJOR > revision of the product is supposed to come from. It's not a playground. > Changes made there are changed intended to appear in the next major release. Well, I don't know if the TB trunk is anything like the FF trunk (I assume it is), but things are frequently checked into the FF trunk that aren't necessarily going to stay; for example, recently there was a patch checked in to grey out the non-hostname parts of the URL. That was removed after a bunch of complaints. They hadn't properly consulted before checking it in, and it was a bit of a 'playground' addition. Frankly, I don't see the problem. Trunk is meant to be for testing, and as long as a patch is relatively sound (which 1.8 is), it should be fair game for checking in.

I applied this patch, minus the "upgrade" code to add the received column, but the received column is still getting displayed, even after I removed it using the column picker. So I need to figure out why that is. I know you can't see any possible reason someone would want to remove the Received column, but I need to fix that before I can check this in :-)

So the problem is the function ShowReceivedColumn - whenever it's called with true, the received column is shown. My bad for not noticing how broken that was before. It basically copies the logic for ShowLocationColumn, but that's not the right thing to do. I'm not sure that this is even needed, since we show the Date value in the Received column, when there's no received value. It would be nice to switch in the Date column for those folders where Received doesn't make sense...but I'm not sure if we can do that easily.

(In reply to comment #219) > So the problem is the function ShowReceivedColumn - whenever it's called with > true, the received column is shown. Whoops - well spotted. Patch v1.9 is the same as v1.8, except it uses a persistent 'washidden' attribute to only show the column if it was selected before.

... and of course I manage to screw up msgmail3panewindow a bit in v1.9. This one is fixed.

the other issue I saw last night was that threadpane.xul by default hides the date column and shows the received column - that's not what we want right now, so I've fixed it locally.

Comment on attachment 277907 [details] [diff] [review] Patch v1.9.1 the washidden logic isn't quite right. washidden never gets set unless you switch to a folder that requires hiding of the received column. Otherwise, it never gets set. Thus, if I turn off the received column, shut down, and restart, the received column shows up again. I think washidden needs to get initialized with the hidden status of the received column, on startup. I'll try that.

I'm still not sure how to fix this - I think we need to know when the user hides the column, and when the code hides the column. I tried re-working it so the variable is called "temphidden", meaning the code hid it, but I'm not sure when to clear that - maybe when you click on a folder that doesn't require hiding, you clear "temphidden". It's really not a nit - people are going to be annoyed if the columns keep switching back and forth on restart based on what the last folder they shut down with. I.e., as it stands now, imagine you have chosen to show the received column. If you read a newsgroup, and then shut down, when you restart, the received column will not show on the inbox.