I have this problem now - ever since I removed and then recreated my Thunderbird profile, in an attempt to escape bugs. I despair.

Things are now thus: emails that should have been moved by filters from my inbox are still there; when I try to delete them, some other e-mail gets deleted. This is just a disaster.

Also: why is this bug report so hard to find on Bugzilla?

No, filters do not help the situation at all.

ABSOLUTELY NOTHING helps. Including the folder compression/restructuring and everything else discussed here.

What is worse is that developers are in complete denial of this problem. They must be lucky enough to not have those zero-length shadow letters.

I should like to know, developers, what your attitude to this problem is. Must I and others (1) hope that the Gods smile upon us and hence that filters start working properly - as against creating phantom emails and, indeed, in my case, making it the case that when one tries to delete such an e-mail, another e-mail - which one may want to keep - disappears from view?

Or perhaps (2) we should give up on Thunderbird, after (in many cases) many years and with much of our lives tied up in it, and despite all its (other) virtues, and move to some possibly closed-source, possibly data-mining alternative?

Or is there perhaps (3) the option that this bug - or these bugs - dating back, in this thread alone, sixteen years (sic) might actually get fixed?

I imagine there might be some professional pride involved as well.

Please do not tell me to make a new profile. For, I started having the problem - which I had had once before, years ago - just after . . I had created a new profile. Moreover, even if there is something unique and even user-caused about the problem in my case, there are I do not know how many contributors to this thread. So, we have here, at the least, a problem of Thunderbird robustness - and a rather serious one too, even aside from the boggling amount of time this problem has been around.

I appreciate that there has been some intemperance on the part of the users. But, really . .

AFAIK, we don't have any steps to reproduce this problem. Once we have, the problem should be fixable.

True. Please add steps-wanted when we need reprodicble steps

There are no specific steps. Those phantom mails sometimes appear, and sometimes not.

The problem can be fixed once and for all forever if Thunderbolt had just like three lines of code to check if mail has zero length (hence e.g.date defaults to 1970) and then not to add it to the list of mails, to never render it.

Can anyone with access to Thunderbird code please do this?

Re specific steps and possible causes: please see the extensive technical discussion in this thread between WADA and me in the fall of 2014. A fair amount of debugging was attempted, considerable information about the affected folders and .msf files was provided (including screenshots and other uploads), and WADA had a number of ideas as well as pointing to other bugs that can in fact be used to reproduce the phantom messages.

I can no longer help because it only ever happened for me with Junk processing and auto-expiry into Trash, and these days I see essentially nothing that ends up as Junk. The ISPs for my older email accounts (in use at the time of those earlier discussions) basically block everything now, and my new privately-hosted email address, for which I use IMAP instead of POP3, either is treated the same way or is simply too new to have been scraped from anywhere (or IMAP doesn't display the problem). NB This black-box spam-handling approach annoys the hell out of me because I like to see what spam is being sent to me. Not only does the current situation mean I can't tell whether there are any false positives (far from unheard-of), the subaddressing scheme I use also identifies who leaked my address, and then I get to yell at them. If I don't see anything, I can't deal with either of those problems.

As a result of the above email changes, I never see the phantom message problem any more. But I think it would be really helpful if the TB developers could acknowledge that the problem is real, that it has affected a large number of users for a very long time, and try to find someone who can work with them to debug it instead of just saying "we can't fix it without more info".

(In reply to Ruth Milner from comment #227)

try to find someone who can work with them to debug it instead of just saying "we can't fix it
without more info".

Isn't that what I just said? If anybody has steps to reproduce, we can work on fixing it. That someone must step forwards.

I wonder why Thunderbird's code allows zero-length mails to be added to the list.

No need for a debugger to check this. (Besides, to run it you have to set up an environment, download fat Thunderbird source code, run it 24/7, etc.)

There is a point when the algorithm fetches new mails and, if it was a success, adds them to your synchronized mail folders in bold font, fires "new mail" to show an icon in notification area, and so on.

The whole issue can be fundamentally solved forever by implementing a simple check that would prevent all those events from happening. It is sad that no Mozilla programmers attempted to do that despite decades of the issue existing. Let's hope that one day someone will do it.

An empty mail (body) is not an invalid email. Some people put all the message into the subject.

There are also other valid reasons why an "empty" mail can be a valid, like when you have set up Thunderbird to only download headers at first.

Any "simple" fixes will just break some other use case, that's why we need to know the cause, to fix that and not break things for others.

It is not just the body of the phantom message is zero-length, it is is ALL of it, including the subject, the dates, the headers, etc. There is literally nothing there (people can see it by themselves by selecting "Show source" on such phantom messages).

So why it is impossible to built-in a tiny check for the whole message's object? It can not disrupt anything as at no other cases zero-length messages exist or should exist.

(In reply to Magnus Melin [:mkmelin] from comment #228)

If anybody has steps to reproduce, we can work on fixing it.

There are ways it can be reproduced described by WADA above, and there are various types of logs and other data posted above as well. It is complicated by the fact that it seems to happen in more than one way. In this bug thread, people mostly refer to a correlation with message deletion. At the very least these reports/comments present lines of inquiry. A few thoughts:

o it reportedly happens with IMAP accounts as well as POP3, but is there any feature in
particular that the setups have in common, e.g. offline storage?
o Does it occur with any folder or only a limited subset? So far Inbox, Trash, and Junk
have be mentioned; is that's all? If so, it may be meaningful.
o I documented above that, when a phantom appears, there is no difference in the content
folder before/after, but there is a change in the .msf file before/after. There is
even an uploaded screenshot and some pasted text of these differences.
o It appears there may be a race condition of some sort involved, since the problem
occurred reliably with normal execution, but not at all when NSPR logging was used.
o It first showed up for me when upgrading from TB 1.x -> 3.1 and has continued with
every version since. What changed in 3.x that might be relevant? WADA mentioned
something, but I don't know if it is related or is even still in the code.

WADA's extensive contributions and results from a lot of testing begin with comment #162.

(In reply to User Dderss from comment #231)

So why it is impossible to built-in a tiny check for the whole message's
object? It can not disrupt anything as at no other cases zero-length
messages exist or should exist.

But that likely isn't the problem, since no such mails would exist. It is shown as a mail, but it's showing something that is nowhere. The problem is finding why it's showing something that isn't there.

Ruth, thx, but unfortunately non of that give much clue. What we can say is that there seems to be some problem with the .msf file (meaning the folder cached database).

(In reply to Magnus Melin [:mkmelin] from comment #233)

But that likely isn't the problem, since no such mails would exist. It is shown as a mail, but it's showing something that is nowhere. The problem is finding why it's showing something that isn't there.

There is no mystery in that either. Thunderbird shows last previously rendered subject/body as the phantom mail itself has nothing to render (msg file from phantom mail is zero), and its zero date defaults to 1970.

This is why to solve the issue the mail's total (!) length should be checked before it is added to any folders. Weirdly enough, it is not done in the code. And I can not understand why such a simple check can not be implemented.

(In reply to Magnus Melin [:mkmelin] from comment #233)

What we can say is that there seems to be some problem with the .msf file (meaning the folder cached database).

And none of the previously-provided .msf snippets suggest anything? They are the parts that were different when the phantom message was shown/not shown.

Thanks to Ruth Milner and all those, especially the Thunderbird developers, who have responded helpfully to my call for a fix.

Even though I am not a developer and even spend what I regard as too much time reporting bugs for open-source software, I would be happy to supply whatever the Thunderbird developers need by way of logs. (I have the problem on my Windows computer. Also, I might - though, frankly, I hope not - be able to reproduce it on my Linux computer. The latter has a lighter TBird installation: fewer stored messages, fewer filters.)

So . .

Currently on my system the filters get (visibly) applied as soon as one refreshes the painting of the inbox by switching the view to a different folder and then switching back again. So surely some sort of automatic refresh would fix this, at least in the version I suffer from it currently?

Also: come on devs, you showed some willingness, recently, to engage with the problem, but now you have gone quiet again.

None of the comments above are labeled STR. Data is needed. So here is how you can assist moving this bug forward
a) which of Ruth's and/or Wada's comments reliably reproduces the issue, with the barest amount of steps
b) provide msgdb log to go with a) above
c) identify whether it happens with pop-only, imap-only, or both

Thanks.

I do not know the meaning of 'STR'.

Re your point (a): you wish me to identify which of those comments allows one to reproduce the issue and, if there is more than one such comment, identify that which contains fewest steps? Alright, I will try.

Re your (b): provide the 'msgdb' log corresponding to the procedure identified in step (a)? OK. (I will have to look up what and where that log is. I have a Windows installation of TBird and a Linux one but only the former has the problem.)

Re your (c): I use only IMAP. But I suppose I have e-mail account on which I can enable POP. Subsequently though I would have to ensure that that account receives e-mail - e-mail, I mean, to which one of the problematic e-mail filters applies. I would like it were someone else to try this step; it's a lot of work for me.

STR
https://wiki.mozilla.org/BMO/comment_tagging#Highlighting_Important_Comments
https://developer.mozilla.org/en-US/docs/Mozilla/QA/Bug_writing_guidelines#Writing_precise_steps_to_reproduce

C) both POP and IMAP have been reported in previous comments

I should have said, C is tied to A - which is to say, each set STR someone provides must specify whether it reproduces with pop, imap or both.

Note STR must also be reproducible for the other traigers and developer who potentially work toward identifying cause and fix - otherwise, the STR doesn't get us to a solution.

The following steps may suffice to reproduce at least one version of the problem. ('May?' See below.)

1. Install Thunderbird.
2. Create a Thunderbird profile (via any normal means, including via running Thunderbird for the first time and then following prompts) and as part of that specify an IMAP e-mail account with Thunderbird and set the 'message store type' to 'mbox'.
3. Create an e-mail filter (in any normal way) that: moves an e-mail (received via the aforementioned account) to some local folder; has 'subject contains' as its criterion; and that can be run manually (as to the 'filter before junk classification', how one sets that option seems not to affect whether the bug manifests itself).
4. Receive an e-mail that fits the filter.

Should these steps not suffice to reproduce the problem (on a developer's installation of Thunderbird) then I invite suggestions about missing necessary steps. Possibly one needs more than one e-mail account. Also one might try converting an existing Thunderbird email account to use 'mbox' (from using 'maildir').

It may be hard to identify all the necessary steps - because of the number of Thunderbird options and add-ons. Such difficulty does not entail that this bug is not a (deeply irritating) bug. Nor do I find it clear that it is users' responsibility to identify the extra prerequisites. I do realise that someone has to in some way identify those steps, if the bug is to be fixed. Yet: if you, developer, cannot reproduce the problem, then why don't we keep adding specifics of my installation - adding them to your test installation - until you can reproduce it? Thanks.

I will provide a log as soon as I know how to provide one.

It seems the wanted log is discussed here (https://wiki.mozilla.org/MailNews:Logging) and here (http://kb.mozillazine.org/Session_logging_for_mail/news) but the procedure involves setting environmental variables and, on Windows (as against Linux - and, as said, I have problem on Windows) I am not not familiar with such things and so could do with more information than provided by the pages that I cite.

Must I write a batch file that sets environment variables and that runs Thunderbird? Or will it suffice to set variables using say Rapid Environment Editor (https://www.rapidee.com/en/about) and then run Thunderbird normally?

I could try to answer this question about logging myself, but of course the developers know such matters inside out; that's why I am asking.

If it is only environment variables, there may be command-line options to set them when starting TB. On Windows, you can edit the Properties of a shortcut to run the command with extra stuff.

If this is special logging, note what I documented above: when that was enabled in my testing, the problem never occurred. That testing was done at a time when the phantom messages showed up many, many times a day. IIRC, it ran for two days with none. As soon as I went back to normal execution, they showed up within minutes.

This is why I noted a possible race condition in my summary above. WADA commented that some types of logging force TB to do some operations serially instead of how they're normally done, which might prevent the problem from being triggered.

I have clarified logging at https://wiki.mozilla.org/MailNews:Logging#Environment_Variables_to_set
You choose whether the use environment variable, or a batch file. Batch file is simpler.
Note, MsgDB is case sensitive.

I can confirm - albeit on the basis of a small data-set - that, as Ruth said, when logging is enabled, the problem does not occur (!), although I do get a flash near my system-tray when a filter-meeting e-mail arrives, as if to suggest the creation and immediate removal of a new mail notification. So here, strangely enough, we have a work-around of sorts, especially if a very log level prevents the problem occurring. (I used log level 5 for the test, advised.) I will try now to attach the log.

I can't see how to add an attachment (and https://wiki.mozilla.org/BMO/UserGuide did not seem to help).

Also, I discover that my log contains the full text of at least one e-mail that I wrote.

So, please advise (1) where / how to post the log (ideally somewhere that it will expire after a certain period), (2) how to redact it / supply only what is needed. Thank you.

I have (re)discovered how to attach files here. However, the log is absolutely full of confidential information - so do please advise.

I have not seen this problem since I got a more reliable broadband ISP.

But based on past observations:

Set up a dummynet between thunderbird and your mail server.
Start increasing the packet loss rate.

https://www.freebsd.org/cgi/man.cgi?query=dummynet&sektion=4&manpath=freebsd-release-ports

Eventually you should see this problem and probably reproduce a number of other open problems.

It looks like both Firefox and Thunderbird leak memory when there is a network reliability issue. Eventually this memory leak will cause this issue to show up.

This is the same user as malmberg@encompasserve.org. I reset the password for that account and it still would not let me log in.

Definitions matter here. I've had the same Internet Service Provider - i.e. physical connection to the Internet - for 12 years (CenturyLink,the only one available that isn't $$$ satellite). The phantom problem almost completely stopped for me when the spam I receive through my then-primary email provider (Yahoo) went almost to zero. Spam is the target of almost all of the filtering I use in TB, and that is one of the deletion methods multiple people have reported as a trigger for the phantoms. (They would first appear in Junk and then quickly get auto-expired to Trash because the interpreted date was >30 days before.)

In the past few months, i.e. long after the time spam filtering/blocking got tougher (which I'd turn off if I could), my DSL quality has gone from mediocre to pathetic, but the TB phantom problem is still extremely rare. So I don't think it's directly related to connection quality. You might check whether your provider(s) happen to be doing a better job of spam reduction as well as link reliability.

Firefox leaks memory constantly, even on very fast reliable networks like I had at work. TB has far less of a problem with this, even on my crappy home DSL. When I was seeing the problem, it would show up within minutes of starting TB - before it had any opportunity to leak.

Perhaps ISPs are involved in the causation of the problem.

Yet, we know that enabling a certain form of logging suppresses the problem. Can this not be turned into a (less hacky) solution?

I have used the Windows control panel (on my Windows 8.1 computer) to set the user environmental variables as follows: MOZ_LOG to IMAP:1 and MOZ_LOG_FILE to R:\imap.log. I will see whether this fixes the problem.

Another possibility: we pay someone outside Thunderbird - pay one of those will-fix-open-source-for-money companies - to fix this problem.

Feel free to send me MsgDB log for when the problem happens. No promises on when I'm going to be able to look closely at it.

@mkmelin]

The MsgDB log is the one captured by setting the environmental variables, right? If so, what loglevel do you want set? For, when that level is 5, then - as reported previously - the problem does not occur. I discover also, today, that setting the level to 1 does allow the problem. Levels 2-4 are as unknown territory. Also, and as I've mentioned before: at least with level 5 logging, the log contains a lot of confidential e-mail. Also - are you asking for payment, Magnus?

As high level as possible (5). Feel free to remove any confidential information.

And no, not asking for any payment. I'm working for the Thunderbird project.

Magnus: thank you for clarifying that one matter. However, consider the following.

(1) At level 5, the problem does not occur. That is, and as more than one person has reported on this thread: when logging (level 5) is on, the problem is gone - and I would have thought that, thwarting as this is, it itself provides some clues to the would-be bug-fixing programmer.

(2) At least at log level 5, it is non-trivial to extract from the reams of data the text of e-mails I have sent and received. So it would be easier to tell me what you do need from the log.

So: I like you enthusiasm, but I need help in proceeding.

It doesn't necessarily need to be level 5. What we need is a way to reproduce, or potentially just logs can clear up when it happens. I'm just sceptical anything other than level 5 will have enough data to give us the needed clues. Even with level 5 there is no guarantee.

What you can look for in the log is any errors or warnings, or if there's a certain pattern that could be of interest (hard to say exactly what).

signupemail, also, you will want to write down the precise time of your actions in Thunderbird, so these actions can been directly correlated to data in the logs

Wayne: which actions, please? Do you mean automatic actions, i.e. when the filters run automatically? If so, well, it's hard for me to know when that happens. If instead you mean when I initiate actions, then, which actions, please? Begin to compose an e-mail? Send an e-mail? Apologies if I misunderstand.

Only the actions related to you reproducing the issues of this bug report

Only the actions related to you reproducing the issues of this bug report

Well, to be brief: that's hardly any actions. For the problem occurs, mostly, when filters run automatically on mail that is fetched automatically.

(please take care to needinfo the correct addresses)

Wayne Mery: sorry.

I have some results - as follows.