Against an ISP known to be unrealible I wouldn't set mail checks to a low value like two minutes, that's just multipying the probability of causing a problem. 10 or higher is more reasonable.

Many people comparing against the behavior they see on phones, but it's not helpful, there's no useful basis for comparison.

You've been tenacious and helpful, but for now let's close this until we have evidence of a Thunderbird bug

Same problem with IMAP and Hotmail. Every now and then (about once a fortnight) Thunderbird starts re-downloading the entire folder set, even though they are all present. Click on a folder, the emails are all visible, then suddenly disappear and reappear as it re-downloads the headers. Status bar shows it's downloading message 345 of 6789 in XYZ folder.

It does not appear to be related to a crash, it's happening now and neither Thunderbird nor Windows have crashed recently.

This appears to me a long-standing bug and annoyance with Thunderbird all the time I've been using it (about 6 years). It eats CPU time, makes the app unresponsive whenever it happens, and takes hours to re-download the messages (much longer than it should).

Please re-open.

I've been having the same problem over the last few years. For me, it's happening with Hotmail (Outlook.com) over IMAP. It doesn't seem to happen all of the time, so I wonder if it has anything to do with something periodic, like how the state of Daylight Saving Time is tracked? It may not be DST specifically, but something like that. Maybe when an e-mail sub-folder is added, removed, or moved? Fortunately, I only use Thunderbird as an offline backup of my Hotmail account, so I only sync it occasionally, but it's absurd and a total waste of time and resources that it has to re-download my entire Hotmail account each time this bug occurs.

Hard to explain, but the problem hasn't happened to me for over a year (Verizon domain email address, now handled by AOL, POP). I'm the OP. I thought that Mozilla had fixed the problem, but from the last two posts it appears they haven't. I can't explain why my problem went away.

Sounds to me like this bug should be re-opened.

While your original report was with POP3, I'd think the bug could be something not specific to POP3 or IMAP. I'm going to re-open it, and monitor mine more closely to see if it happens again.

@MMortal, we still do not have evidence of a Thunderbird bug. So I am going to close this report again. Please do not reopen it. IF you do get evidence of a bug, feel free to open a new bug report with the steps to reproduce and cross reference this one.

@Matt, will do. My only thing is that Jake (Comment 11) and myself have both experienced it in the last six months, myself having it happen with the latest version of Thunderbird in the last week (68.3.1 32-bit Thunderbird, Windows 10 64-bit 1909).

I'm genuinely interested in what threshold of evidence might be enough to just have someone more knowledgeable theorize on what it could be. I ask because, while I understand your request for steps to reproduce, it's hard to even begin to come up with steps here without having some underlying Thunderbird knowledge on what we might try.

If the implication is that it's somehow not Thunderbird, but some misconfiguration on Microsoft Hotmail's servers... well, playing devil's advocate, what miscommunication from a mail server should ever cause Thunderbird to react by re-downloading everything from scratch? Maybe not a bug, per se, but definitely bad error handling on Thunderbird's part to not be able to avoid such a significant data loss situation. In my mind, Thunderbird should never be, without prompting the user, re-downloading the entire account from scratch. In other words, Thunderbird should clearly notice that its ImapMail folder is not empty, but 7GB in size, lol, and then stop and re-evaluate whether it should be taking this destructive action.

What I've witnessed every now and again is (from memory):

1. New mail arrives in Hotmail IMAP account
2. A Thunderbird rule moves this to another folder
3. I see there's new mail in that folder, and click on the folder
4. All messages in the folder are initially displayed but immediately disappear, then start reappearing as the headers are re-downloaded

What might be happening at some stage (either during the rule processing, or when clicking on the folder, or some time between) is that an 'error' is received from the server which Thunderbird interprets to mean that the folder is empty. But that's just hazarding a guess.

Thunderbird is "deciding" the emails are out of date and need resyncing. 10 years ago that was because Microsoft servers lied about the message size, bug 534835 and there have been plenty since. Currently repairing a folder causes a resync. But that is another bug.

However you appear to have partial steps here.

New mail arrives in Hotmail IMAP account
A Thunderbird rule moves this to another folder
I see there's new mail in that folder, and click on the folder
All messages in the folder are initially displayed but immediately disappear, then start reappearing as the headers are re-downloaded

Now create a new folder and send some mail to it using a filter. Backup the relevant MBOX file with the mail in it after the resync is done.
Enable IMAP logging. https://wiki.mozilla.org/MailNews:Logging
Send another message that will cause another resync via the filter. (The filter use might or might not be relevant.)

File a bug with the steps outlined here in your new bug and the two small mbox files demonstrating that yes there was a change or no there was not. Add the IMAP log.

Then there will be facts to look at instead of a vague "I see this" The log is probably the key, but it might be that Thunderbird uploads the mail to complete the move and Outlook.com changes the order of the headers in their IMAP version so invalidate the folder. I do not know. I am guessing because there are no real facts.

I'm the OP and an ex-programmer. The occurrence for me was totally unpredictable. I tried to determine some kind of pattern, but was unable. I could not cause it on-demand. For me, no filters were involved. Just two verizon.net (using AOL servers) accounts using POP. It would happen to one account, and the other would be fine. And, like I said in my original post, I had one Android phone & one iPhone accessing the same accounts and the problem never occurred. Also, a W7 PC with Windows Live Mail and the problem never occurred. Just the W10 machine w/Thunderbird.

And it would go away immediately 100% of the time by just re-starting Thunderbird. But it was a PITA to clean up after.

It looked like some kind of timing window to me, but I have zero knowledge of Thunderbird internals.

@bigbirdPhila
One of the issues that is appearing in Support are a number of reports about Yahoo simply not supporting leaving messages on the server. We know they do, but their servers are not reporting that. Like many things with the Yahoo/AOL/ Verizon amalgam. What happens makes little or no sense and getting through to people that work there are actually have a clue is not for the faint hearted.

Can we consider the three to be one? Now that is something they are reluctant to tell anyone. But some of the things I have seen reportedly happening with what are supposed to be AOL accounts leads me to think they are using the same server software, if not the same servers.

have a look as the popstate.dat file in your account. It is from that Thunderbird establishes what messages it has downloaded and therefore when a new pop session starts, that file is referenced to work out which messages NOT to ask for. If the references change then the offline reference Thunderbird has is useless. If Thunderbird can not open the file because it is locked or corrupt, then it is also useless. But shear size might have an impact. I have no familiarity with the code, but it is old. So it may wall only define the array holding the mail not to request as a number smaller that the number of mail you have on the server. I do not know. That is why Wayne said

You've been tenacious and helpful, but for now let's close this until we have evidence of a Thunderbird bug

Because we do not know. My last comments were directed to Jake in the hope he could put together a fully formed steps to replicate for an IMAP account. A completely different issue to your POP one and not a reason to reopen this POP bug.

Now ... Send another message that will cause another resync via the filter.

File a bug ...

I'm not sure it will be that straightforward as the problem happens only very occasionally. However, I have enabled IMAP logging so at least will have a log file available the next time the problem occurs (if indeed it does!).

I just tried syncing my Hotmail account for the first time in months, first updating to Thunderbird 68.6.0, and it has started downloading everything yet again. I did start it with that IMAP logging command from the instructions above, but the log output is already massive, and not going to be specific enough for sharing. Plus, there's no way to remove personal information from the contents of it, which is going to make this much harder for anyone to provide a log to diagnose this.

One thing I do see is that the STATUS requests on the folders (with the counts of the e-mails, i.e. MESSAGES, UNSEEN, and RECENT) come back with the same number being RECENT as the total MESSAGES count, even though they should, of course, not be the same.

I also noticed that, occasionally, some of the e-mail dates on really old e-mails come up as new, especially in the Drafts folder, but this could be a separate issue, because Thunderbird doesn't always start downloading from scratch, even though some of the e-mail dates are off.

I'm not sure it will be that straightforward as the problem happens only very occasionally. However, I have enabled IMAP logging so at least will have a log file available the next time the problem occurs (if indeed it does!).

Now captured. However, the logfile (covering less than 20 minutes of activity) is almost 1Gb and contains lots of personal information and email message content. What are the salient things in the logfile?

Evidence of the issue with 68.12.1 (64-bit) on Windows 10.

PS. After restart, it's continuing to download all messages, which it already has, from various other folders.

(In reply to jake from comment #23)

However, the logfile (covering less than 20 minutes of activity) is almost 1Gb and contains lots of personal information and email message content. What are the salient things in the logfile?

Yep, same here. I think expecting us to share that log is a non-starter for a private e-mail account.

I believe some Thunderbird dev who actually understands the code is going to have to do some sort of dry analysis on why it would ever be the case that Thunderbird's logic would be to re-download all e-mails in a folder from scratch unless it were explicitly requested. That should just be a huge red flag in the downloading logic. There should be some error handling built in, even if it's technically a problem with the mail server. I mean, Thunderbird should clearly notice that its ImapMail folder is not empty, but is 7GB in size, and then stop and re-evaluate whether it should be taking such a destructive action.

I currently only use Thunderbird as a backup of my e-mails, but have resigned to only run it once a year or so now, because it, inevitably, re-downloads everything from scratch at some inopportune moment. Today I decided to give it a go again, and, yep, it's downloading everything from scratch, with version 78.5.1 (32-bit) on Windows 10. Even though I'm using it as a backup, I'm sure Microsoft would rather not have their bandwidth wasted, and there are also occasions where it'd be great to actually use Thunderbird as it was intended.

If the server says the contents of the folder are out of date and need to be redownloaded, that is what happens. You appear to be saying Thunderbird is initiating the download. Without a log, no one can know what initiates the download so you are guessing. Guessing is not sufficient to determine what initiates the download, so as a bug report this is still invalid. Have a nice day.

Without a log, no one can know what initiates the download so you are guessing.

I have a log. But it contains lots of personal information including the contents of emails that were re-downloaded, along with email addresses of sender, recipients, etc.

Would you be able to advise what the relevant entries in the log are, so I can provide only these?

I'm in agreement with jake that we're going to need some ability to create such a log that narrows down the massive amount of information that's logged, and also doesn't include personal e-mail contents. I guess that that would be yet another bug report or feature request to file -- to enable the practical, sanitized logging of these situations?

"If the server says the contents of the folder are out of date and need to be redownloaded, that is what happens."

That may be how it's currently programmed, I can only trust your description, but Is there no potential programmatic or algorithmic way for Thunderbird to detect this destructive edge situation and postpone such a command from the server when the result will be a complete redownload of entire folders -- and at least prompt the user before destructive action? I imagine there could be some algorithmic way for Thunderbird to know to keep the previously downloaded message database untouched at a point prior to destruction unless the user chooses to allow it. I don't know enough about IMAP to say, but is there no way to non-destructively compare e-mail contents for changes and not ultimately carry them out whenever the entire folder will be wiped?

some ability to create such a log that narrows down the massive amount of information that's logged

There is, by not setting the logging level so high. I set

MOZ_LOG=IMAP:5,timestamp

to make sure everything that might be useful was captured. Now it is, but what to do with it?

destructive action

YMMV, but it's not destructive AFAICS: it simply re-downloads everything that was already downloaded (resulting in a huge waste of bandwidth and CPU time - but no loss of data).

I did wonder if DST might be a cause, as I last had the problem at the end of October. However, this was about 5 days before the UK moved to 'winter time'. Unless Redmont or some other location of Microsoft's servers switched the weekend before, this is probably unlikely. Also, it appears I've experienced the problem at unrelated times of the year (such as January and June).

It might be possible to process the log file with some kind of regex, to eliminate the personal information, but would require quite some effort. The messages are logged one line at a time, with a log prefix, e.g.

2020-10-27 23:42:26.154000 UTC - [(null) 10708: Unnamed thread 0000020C1F2C1C00]: D/IMAP ReadNextLine [stream=0000020C1BD7A400 nb=16 needmore=0]
2020-10-27 23:42:26.154000 UTC - [(null) 10708: Unnamed thread 0000020C1F2C1C00]: I/IMAP 0000020C23102800:imap-mail.outlook.com:S-Events:CreateNewLineFromSocket:    color: #fff

Surely there's a better way of solving this. Thus I/we need to know what to look out for in the logfile. Hence the needinfo request. TIA.

The bug was originally reported against Verizon/AOL, but has also been found with Microsoft/Hotmail so does not seem specific to a particular email provider - unless they are using the same software internally. This suggests an issue with Thunderbird. Have a nice day.

"YMMV, but it's not destructive AFAICS: it simply re-downloads everything that was already downloaded (resulting in a huge waste of bandwidth and CPU time - but no loss of data)."

A reason why I see it as destructive is that, once it happens, you immediately no longer have all your e-mails to work from in offline mode in those particular folders that it's started downloading from, until it downloads them all again; so you no longer have that full backup of your e-mails up to that point that you can search through and do with what you want, until it completes the re-download of everything. Am I wrong on this?

Regarding the log and playing devil's advocate, to properly debug the problem, it might, theoretically, require that we know which e-mails (i.e. their subject line) or which folder names are the ones where it's first being triggered to act up. However, given that once it happens, it happens on every folder, I think the odds are that it's something independent of particular e-mail subjects, contents, from/to information, or folder names. It'd be very useful to be able to produce a log that filters all of that out, and maybe replaces such personal info with generic identifiers? (e.g. Sub-Folder 1, Message 400).

(In reply to jake from comment #28)

Without a log, no one can know what initiates the download so you are guessing.

I have a log. But it contains lots of personal information including the contents of emails that were re-downloaded, along with email addresses of sender, recipients, etc.

Would you be able to advise what the relevant entries in the log are, so I can provide only these?

I would guess the relevant part would be that which initiates the download of new mail. That is highly likely to be a UIDVALIDITY returned by the server that is not the same as the one Thunderbird has. See https://tools.ietf.org/html/rfc1730#section-7.4 and look at the sample transcript.

Given this discussion appears to revolve around providers with known issues with actually providing a standards-compliant service (Outlook, Yahoo, Gmail, AOL et. al.) is may be that they really can not do the job they have set out to do. Gmail do not provide IMAP standards compliance, and their remaping of labels to IMAP folders is somewhat problematical. Outlook is based around Exchange, where IMAP is so important that it is shipped by Microsoft as disabled. Since the Verizon acquisition, nothing about mail from their stable is consistent.

A compact can also cause a complete redownload of the affected folders as can a folder "repair" undertaken from the user interface. (in IMAP accounts that is taken as a full resynchronise with the canonical source.

Then there is the fact that retrieving message 1 or XXX can actually mean just the message header/ List not the entire messages. Hence the need for logs to actually see if it is while messages or just headers.