User Agent: Mozilla/5.0 (Windows NT 6.1; rv:102.0) Gecko/20100101 Firefox/102.0

Steps to reproduce:

A few days ago (using HELP, ABOUT as usual), I manually upgraded TB from 91.10.0 to 91.11.0 and then immediately to the new 102.0 major release. Then I just let it run as always. Message fetches happen automatically at various times, or it gets run manually by the GET MESSAGES button, or specific accounts get polled on demand.

Running Windows 7 Pro (32-bit) with Norton 360 AV.

Actual results:

Various weird things started happening since upgrading, mostly display bugs & irritants. But today, TB 102 started to download the same emails multiple times from various accounts. (All are POP3.) I can sort by the NEW/UNREAD flag to delete the duplicates & triplicates, but they keep coming back! Polling a specific account on demand duplicates only that account. A full GM maybe tries to duplicate everything.

Expected results:

TB Release 102 should run as good as (or better than) anything previously. It should not be full of bugs, given all the testing it has gone through.
Searches for solutions only come up with old stuff circa 2019 and don't make sense in my case.

Sorry, I realize these four items are probably separate bugs to be reported individually, but they may also be tangential to THIS bug, and out of urgency, I'm including them here. If need be, I'll write them up later...

I can further say that some simple filters which have always worked ("if subject = something, move here") intermittently do not appear to work in TB 102, so duplicated messages may sometimes appear in INBOX and other times appear in the target folder. May be related or might be a separate bug. The intermittency is a concern.

Also, clicking on any large folder such as INBOX, TRASH, etc, occasionally causes the top-level Local Folders screen to appear for many seconds while a message says "Building Summary..." Might be indicating something is amiss while TB is deciding what to download next -- and may be taking awhile if TB is checking how much to download.

When selecting specific messages to view, I had display issues whereby Msg 1 & 2 displayed OK, then Msg 3 was indicating from Sender 4, Msg 4 from Sender 5, etc. Switching folders and coming back didn't help. I ran REPAIR FOLDER and the problem went away. I also noticed that the COMPACT FOLDERS process now runs automatically (nothing in the Release Notes about that!), which is a likely candidate to have caused this mis-aligned display issue.

First time a file is deleted after loading TB, the entire message display window goes blank as though everything in the folder has been deleted. Does not refesh or restore itself until I click any other folder and go back -- then all is displayed normally (and the deleted item is gone as expected). Worrisome when it happens -- first time, I thought I had deleted my entire Inbox! Does not recur until you close and restart TB.

Sorry Dan, that's not good at all. I think I have seen another report for this, but can't find it atm.

Thomas, perhaps you mean Bug 1762327 ("Thunderbird repeatedly downloading 1000"s of old pop messages. Caused by corrupted popstate.dat?")
I've been following that bug closely, and I thought it was just a local instance. It worried me a little when I upgraded to 102 -- and then when it happened here, wow. Thankfully my main machine is still at version 91.8.

Is there a way I can go back to 91.8.10 (since it still works OK for me)?

Today things seem to be more stable, in that polling brings down only whatever new messages there might be -- but it takes a very long time checking! Almost like it might be scanning the entire mailbox for something to download. But that has happened since the beginning -- looks normal, but then later starts downloading the dups. I can't blame any of the servers we poll, because my Tower polls exactly the same servers in exactly the same way (except for two respective device-specific mailboxes that both work the same way). And with no Progress Display bar (Bug 1777765), it's impossible to know what's happening.

BTW, an oddity related to scanning all messages: As soon as v102 ran for the first time, it seems to have created an ARCHIVE folder with subs for three years -- 2012, 2013, 2014 -- each containing one or two files. (I always delete the ARCHIVE folders when they usually appear after New Year -- they're always empty. But this time there were files present. In any case, the Copies & Folders settings for ALL of my accounts specifically have the ARCHIVE radio button turned OFF (and with no path spec where to put it anyway).

URGENT UPDATE

Regarding my Comment 1 above:

When selecting specific messages to view, I had display issues... (mis-aligned info)
First time a file is deleted after loading TB, the entire message display window goes blank...

These items were mentioned here in only passing, but they are related to each other (and to automatic compacting), and they are NOT benign -- the process causes actual loss of messages!

See Bug 1778248 "Deleting message erases folder display window in TB 102 Release" for full details & STR.

Are duplicated messages still downloaded? Can you get some logs and maybe provide the popstate.dat as well (bug 1762327 has some instructions)? Thanks

It seemed not, but then late this afternoon it started up again -- dups and some trips. Not sure if they are repeats of ones deleted previously or new dups. I checked POPSTATE files earlier today -- no duplicated files, but there are multiple files (one file per folder) which as I recall has always been the case. I'll work on this fully later.

Each message has a unique UIDL, downloaded messages are recorded in popstate.dat, like this

k 000002155f7aa7cb 1648538196

Duplicates happen when a downloaded message is somehow not recorded. And "Get Messages" will download it again next time.

Comparing logs with popstate.dat can possibly find two things:

1. a previously downloaded is not in popstate.dat (duplicates)
2. a newly downloaded is not recorded in popstate.dat (duplicates next time)

Bug 1777387 also reports repeated POP downloading.

Based on multiple reports, let's assume there might be a regression.

Here's a news flash! Of the several similar duplicate-download bug reporters, I seem to be the only one using "Leave Files On Server For At Most so-many Days".

The Laptop (now running TB 102.0.1 as of overnight/early this morning) has every account (except one device-specific acct) set to 9999 days (over 27 years, essentially forever). The Tower (still running TB 91.10.0) limits message longevity by having (various) short-term settings, typically 60 to 90 days, longer on important stuff, shorter on others. (I'm from a time where ISP server space was expensive -- we were charged penalties for being over certain overall size limits!)

I know my concern was that, if I deleted a message locally on either machine, they would disappear from the server too, which messed up future syncs & reference -- ie, deletions on one machine would preclude other machines from downloading those messages. So I never checked "Until I Delete Them" on any account. I also foresaw instances where re-fetching lost or deleted messages would be impossible after some local failure -- including human stupidity here. :)

So that explains why my duplicate downloads often repeat to become triplicates, maybe quads, etc -- whereas most other reports have just duplicates but in the thousands.

Also (for completion), all of my 20 active accounts are POP3. (I thought I had one IMAP account but can't seem to find it...)
About 15 are polled automatically and included in GET MESSAGES. The others are polled on demand. (Except in one case, it's the default FROM account, set to a null SMTP server -- to force selection of a FROM rather than sending by accident -- and it has no valid inbound settings.)

In an attempt to minimize unwanted duplicate emails, I turned off (unchecked) "include this server when getting new mail" for all accounts. (The idea was to poll on demand only the accounts I wanted -- doing so hasn't caused duplicates from what I've seen so far.) But emails are still leaking through! See Bug 1778611 for details...

become triplicates, maybe quads,

As I said in comment 8, this means newly downloaded messages are not recorded in popstate.dat. Can you focus on one server, backup the popstate.dat first, after a "Get Messages", backup the new popstate.dat again.

It's very helpful if you can upload both the popstate.dat files and logs. Thanks.

Sorry, got this wrong -- thanks, Ping Chen for pointing this out. So I restored the checks in Comment 13 and unchecked the "Check for new messages every so-many minutes" instead. Works, suppresses the automatic repeated downloads, at least limiting the duplications.

Magnus, I'm not sure specifically "9999 days" matters because -- as of Comment 15 -- the boxes are now unchecked, so (theoretically) TB never sees any values there. In any case, at least one was a more reasonable number, like a few hundred days. Yet the symptoms remain for all.

What I did discover just recently is that using the GET MESSAGES button (to fetch all) makes a big difference in the pathology. Whereas by itself selectively downloading from any mailbox hasn't caused any duplicates as far as I've seen, once we use the GM button, then duplicates start to happen! That is, it looks like clicking on GM sets the stage. GM will poll everything first time without dups, but then forever after (until TB is re-launched), there is a likelihood of dups. A manual GM, the automatic timed GMs, and any selective GM will produce dups! Maybe not every time, but any time you aren't watching. :) This is why for now I exclude all accounts (except the one) from the timed fetches and from get all msgs.

So I have the following configuration right now:

• Check so-many days is unchecked (as stated above);
• All accounts (except one) are excluded from "Include this server when getting new mail";
• That one account is polled every 8 minutes.

That one account normally produces no dups. But, if I click the GM button, being the sole account included on the button, I will see dups, either right away, or soon on a timed poll. But I could now see dups on the next manual fetch from ANY account! Again, not every poll, but once started (simply by clicking the GM button), dups keep at it until TB is re-launched.

BTW, now running TB 102.0.2 as of this morning, including some time in Comment 16.

102.0.3 ?

I updated to TB 102.0.3 this morning. Good news, no dups!

Background Recap:
I had only one mailbox (containing only a few recent messages) attached to / included in the GET MESSAGES button. Previously, firing GET MESSAGES would poll this account and it may or may not find anything, but it would then forever after start to bring down dups (whatever was in this mailbox) every so-many minutes. This could be by the timed fetches, or could be manually clicking GET MESSAGES, or could even be clicking this one specific account -- any of these would usually give dups. If I only clicked specific accounts during a TB session and never clicked GET MESSAGES during its lifetime, then there would be no dups. And early on in the Bug, when I had all accounts attached, hundreds of recent messages would come down. (Seems that the GET MESSAGES button was triggering the dups...)

Results:
Today I re-attached / re-included all relevant accounts to GET MESSAGES. I'm happy to report that I've clicked & run GET MESSAGES several times today -- and left it running with timed fetches enabled. So far, no dups! All appears normal.

However, two recurring issues:
(i) The progress display bar is still missing (Bug 1777765), which makes it hard to know if all fetches are completed, or if only the most recent is done with more to come.
(ii) I'm still getting the occasional popup warning that certain accounts are "busy" (Bug 707933, Bug 827897). Happened when I clicked GET MESSAGES first time and the timed fetches started too. In fact, for the first time I had two popups simultaneously -- one immediately on top of the other -- as the two accounts competed for attention, and a total of three popups altogether (out of ten accounts actively polled on this machine).

...And tonight, the news is bad again!

Machine was left sitting with TB just running in the background as it always does, then suddenly there were something like 226 duplicate emails, from all the various accounts which I had reactivated. Recall from Comment 19 above that I had clicked GET MESSAGES in a deliberate attempt to trigger the duplicate downloads, but they hadn't materialized.

Now see previous Comment 16 above: Maybe not every time, but any time you aren't watching.
So out of the blue, dups reappeared again.

I deleted all dups, and manually polled all accounts one by one. ONE account still re-downloaded its 29 dups yet again! Then no more. Keep in mind in Comment 19 I had already tested polling of specific accounts (all of them) looking for dups -- there were none, so if some log of downloads was scrogged, shouldn't the dups have shown up then rather than later?? And what did the recent REPAIR and COMPACT do? Again, I would have expected the dups to reappear immediately if they were going to, not hours later unannounced and unexpected.

Interesting Observation:
Sometimes -- I can't say every time, but several times recently -- running TB 102.0.2 and now 102.0.3 releases, when a series of duplicates came down, they were all tagged with little yellow diamonds at the start of the SUBJECT field, without a column header of its own. (Subject text was not indented -- all subjects were perfectly aligned, with enough leading space to accommodate the diamonds for those that had them.) Dups could be contiguous or scattered throughout the folder.

Is this a flag of some kind to acknowledge duplicates? Is it an alternate NEW (UNREAD) status indicator? Does it indicate a potentially damaged (or damaging) record? I can't say if they persist or not, as I've always deleted the dups right away, and I don't have any diamond-tagged ones at the moment to test further.

That sounds like the NEW indicator yes (==unseen before, not the same as unread)

Currently still running TB 102.0.3 (and waiting imminent release of TB 102.1.0)...

I had said previously (and elsewhere) that clicking GET MESSAGES seemed to trigger something that eventually led to repeated polling & fetching of duplicate messages. Yesterday we had dozens of dups from the ONE account which is "included when getting new mail" and is set to be polled "every 8 minutes". I suspected that the main laptop user had clicked GET MESSAGES recently, despite a protest to the contrary.

Today, TB had been relaunched and was just... running. The user had always polled her account manually, and I had polled maybe four specific accounts (including hers) a few times -- but neither of us clicked GET MESSAGES. Out of the proverbial blue, her account suddenly downloaded 72 dups! Her account is not set for auto polling or inclusion when getting new mail, nor are the others (except the one), and I repeat, we did not click GET MESSAGES.

So... I guess something is still lurking that invokes the duplicates when we aren't watching. (But I stand by previous statements that GET MESSAGES will trigger dups -- only sooner & quicker.)

Now running TB 102.1.0...

I re-attached several accounts to GET MESSAGES, restarted TB, and clicked the GM button. Only a current message came down. Clicked again several times with no additional messages forthcoming, so I left it for the night. This morning, at some point after laptop was awakened, a dozen or so messages came in automatically from one account, ten of which were duplicates. All accounts were being polled automatically as timed (THE ONE every 8 minutes, a few with times in a similar range, and some every hour or two), but only this one account duplicated -- as it has in the past -- suddenly and without warning. (And ironically, again, it's the account that handles my Bugzilla email.) Another account also famous for dup'ing did not do so this time.

So I don't know...
When TB is left on its own, it suddenly and without warning misbehaves -- maybe not every time, but at some time when I'm not watching.

Now running TB 102.1.2...

Basically same as my Comment 24 above -- re-activated accounts on GET MESSAGES button and clicked it. Nothing current came down over a time. Left for the night. By noon, 16 duplicate messages from at least two different accounts suddenly appeared in the Inbox. Yet I left the system for a few days without having clicked the GM button -- and there were no dups during that time!

Another ongoing oddity while testing the above:
When I manually poll specific accounts, there is sometimes a pause or long pause before saying no files. Sometimes it's a very long pause of almost a minute. Or, I will poll and the status line says downloading 1 of nn messages, then take forever before it finally comes down. Occasionally something times out and only the one (or first) message of the list comes down, then later this one is duplicated as the first of nn messages that come down. It's rather hit-and-miss, unpredictable, not reproducible on demand (because it always works then :)

See resolved Bug 352107 which has (which had) similar overtones and is supposedly fixed. (My Comment 18 was posted minutes after resolution.)

Now running TB 103.0.2...

This is part of my Comment #6 in Bug 1778689:

Today I manually polled the timed account and for no reason it attempted to download 7 duplicate messages. (I know because there was nothing new in the account -- and I know that because I haven't sent to it in a while.) The status line flipped through the usual connection info and said "Downloading 1 of 7" before quickly erasing the status line and popping up the error: "Unable to write the email to the mailbox. Make sure.. [etc]." I cancelled the popup and tried the poll again. This time it attempted 4 messages, and they came down OK -- all dups as expected.

So not only are the dups happening randomly in time & number, but so is throwing an unrelated & misleading error message! (I DO have write authority, and have gobs of space.) And I've seen this "Unable to write..." popup a few times throughout TB 102, always at an unexpected point, seemingly at random, with no particular sequence of steps to trigger it except the obvious clicking an account to get its messages -- and sadly, just when things feel like they're back to normal. :(

Please note correction: I'm still running TB 102.1.2. (I misquoted by giving the version of Firefox I was running to view Bugzilla! :)

...And I'm finding TB 102.2.0 is no better. Still downloading duplicate (older) messages when it randomly feels like it, and throwing 2 or 3 variations of popup error messages occasionally.

Without logs and popstate.dat before and after "Get Messages", it's hard to know what and how it happened.

Thanks, Ping Chen.

Just to be clear...
I once thought that clicking GET MESSAGES predisposed TB to the multiple (duplicate) downloads -- see my Comment 16. But I've since realized that downloading gets triggered at random -- could be right away after launching TB, or could be a day later / could be automatic (timed) fetch, or could be during manual (selective) poll / could be first manual poll after a while, or could be manual poll immediately after a manual null fetch (nothing found) / etc. Clicking GM might increase the likelihood of dups, but no guarantee.

Sorry, it's still not clear to me how to log all this. You had said to Reg (Bug 1762327 Comment 9):

"Can you please upload both popstate.dat? Better with some logs, go to Config Editor, set mailnews.pop3.loglevel to All, open DevTools > Console."

Is that all? I know I hesitated before because you had also said about creating new profiles and "...delete everything from the INBOX..." -- I'm not comfortable with any of that and potentially losing what's there.

The other thing is that I can't "make" this thing happen -- it just does, on a whim. If I start logging and then it takes a day or two, how big will the log be? But I can definitely capture multiple copies of POPSTATE.dat for you and upload them. (I presume you want the one from INBOX -- looks like there is one for each account.)

I also read your thread in Comment 8 (above) about messages being duplicates because maybe they're not in POPSTATE:

Comparing logs with popstate.dat can possibly find two things:
- a previously downloaded is not in popstate.dat (duplicates)
- a newly downloaded is not recorded in popstate.dat (duplicates next time)

That doesn't quite explain the randomness of the duplicates. (1) If an original message(s) is missing, it would/should be polled once next time. But messages can be re-polled and re-fetched multiple times (triplicates and quads). (2) If a new message(s) was not recorded, it would/should only be fetched one more time -- unless it never gets recorded at all -- because I see duplicates of both new (most recent) and old (recent and older) messages!

One last point:
My current procedure follows this pattern:
(i) If I see duplicates, I sort the Inbox by NEW flag, then delete all the new items (carefully avoiding the one or two legitimate new's still flagged). Then I re-sort back to the usual BY DATE, run REPAIR on the Inbox (as precaution) and COMPACT the folder (Inbox).
(ii) If I do a manual (selective) poll of an account and suddenly see duplicates coming in, I immediately close TB -- legitimately, by the red "X" in the top right corner -- and then do (i) above.
Either way, I then re-launch TB. It starts by re-building the Summary File, then carries on "normally". I have to manually fetch any accounts I want to see (only one account is polled automatically and included in the GET MESSAGES button), and nothing ever duplicates -- until it does later, some time at random.
(Perhaps I am "resetting" the conditions here, but I would think that should be a good thing...)

Thanks for the updates, Wayne.

Just to post here the gist of what I cross-posted elsewhere...
I just upgraded to TB 102.2.2 and tested out the GET MESSAGES thing again. TB still suddenly, unexpectedly, and randomly downloads duplicates at some point after clicking the GM button. In recent use of TB 102.2.1, I did not see any dups, but for sure I did not click GM at all, ever. So I would say the issue persists.

Perhaps again send new logs and popstate to Ping.

(In reply to Wayne Mery (:wsmwk) from comment #32)

Perhaps again send new logs and popstate to Ping.

Do we have good news?

Perhaps unrelated, but duplicate messages is being widely reported in support forums in the last few days, for version 102 (not beta). Just in the last 24 hours:

• https://support.mozilla.org/en-US/questions/1394405
• https://support.mozilla.org/en-US/questions/1394359
• https://support.mozilla.org/en-US/questions/1394340

I recently upgraded to TB 102.4.0 32-bit official release, and just now tested out the GET MESSAGES button. So far no dups -- but they may come later, which is how this thing always manifests itself.

In any case, there have been dups occasionally, at random, when polling individual account(s) manually.

Will uplift bug 1793374 and bug 1796903 later this week. The this._uidlMapChanged = true; change in 1793374 can potentially prevent some duplicates.

@Ping, I am not across those bugs, but what would help was if Thunderbird updated the popstate file with the last download in the event of a connection timeout before the POP session completes or other abnormal termination occurs.

The current implementation records nothing in popstate if the antivirus, which appears to frequently be Norton, simply chokes and dies on a malformed email and eventually the hung session times out and eventually restarts post failure on the next times download.

These reports have been appearing in support for more than a decade. With the accepted advice being to either cease antivirus mail scanning until the mail queue is empty before resuming it, or clearing the spam folder server side as most malformed mail also get identified as spam, or both.

Last night I updated to TB 102.4.2 32-bit official release. But immediately before doing so, while still running TB 102.4.1, I manually did a GET MESSAGES for one specific account which we always poll -- and TB suddenly & randomly downloaded 11 messages, 10 of which were duplicates. Polling a few other accounts then worked OK.

Today running TB 102.4.2, I again manually did a GET MESSAGES for the one specific account which we always poll. Got a quick pop-up window warning about inability to write to the mailbox. Tried polling again, and TB suddenly & randomly downloaded 75 duplicate messages (probably the full list). Re-polling this account then gave nothing further, and polling other accounts all worked OK (nothing to download).

Today on another account (polled manually) I randomly got the error, then a download of 52 messages when I re-tried the poll. (Other accounts worked OK.)