User Agent: Mozilla/5.0 (Windows NT 6.1; rv:102.0) Gecko/20100101 Firefox/102.0

Steps to reproduce:

COMPACT FOLDERS process now happens automatically, so just create and move and delete sufficient volume of messages to trigger a compacting
AND/OR
Select "Compact Folders" from the menu and let it run.

Running on Windows 7 Pro 32-bit, with Norton 360 AV.

Actual results:

I started TB and deleted the first two messages in INBOX (ironically, Bugzilla notices), and the INBOX display window went blank. I clicked elsewhere and came back, and the window re-displayed. Then COMPACT FOLDERS automatically triggered on INBOX. I watched it run and it saved 102 mb -- probably reasonable, since there had been several dups downloaded (Bug 1778037) and deleted. The third-now-first message seems to have disappeared -- and never came back. (Maybe it was a phantom to begin with ??) I ran REPAIR FOLDERS but that did not make it re-appear.

Then while intending something else (probably REPAIR again), I inadvertently clicked on CF from the menu. The process ran on the same INBOX as had just been processed and also SENT folder, saving 142 mb. I ran CF manually a second time -- so third CF in total -- and it again processed INBOX and SENT, and again saved 142 mb.

Expected results:

The first automatic CF should have left zero in INBOX. Unlikely that there was 142 mb in SENT for the second CF (but I'd believe 40 mb -- 142 minus 102). The second CF for sure should have left zero in both INBOX and SENT, but the third procedure still found the same 142 mb. What?? The message count was consistent, so I don't think anything was lost, but the arithmetic punched out and went home.

Just for clarity...
I noticed the progress bar was leisurely moving (it doesn't display in TB 102 on some things, like fetching messages -- see Bug 1777765) but there was no indication of what was actually running. Only when the process ended, the status display said Compacting Files was completed and the 102 mb were saved. Then I carried on as documented above.

Are you using Mark as deleted?
Then Bug 1776727 - compact fails with "just mark it as deleted" in 102RC1

No, nothing in Bug 1776727 looks or sounds familiar. I see no such setting in either of my current installs (releases 102.0 and 91.10.0). All of my active accounts are currently POP3.

I see the mark-on-delete concept only in Webmail at my website host (Rebel.ca) which uses a package called HORDE -- but TB doesn't know about that. Messages deleted locally do not delete from the server -- and my TB's are set up that way -- so that all of our machines can poll the messages at any time without them disappearing. Mailbox styles are set to "Files Per Folder (MBOX)" -- and all of them are, AFAIK.

BTW, I noticed tickboxes in TB 102 about Compacting: The new option to suppress/bypass automatic CF has been added (UNCHECKED), and the option to be asked every time for CF is now UNCHECKED (had been enabled until v102).

I now know that compacting files can cause loss of messages!

I have previously mentioned elsewhere that the first message (or selected msgs) deleted or moved after launching TB erases the entire Display Window. When this is done in conjunction with compacting, it results in misaligned info (when the display is recovered) and ultimately the loss of the next-following message(s) when the blank lines are removed and everything below rolls up. (Because compacting was automatic and unannounced, perhaps it was in progress when the deletions occurred?)

See Bug 1778248 "Deleting message erases folder display window in TB 102 Release" for full details & STR.

For the record, all of my TB accounts are POP3 with setting "Leave Files On Server For At Most so-many Days"

• 9999 days (27 years, aka forever) on the Laptop now running TB 102.0.1
• 30, 60, 90 days (typical) on the Tower still running TB 91.10.1

So the Tower's TB limits the longevity of the messages.

xref/dupe bug 1777454

Just updated to TB 102.0.2 -- not sure if this includes the fixes from Bug 1777454.

After the update, I did a REPAIR FOLDER immediately followed by COMPACT FOLDER for specifically Trash and Inbox. There were no apparent disappearances, blank lines, or missing messages as far as I could tell (other than two pre-existing cases) -- but that was also true of prior versions as long as there was no corruption by faulty deletions & moves.

I've been taking evasive action to avoid further message loss -- such as avoiding full GET MESSAGES, and minimizing deletions & moves (and not doing them first when launching TB) -- and I'm hesitant to try what I know used to fail. I also have to say that, before the processing, clicking on Inbox and Trash resulted in a regeneration of the summary tables (Local Folders menu displayed temporarily in the Display Window). This seems to happen occasionally for no particular reason, and it's unclear if it fixes or compounds problems.

I'll keep checking...

Just when I thought I had a handle on all this, they throw a curve! FYI, running TB 102.0.2 this time...

Inbox is sorted descending by DATE -- as usual. I had two pre-exsting weird text/html messages, preceded by two Bugzilla notices, which themselves were preceded by two recent read messages, all preceded by a few new pieces from specific fetches. So new/unread on top, two read, two Bugzillas, the two weirds, and then everything else. Before doing anything further, it so happens that I previewed all of the top messages (and a couple more for good measure) by moving the cursor down, just to make sure everything was OK -- and it was.

So... I copied the two weird messages (to ensure I had them for reference), and then MOVE'd the two Bugzillas to their folder -- and recall I had done the fetches (when TB woke up) plus COPY just now, so the MOVE was not the first thing done. The two messages moved OK and no longer displayed in Inbox -- exactly normal.

Then I inspected the Inbox items again. The next two messages (ie, right below the ones just moved) went weird, now having misaligned headers & subjects -- same subjects as before but different content -- but all the rest seemed fine. I ran REPAIR FOLDER followed by manual COMPACT on the Inbox, and things got stranger! The two moved Bugzillas were gone (as they should be), the original two weird ones now appeared at the top again (strange), followed by the new/unread (all OK). But the two newly-weirded items now had disappeared completely!

Thanks for testing, Dan. It's likely bug 1777454 (will be fixed for 102.0.3) resolved at least the compact issue. Hard to say about the other.

(In reply to Dan Pernokis from comment #7)

Just updated to TB 102.0.2 -- not sure if this includes the fixes from Bug 1777454.

It doesn't. Will be in 102.0.3

Magnus, I'm not convinced that the compacting is totally at fault. In my experience it is more of a casualty. See Bug 1778248 -- I cross-posted Comment #8 there and another from Bug 1778453. Bug 1778248 originally documented the disappearing display window if a DELETE (or MOVE, which includes the same type of deletion) was executed. It now seems the window disappears only if DELETE (or MOVE) is done as the first operation since TB launch (and at least one time here, first time since wake-up).

The main point is three-fold: (i) that the delete function messes up something, which COMPACT then trips over badly if you don't run REPAIR FOLDER beforehand; (ii) the new automatic COMPACT complicates matters because users aren't aware that it is running or will run, so REPAIR hasn't been run first; and (iii) REPAIR sometimes tosses one or more messages anyway but at least the COMPACT works cleanly. Until my Comment #8 kicks in!

Further to the automatic compacting... I've read several bugs (many dups) which show the same basic symptoms -- and many could be explained by an unseen COMPACT running amuck. Many reporters obviously aren't aware of it happening.

Agreed, there might be more factors involved. But while the compact problem is still involved, it will be hard to say. If you can, do try beta.

Oh, this is without Norton.

Early on, developers were thinking the duplication & multiple downloads might have been related to the practice of isolating individual inbound email messages for anti-virus scanning purposes -- so specific infected emails could be quarantined if necessary rather than quarantining the entire Inbox file and effectively shutting down TB. So "Norton" (or other) was tagged in the bug titles. Good to hear -- if it can be said -- that this particular AV isn't the issue. (I'm running Norton on my TB 102 machine. Also running Bitdefender, but it's at TB 98, so can't compare directly.)

I have Avast at home, and McAfee at work.

(In reply to Magnus Melin [:mkmelin] from comment #10)

(In reply to Dan Pernokis from comment #7)

Just updated to TB 102.0.2 -- not sure if this includes the fixes from Bug 1777454.

It doesn't. Will be in 102.0.3

Cross-posted from Bug 1778248...

Since installing TB 102.0.2 Release, I hadn't seen the screen disappear when DELETE (or MOVE) is the very first action taken. Today I installed TB 102.0.3 Release and specifically looked for this. So far, looks fine: deletes delete. Further, I did a REPAIR and COMPACT on my Inbox since it hasn't been done in awhile. So far as I can tell, all is well.

However I still can't say that COMPACT on its own is actually safe without the REPAIR -- ie, I don't know that the deletion process is working cleanly and not leaving something that COMPACT would trip over. (This aspect is not reported as fixed in the Release Notes, and I'm not willing to risk my Inbox to find out.) So I just did a few DELETEs from the Trash folder instead, and then ran COMPACT without doing a REPAIR. I don 't see any corrupted messages (except for a few known weird items deleted early on in TB 102 which still remain -- OK). Looks like the DELETE today did not mess up. But that of course presumes Trash reacts the same as Inbox.

But I don't know for sure what state the Trash was in originally -- there may or may not have been some deletions from prior 102 versions. When I first clicked on Trash, TB decided to rebuild its summary(?) -- it displayed the Local Folders menu screen for awhile, then came up normally.

Let's presume Trash was in good shape. If that's true, then the DELETE was OK since the COMPACT was OK. Had the Trash been bad, COMPACT likely would have tripped over something, I think, since REPAIR was not run to clean it up. On the other hand, I really only know about Inbox compaction problems -- not sure that they pertain to other folders.

Seems we can close this now? Still a problem with compact in latest version?

I really don't know. So many times things look fixed and then break wide open again, sometimes in a new & different way. FYI, my latest version is TB 102.1.1 Release.

I've run COMPACT many times on Inbox -- but always after doing REPAIR first. Then Bug 1783571 hit in TB 102.1.x and filter-MOVE'd messages started to be reincarnated in the Inbox, so obviously they hadn't been completely erased -- or REPAIR and COMPACT are somehow seeing things differently.

I still see commonalities here that makes me distrust deletions and what they leave behind -- so IMO no, don't close yet.

(In reply to Magnus Melin [:mkmelin] from comment #19)

Seems we can close this now? Still a problem with compact in latest version?

"WONTFIX"?

No! This stuff has to get fixed, and the sooner (and earlier in the release chain) the better. Either that, or give us a route back to TB 91 and we'll wait it out.

I know you guys are doing your best to resolve a lot of problems. Fortunately most do not affect me so I can live with them. But these deletion & compacting issues are serious for a lot of people -- just look at the resolved dups to see the scale.

I'm willing to hang in there doing what I can to help the process -- but only provided there is hope for a full fix really soon.

Thanks for helping figure things out!

(In reply to Dan Pernokis from comment #22)

No! This stuff has to get fixed, and the sooner (and earlier in the release chain) the better. Either that, or give us a route back to TB 91 and we'll wait it out.

What exactly is it that you can still reproduce when using 102.4.1? With short numbered steps please, so it is easier to follow. Thanks

I just today updated to TB 102.4.1 Release (from 102.4.0)...

Wayne, this bug is over four months old and the quote you pulled is from the end of July 2022 -- from a time when COMPACTing messed up big time unless REPAIR was run on the folder first. I recently ran COMPACT on my Inbox accidentally without running REPAIR, and things seemed OK. I just now deliberately did a COMPACT without REPAIR, and again all seems well. And I also tried it on one or two random mailboxes without incident.

So I would say this bug is resolved -- presumably fixed along the way with numerous other corrections -- and there is nothing outstanding here.

But just for completeness, the steps to test are:
(1) Inspect the first dozen or so messages in Inbox in reverse chronological order -- that they are present & viable, and that the subject agrees with the content.
(2) Delete one or more items from Inbox, chronologically near the very top of the file.
(3) Run COMPACT on the Inbox.
(4) View each of the messages in reverse chronological order, starting at the top of the file.
(5) Notice whether any of the first 2 dozen messages go strange in any way.
(6) Rerun using any other random mailbox.

Today (and the one time previously), this all went well here.