Tilman Reporter
	Description • 15 years ago

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2) Gecko/20100115 Firefox/3.6
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20100111 Thunderbird/3.0.1

I have put the "Compress" button in the toolbar. Each time I press it, another nstmp file is created in the imap folder, which is 
D:\Thunderbird\ImapMail\XXXX.xxxx.de . (I don't have enough space on C:). This effect does NOT happen if I compress all folders through the item in the file menu "Compress all folders of this account". Note that the german text for the compress button means something like "permanently remove deleted messages from the selected folder", i.e. it is a different functionality than the menu item.

I found this out only because I was wondering why my D: drive was getting full. I had nstmp files up to "nstmp-70", which were using over 10 GB!!!

I press "compress" often because I have thunderbird configured that messages are only marked to be deleted, not deleted immediately.

Reproducible: Always

Steps to Reproduce:
0. Delete something in an imap folder, so that it is marked for deletion.
1. Configure "compress" button in the toolbar (right click on the toolbar and drag it there)
2. in thunderbird, press Compress button
3. open imap folder in Windows explorer, see another nstmp file

Actual Results:  
another nstmp file appears

Expected Results:  
delete the nstmp file after compress is done

I suspect that the implementation for the "compress" button forgets to clean up.

	Tilman Reporter
	Comment 1 • 15 years ago

Possibly similar bug here:
https://bugzilla.mozilla.org/show_bug.cgi?id=544843

	WADA:World Anti-bad-Duping Agency
	Comment 2 • 15 years ago

Problem was partially reproduced with Tb 3.0.1 on MS Win-XP SP3.
> Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.7) Gecko/20100111 Thunderbird/3.0.1

(1) Use [Gmail]/All Mail, offline-use=on, wait for end of auto-syc.
    [Gmail].sbd\All Mail : 11,552 KB, [Gmail].sbd\All Mail.msf : 3,829 KB
(2) Context menu of [Gmail]/All Mail, Compact => nstmp-N(nstmp if N=0) is used
(3) While compaction is running, Context menu of [Gmail]/All Mail, "Compact"
    => nstmp-(N+1) is created
(4) Repeat (3) multiple times => nstmp-(N+m) is creted
=> Phenomenon of multiple nstmp-X was observed.
(5) Wait for end of compaction => all of nstmp-X were cleanly deleted.
=> Garbages of multiple nstmp-X was not observed.

Do you enable "offline use" of the IMAP folder?
(Folder Properties/Synchronization)
If yes, how many mails do you have in the folder? What is file size of offline-store?

Does "Compact" of the IMAP folder never ends normally, even if you don't disturb comaction by "next comact request while previous compaction is running(file size of newest nstmp-N is increasing)"? 

Do you use auto-backup software for Tb's mail directory?
Do you run virus-scan periodically for files under Tb's mail directory?

AFAIR, "next comact request while previous compaction is running" didn't invoke next compaction with message like "compaction is runnning", "other process is in progress", etc. AFAIR, it was true during my testing with nightly of Tb3.0XNPre. Regression?

	Tilman Reporter
	Comment 3 • 15 years ago

(In reply to comment #2)

Hello,

My bug happens when I do wait until done.

I just tried what you did, i.e. several compress without waiting. In that cases, all nstmp files are deleted correctly. However only one folder had something to delete. Then I deleted something in several imap folder, and then pressed compress quickly in all five of them. Two nstmp files were left, although not from all folders. The inbox folder is 200MB large, and "his" nstmp file was properly cleaned up this time.

I do not use auto-backup software for Tb's mail directory.

I do have an antivirus product but don't know when it runs. It does have a realtime monitor, however.  (Corporate environment)

Offline use is enabled, see screenshot.
I have 1126 mails in the inbox folder, the file is 200MB large.

Tilman

	WADA:World Anti-bad-Duping Agency
	Comment 5 • 15 years ago

I could see the problem with many compact requests of [Gmail]/All Mail only.
Interfere by "new mail check" and "mail move" were added to previous test.
(1) In addition to previous setups of Tb 3.0.1,
    Change number of cached connections to 5 (I set 1 in preious test)
    Enable "Check new messages at startup/every 1 minute",
    Enable "Check this flder for new messages" of [Gmail]/All Mail
(2) Move 2 mails in [Gmail]/All Mail to [Gmail]/Trash,
    5 to 10 times of "Compact" of [Gmail]/All Mail,
    Move 2 mails in [Gmail]/Trash back to [Gmail]/All Mail.
    5 to 10 times of "Compact" of [Gmail]/All Mail
(3) Repeat step (2) many times.
(4) File size of some of nstmp-X = 11MB (all mails are copied)
    File size of other nstmp-Y is less than 11MB, and is increasing.
(5) Wait for a while, all of nstmp-X becomes 11MB, CPU utilization of Tb=0%
    => 15 nstmp-X of file size=11MB were kept.
       (minimum X: nstmp(X=0), maxmum X: nstmp-20 in my test)

Problems looks next for me.
(Problem-1)
  If "Compact" is requested multiple times for an IMAP mail folder,
  "Compact" task is scheduled for each "Compact" request.
(Problem-2)
  Some of scheduled "Compact" tasks don't delete nstmp-X file,
  if something wrong is detected after end of copy to nstmp-X.
  ("replacing of .msf" is probably fails due to contention of ".msf")

	WADA:World Anti-bad-Duping Agency
	Comment 6 • 15 years ago

Confirming based on our test results, although solid/crisp/mimimum problem recreation procedure is not found yet.

	WADA:World Anti-bad-Duping Agency
	Comment 7 • 15 years ago

Change "Compress" in bug summary to "Compact"(term of en-US build) to avoid confusion.

	Wayne Mery (:wsmwk)
	Comment 8 • 15 years ago

regression?

	Tilman Reporter
	Comment 9 • 15 years ago

1) One factor might be that this is an old PC. It is a Pentium 4 although with 3GB of RAM. The compacting of the inbox takes about 20 seconds.
2) Due to the last comment on https://bugzilla.mozilla.org/show_bug.cgi?id=544843 , I'd say that we both have the same bug.

	WADA:World Anti-bad-Duping Agency
	Comment 10 • 15 years ago

(In reply to comment #9)
> The compacting of the inbox takes about 20 seconds.

A facter of it is probably Bug 539389. Even when local file, "write request per each mail data line" of Tb3 makes file copy(copy mail data to nstmp-X in this bug's case) time longer than Tb2.   

> Bug 544843, I'd say that we both have the same bug.

I think so, but I believe that your this bug has far better bug summary and far better observation of problem/additional test results by you than that bug.

	David :Bienvenu Assignee
	Comment 12 • 15 years ago

I have a fix for this.

	David :Bienvenu Assignee
	Comment 13 • 15 years ago

This patch does several things:

1. Make compacting offline store handle errors trying to stream messages. Now we continue to the next message. Previously we would just abandon the compact attempt, leaving nstmp files lying around.

2. Make compacting offline stores hold onto the folder lock for the whole compact process instead of just grabbing it for every message. This prevents the user from getting two compacts going on, or autosync sneaking in and appending to the offline store during a compact. 

3. Make compact of an imap folder with an offline store only send a notification when both the online expunge and the offline compact are finished.  This is required to make a remotely reasonable unit test.

4. Added a test to the offline store compact unit test to try to test this. It's a bit tricky because it's essentially a race between the online expunge and the offline compact so I had to tweak the unit test to give us a better chance to lose that race.

I'd really like this to get some testing during beta 2. I'll also generate a try server build in a minute...

	David :Bienvenu Assignee
	Comment 14 • 15 years ago

Andrey, this is where I'm working on a fix that might help with bug 555171 - try server builds I requested should show up in a couple hours here - http://tinderbox.mozilla.org/showbuilds.cgi?tree=ThunderbirdTry

	David :Bienvenu Assignee
	Comment 15 • 15 years ago

Gerv, you might be interested in the above try server build, because you've had issues with compacting your imap folders&offline store.

	David :Bienvenu Assignee
	Comment 16 • 15 years ago

I hope this is related to bug 555171, or rather, this fix will help with that bug, but this is the bug I was able to reproduce and fix.

	Andrey Cherezov
	Comment 17 • 14 years ago

(In reply to comment #14)
> Andrey, this is where I'm working on a fix that might help with bug 555171 -
> try server builds I requested should show up in a couple hours here -
> http://tinderbox.mozilla.org/showbuilds.cgi?tree=ThunderbirdTry

What exact file from there should I try? The last file from bienvenu@ dated 2010-03-15.

	David :Bienvenu Assignee
	Comment 18 • 14 years ago

Andrey, there were errors in my try server build, so I need to fix them and request a new one.

	Mark Banner (:standard8)
	Comment 20 • 14 years ago

Comment on attachment 436191 [details] [diff] [review]
fix with header file changes

>-nsresult nsOfflineStoreCompactState::CopyNextMessage()
>+nsresult nsOfflineStoreCompactState::CopyNextMessage(PRBool &done)
> {

This should have a comment as to why it is potentially trying to copy more than one message.

>+    nsCOMPtr<nsISupports> thisSupports;
>+    QueryInterface(NS_GET_IID(nsISupports), getter_AddRefs(thisSupports));

I think this is better:

nsCOMPtr<nsISupports> thisSupports = do_QueryInterface(this, &rv);

	David :Bienvenu Assignee
	Comment 21 • 14 years ago

(In reply to comment #20)

> >+    nsCOMPtr<nsISupports> thisSupports;
> >+    QueryInterface(NS_GET_IID(nsISupports), getter_AddRefs(thisSupports));
> 
> I think this is better:
> 
> nsCOMPtr<nsISupports> thisSupports = do_QueryInterface(this, &rv);

That doesn't compile, unfortunately.

	David :Bienvenu Assignee
	Comment 22 • 14 years ago

fix checked in.

	Wayne Mery (:wsmwk)
	Comment 23 • 14 years ago

Dominique (bug 544843), Tilman, are your issues related to this completely gone?

Andrey (bug 555171) is no longer using thunderbird, but I think we can safely dup his bug to this.  (does autosync issue in bug 555171 comment 9 sound like bug 541917/bug 531033?)

	Tilman Reporter
	Comment 24 • 14 years ago

I remember that one or two releases after "fix checked in", the bug no longer happened. In the meantime I have also moved to a faster PC. I just looked in my imap dir, and it did have a few leftover nstmp files - but they were a month old. So the only thing that is sure, is that I can no longer reproduce the error at will. (I am compacting the folders several times a day) So I'd consider it fixed. Mostly :-)