User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 2.0.50727) Build Identifier: version 2.0.0.6 (20070728) Thunderbird is extremely slow when deleting lots of emails, hundreds or thousands of them. I am not using IMAP; just deleting emails fetched from POP3 and stored on my local PC. The particular task I was trying to accomplish was deleting a few thousand emails (from a folder into the trash, although it also seems slow deleting selectively those emails from the trash also). CPU seems to spike to 100% and stay pegged while emails are being deleted. Thunderbird tends to hang until the deletions have completed. I've had problems doing this for various Thunderbird releases for the last year or so, not just the current release. I've tried timing the behavior to quantify the severity of the issue. In a large email box, I timed the following was finding that deleting N consecutive emails took the following amount of time: 10 emails- 3 seconds 30 emails- 9 seconds 100 emails- 32 seconds 1000 emails- 400 seconds (8 minutes) The emails are all small around 2k and have no attachments. I have an 800 MHz CPU which is admittedly old. My testing has also curiously revealed that deleting the last 100 emails from a 20,000 email box is much faster than deleting the first 100 emails, by a factor of about 5x(!). Viewing the file accesses via Sysinternal's filemon (I can post logs if you'd like), the cause does not appear to be due to any difference in filesystem READ/WRITE/QUERY activity. Reproducible: Always Steps to Reproduce: 1. 2. 3.

> 10 emails- 3 seconds > 30 emails- 9 seconds > 100 emails- 32 seconds > 1000 emails- 400 seconds (8 minutes) Linear till 100, but becomes non-linear when number of mails increases. Did you observe Virtual Memory Size/Memory Usage(real memory) column for thunderbird.exe in Process Tab of Task Manager? When I tested Shift+Delete of 10,000 mails, I saw that Virtual Memory Size column for thunderbird.exe became larger than my expectation. I think this is a result of faster new virtual memory requests than freemain by garbage collection. If MS windows detects real memory shortage, severe swapping will usually occur. (Q1) Slowdown by swapping when your case? (Q2) Will same result be obtained when config.trim_on_minimize=true? (Is slowdown when 1000 mails observed?) 1. Put next line in user.js user_pref("config.trim_on_minimize", true); 2. Restart Thunderbird with -safe-mode 3. Execute a test case 3-1. Minimize Tb's window 3-2. Return to normal window size 3-3. Do test case 4. Repeat 3. (Q3) What is number of mails / file size? - Trash folder, file of Trash, file of Trash.msf - Source mail folder, file of <folder>, file of <folder>.msf "Delete of mails" is "Move mails to Trash", and consists of; (1) Copy to Trash 1-1. Copy all selected mails to file of trash (append mail data at end of file) 1-2. Update ".msf" for Trash folder (2) Delete from mail folder 2-1. Flag as deleted in the mail folder for all selected mails 2-2. Update ".msf" for the folder (3) Follow-up actions. e.g. count update in folder pane, thread pane refresh 1-1. is file read&file write with large buffer, so CPU 100% itself is usually an evidence of efficient file I/O. Observe behavior of each step. (1) Restart Thunderbird with -safe-mode & config.trim_on_minimize=true (2) Create two local mail folders (say FolderA and FolderB) (3) Copy all mails in a big mail folder to FolderA (4) Copy N mails from FolderA to FolderB (<=Copy step) (5) Select all mails in FolderB, then Shift+Delete (<=Delete step) (Q4) Is there any difference from test cases in Comment #0? By the way, newer "Process Monitor" may be better than filemon to see Tb's behavior.

> 100 emails- 32 seconds > 1000 emails- 400 seconds (8 minutes) One of the reasons why "time to delete 1000 mails" is greater than 10*"time to delete 100 mails" was : (See Bug 226730 Comment #6) > for every 300 messages deleted, we're committing the .msf file and that's taking close to a second And, if very large mail folder, "committing the .msf" will take longer than a second. (When my test folder of 80,000 mails, 2GB file size, ".msf" is 60MB)

(In reply to comment #2) "2GB file size, msf=60MB" was for test folder of 320,000 mails. Sorry for spam.

I couldn't find a bug which discusses the commit frequency. Does the commit value need to be tweaked? Are the assumptions used determine the 300 message per commit (from x years ago) still valid? In reply to comment #2) >... > One of the reasons why "time to delete 1000 mails" is greater than 10*"time to > delete 100 mails" was : (See Bug 226730 Comment #6 Bug 226730 Comment #6 reads: David Bienvenu 2003-11-27 08:35:45 PST when you delete messages from a local folder, we have to go mark the messages deleted in the local mail folder, adjusting the x-mozilla-status line of each message to indicate that it has been deleted. This involves seeking to each message and rewriting the x-mozilla-status line, as you can tell from your log. It's not really mozilla's fault if an individual seek take a second, but I'm not sure that it's really the seek that's taking a second. I suspect what's happening is that for every 300 messages deleted, we're committing the .msf file and that's taking close to a second. I can't tell how many messages you're deleting, but if it's 1000, for example, you'll have 3 db commits. We were trying to keep the db in sync, but in this particular example, it's the wrong thing to do, and I'll try to attach a patch to fix it.

Is it possible that this same issue would affect IMAP mailboxes, or is that likely to be a different problem? I experience this frequently with IMAP where Thunderbird will eat 100% of a CPU for 20 or 40 minutes when deleting a few thousand messages from an IMAP mailbox.

(In reply to comment #5) > I experience this frequently with IMAP Delete process is different in POP3 and IMAP, although action on ".msf" is similar. (next is inaccurate and very rough, but conceptually not-wrong) (A) POP3 1. Copy of all mail data to be moved to file for move target folder This is physical file read & write of local file by Tb. 2. Refresh of ".msf"(mail db) for move target folder(Trash) 3. Set of "deleted flag" of X-Mozilla-Status: in file for source folder 4. Refresh of ".msf"(mail db) for source folder (B) IMAP (care on "move to a folder on same non-Gmail IMAP server" only) 1. Request "copy" to IMAP server for all mail to be moved File I/O(physical file read/write==copy) occurs at IMAP server IMAP server has to update "index data" for move target folder 2. Re-sync and refresh of ".msf"(mail db) for move target(Trash) 3. Request "flag \Deleted" to IMAP server for all mail which is moved IMAP server has to update "index data" for source folder 4. Re-sync and refresh of ".msf"(mail db) for source folder When Shift+Delete, step 1 & step 2 are bypassed. To Dave Miller: Your case looks for me phenomenon just before Bug 340265 starts to occur. > Bug 340265 can't remove/delete large number of messages at the same time, timeout error (imap) > Bug 368685 Large IMAP (shift+)Delete causes Memory Exhaustion I recommend you to open separate bug for IMAP case, after check real protocol level flow by IMAP protocol log, and after search bugzilla.mozilla.org well for "open(and DUP'ed) bugs with 'IMAP & delete' in bug summary".

How is that process any different if I sync for offline use frequently? I would then have local storage to be dealt with, right?

at the point when I hang, all traffic to the IMAP server seems to have stopped. Process Sampling indicates Thunderbird is stuck in nsMsgDatabase::ReparentChildrenOf when it hangs.

(In reply to comment #7) > How is that process any different if I sync for offline use frequently? > I would then have local storage to be dealt with, right? When I deleted mails in Gmail IMAP folder of offline-use=on while online mode, content of local mail data file was not changed(X-Mozilla-Status: was not modified). When "Compact Folder", the local mail data file was updated(mail data for deleted mail was removed) in addition to Expunge to IMAP server. I think there is no difference in general flow of "delete" between offline-use=on and offline-use=off, although I think what is done at step B-2/B-4(what field/data of msf is updated) depends on offline-use option. (In reply to comment #8) > at the point when I hang, all traffic to the IMAP server seems to have stopped. > Process Sampling indicates Thunderbird is stuck in nsMsgDatabase::ReparentChildrenOf when it hangs. It looks that your problem is never "Thunderbird is slow to delete"(issue of this bug). It sounds hang/loop while nsMsgDatabase::ReparentChildrenOf (perhaps for threading of mails), then communication with IMAP stopped("communication with IMAP" is victim). Bug 428614 seems to be trunk crash when recursive threads in news. Problem on specific IMAP folder? Problem occurs even after "Compact Folder"? To Dave Miller: I can't think your problem has relation to this bug. Open separate bug with sufficient data such as protcol log, msf file backup, with clear problem description, after search bugzilla.mozilla.org well for already opened/dup'ed bugs for same problem as yours.

(In reply to comment #9) > It looks that your problem is never "Thunderbird is slow to delete"(issue of > this bug). It sounds hang/loop while nsMsgDatabase::ReparentChildrenOf > (perhaps for threading of mails), then communication with IMAP > stopped("communication with IMAP" is victim). a) The folder in question has threading disabled, they're just in the order received and that's it. b) Thunderbird shouldn't be consuming 100% CPU while it waits for a response from the IMAP server if the problem is really that communication has stopped. There aren't any other bugs, I searched for a long time before I started poking this one. :) If you're sure it's not the same issue, I'll go ahead and file another bug.

Bug 429753 has been created for the IMAP version. You'll note the description is almost word-for-word identical to this bug except for s/POP3/IMAP/. The symptoms are *exactly* the same. That's the reason I'm so hesitant to believe it's really a separate problem. (But then, I don't know the code, so maybe it's just broken in the same way in both places or something)

I am very confident that Dave Miller's problem is the same problem noted here; his bug (bug 429753) ended up duped to bug 296453. The 100% cpu usage fits and the fact that deleting e-mails from the bottom instead of the top has a serious performance impact also matches. This also jives with the code and statistical profiling of deletion slowdown in POP3.