User Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36 Steps to reproduce: Automatic compacting folders isn't working for me. For all my installations (Windows XP, Windows 7 x86, Windows Server 2003 x86 Terminal Server, Windows Server 2008 R2 Remote Desktop Services) exclusively imap accounts are used with the following settings: - mail.purge.ask = true - mail.purge_threshhold_mb = 20 - mail.prompt_purge_threshhold = true But the MBOX files in the profiles folder (and the Terminal Server profiles!) growing permanently. The prompt "Do you wish to compact ...?" don't appears, too. A manual compaction (a single folder or all folders) is working without any issues. Then a single MBOX file (not only in total!) reduces the size over 20 MB after compaction. Actual results: Nothing, no prompt, no compaction. Expected results: The MBOX files should be reduced by the specified settings (mail.purge_threshhold_mb = 20).

Are you seeing this also with version 38.3.0?

I can confirm that this happens with Thunderbird 38.3.0 (thunderbird 1:38.3.0+build1-0ubuntu0.14.04.1) under Ubuntu/Linux as well. - mail.purge.ask = true - mail.purge_threshhold_mb = 20 - mail.prompt_purge_threshhold = true INBOX size before compacting the folder (but after closing thunderbird): 4.1 GB INBOX size after starting thunderbird and doing a manual "Compact Folder": 280 K

I think the folder compactor should cope with the size to be shrinked to be above 4GB since TB 38 (bug 894012). I think this could have happened in your case: 1. the folder was accumulating expunged bytes to be compacted (from deleted messages) in version TB 31. 2. the 32bit counter wrapped to 0 when reaching 4GB of expunged bytes. When when the counter upgraded to 64bit (to carry more than 4GB), that will not fix the 0 (or very low amount) stored already in the database. The fix will only take effect only the next time you reach 4GB again (which you should not as after 20MB you should get automatic compact). What you could try as an experiment, if you still have a backup of the 4.1GB folder: restore that folder back into TB profile, but delete the corresponding msf file. I am not sure how IMAP will behave in this case. So do this in your testing environment (a test profile or a copy of your profile). But if it were POP3, it would rebuild the msf file (the database) and accumulate the size to be expunged again, now into the larger 64bit counter. Then it could see that that is more than 20mb and initiate a compact.

I have a backup of the profile and followed your instructions noticing the following a) quit thunderbird, delete INBOX.msf and start thunderbird again: before the restart: -rw------- 1 albert albert 4.1G Nov 19 07:58 INBOX -rw-r--r-- 1 albert albert 7.4K Nov 19 08:01 INBOX.msf afterwards: -rw------- 1 albert albert 4.1G Nov 20 07:42 INBOX -rw-rw-r-- 1 albert albert 5.1K Nov 20 07:42 INBOX.msf => it looks like only the .msf has been re-generated. folder info shows: number of messages: 4 size on disk: 254 kb manually repairing or compacting the folder removes the old messages and reduces the folder size to 255 kb. I am afraid that the 4 gb limit might not be the only problem. Is it possible that folders != INBOX are not expunged automatically? b) there is another folder called _action which only uses 391 mb: folder info: number of messages: 11 size on disk: 6.6 mb an ls in the ImapMap directory reveals the following: -rw------- 1 albert albert 391M Nov 18 16:56 _action -rw-r--r-- 1 albert albert 15K Nov 20 07:30 _action.msf quitting thunderbird, removing_action.msf and starting thunderbird again yields: -rw-rw-r-- 1 albert albert 13K Nov 20 07:36 _action.msf -rw------- 1 albert albert 393M Nov 20 07:36 _action (it looks like thunderbird is re-acquiring the current messages from imap but not cleaning up the local mail folder). manually compacting or repairing the folder finally cleans up the old messages: -rw------- 1 albert albert 1.2M Nov 20 07:36 _action -rw-rw-r-- 1 albert albert 14K Nov 20 07:36 _action.msf

> I am afraid that the 4 gb limit might not be the only problem. Is it possible that folders != INBOX are not expunged automatically? Albert, Same in version 45?

(reporter's email bounces - so (s)he's no longer in the picture)

I have the same bug : automatic compacting doesn't work at all and had never worked since I installed Thunderbird (last year). I'm currently using TB 45.3.0 and there is still the bug. I must compact folders manually. I have Windows 8.1 x64. Here are my infos : Paramètres de base de l'application Nom: Thunderbird Version: 45.3.0 Agent utilisateur: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.3.0 Dossier de profil: Ouvrir le dossier correspondant (Lecteur local) Identifiant de compilation de l'application: 20160825102941 Plugins activés: about:plugins Configuration de compilation: about:buildconfig Utilisation mémoire: about:memory Comptes courrier et groupes account1: INCOMING: account1, , (imap) imap.free.fr:993, SSL, passwordCleartext OUTGOING: , smtp.free.fr:25, plain, none, true account2: INCOMING: account2, , (none) Local Folders, plain, passwordCleartext account3: INCOMING: account3, , (imap) imap.free.fr:993, SSL, passwordCleartext OUTGOING: , smtp.free.fr:25, plain, none, true Rapports de plantage Extensions Signal Spam, 0.6.1, true, signal-spam@signal-spam.fr Préférences modifiées importantes Nom: Valeur browser.cache.disk.capacity: 358400 browser.cache.disk.filesystem_reported: 1 browser.cache.disk.smart_size_cached_value: 358400 browser.cache.disk.smart_size.first_run: false browser.cache.disk.smart_size.use_old_max: false dom.apps.reset-permissions: true extensions.lastAppVersion: 45.3.0 font.name.monospace.el: Consolas font.name.monospace.x-cyrillic: Consolas font.name.monospace.x-unicode: Consolas font.name.monospace.x-western: Consolas font.name.sans-serif.el: Calibri font.name.sans-serif.x-cyrillic: Calibri font.name.sans-serif.x-unicode: Calibri font.name.sans-serif.x-western: Calibri font.name.serif.el: Cambria font.name.serif.x-cyrillic: Cambria font.name.serif.x-unicode: Cambria font.name.serif.x-western: Cambria font.size.fixed.el: 14 font.size.fixed.x-cyrillic: 14 font.size.fixed.x-unicode: 14 font.size.fixed.x-western: 14 font.size.variable.el: 17 font.size.variable.x-cyrillic: 17 font.size.variable.x-unicode: 17 font.size.variable.x-western: 17 general.autoScroll: false gfx.crash-guard.d3d11layers.appVersion: 45.3.0 gfx.crash-guard.d3d11layers.deviceID: 0x0416 gfx.crash-guard.d3d11layers.driverVersion: 10.18.14.4414 gfx.crash-guard.d3d11layers.feature-d2d: true gfx.crash-guard.d3d11layers.feature-d3d11: true gfx.crash-guard.glcontext.gfx.driver-init.direct3d11-angle: true gfx.crash-guard.glcontext.gfx.driver-init.webgl-angle: true gfx.crash-guard.glcontext.gfx.driver-init.webgl-angle-force-d3d11: false gfx.crash-guard.glcontext.gfx.driver-init.webgl-angle-force-warp: false gfx.crash-guard.glcontext.gfx.driver-init.webgl-angle-try-d3d11: true gfx.crash-guard.status.d3d11layers: 2 gfx.crash-guard.status.glcontext: 2 gfx.direct2d.disabled: false gfx.direct3d.last_used_feature_level_idx: 0 layers.acceleration.disabled: false mail.openMessageBehavior.version: 1 mail.winsearch.firstRunDone: true mailnews.database.global.datastore.id: 8ee6a6c7-37a0-4865-ae80-f51820864d9 mailnews.database.global.views.conversation.columns: {"threadCol":{"visible":true,"ordinal":"1"},"flaggedCol":{"visible":true,"ordinal":"3"},"attachmentCol":{"visible":false… mailnews.database.global.views.global.columns: {"threadCol":{"visible":true,"ordinal":"1"},"flaggedCol":{"visible":true,"ordinal":"3"},"attachmentCol":{"visible":false… network.cookie.prefsMigrated: true network.predictor.cleaned-up: true places.database.lastMaintenance: 1472053788 places.history.expiration.transient_current_max_pages: 104858 plugin.importedState: true plugin.state.flash: 0 Accélération graphique Description de la carte: IGFX ID du vendeur: 0x8086 ID du périphérique: 0x0416 RAM de la carte: Unknown Pilotes de la carte: igdumdim64 igd10iumd64 igd10iumd64 igdumdim32 igd10iumd32 igd10iumd32 Version du pilote: 10.18.14.4414 Date du pilote: 3-23-2016 Description de la carte (GPU 2): AMD Radeon HD 8600M Series ID du vendeur (GPU 2): 0x1002 ID du périphérique (GPU 2): 0x6660 RAM de la carte (GPU 2): 1024 Pilote de la carte (GPU 2): aticfx64 aticfx64 aticfx64 aticfx32 aticfx32 aticfx32 atiumd64 atidxx64 atidxx64 atiumdag atidxx32 atidxx32 atiumdva atiumd6a atitmm64 Version du pilote (GPU 2): 15.200.1062.1004 Date du pilote (GPU 2): 8-3-2015 Direct2D activé: true DirectWrite activé: true (6.3.9600.18123) Paramètres ClearType: Paramètres ClearType introuvables Rendu WebGL: Google Inc. -- ANGLE (Intel(R) HD Graphics 4600 Direct3D11 vs_5_0 ps_5_0) -- OpenGL ES 2.0 (ANGLE 2.1.0.316930d51ea9) Fenêtres avec accélération graphique: 1/1 Direct3D 11 AzureCanvasBackend: direct2d 1.1 AzureSkiaAccelerated: 0 AzureFallbackCanvasBackend: cairo AzureContentBackend: direct2d 1.1 JavaScript Ramasse-miettes incrémentiel: 1 Accessibilité Activée: 0 Empêcher l'accessibilité: 0 Versions des bibliothèques Version minimale attendue Version utilisée NSPR 4.12 4.12 NSS 3.21.1 Basic ECC 3.21.1 Basic ECC NSS Util 3.21.1 3.21.1 NSS SSL 3.21.1 Basic ECC 3.21.1 Basic ECC NSS S/MIME 3.21.1 Basic ECC 3.21.1 Basic ECC

Julien's issue is also reported as https://support.mozilla.org/en-US/questions/1137007 I do not see anything unusual in the settings. Could we unearth what's wrong with a special build?

This bug doesn't seem to be fixed in TB 52 beta. What could I do in order to see it fixed? I have personnally NEVER seen the prompt window asking if I want to compact the folders. If you want to do something about this, this should be for TB 52.

Hey :) I'd like to "wake up" that bug report in order to see it fixed in TB 52. Should I open another bug report to prevent other mozilla developpers? thank you!

Sorry, I wanted to say : "Should I open another bug report to warn other TB developpers?"

We do not have a cause, therefore we do not have fix. So it most certainly will not be solved in early versions of 52. Please do not open another bug report

I Think the new Thunderbird Staff (see https://blog.mozilla.org/thunderbird/2017/12/new-thunderbird-releases-and-new-thunderbird-staff/ )should worry about this huge (and perhaps rare) bug. The next generation of Mozilla Thunderbird should focus on important bug and not only on lifting interface. Hey Wayne, do you know a member of Mozilla that could be involved in that bug? Thank you.

The challenge here is not staff, but someone other than the reporters being able to reproduce this issue. https://support.mozilla.org/en-US/questions/1172905

This is also described in Bug 1432879 Comment 18. This only addresses the lack of auto-compact triggering, for imap folders, in online mode. Without this change, compacting is only prompted when transition to offline occurs. When I first started tb daily after adding this change, I was prompted to confirm that I wanted to compact all my folders. I did and it worked properly. On subsequent restarts there was no prompt. If the prompts become annoying to the user, they can disable it or increase the MB threshold in the advanced settings, or just let it occur automatically with no prompt. It may help in avoiding "disk full" problems when the size of uncompressed folders build up over the years.

Comment on attachment 8953892 [details] [diff] [review] 1157256-review-changes0.patch Sadly what the commit comment says is true: > For unknown reasons, imap folder were only auto-compacted in offline mode. We have no indication why this is so since manual compaction works. So all we can do is to try what happens. Many user will never have gone offline in their lives, so if this change isn't right, we will hear about it real soon. As always, I'll adjust the comment slightly: + // Allow imap folder auto-compact to occur when online or offline like + // local folders and pop3 folders. Technically there are no "pop3" folders. Users using POP3 download their e-mail into local folders. BTW, thanks for tracking this down and sorry about the delay. TB was quite busted in the last couple of days and that work had absolute priority.

Pushed by mozilla@jorgk.com: https://hg.mozilla.org/comm-central/rev/b76b8ab0796b Allow auto-compact to occur for IMAP folder when online. r=jorgk

(In reply to Jorg K (GMT+1) from comment #19) > Comment on attachment 8953892 [details] [diff] [review] > 1157256-review-changes0.patch > > Sadly what the commit comment says is true: > > For unknown reasons, imap folder were only auto-compacted in offline mode. Yes, and the "blame" feature of DXR indicated, I think, that it was like this at day 1. > We have no indication why this is so since manual compaction works. So all > we can do is to try what happens. > > Many user will never have gone offline in their lives, so if this change > isn't right, we will hear about it real soon. I would sometimes see a prompt to compact all my folders when I looked at nntp (usenet gmane) with tb. It always surprised me when it happens. But never saw it when I just stayed on imap accounts. Probably should be fairly high in the beta release notes since it might be surprising if and when it occurs.

(In reply to gene smith from comment #22) > Probably should be fairly high in the beta release notes since it might be > surprising if and when it occurs. Absolutely!

I'm trying to wrap my head around how this could have been broken forever/not be a regression. But looking at annotate of nsImapMailFolder.cpp [1] there does not appear to be a "change" that caused this. I'd be interested in any theories - was this always broken? That asside, I have some questions and observations, partly directed at Gene but all feedback welcomed ... (In reply to gene smith from comment #18) > Created attachment 8953892 [details] [diff] [review] > 1157256-review-changes0.patch > > This is also described in Bug 1432879 Comment 18. This only addresses the > lack of auto-compact triggering, for imap folders, in online mode. Without > this change, compacting is only prompted when transition to offline occurs. 1. My reading of this bug, is only users who have ONLY imap are affected, i.e. auto compact kicks in for users with any pop or news accounts. Including if a user starts Thunderbird in online mode. My profiles have always had at least one news or pop account, and when I opened a newsgroup I would often get a compact prompt - so presumably that is a symptom of this bug. Does this agree with your understanding of the code? 2. And users who only have imap accounts would compact only if the go offline. But desktop users probably never go offline and therefore not compact. And laptop users also might never get compact depending on their OS and their offline config settings (which came to our UI in version 37 via bug 731145). The current default for Mac and Windows is online/offline automatically follows "detected online state". But it hasn't always been the case (I forget when mac and Windows changed - version 41 per bug 654579?). And Linux users don't get automatic offline preference by default prior to version 55, via bug 1340831 a year ago. To summarize, if this bug is not a regression, and many users only have imap accounts, then many users never had their folders compacted. From a support perspective that's pretty racy. Have a stated any untruths? > It may help in avoiding "disk full" problems when the size of uncompressed > folders build up over the years. Sure as heck should. So glad to see this patch. We should probably uplift to 52. Perhaps also bug 1340831? [1] I don't see a smoking gun around the line in question. Prior to bug 1294260 if (rv == NS_MSG_ERROR_OFFLINE) was switch (rv) { case NS_MSG_ERROR_OFFLINE: if (aMsgWindow) AutoCompact(aMsgWindow); ... and prior to bug 439480 if (aMsgWindow) was if (msgWindow) per https://bugzilla.mozilla.org/attachment.cgi?id=383539&action=diff I don't see any history prior to that

(In reply to Wayne Mery (:wsmwk) from comment #24) > That asside, I have some questions and observations, partly directed at Gene > but all feedback welcomed ... Upon reading this, I believe I do have this problem on my Windows work computer. - only one IMAP account - Even though there is a news account, it's not being used at all, and it cannot reach the server from inside the office network. Beyond that I suppose RSS feeds don't matter for this problem. - never going offline At times I was wondering why I never saw a prompt for compacting, but I guess I just shrugged it off. I didn't become a problem because I'm compacting religiously by manually triggering it. So I guess I should see compacting prompts after updating to TB60? Updating is a different story though, due to add-on (in)compatibility.

(In reply to Christian Riechers from comment #25) > ... > - Even though there is a news account, it's not being used at all, and it > cannot reach the server from inside the office right - and at least for me, the compact would not happen unless I actually clicked a newsgroup - whereupon the dialog immediately appeared > At times I was wondering why I never saw a prompt for compacting, but I > guess I just shrugged it off. I didn't become a problem because I'm > compacting religiously by manually triggering it. Also many users who have visited support forums will be in this boat, having been advised or seen support articles to disable automtic compact. > So I guess I should see compacting prompts after updating to TB60? Yes, if you don't have prompts supressed

Comment on attachment 8956607 [details] [diff] [review] 1157256-review-changes0.patch - modified comment, patch for checkin [Approval Request Comment] Regression caused by (bug #): none afaik (not 100% clear) User impact if declined: performance issues, high disk space usage Testing completed (on c-c, etc.): been on 60 beta for ~two months Risk to taking this patch (and alternatives if risky): should be zero risk

Comment on attachment 8956607 [details] [diff] [review] 1157256-review-changes0.patch - modified comment, patch for checkin Wayne, I think TB 52 is coming to an end very soon now. This came with a big bold release note: "Thunderbird will now prompt to compact IMAP folders even if the account is online". So I prefer not to sneak something like this in at the very last minute.

(In reply to Jorg K (GMT+2) (bustage-fix only, NI for urgent reviews) from comment #29) > I think TB 52 is coming to an end very soon now. ... So I prefer not to sneak something like this in at the very last minute. If we were weeks away from the end of 52 I would agree with not uplifting, in fact I would not have asked for uplift. But we are not weeks away from the end. We are likely 3 months from being fully unthrottled for updates. And given past experience we are 4-5 months until to 80-90% adoption - which is probably optimistic given the version 60 add-ons situation. We also know some enterprises will hang on 52 much longer. > This came with a big bold release note: "Thunderbird will now prompt to compact IMAP folders even > if the account is online". This note implies users are being compacted but not being prompted. Regardless, the real impact here is the several side effects of not being impacted. As we know, prompting already happens for pop and news users, and imap users who go offline for some reason. So the prompt is not new to us and the checkbox allows user to forever disable. There is no signficant burden here. Even if one choses to make prompting relevant, deciding to take the patch must be balanced against the user impact of not being compacted for several years, which include: * performance - higher memory usage impacting overall system performance, longer folder open times, more battery (from more IO), jank during compose and sending mail (large folders - inbox, draft and sent - are known to have user visible effects). For example bug 1432879 * stability - we've had cases where users with large folders crash OOM - unless they discovered 64bit builds * higher disk space and longer backup times (especially important to enterprise users) (sorry for not posting this larger list earlier) We will have a significant impacted user population on 52 for many months. Please make this very important patch available for imap users of 52.

Comment on attachment 8956607 [details] [diff] [review] 1157256-review-changes0.patch - modified comment, patch for checkin I'll send complaints your way ;-(

TB 52.9: https://hg.mozilla.org/releases/comm-esr52/rev/2fa8e07e4394de761c145112825f1f771cb99e36 I sincerely hope we won't get into trouble with this: - Users will see an unknown popup asking them to compact. - If only 0.01% of 5 million users have problem with compaction, that's 500 unhappy users :-(

No doubt this change won't be 100% clean, because we still have bugs*. But (I'm citing this not to be persuasive, but rather to add to the record) https://support.mozilla.org/en-US/questions/1225116 is yet another example of why we need it .... "Having problems with laptop having no storage and it turns out Thunderbird is taking up a HUGE amount of memory [actually he means disk] on it - despite me deleting all content and removing from spam and bin (only keep 6 x months of emails)" * We can expect to see reports about the following (to name just a few): ** bug 716412 compact dialog prompt: unchecking "Always ask me before compacting folders automatically" fails to change or keep mail.purge.ask preference to false ** Bug 139215 - mail not filtered while compacting folders ** Bug 479064 - When a deleted message causes 'compact folders', the message list pane loses focus/selection and scrolls back to the top ** Bug 540857 - Sort by thread is forgotten (probably after compacting folders)