User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:16.0) Gecko/20100101 Firefox/16.0 Build ID: 20121010234852 Steps to reproduce: Reopening bug #802217 as it is a separate issue from #803843 as per https://bugzilla.mozilla.org/show_bug.cgi?id=803843#c33 This is happening (at least) for 16.0.2 as downloaded from Ubuntu.

The continuous downloading is happening in the same folders where the size growth from bug 803843 was happening, so the same messages are probably triggering it. To trigger the behaviour I only need to go to the folder. Leaving the folder and going back into it right away triggers it again.

The test message from the other bug does not trigger this behaviour for me. I took it, forwarded it to myself with a 10mb PDF attachment to increase its size, and have no problems opening it in a folder that contains just that message, and it doesn't redownload. What seems to help is repairing the folder. After I repair them, switching into it doesn't trigger the behaviour anymore. I have one folder with 10 messages where it is still triggered (its only 500kb or so). I would guess that if I repair it it will fix the problem as it did in the several other folders before.

Can you save the .msf from that folder ?

I see the issue in SeaMonkey 2.13.1 as well. We updated from 2.10.1 to 2.13.1 and now we have several IMAP issues, including: - slow opening of messages with big attachment (should retrieve only text when initially showing the message and retrieve attachment only when it is saved) - wrong information in Size column of the message list, changing when attachments are opened - attachments sometimes cannot be opened, the dreaded "This part will be downloaded on demand" problem that has been lingering for a long time. repair of the .msf file fixes it, but it re-occurs when visiting another message and returning - apparently depending on details of the MIME structure of the message, some messages show consistent problems and on other messages it is not reproducible I will investigate with 2.13.2 just released, and with 2.12

I have done a lot of testing and decided to revert to 2.12.1 which solves our issues. 2.13.2 is better than 2.13.1 but there are still problems.

I have the same issue in Tb17 on Mac 10.8.2 Interesting fact: When compacting the affected folder, *all* instances of that duplicated email are removed from the file. If you need additional info, I'm happy to help/test.

I am having this issue on TB16.0.2 with Win7x64. I am not experiencing any IMAP mail growth. My inbox is about 40mb and is not growing. The folders that are constantly being redownloaded do not appear to be growing without bound, either.

@Eric: does the issue go away when you repair (right click -> properties) the folder?

Additional information: With SeaMonkey 2.13.1 when an attachment cannot be opened (doubleclick on attachment, the saved temporary file is not correct and the started application throws an error), a repair fixes the problem for 1 try, the problem comes back when re-visiting the same message and another repair is required. With SeaMonkey 2.13.2 and a freshly created .msf it works OK, but when repair is selected the problem then occurs. (inverse of 2.13.1) This is a setup where IMAP is used (UW IMAPD) and folders are NOT synchronized for offline use. So the file growth is not visible, but there still are problems.

I just found myself with a 208gb /home/user/.thunderbird folder. Compacting the affected folders does drop the size by around 50gb, and doesn't seem to grow as much as it used to prior to 16.0.2 but I'm still getting banned by Google for a few hours at a time due too much bandwidth. Its still re-downloading the messages. Will grab anything you need or more information. Ubuntu 12.10

Folder repair is a local phenomenon. In other words, the folder being repaired is the *local* filesystem folder. I can create a brand new Thunderbird profile and have this problem. It will act like it is downloading all the messages and eventually finish. Once I close Thunderbird, it starts all over on the next restart. Additionally, a message that I just clicked on and viewed in the preview pane will be re-downloaded the next time I click on it. The little status area at the bottom says "loading message." In TB15, when the message is downloaded it does not attempt to re-download it the next time I view it.

Join bug 805830 if you are banned from gmail.

I am experiencing the same behavior running TB 16.0.2 under Ubuntu 12.04. IMAP keeps downloading but not writing into the folder and after a few hours google is banning my account for excessive usage. While I was using TB 16.0.1 the filesize of the IMAP folder was constantly increasing. This is a serious problem rendering such a critical program mostly unusable.

Just for giggles I have repaired my inbox folder and am still noticing the same issues: * Clicking around on various messages, every time I return to the same message it appears to be downloaded again. * Going to File->Offline->Download/Sync Now and selecting to sync the Inbox results in a status message being displayed that it is downloading the various messages, but no changes to the filesize of the inbox file actually occur - after the repair it's still sitting at 0kb. * Restarting Thunderbird results in an immediate activity in the activity manager of "Bringing Inbox up to date" -- right after I've just told TB to sync/download the folder! * After clicking around again and deleting a few messages from Inbox, nothing has changed with the filesize. It also is still downloading the message I click on every time. * I then saw another activity "Bringing Inbox up to date" where it is showing that it is downloading all 110 messages again, but nothing much is happening to the inbox IMAP file on disk. I have another mail client attached to this same account (TB15 on Fedora 16) and the Inbox file is approximately 100meg. Before the repair on TB16/Win7x64 the Inbox file was only up to about 46 meg. Something is definitely completely bonkers with the IMAP engine in TB16+. 16 and 16.0.2 seem to do the same thing, and it's gnarly.

This should probably be a blocker for bug 805830 (banned from gmail). And I would suggest raising the importance a notch.

I finally got Glodaquilla to work against 16.0.2. Guess what? Not a single message in inbox is being marked as "on disk." In fact, after looking through every folder I have, I would say that less than 20 messages are marked as being "on disk." This is despite the fact that the "Imapmail" folder for this profile is ~117MB. So, something is definitely going on here.

Ah, nice addon! The one folder where the continuous downloading happens for me has one mail marked as not on disk (all other folders have all mails downloaded after repairing them), and that is the only mail with the gloda dirty flag as well. The flag clears after 30 seconds or so, and then when the next re-download happens another 30 seconds or so later, its obviously set again. Rinse and repeat.

What does the dirty flag imply? What is 0 and what is 1? I'm seeing odd behavior in inbox.

I can provide a sample mail account for anyone who needs to test this against a server that actually causes it.

Thanks for all your triage work, everybody. The information from Glodaquilla is very helpful. I can reproduce this in my development environment and I'm working on a fix; I'll reach out to you if I need any additional information.

http://mesquilla.com/extensions/glodaquilla/ describes the flags. more detail - https://developer.mozilla.org/en-US/docs/Thunderbird/Gloda_indexing describes indexing - https://developer.mozilla.org/en-US/docs/Thunderbird/gloda gloda generally dirty means needs to be reindexed 0 means not indexed 1 means bad message / dont' reindex (iirc)

(In reply to Wayne Mery (:wsmwk) from comment #22) > http://mesquilla.com/extensions/glodaquilla/ describes the flags. > > more detail > - https://developer.mozilla.org/en-US/docs/Thunderbird/Gloda_indexing > describes indexing > - https://developer.mozilla.org/en-US/docs/Thunderbird/gloda gloda generally > > dirty means needs to be reindexed > 0 means not indexed > 1 means bad message / dont' reindex (iirc) sorry, correction (I shouldn't trust memory) - dirty flag - 1 means dirty, needs index update - gloda id - 0 not indexed 1 means bad message / dont' reindex 2 to 15 or 32 (reserved)

Disabling download of MIME parts on demand may work around this issue - could some of you try setting the advanced preference mail.imap.mime_parts_on_demand to false, restart Thunderbird, and see if the problem goes away?

(In reply to Irving Reid (:irving) from comment #24) > Disabling download of MIME parts on demand may work around this issue - > could some of you try setting the advanced preference > > mail.imap.mime_parts_on_demand > > to false, restart Thunderbird, and see if the problem goes away? I had tested this to see if it fixes the issues mentioned in comment #9, but it doesn't.

I saw no change. I am not seeing as much "bringing up to date" as I was before, but no messages are flagged as being on disk and it tries to download every one every time I click on it. So something is still wrong.

Redownload also happens with: Thunderbird 16.0.2, WinXP and Win 7 X64, Server: Zarafa 7.1.1. After repairing the problematic folders, the problematic mails are not offline available any more! In my inbox are only such mails, thus the local inbox file is always empty (file size = 0 bytes) now... mail.imap.mime_parts_on_demand = false didn't change anything.

I have the same issue, beginning with 16.0, now with 16.0.2. Dovecot 1.2 is the remote server, I can provide access to it if that is helpful. If it helps, here's my report: I'm configured for offline folders with a limit of 50 KB per message. It seems that since TB 16.x, messages from several folders are reloaded periodically, even if I hadn't opened them for weeks. Also, the status line is wrong, e.g. it shows "Loading message 1 of 1 from Folder XYZ", although in the folder there are e.g. 500 messages. Compacting messages and repairing folders didn't help at all. What is interesting, though, is that the problem seems to vanish for the current session only, when I sync all messages, even if new messages arrive after that. After a TB restart, the same problem occurs, until I sync again.

We've started having IMAP issues with our Exchange server at work. I don't think our IT group has thought of it yet but it could very likely related to the fact that we have a large number of Thunderbird users that have upgraded to this issue. As Thunderbird user I'm also having similar issues with my Gmail but it doesn't seem as bad. Either Google can better handle the extra demand or have been temporarily blocking the worst offenders. I hope this gets fixed before our IT department bans Thunderbird.

I offered to irving to look at this, and I played around with this for a few hours, using a test mail he sent me. I really cannot reproduce anything though - but I am not really sure what I need to do to get there. It would be very helpful to me if those who are seeing this would try to create a reproducible case on a fresh profile. Alternately, if you are seeing this on a working profile but cannot reproduce on a fresh profile, that would also be interesting to know. I'm happy to fix a problem that I can reproduce, but I need help to get to that reproducible condition.

When you say "fresh profile" you mean a brand new Thunderbird profile, right? 100% of my testing was done with a new Thunderbird profile that had zero mail accounts in it, that I then attached to an existing Zarafa 7.0.x account via IMAP. There are multiple people in this thread offering to set up accounts on servers that cause this behavior, myself included.

I'll just privately give you my account where it can clearly be duplicated. (Gmail account). Thunderbird was banned Campus wide today, due to basically crashing IMAP servers for constant re-downloading.

Erik, could setup a test account that shows the issues that you are having (including having sample problematic emails in the folders), and then tell me what configuration I need to use to connect Thunderbird so that I can duplicate this? I would also appreciate either a detailed summary STR post, or reference specific comments that you think do that already.

Connor, if you have a setup that you think that I could use to reproduce your issues, I would be happy to try that as well. Private information you can send to kent@caspia.com

Kent - I sent you an email directly to your caspia email address and provided login details for our mail system.

I tried the repair folder trick, and it seemed to work. But the operation reset the folder settings; I like to see additional columns such as "Recipient". As soon as I set that, I noticed the issue started again. Win7 (all updates applied), x64. Sorry I didn't test on a new profile, and the timing of my change and noticing the issue may be coincidental. But I figure every bit of info may help you. What also should be mentioned, is that there are known IMAP problems with recent version of MS Exchange. So much so, I ended up installing the DavMail gateway.

The account that Erik provided me is failing to set the offline status for any message at all. So at least the problem that he is having is reproducible, and we are investigating. I suspect there is more than one issue though, but I'll know more after I have investigated some more.

Kent - if you test the account with TB15, you will have no problems. That may provide some comparison/reference.

Did anyone check what changed in IMAP or the folder management between TB15 and TB16 (or SM2.12 and SM2.13)? Is it just a small change that causes these problems or has there been a rewrite of some sort? Any chance of just backing out the change(s) and release a new update?

Using Erik's server, irving and I looked at this, and came to some conclusions about root causes, at least for that server. Here are excerpts from the conversation: [08:18] rkent irving: But the issue I am looking at is much bigger than line ending counts. The debug message you added says this: [08:18] rkent 0[100f140]: ###!!! ASSERTION: Offline message too small: messageSize=19812 curStorePos=5645 numOfflineMsgLines=148: 'Err [08:18] rkent or', file c:/tb/beta/src/mailnews/base/util/nsMsgDBFolder.cpp, line 1735 [08:20] rkent The check that is failing is: [08:20] rkent if (messageSize > (uint32_t) curStorePos && [08:20] rkent (messageSize - (uint32_t) curStorePos) > (uint32_t) m_numOfflineMsgLines) [08:44] irving it's clearly messageSize that's bogus at the "ASSERTION: Offline message too small" line [08:46] irving (could be some other reason we have a bad messageSize, multiple message download might still be a distraction) ... [09:15] irving rkent: ermahgerd we're detecting it as an AOL mail server, requesting XAOL.SIZE when we pull the headers, and getting the bogus number [09:19] irving rkent: I have 7852[c4eaef0]: ce55000:www.jumpshipservices.co:S-INBOX:CreateNewLineFromSocket: * 1 FETCH (UID 549697 XAOL.SIZE 18517 BODY[HEADER.FIELDS (FROM TO CC BCC SUBJECT DATE MESSAGE-ID PRIORITY X-PRIORITY REFERENCES NEWSGROUPS IN-REPLY-TO CONTENT-TYPE)] {551} [09:20] rkent OK, but we have some line numbering difference [09:22] irving rkent: when we log into the server it includes XAOL-OPTION in its capabilities, so we're not making a mistake on that [09:25] irving rkent: i'm guessing that the trigger is, we trust that size header more in the current build than we did before the bug 92111 patch ... [09:31] irving before bug 92111, we would update the message size on the first block of body data we received (but with an incorrect size if the server had the 92111 bug) [09:32] irving I think that was covering for the XAOL-SIZE issue, and maybe for whatever is bothering gmail too [09:33] irving problem is, we can't know the correct size until we download all the chunks, in the case of chunked message download

Rob: The regression range we expect is described here: https://bugzilla.mozilla.org/show_bug.cgi?id=803843#c11

This undoes a behaviour change introduced in bug 740453, where we changed from updating the message database's size field (with a sometimes incorrect value) on every offline message download, to only updating when the server's RFC822 size didn't match the real message size. With this patch, we update the DB on every complete message download (including messages not stored offline), with the exact message size as downloaded. If you would prefer the patch to more closely match the old implementation (only updating the size for offline messages) the change won't be too complex.

There are now debug "try" builds available with this patch at http://ftp.mozilla.org/pub/mozilla.org/thunderbird/try-builds/ireid@mozilla.com-fcd5c6c5168e/ If a few of you who have reported this issue could please download and test these builds it would be a big help to us.

I have 19.01a running now. I selected all folders and did a Folder Compact. My .thunderbird folder went from 55G to 4.2GB after 20 minutes of running. So thats good, and for the first time ever thunderbird said "No messages to be downloaded" and the progress bar isn't doing anything (until of course a message arrives). I will monitor for remainder of day.

Connor, are you testing against Zarafa or gmail (or both, or other)?

Irving - I tried the "try" build. After thunderbird started it began indexing 11000+ emails and started downloading from different folders. I had to exit it before it stabilized because it consumed an extremely large amount of memory and my system became extremely unresponsive. IO wait was around 50% and load average was over 7 by the time I exited thunderbird. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 26208 pcullum 20 0 12.4g 6.0g 18m D 2.7 75.1 7:04.68 ./thunderbird Activity manager showed that it had indexed 19% of messages but it wasn't progressing anymore. I don't know what this means but the last lines logged on the console were: nsStringStats => mAllocCount: 1232321 => mReallocCount: 139139 => mFreeCount: 1192060 -- LEAKED 40261 !!! => mShareCount: 2426787 => mAdoptCount: 298338 => mAdoptFreeCount: 298332 -- LEAKED 6 !!!

Yes gmail accounts only. I figured the IO was due to messing around with around 50GB of IMAP folders. It stablized afterwards and still working great. Havent been banned from gmail today so I'd say its very good progress if not completely fixed in my case.

Paul, the closing output is normal from a debug build (it's turned off in final builds). I suspect you're now hitting another open issue - can you check the following bugs and see if you fit any of them? Bug 891334, Bug 795590, bug 796989, bug 793792 If so, get in touch with me or comment on the bug, and it would be great if you could try a run with the Thunderbird profiler to help us isolate the problem.

After Connor (comment 32) offered to let me test his gmail account, I loaded his account using a TB 17 beta debug build. It ran for hours (this is a massive account with >6000 messages that results in "Throttled" comments from gmail), but eventually finished. After it finished, it seemed fine. This morning, I opened it again. "All Mail" started downloading from the beginning. It got halfway through (in just a few minutes, must not be Throttled), then seemed to stop, then started over from the beginning again. Now I'm back to Throttled, downloading slowly again, and at message 995 of 6833 in All Mail. So I think there are still some issues here unfortunately. Irving, I think that we should proceed with the fixes that came about from examining Erik's issues in this bug, and file a new bug (or adopt bug 805830) for the gmail issues. I also now have a Debian account, kindly provided by Florian (comment 28), that I hope to also test. I'll also test on my Exchange 2007 accounts.

I won't be able to test until later this evening against my Zarafa account with a lot of mail. I also have a GMail account that I can test against, but it sounds like we've got that covered. I've been doing all my testing on Win7x64 but I do have Fedora 16 x64 that I could test with if anyone desires.

Continuing with my comment 49, I'm not so sure there is an issue. Downloading of offline messages can stop after awhile, then seems to restart later, and this can appear to be redownloading unless I am really paying attention. So I am not 100% confident that "All Mail" was downloaded previously, for example. And now Connor's profile has decided to download 2853 messages in "Important" which I don't think have been downloaded before. I should point out that my experiments do not include the patch for this bug. I think what I really need at the moment is a much more detailed description of the problems that people are experiencing (except for Erik's issues, I think we have those covered). For example: 1) Do you have a folder that was completedly downloaded previously, and is then downloaded again? If so, what triggers the new download (restart of program, selecting the folder, next day, if just random then how frequently does this occur). 2) In the previous issues, there was evidence that a particular type of email caused issues, and this email was repeatedly downloaded. So are particular emails being repeatedly downloaded, or the entire folder? The big issue here is, will the patch that Irving has proposed for Erik's issues also fix other issues, or are there some additional problems? I am particularly suspicious that there may be unrelated issues still plaguing Gmail.

Hi all, having exactly these same symptomps against my Zarafa 6.4.x IMAP server. Worked always well untill TB 16.x was installed as part of my regular updates. To make sure I have no corrupted messages or so I have been through the repair, compacting etc. but no luck. This afternoon I deleted my local TB profile (after having placed my mail folders in a local TB folder) then deleted my Zarafa mail account and all associated from the server. Booted Zarafa server freshly and recreated my email account (got a few calls in the mean time from people getting bounces sending me mail since my mail account was really completely removed for the sake of the test) and then recreated my TB profile. Reimported only my inbox folder with some 1100 msgs and synched completely being connected with eth in the office with LAN connection to server. During the afternoon, saw the same behaviour again (ie. redownloading all msg) and now being connected remotely over VPN it's even more obvious with the slower connection. Let me know if I can provide more/better details to help on this issue. It is quite a big issue, and as we update with Ubuntu regular updates as they come out automatically it is starting to hit other people in the office as well.

I tested today with: - thunderbird-19.0a1.en-US.win32.zip - a existing thunderbird 16.0.2 profile - at Window XP - and Zarafa 7.1.1 IMAP-Gateway - LAN connection Results are: - TB downloaded the messages (~300 in several subfolders) now only one time - The messages are offline available - No growing mbox files - Later TB crashed sometimes when I deleted are moved messages...

Using Kent's glodaquilla extension and turning on the "onDisk" column is very helpful to see whether messages are considered to be properly downloaded for offline storage

So what I am hearing is that there is an issue with TB 16.x with Zarafa servers, and that issue is resolved with Irving's patch.

That might be (but I hear it happening also against other IMAP servers in this thread) but I'm not sure how to solve this. How to implement Irving's patch ? It doesn't seem to be installable on my Ubuntu 12.04 or should I replace some files manually ? I think this is broader than just TB against Zarafa. Remember a few years ago, between v 2 and 3 of TB the same kind of issue arose at some stage (and was solved subsequently but I can't remember in what version exactly).

Joris: I probably agree with you, but the problem is that we do not have a reproducible issue yet with other servers. The Zarafa server issue is real, and because a bug should be limited to one issue, at some point we will need to switch this discussion to other bugs, and declare this bug as specific to the Zarafa server, and "fixed". But let me emphasize again how important it is for anyone experiencing this to develop a test case, and provide access to that test case to a core developer. Erik's issue was easily reproducible, and it did not take long to locate the problem. Irving or I can quickly pin down a fix if we have a reproducible case, but it is almost impossible without one. Currently, we only have a reproducible issue with the Zarafa server.

OK agreed, let's stick to the Zarafa issue, which is my issue as well. So Irving's patch should work for me as well likely ? How do I apply it though ? Went through the ftp link but I'm not sure what's needed exactly. I would like to help to provide reproducible results but our server is behind a corporate F/W... Let me know what you like me to provide and I will try to do so.

For me, the issue arose with a TB<->Gmail/IMAP connection, and I just got an email confirmation from Google that they do not use Zafara but some self-written mail server. Since the last two days I always let wireshark run in the background. Unfortunately, the problem did not show up again as massive as a few days ago (and I did not install the patch). Only sometimes I can observe that TB redownloads either 30 or 50 messages from the "all messages" folder - with no evident need (because I only deleted one email - why must TB now synchronize some 30 other emails?). Sorry I can't help more than this ...

I had the issue with dovecot locally, and it persisted until I repaired the folder, then it was OK. I saved the broken .msf and mailbox... if I can send that to someone privately, please let me know.

Wolfgang: I also observed this "TB redownloads either 30 or 50 messages" with the gmail account that Connor gave me, and I have one possible explanation. During the initial download with the debug version, I can see a number of cancelled downloads due to failures to acquire a folder lock here: c:/tb/beta/src/mailnews/imap/src/nsAutoSyncState.cpp, line 643 After some period of time, there were additional attempts to download messages from folders that I thought were already done, and it is possible that those earlier failures were being downloaded. I have not fully confirmed that this is the cause, but it is a possible explanation.

I second the GMAIL experience. I have about 15,000+ messages in the INBOX. TB persistently downloaded 3 messages, possibly with large attachments over and over til I hit the GMAIL daily limits, and drew 2GB+ of usage off my mobile hotspot. Since I was only watching activity monitor afterwards, I dont know what messages it was downloading. Compacting the mailbox did not help. Repairing the mailbox seemed to have done the trick, although I am not certain if that was the fix, or new messages flowing into the mailbox was the real fix. I have several other mailboxes that do not appear to exhibit this problem (including another GMAIL one that appears to work ok, at least according to activity monitor).

Art (or others): if you can identify a particular message that is repeatedly downloading in gmail, it would be very helpful to either post the message in bug 805830, or send it privately to Irving or myself for analysis.

It would also help us a lot if you could run from the command line with IMAP and related logging turned on; set the environment variable NSPR_LOG_MODULES=imap:5,ImapAutoSync:5,msgdb:5,timestamp,sync before starting Thunderbird, and capture all the output to the console (stdout/stderr for you Linux types). This will include headers and contents of email messages, so if you don't want to publicly post it on Bugzilla you can either send it directly to me or get in touch on irc.mozilla.org #maildev and we'll work together to extract the relevant info.

I am trying to run: http://ftp.mozilla.org/pub/mozilla.org/thunderbird/try-builds/ireid@mozilla.com-fcd5c6c5168e/try-comm-central-win32-debug/thunderbird-19.0a1.en-US.win32.installer.exe I am using Win7x64. When I attempt to start Daily, I get the following error: "The program can't start because MSVCR100D.dll is missing from your computer. Try reinstalling the program to fix this problem." According to this thread: http://social.msdn.microsoft.com/Forums/en/vcgeneral/thread/3c9a5b9b-1a7d-4d86-bc82-85652448e0c9 It sounds like this is some debug library/DLL provided by Visual C++ -- do I need that?

Comment on attachment 679311 [details] [diff] [review] Update message size in database on every full message download I've tested this on Erik's account, and others have tested it, and it seems to work. Code looks good as well.

(In reply to Kent James (:rkent) from comment #66) > Comment on attachment 679311 [details] [diff] [review] > Update message size in database on every full message download > > I've tested this on Erik's account, and others have tested it, and it seems > to work. Code looks good as well. Did you also test for the case where folder data is NOT cached for offline use? In the past, there have been issues causing corruption of attachments when offline caching is not used. For example, receive a mail message with a largish attachment, read only the text part (attachments are never downloaded), then forward the whole message including attachment to another address. Receiver gets only "This part will be downloaded on demand" instead of attachment. Other example: receive message that fulfilled the criteria for the bug discussed here to cause repeated downloading, but in a folder that has offline caching disabled. Open attachments. Attachments are corrupt. Those errors resurfaced at the same time the issue discussed in this bug appeared, and showed the same phenomena (like a wrong message size in the message list, a repair of the folder temporarily fixing it). I think it is the same bug but manifested in a different way because of the different setting of offline caching.

Sorry for a silly question, but where can I download a version of tbird that includes this bug fix? I need both Linux and Windows. Thanks, Doug

The fix is in the latest Nightly builds at http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-comm-central/. If we can get some confirmation that this improves things for people, and doesn't cause any other issues, we'll try to pull it into the last TB 17 Beta build early next week. Rob, we've been focused on the offline storage case because we had the most problem reports about that, and were able to reproduce it in our test environments; now that we have a candidate fix for offline, I'll take a look at your issue.

Been about 12 hours, no problems with the build in comment #70. No bans from Gmail and messages aren't re-downloading like before. ~/.thunderbird is at 4.2GB so no overgrowth. All is well so far :)

Irving, I got this version from the URL above: 19.0a1 (2012-11-10), and so far so good. :) It certainly doesn't seem to be doing any harm, and over a few hours last night, and so far this morning I haven't seen a repeat of the repetitive downloading. I have now visited every folder on the problematic account, just to be sure. If I see any problems I will update here immediately, but I definitely vote that this fix go into 17. Thanks, Doug

It indeed seems that the current nightly fixes the issue for me. Thanks!

Comment on attachment 679311 [details] [diff] [review] Update message size in database on every full message download Review of attachment 679311 [details] [diff] [review]: ----------------------------------------------------------------- Sorry for late sr. This looks like the right thing to do. Given all the positive comments so far, I think we should take this into 17 as well, so a=me for branches.

The nightly in comment #70 worked against my Zarafa instance with my personal account. Glodaquilla indicated all messages in Inbox got downloaded and were "on disk". Clicking around through the message pane didn't result in messages trying to be fetched from the server. Manually forcing an offline sync of all folders seemed to get everything to be indicated as "on disk" with Glodauilla. Looks like it is working.

Interesting spot: It seems that since I ran the nightly build, also my regular 16.0.2 stopped downloading over and over again. Maybe a coincidence, but maybe the nightly also changed the mailspool's database.

Maybe once the messages are in the local database, they are no longer refetched from server even with the logic of TB16. The bug was that under some circumstances (size differences) TB16 refused to store the msgs into the database (or did not mark it as successfully downloaded/stored). So had to refetch them again and again.

Aurora: https://hg.mozilla.org/releases/comm-aurora/rev/5a84996bc6cb Beta: https://hg.mozilla.org/releases/comm-beta/rev/9e078e25d475

Do I assume right that the fix is not included in 17.0b3? Tried it, and still has that bug.

(In reply to Florian Effenberger from comment #79) > Do I assume right that the fix is not included in 17.0b3? Tried it, and > still has that bug. Have you repaired the folder? Right click on the folder, select "Properties...", in the "General Information" pane click "Repair Folder". Thunderbird should download the messages one more time and then stop.

FWIW I just switched to 17.0b3 and it's working fine for me. I already repaired all my folders when I moved to the 19 nightly to test the fix. Also, as an Ubuntu user I found this very helpful: https://launchpad.net/~mozillateam/+archive/thunderbird-next I'm still starting tbird on the command line to make sure that there are no problems, and I see this error periodically: autosyncActivities ERROR OnDownloadError: Folder of Account When I saw that error with 16 it always heralded another round of repetitive downloading, however with the fixed versions it seems to simply be an indicator that the folder is out of synch on my local system, and once it gets updated it seems to stay that way. I use both a laptop and a desktop to access the same mail accounts, so it's expected that the folders on one or the other will be out of synch if I downloaded mail in that folder to the other system already. hth, Doug

Very strange... repaired and compacted the folder a view times, but it seems it was not until I deleted the local folder file, redownloaded again, and repaired and compacted again, until it worked. I'll have a look and see if it works now :)

I have tested thunderbird trunk now with two different Imap server. Thunderbird against Dovecot works as expected, emails are downloaded to local Imap folders and everything is fine. Thunderbird against Zarafa downloads emails but when finished the local Imap folders are reset to a very small size, so emails are not kept. This causes thunderbird to download over and over again. I have started completely fresh with thunderbird trunk. Anything I can to help debugging? I.e. this issue is not fixed... I have also reported this issue in Zarafa forum, https://forums.zarafa.com/showthread.php?8211-A-few-users-with-IMAP-performance-issues-via-Thunderbird.

It seems it is not fully fixed in 17.0b3 (w/ Dovecot 1.2) It works much better than 16.0.2, however, from time to time, a message folder is loaded forever and forever. The folder changes, it seems the pattern is MIME/multipart messages (HTML mails, to be precise) Compacting and/or repairing the folder, as well as downloading all messages (synchronizing) does NOT help. It seems it helps to manually open each message. Right now, I have a folder where "1 of 2 messages" are continously being loaded, it always stucks at message #1. The folder, however is definitely empty, has been compacted, repaired, and downloaded. To no avail. Error console shows, but might be unrelated: Zeitstempel: 17.11.12 20:28:56 Fehler: 2012-11-17 20:28:56 moveCopyModule ERROR Exception: [Exception... "'Component is not available' when calling method: [nsIActivityManager::removeActivity]" nsresult: "0x80040111 (NS_ERROR_NOT_AVAILABLE)" location: "JS frame :: resource:///modules/activity/moveCopy.js :: <TOP_LEVEL> :: line 133" data: no] Quelldatei: resource:///modules/gloda/log4moz.js Zeile: 687 and Zeitstempel: 17.11.12 20:28:21 Fehler: 2012-11-17 20:28:21 moveCopyModule ERROR Exception: [Exception... "'Component is not available' when calling method: [nsIActivityManager::removeActivity]" nsresult: "0x80040111 (NS_ERROR_NOT_AVAILABLE)" location: "JS frame :: resource:///modules/activity/moveCopy.js :: <TOP_LEVEL> :: line 133" data: no] Quelldatei: resource:///modules/gloda/log4moz.js Zeile: 687

Quitting and restarting Thunderbird helps, it seems.

Does this patch apply clean to 16.0.2? Its disappointing that such a critical fix hasn't been pushed to stable release yet...:(

It does seem that a beta from Thunderbird 17 (from ubuntu mozillateam) fixes the issue for me. The fix does not seem to be in Thunderbird-trunk yet, I hope it gets there... I re-downloaded all my email from zarafa-IMAP with thunderbird 17 beta from mozillateam and after many hours of waiting, and downloading some messages twice, it now seem stable and calm. Thanks! I can now use thunderbird again. It was depressing not being able to use it for a couple of weeks.

It seems this issue is indeed not fixed yet. This morning, I had a folder downloading over and over again, with wrong message count (19 messages were in and about 7 were to be downloaded according to the activity window). I have, previously, for several times compacted, downloaded and repaired that folder, and also have deleted all sbd and msf files belonging to it, to no avail. Had to repair the folder again today to make the download stop.

(In reply to Sunil from comment #86) > Does this patch apply clean to 16.0.2? Its disappointing that such a > critical fix hasn't been pushed to stable release yet...:( Thunderbird 17 is being released tomorrow with this fix.

(In reply to Florian Effenberger from comment #88) > It seems this issue is indeed not fixed yet. If you are still seeing issues, please file a *new* bug. It is obvious from comments here that we've fixed at least some of the issues, hence we won't be re-opening this bug. Further comments on this bug may get lost as it isn't open. If you are filing a bug, it would also be useful to have prepared a log of what Thunderbird is doing: > It would also help us a lot if you could run from the command line with IMAP and related > logging turned on; set the environment variable > > NSPR_LOG_MODULES=imap:5,ImapAutoSync:5,msgdb:5,timestamp,sync See https://wiki.mozilla.org/MailNews:Logging for more info.

Why was the change that introduced this issue not rolled back? Why not make 16.0.3 release? How do we know we haven't gotten plethora of new bugs with TB17? Are you folks trying to kill off Thunderbird as an email client? This is a widespread breakage and its killing servers with redundant traffic.

I'm sorry to say, but for me it got *MUCH* worse with 17.0 now. Several times, messages are redownloaded, and when that's done, the activity window shows nothing, but Thunderbird still consumes 100% CPU power. I deleted my whole ImapMail folder, to no avail. I'll try to get some debug information soon and open a new bug, because it's really unusable for me now. :(

Please leave the bug number for new bug here. TB has not been usable for me for more than a week now, and I have been using alternatives. Someone at TB development has messed up change management really badly!

Im running the latest stable version, and it is downloading the same 4746 emails in the "All Mail" folder.... constantly. Seems to start over again almost as soon as it finishes. Im surprised gmail hasn't shut me down yet. This is the same thing it did on my prev laptop, moving to a new one hasn't helped at all. This needs to be resolved now.

Just noticed it is set to resolved/fixed. It most definitely is not.

(In reply to boomhauer from comment #96) > Just noticed it is set to resolved/fixed. It most definitely is not. The specific issue described in this bug is indeed fixed in TB17. What you are seeing, as a gmail user, is something different described in bug 816327.

Florian, Sunil, Can you please post the imap protocol log. That would be really helpful for us to analyse what is going on. Thanks :-)

Sorry for this delay - finally I was able to produce a log when TB got sluggish. I tested it with TB 20 as well, same problem. Can I send the log to someone in private, because I assume it may contain sensitive data?

Yes, you can send it to me - is it OK if I also share it with Atul, if he has more time to work on it? Just click on my name in the Bugzilla comment and you'll get the right address.

Yes, I'll surely look upon the protocol log.

I'm still having this problem, but I am stuck on Thunderbird 17.04 on Ubuntu Linux. Even though there are newer versions available for Windows. I previously had set the Default Character Encoding to UTF-8 the Character Encoding had been set to Western (IS0-8859-1). If this is being reset by an upgrade, it is possible that other things are being changed that affect this issue.

(In reply to Thomas Sisson from comment #102) > I'm still having this problem... Please see comment 97. If your problem sounds like the bug linked there, you can help by posting more detailed information on that bug, such as the exact wording of messages that your are seeing or screenshots. If you are experiencing something different, please search for another open bug or file a new one. Thanks and good luck.

Thomas, Can you confirm if your problem was solved? Thanks :)

Not fixed! Using 17.0.5 on Win 7 and Ubuntu 12.04 and copied profile from Win to Ubuntu. Ubuntu "synchronizing)reloading and reindexing all my IMAP folders repeatedly. I let updates run to completion overnight, then first operation on inbox started another reload and reindex of all IMAP folders. Had same problem a year or more back when bringing up my profile on my notebook and other Win machines. FWIW found that my User Name had to be the the same or thunderbird exhibited the same symptoms. I currently have multiple Win 7 machnes running this profile with no problems. I have a Ubuntu machine log file capturing thunderbird actions starting with a fresh copy of my profile from the Win 7 machine.

This is still happening in TB 24.0.1 I have a user who's sent folder downloads so much it fills up her hard drive. I turned off the 'save for offline' checkbox, but basically I need to rebuild this sent folder. The folder is 8GB on the server and about 24GB on her local machine. Is there anyway to just rebuild one folder, rather than rebuilding her whole mail profile? She uses many add-ons and would be a big job to rebuild the entire profile. Any ideas?

> Is there anyway to just rebuild one folder, rather than rebuilding her whole > mail profile? She uses many add-ons and would be a big job to rebuild the > entire profile. That is easy. Just close down the program completely, open the profile in the explorer and drill down to the folder where the mail is stored (Profile/ImapMail/ servername) and delete the Sent and Sent.msf files. When you start the program again, it will download the headers of the Sent folder into Sent.msf again and will not create Sent (when you have switched off the offline copy and there is not some wiseguy programmer who thinks it is a good idea to save everything for offline just in case).