This is on the latest thunderbird on linux a few days ago (Saturday, Feb 11). I was away from this particular computer and tried to download about a couple of weeks of e-mail messages during my absence. During the download, a folder seems to have reached the upper limit. OK, the warning mechanism to refuse to let the file size grow to the real hard limit kicked in. 1. Problem. I can't break out of the warning and download/filtering. BUT, when the warning dialog appears to warn me to shrink the folder by dividing the folder or by compaction appeared, I had no way to stop the download (or filtering that moves the article to this folder.) Every time I said OK to the dialog, TB re-tried to move the article to the almost full folder and so the dialog appeared again. In the end, I gave up and exit TB by cliking the program exit button in the upper right corner. (This abrupt exit might be the cause of the strange behavior below.) I am not sure what happened to the particular articles that caused the warning. Since I could not stop TB when the warning appeared, I can only hope TB didn't remove it from POP account and retrieved it again. (But see below. the article may be already in Inbox, which is rather small, before it was filtered and moved into the almost full folder.) 2. Now, after the exit, and re-invokes TB, I make it off-line, and tried to compact the said folder (close to 4GB limit. And this is on linux. I see a mention of limit 2GB for linux, is it true? Maybe the code was buggy and let the folder to grow over 2GB limit somehow.) the compaction never happens on this folder. On a hunch, I removed the .msf file to see if it helps. No, it was a bad mistake. Now TB enters the following loop. It says re-creating index file for the said folder, and the progress bar is shown, but when it finishes (the bar reaches 100%), it again says creating the index and starts over. This is repeated. In the end, I had to split the said mail folder file into smaller chunks and all is well now, but this should be improved. Cure 1: Offer a way to move out of the operation download( or subsequent moving of articles caused by filtering?) when a folder becomes full (or almost full). Cure 2: Make sure an almost full folder can be compacted. I think the download should have stopped when it encountered this situation. (But It may be that the downloading has stopped and the filtering of the incoming e-mail messages are run, and during the moving caused by filtering AFTER the whole download ended may be going on. I am not sure. We have to figure out in what state TB should leave the updated Inbox, and the affected folder. It may be that there are unfiltered mails left behind if we stop short. But again, I observe that the articles left in Inbox seemed to be filtered in the next invocation of TB. So there may be a mechanism to run filter on non-filtered messages [due to abrupt exit of TB] ???) Note: when I say "full" or "almost full" above, the mail folder file is close to allowed limit. (I believe we have worked on the size limit the year before, and at least the folder can't grow more than the size limit - a predefined size). So if linux's size limit is 2GB, then there is a bug. My folder somehow went over that limit and went over 43000...000 bytes in decimal notation. So maybe the code to look at the size limit now uses 4GB as the upper limit? Oh wait, maybe it is the compaction code that looks at the size and gave up silently since purportedly 2GB is the size limit and it doesn't want to touch anything larger (but then it should warn in a visible manner.) Sorry, I tried to compile TB3 for the last 6 months on my PC and could not due to the major bumping of COM infrastructure, strange Debian-specific bug in the tool chain, and mysterious failure of git/csv fetch which affects one of often used PC. I could fetch the latest on another PC last week, but it didn't compile. So I am asking for help to look into this matter.

After fix of Bug 608449(fixed by Tb 3.3, already imported to Tb 3.1.7), "Compact Folder" can compact mail folder file whose file size exceeds file size limit(4GB or 2GB), as far as fil size after compact is within the limit(i.e. deleted but not expunged/purged mails exist). ISHIKAWA san, are you talking about "file size is still greater than limit even after compact" case? > My folder somehow went over that limit and went over 43000...000 bytes in decimal notation. > So maybe the code to look at the size limit now uses 4GB as the upper limit? Bug 598104 existed on Sent folder, and mail data is being appended even after file size exeeded file size limit. But that bug says that it was fixed by Tb 3.3 and Tb 3.1.7. Does problem still remain on Linux?

> On a hunch, I removed the .msf file to see if it helps. > No, it was a bad mistake. Now TB enters the following loop. > It says re-creating index file for the said folder, and the progress bar > is shown, but when it finishes (the bar reaches 100%), it again > says creating the index and starts over. This is repeated. It's not surprizing. Compact opens folder, and file open process detects deleted .msf and starts Rebuild Index internally, and Rebuild Index finds something wrong and fails, then go to step 1, thus infinite loop happens. However, I thought it's merely wrap of offset on Win, because offset is held in unsigned 32 bits integer if Win, and such ifinite loop is avoided. If Rebuil Index is successful, mails of correct offset can be deleted(EXPUNGE bit can be written), then Compact after fix of fix of Bug 608449 can compact the folder. On Linux, IIRC, offset is still saved in signed 32bits integer. Negative offset may be generated and it may produce Rebuild Index failure which causes infinite loop of Rebuild Index. Another concern, "wrap around twice". Is the "over 43000...000 bytes" greater than 2*2GB(=4GB) in your case? > Cure 1: Offer a way to move out of the operation download( or subsequent moving > of articles caused by filtering?) when a folder becomes full (or almost full). "Work Offline" doesn't work? How about pull off of LAN cable?

Thank you for the comment. (In reply to comment #1) > After fix of Bug 608449(fixed by Tb 3.3, already imported to Tb 3.1.7), > "Compact Folder" can compact mail folder file whose file size exceeds file size > limit(4GB or 2GB), as far as fil size after compact is within the limit(i.e. > deleted but not expunged/purged mails exist). > > ISHIKAWA san, are you talking about "file size is still greater than limit even > after compact" case? I am away from the particular computer on a business trip for a week. So I may not be able to figure this out until next week. (I can verify what is like the resulting file by trying to run a compaction to the newly created divided files from the original large folder, and then sum up the compacted file sizes.) > > > My folder somehow went over that limit and went over 43000...000 bytes in decimal notation. > > So maybe the code to look at the size limit now uses 4GB as the upper limit? > > Bug 598104 existed on Sent folder, and mail data is being appended even after > file size exeeded file size limit. But that bug says that it was fixed by Tb > 3.3 and Tb 3.1.7. > Does problem still remain on Linux? I read 598104 and was surprised to find that there is a different mechanism for copying e-mail articles for Sent folder. I wonder if there can be another different method while article is moved by filtering invocation. If so, that is another path that may need fixing. In the patch for 598104, I noticed that the refreshing of file size in the original lines 2192-2193 (nsMsgLocalMailFolder) are removed. I wonder if this may be the cause of the bug? One other thing, in the if(!CheckIfSpaceForCopy(...)) we return NS_OK for now. Since if CheckIfSpaceForCopy fails due to the size over error, it throws an exception, it seems, so the return value NS_OK probably is irrelevant, but I feel an error value might be more readable IMHO. I have to do some serious debugging after creating a test environment using the original big folder to figure out if the problem persists under linux.

(In reply to comment #2) > It's not surprizing. Compact opens folder, and file open process detects > deleted .msf and starts Rebuild Index internally, and Rebuild Index finds > something wrong and fails, then go to step 1, thus infinite loop happens. OK, I figure this was the case. But what error TB3 detected which caused it to loop forever? > However, I thought it's merely wrap of offset on Win, because offset is held in > unsigned 32 bits integer if Win, and such ifinite loop is avoided. If Rebuil > Index is successful, mails of correct offset can be deleted(EXPUNGE bit can be > written), then Compact after fix of fix of Bug 608449 can compact the folder. > On Linux, IIRC, offset is still saved in signed 32bits integer. Negative offset > may be generated and it may produce Rebuild Index failure which causes infinite > loop of Rebuild Index. I see. This signed 32bits integer under linux could be the culprit. Hmm... I will target my debug effort on this point if I can compile TB3 under my Debian GNU/Linux environment hopefully soon. > Another concern, "wrap around twice". Is the "over 43000...000 bytes" greater > than 2*2GB(=4GB) in your case? I ran bc (big number calculator) and found this: bc 1.06 Copyright 1991-1994, 1997, 1998, 2000 Free Software Foundation, Inc. This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. 2^32 4294967296 So obviously, 43000...000 (actually there were some non-zero numbers where I typed periods) is larger than 2^32 (!). > > Cure 1: Offer a way to move out of the operation download( or subsequent moving > > of articles caused by filtering?) when a folder becomes full (or almost full). > > "Work Offline" doesn't work? How about pull off of LAN cable? Actually, there was no timing window where I could hit "Work Offline" button. This is because as soon as I hit "OK" to the warning dialog, and before I could do anything reasonable, (I think because of the moving an article to the said folder due to filtering) TB3 again showed the modal warning dialog instantly (I tried moving mouse very quickly, but I can't win against CPU) and so I could not invoke the file -> offline menu at all. Pulling off of LAN cable: It it is a standalone computer in an open space, it would have been easy. Unfortunately, the said PC was installed in a very difficult to reach position, and its wiring is was very hard to reach. The only easy to access button was POWER switch(!). I didn't want to hit power switch because there were other documents open at the time this problem occurred. I think pulling off LAN cable may have worked if and only if the problem was caused by saving the downloaded file into a said folder. But I am not sure because it seems to me that the download of articles seems to have already occurred to a (tentative or inbox?) folder and TB3 was running a filter and tries to move the article to another folder (it is not Inbox, or Trash). If TB3 was showing this error dialog during moving and after the downloading is done on a particular article that caused the problem, then pulling off the wire may not have any effect at all. TIA PS: I should be able to investigate once I come back from the business trip.

FYI. As I wrote in bug 608449 comment #19, rebuild-index is normally executed on Win even if mails of offset="over 4GB" exist. Mails of offset="over 4GB" looks simply ignored. If rebuild-inex fails on Linux when size>2GB, it may be caused by negative offset due to 32bits signed integer.

(In reply to comment #4) > But I am not sure because it seems to me that the download of articles > seems to have already occurred to a (tentative or inbox?) folder and TB3 was > running a filter and > tries to move the article to another folder (it is not Inbox, or Trash). > If TB3 was showing this error dialog during moving and after the > downloading is done on a particular article that caused the problem, > then pulling off the wire may not have any effect at all. If error upon "move/copy to target-folder by filter" and the target-folder>2GB, "move/copy by filter" fails by "Cancel" reply to error dialog, and "move/copy" is skipped even though "move/copy" is logged in filter log(It looks successful "move/copy" by filter in filter log. Mail remains in Inbox. It's already known issue). It could be observed by "Run filter on folder" with copy-target-folder of size>4GB on Win. I guess infinite retry-loop may happen in download step when Inbox>2GB(4GB on Win).

FYI. "folder can't compacted..." part in bug summary is already fixed by bug 608449 (fixed by Tb 3.1.7).

I checked mail download with next Inbox folder, using Tb 3.1.9 on Win. > Size of each mail is 13MB. 316 mails in Inbox. > +-- Offset=0 +-- 4GB boundary > V V > <---------------------- ... --------> <----------------- > <--------------------> ..... <--------------------> > mail-1(13MB) mail-316(13MB) > <-------> <----------> > 1MB 12MB(this 12MB data over 4GB is lost by rebuild-index) (1) Rebuild-Index normally ends. mail-1 to mail-316 is shown at thread pane. Mail size of mail-316 shown at thread pane after rebuild-index is 1MB, and view/source shows the 1MB only. (See bug 608449 for detail) (2) Restart Tb, clear password, Get Mail of POP3 account. => following alert was shown, and no password prompt was shown. i.e. Download was not initiated. > Alert > ! The folder Inbox is full, and can't hold any more messages. To make room for > more messages, delete any old or unwanted mail and compact the folder. > [ OK ] I guess mail download is terminated due to above failure on Inbox, when above failure occurs during download of mails to Inbox. And, I guess phenomenon you saw was many "failures of move/copy to target-folder by message filter due to file size limit on target-folder" for many mails which are already downloaded to Inbox normally.

My guess on "Inbox file size exceeds limit during download" was wrong. As I wrote in bug 640371, Tb silently continued to append all downloaded mail data to file of Inbox even after file size exceeded 4GB on Win, and Tb produced "wrap around of offset". Because of garbage by "wrap around of offset", "file size after compact" may still exceed size limit. In this case, "delete of other mails" is required to execute compact normally.

Obviously, I forgot to follow this up since I could not compile TB on my Debian GNU/Linux PC well later in 2011. I am sorry for the lapse. But I have hit a similar bug (but slight different scenario) and TB stopped downloading this time, and so will look into this problem again with a fresh eye. (Yes, I can now compile TB on my PC. The new experience is going to be posted in a comment to Bug 789679. [I will use that entry because this new experience seems more related to 4GB size limit. In the process, I hope to cover the issue here also.] TIA

The following is posted to Bug 640371 where problems of the folder size over 4GB are discussed right now. > > (But I think we need to make sure the following case is handled too: > > Downloading to Inbox, then filtering tries to move the article to a > > different, almost full folder. In this case, POP3 downloading works, > > but the problem was that I could not get to stop downloading since the > > modal dialog about the full folder continued popping up and did not > > give me the chance to stop downloading (!?) at all. If pop3 download > > continues with hundreds of e-mails which are then being fed to the full > > folder due to filtering > > then we may still have a problem. But changing the status of download may be > > a way to go as mentioned somewhere. > > Gee, I can not find the bugzilla entry handy, but I reported this problem > > before.) > This should be bug 634544. But did you experience this problem with this > patch? [...] Well, the new behavior in 22.0a1 (2013-03-21) is a mixed blessing. I tested as follows. I copied almost full Inbox to a differently named folder "FilteredFolder", and created a mail filter to move e-mail article that has "FILTER to other folder" in its subject line. And, I injected e-mails with such subject every two e-mails and see how TB handles the situation. STEP-1. (I forgot to re-create the .msf file for FilteredFolder folder by simply renaming Inbox to this name.) Empty Inbox received the incoming e-mails, but received e-mails are not filtered to the intended destination. It seems that if the .msf file is not there, the filter doesn't work. I see something like the following in the console where thunderbird was started. I think the message is stored into Inbox, but then somehow when the filter was run, destMailDB (i.e., .msf file?) was not accessible and thus the filtering SILENTLY failed! Incorporate message begin: <--- this is not filtered. uidl string: n'$#!0HO!!8+#"!QbI"! Incorporate message complete. Incorporate message begin: <--- every 2nd message is filtered. uidl string: W~\!!Y~d!!d9'#!I~k!! GetDiskSpaceAvailable returned: 8276869120 bytes WARNING: failed to open mail db parsing folder: 'destMailDB && NS_SUCCEEDED(rv)', file /COMM-CENTRAL/comm-central/mailnews/local/src/nsParseMailbox.cpp, line 2540 Incorporate message complete. <--- It seems filtering was ignored. Incorporate message begin: But there was no visible feedback. uidl string: m2F!!$#m!!ZYQ"!YB#"! Incorporate message complete. Incorporate message begin: uidl string: A*\!!/j~!!QQj"!2c;!! GetDiskSpaceAvailable returned: 8275832832 bytes WARNING: failed to open mail db parsing folder: 'destMailDB && NS_SUCCEEDED(rv)', file /COMM-CENTRAL/comm-central/mailnews/local/src/nsParseMailbox.cpp, line 2540 Incorporate message complete. Incorporate message begin: uidl string: UAm!!mTk!!Xlp!!o(D"! Incorporate message complete. I am afraid that this is a buggy behavior. We should be warned (ONCE at least). STEP-2: I now re-created .msf file and deleted enough e-mails to carry out more testing. OK, what happened is this. Now filtering works up to the point where "FilteredFolder" reaches the size limit. After that, again, filtering SILENTLY FAILS and the message is left in the Inbox WITHOUT ANY VISIBLE FEEDBACK to the user. (and for that matter no discernible console message either on the original terminal console.) I am sure this will confuse the users to no end eventually. I don't want the error dialog to pop up for EVERY download, but at least this should be warned JUST ONCE during the whole POP3 download session to tell the user - the target folder of a filtering is full and e-mail can not be moved/copied. So making space in the folder by moving e-mails to somewhere else or delete) and compact it, AND THEN * - RE-RUN FILTER on Inbox so that the intended filtering takes place then! The second point marked (*) is important. And showing this JUST ONCE, and not for every single failure of a filter to this destination, is enough. Otherwise the user would get panicky like I did. (Hmm, maybe the POP3 download session needs to keep track of a destination folder where move/copy to it fails.) But I will move this discussion to Bug 634544 TIA

I think the filtering problems should go into other bugs (1)when .msf is missing and 2) when the mbox is full). Looks like bug 261419 is the one for missing .msf.

"Filter silently fails when copy/move error occured" is known phenomenon. Because filter log of moved/coped is written before actual moving/copying, and because filter doesn't write log of move/ failure/copy failure, phenomenon of "filter log says moved, but no mail is found in movee target folder" is observed. Bug for this is already opened. AFAIK, it's applicable to Junk filter move, so bug for phenomnon of both "mail was not moved by message filter" and "mail was not moved by Junk filter" were opened sometimes, although it looks rare than before in these days. Bug 497804 is for "Filter and Compact interferes each other" case. In test for such problem, I usually use "interfere by write open of mail folder file from other software", but "Folder full" sounds a very good test case to reproduce the problem consistently without artificial error and to force "very long time to rebuild index".

FYI. Bug 495760 is for "rebuild index is not invoked by filter copy/move when outdated-msf conditon".

Did we not fix this with bug 640371 and bug 794303 ?

(In reply to :aceman from comment #15) > Did we not fix this with bug 640371 and bug 794303 ? I will check by creating a large existing folder and then try to cross the 4gB boundary by injecting, say, a few hundred e-mail messages.

Depends on what kind of injecting you mean.

(In reply to :aceman from comment #17) > Depends on what kind of injecting you mean. Interim report. I injected e-mails by POP3 download. This is lengthy, but I leave this for posterity. Well, I thought 4GB limit is gone, but... Initial Failure : I need a large folder to create the situation reported in the bugzilla. To reduce the time to reach the overflowing condition, it is wise to start with a large enough folder. I thought I was going to create a large folder by receiving many e-mails. But it turns out to be very time-consuming. So, 4.3GB Inbox file, a remnant from an earlier testing, was copied into mail directory (from external source) while TB was not running. It is already ABOVE the limit. TB was started. Since Inbox is updated externally, and Inbox.msf was out of date, I rebuilt .msf file (it seems to have succeeded). [Somehow TB did not offer to rebuild it itself automatically.] (Actually, I needed a large enough destination folder ("FilteredFolder" in my test setup), but I started with large enough Inbox by mistake. This uncovered an issue as below.) Then I sent a few big files (512KB) to this TB account, and TB seemed to have received them. (So no 4GB limit for receiving POP3 e-mails to Inbox.) I moved a several dozen messages (or maybe more) from Inbox to a different folder using filtering (hoping that Inbox went below 4GB limit by this.) I wanted to check the cross-size-boundary behavior. Then after the moving of e-mail messages, TB offered to compact the Inbox folder [not sure what is the criteria for offering such compaction.]. I hit [OK] (to proceed with compaction.). PROBLEM: However, due to the file system overflow (which was not quite notified by proper visible feedback at the first moment when it happened [that the error message/dialog was not shown (see *1 below) according to my WARNING message printout since the ThrowAlertMsg routine was passed NULL msgWindow parameter. I am running the DEBUG BUILD version of TB]), the compaction failed, and the failure of compaction *WAS* reported properly eventually. === begin footnote (*1) Messages from some additions of my own: ###!!! ASSERTION: show stack trace of ThrowAlertMsg once when msgWindow is not null.: 'false', file /COMM-CENTRAL/comm-central/mailnews/base/util/nsMsgDBFolder.cpp, line 5208 ThrowAlertMsg: msgName=<<compactFolderWriteFailed>> string retured from bundle = <<T>>: msgWindow = 0xac9e4f8 ++DOCSHELL 0xd1473c0 == 12 [id = 12] ++DOMWINDOW == 24 (0xed5219c) [serial = 31] [outer = (nil)] ++DOMWINDOW == 25 (0xd02f564) [serial = 32] [outer = 0xed5219c] ThrowAlertMsg: msgName=<<mailboxTooLarge>> string retured from bundle = <<T>>: msgWindow = (nil) ###!!! ASSERTION: msgWindow should NOT be null: 'msgWindow', file /COMM-CENTRAL/comm-central/mailnews/base/util/nsMsgDBFolder.cpp, line 5219 throwAlertMsg failed to show Alert() (I have no idea why stacktrace is not printed... Maybe because I am invoking TB under a different user [mtest2] for testing, and maybe the dynamic libraries could not be located during run-time? I invoked thunderbird-bin instead of thunderbird, and come to think of it I have not set LD_LIBRARY_PATH for this account. Mea Culpa I should have set: LD_LIBRARY_PATH=/usr/local/X11/lib:${MOZ_OBJDIR}/mozilla/dist/bin:$LD_LIBRARY_PATH So many tiny details to take care of :-( === end footnote Afterward, whenever I hit [Get Message], I get the following message: The folder Inbox is full, and can't hold any more messages. To make room for more messages, delete any old or unwanted mail and compact the folder. It is funny. I thought the limit was gone! First of all, I wonder where the folder size limit is still there. (Or have I somehow forgotten to remove the artificial testing limit that was embedded for testing purposes [or for stability purposes for now]?) We have inserted such artificial limit for testing, :aceman, haven't we? Maybe I forgot to remove it from my test source.) Or maybe the code for compacting the folder may have left the Inbox in a strange state (?) [Like the index file, .msf, for Inbox in an inconsistent state?] (Can it be there are places where we may have overlooked improper message key generation(s) in a few unrelated places?) Grrr, silly me. I removed the large ( 4GB ) test Inbox file due to issues caused space limit. I should have saved it in a different place, but there was none for the moment to hold this big file, and have a large destination folder, too. :-( ANOTHER TEST IN PROGRESS. REPEATING the test again after removing Inbox. I am creating a new large destination folder manually since sending e-mails is too slow [I have to throttle e-mail delivery so as not to overflow local mail and pop3 spools.] COPYing some articles manually to Inbox, and then MOVing some articles by filtering to the destination folder. Repeat this several times. Now I have a 3+ GB destination folder. (During that time, I got a folder lock failure "filterFolderDeniedLocked": obviously during filtering move when the POP3 message download kicked in. I wonder if filtering move succeeded. Some messages that should have been moved from Inbox to the same destination folder (FilteredFolder in my case) failed to be moved (!). (Likely the articles downloaded from POP3 server during the operation.) Doesn't TB re-try filtering move/copy when it failed due to locking? I am afraid that this behavior has been with TB for a long time.] Now I am sending e-mails to this account to see what will happen when destination folder crosses the size limit when filtering of messages downloaded from POP3 server occurs. Stay tuned... [OK, I left the test running and FilteredFolder is now 3994MB and I should be able to report the behavior tomorrow morning.] BTW, there are a few things I noticed during testing. These can be RFE (Request for Enhancement) entries in bugzilla. These are lengthy... - Canceling of large move/copy of messages. There is no way to cancel a copy of message(s). That is if I am copying 4989 messages (after selection), and realize, "Oh, I will make the destination too large", there is no way to cancel it. Maybe we should by displaying a dialog "Copying is in progress [Cancel]" and when it finishes "Copying is finished [OK]". I know there is an underlying DB issue. - No real-time update for read, unread, size column: One other thing, while the copying is in progress, there is an update, e.g "Copy of 3922 of 4989 messages in Inbox" in the lower-left corner of the TB window pane. But the column entries for Unread Total Size are not updated in real-time. Maybe it should. (Even if it is updated every 10 copying operations, or each second or two, that would be better and suffice for human feedback.) This update of "read, unread, size column" is especially desirable during filter move/copy (Invoked by Run filters on folder] since there is NO INDICATION of move/copy by filtering in the lower-left corner of window pane at all, and so there is absolutely no visual feedback. [Only the sluggish mouse response, etc. tells the user that something is going on.] - Failure to show Error messages during filtering: It is understandable to limit the amount of messages during filtering, etc. I came to know the desire to explicitly suppress error (some?) messages during filtering from discussions that took place in other bugzilla entries. But some fatal errors (not related to locks), e.g. space overflow, etc., should be shown at least ONCE during large copying/moving. (Definition of ONCE may be difficult if filtering is invoked by POP3 download.) [Definition of "Large copying/moving": Selection of "large area" for DB locking would indicate this operation. Say the number of articles, or the total size of messages.] - Failure to show Error messages during compaction. Actually, the particular error of concern occurred when compaction took place during the testing. Namely that of running out of space in the file system. I have no idea why it was not reported at all DURING compaction. (Yes, eventually, the failure to compact the folder was reported at the end, but I think it would be more informative to tell users WHY it failed exactly. The suppressed error message (which was passed to ThrowAlertMsg, but not shown since MsgWindow was null) would indicate the exact cause at that point.) == Comment on 4GB (or for that matter NNN bytes) limit Testing 4GB limit is difficult and time-consuming. - time consuming if we create the Inbox file/folder from scratch by sending many e-mails (this can be automated), and receiving them in test account. [I have come to know that a certain test harness mechanism uses sparse files to create a large file, but this would not catch the subtle message key issues aceman uncovered early this year.] - Incremental creation using e-mail is good in this sense. However, sending this many e-mails may cause the system mail spool area, or pop3 server spool area to overflow. This happened easily on my linux setup :-( [Unless you set up your linux installation for a large mail server, you end up with not so large /var partition if you follow a typical installer's advice. I set up mine as a general development environment and /var is not that large ;-( .] So I have to throttle the sending side reasonably, and make sure that TB checks the incoming e-mail every minute (which is the smallest time interval I can set using the account option.) - Keeping around a few large files (~ 4GB) to test receiving e-mails from POP3 account, and then moving/copying to these folder using filtering is VERY space consuming, and I ran out of space for a partition during testing very easily :-( (I recall WADA uses a note PC with only 10GB partition. It is impossible to test this type of issues comfortably in the environment.) Despite these difficulties, we really should include tests to deal with issues regarding 4GB boundary. Why? When size-related issues cause our e-mail messages to be at risk, the user is well advised to proceed carefully. But often times, due to the file system space issues, and the thought of losing articles (or thinking mistakenly that they have lost articles already) may cause users to become panicky and do things that may cause more damage to the folders. So we really need to make sure bugs regarding size-limit should be eradicated, especially the ones that cause data loss. At least, we need to uncover more subtle issues (even if the fix needs waiting.) We need more testers with time and file space (both under windows and linux, I think.) TIA

Chiaki, the 4GB limit is still in place (until the msgkey problems are solved). I actually add checks so that the limit (any limit we need) is properly enforced. The specific case of TB downloading mails and crossing the 4GB limit should have been fixed in bug 640371. If you can find steps how TB still ignores the check and crosses the limit AT POP3 DOWNLOAD then please tell us. There are tests for testing the limit in the file test_over4GBMailboxes.js . But that test fails on Windows right now. If you could help me on that platform, there is bug 872357 for it.

(In reply to :aceman from comment #19) > Chiaki, the 4GB limit is still in place (until the msgkey problems are > solved). I actually add checks so that the limit (any limit we need) is > properly enforced. The specific case of TB downloading mails and crossing > the 4GB limit should have been fixed in bug 640371. If you can find steps > how TB still ignores the check and crosses the limit AT POP3 DOWNLOAD then > please tell us. Thank you, aceman. The 4GB check seems to work. I think the particular behavior I saw and reported in the previous post was that externally copied Inbox folder(file) was ALREADY over 4GB (created by an earlier test when 4GB limit was lifted for testing purposes). TB did not have the chance to check the crossing of 4GB boundary. My test ran and right now the folder files have these sizes. -rw------- 1 mtest2 mtest2 87381 Jun 17 04:17 Drafts -rw-r--r-- 1 mtest2 mtest2 3573 Jun 27 10:25 Drafts.msf -rw------- 1 mtest2 mtest2 4290475366 Jun 28 12:15 FilteredFolder -rw-r--r-- 1 mtest2 mtest2 4780299 Jun 28 12:22 FilteredFolder.msf -rw------- 1 mtest2 mtest2 423613771 Jun 28 12:22 Inbox -rw-r--r-- 1 mtest2 mtest2 657887 Jun 28 12:22 Inbox.msf -rw------- 1 mtest2 mtest2 4722922 Mar 18 22:10 Sent -rw-r--r-- 1 mtest2 mtest2 4694 Apr 12 04:55 Sent.msf -rw------- 1 mtest2 mtest2 58040093 Jun 27 23:23 Trash -rw-r--r-- 1 mtest2 mtest2 69282 Jun 27 23:28 Trash.msf -rw------- 1 mtest2 mtest2 4789250 Jun 28 12:15 filterlog.html -rw-r-xr-- 1 mtest2 mtest2 236 Jun 27 10:01 msgFilterRules.dat -rw-rwx--- 1 mtest2 ishikawa 61 Jun 28 12:22 popstate.dat -rw------- 1 mtest2 mtest2 2072061 Mar 24 00:50 xxxyyyzz -rw-r--r-- 1 mtest2 mtest2 5487 Jun 17 04:16 xxxyyyzz.msf Let me see. t = 2^32 = 4294967296 f=4290475366 (size of FilteredFolder) <- this receives large e-mails by filtering. t-f = 4491930 So folder size is not over 4GB and the checking of 4GB (minus fixed slack) kicks in. Anytime a new e-mail (about 507KB) is received to Inbox and filtering happens, the following messages are shown in sequence. So the check works (albeit the first message being a little cryptic.) 1st Dialog message (immediately after download): The messages could not be filtered to folder 'FilteredFolder' because another operation is in progress. [OK] 2nd Dialog message: (immediately after hitting [OK] in the dialog above.) Moving/copy to folder 'FilteredFolder' failed: it is possible that the DB file is corrupted or missing. Please check that you have enough space and you have write privileges to the file system, and then repair the database file context menu [properties], [fix folder]. And re-invoke filtering manually to let the filtering operation to take effect. [OK] (It may be that I have modified the local copy of message bundle for the second message, especially "re-invoke filtering" stuff at the end.) So :aceman, the particular issue this bugzilla entry originally addressed is indeed solved(!) Oh wait, a tiny doubt. Well, come to think of it, I have no idea what happens when Inbox approaches 4GB limit. As of now, the incoming message is added to Inbox (400+ MB size now), but not filtered to "FilteredFolder". If we continue doing this, Inbox will fill up eventually. I am not sure what happens then. But I suspect that we have already tested this (POP3 download to Inbox when 4GB is reached) earlier this year. Didn't we? > > There are tests for testing the limit in the file test_over4GBMailboxes.js . > But that test fails on Windows right now. If you could help me on that > platform, there is bug 872357 for it. I will check 872357. Thank you again!

(In reply to ISHIKAWA, Chiaki from comment #20) > Oh wait, a tiny doubt. Well, come to think of it, I have no idea what > happens > when Inbox approaches 4GB limit. > As of now, the incoming message is added to Inbox (400+ MB size now), but > not filtered to "FilteredFolder". > If we continue doing this, Inbox will fill up eventually. I am not sure what > happens then. But I suspect that we have already tested this (POP3 download > to Inbox when 4GB is reached) earlier this year. > Didn't we? Sorry, I did not understand this. All incoming messages are first downloaded into Inbox (and checked against the limit). Only then they are copie/moved by a filter to the destination folder. Even if moved, the real msg data still stays in the Inbox file, only marked as deleted. So whether filtering works or not I think it makes no difference here. And I think you were looking at the filter problems in another bug. So what do you mean with Inbox filling up?

(In reply to :aceman from comment #21) > (In reply to ISHIKAWA, Chiaki from comment #20) > > Oh wait, a tiny doubt. Well, come to think of it, I have no idea what > > happens > > when Inbox approaches 4GB limit. > > As of now, the incoming message is added to Inbox (400+ MB size now), but > > not filtered to "FilteredFolder". > > If we continue doing this, Inbox will fill up eventually. I am not sure what > > happens then. But I suspect that we have already tested this (POP3 download > > to Inbox when 4GB is reached) earlier this year. > > Didn't we? > > Sorry, I did not understand this. All incoming messages are first downloaded > into Inbox (and checked against the limit). Only then they are copie/moved > by a filter to the destination folder. Even if moved, the real msg data > still stays in the Inbox file, only marked as deleted. So whether filtering > works or not I think it makes no difference here. And I think you were > looking at the filter problems in another bug. So what do you mean with > Inbox filling up? I was not clear enough. The test I performed in the last couple of days checked that the 4GB limit is properly observed for the data path (shown below marked with *) by sending a few 500KB e-mail messages at a time (and let TB to check the messages every minute), and repeat the sending many times. : mail server -- (POP3) --> TB's Inbox ---(*: move by filter*)--> desintation folder The 4GB check during move/copy by filtering worked. Now, during this particular testing, TB's Inbox only increased to about 400+MB because I started out with a destination folder with the initial size of 3.6 GB (or so. The numbers are approximate.) I was wondering whether the POP3 download path properly checks for the 4GB limit of TB's Inbox before inserting the new message into Inbox: mail server -- (*　POP3 *)-> TB's Inbox I think we did check this before, and the code to check is in place and works (correct?) OTOH, my original problem was the usability issue: too many error dialogs appear in rapid succession due to the overflow of destination folder caused by many messages coming in via POP3 download and being moved to the overflowing destination folder (and the inability to stop download after the issue was noticed). [I am not sure whether the filer is invoked after each message is downloaded, or if the filter is invoked after one POP3 session to download many message finishes. My experience two years ago suggested that the filter is invoked after each message is downloaded, and thus I want to stop download in midway if possible.] In the original case, I had thousand of e-mails being downloaded, and the destination folder reached the size limit (of that time: two years ago) during the download. The error dialog showed up for each message after the failure to move by filtering. Each time I closed the dialog, immediately TB kept on showing the dialog for the next message's failure to be moved to the destination folder (the message shown was not quite correct, but it was shown due to the failure to move the message by filtering) and I had no way to stop downloading (since the dialog appeared almost instantly after I close the previous dialog. Thus I had no chance to hit [offline] or other manner of stopping download, etc.) Eventually, I had to kill TB. It is certainly a usability issue when a extreme condition is reached. I think I re-create the original situation, and repeat the test: - the destination folder in my setup is already close to the size limit. (Trying to move 500KB e-mail message fails.) - So, instread of sending a few 500KB e-mail messages at a time, I will see what happens if I send 500 1KB messages at a time, and repeat it a few times. This closely matches the original situation. Stay tuned. TIA

(In reply to ISHIKAWA, Chiaki from comment #22) > I was wondering whether the POP3 download path properly checks for > the 4GB limit of TB's Inbox before inserting the new message into Inbox: > > mail server -- (*　POP3 *)-> TB's Inbox > > I think we did check this before, and the code to check is in place and > works (correct?) Yes, this exactly is bug 640371. If the check is not enough, please say so in that bug.

(In reply to :aceman from comment #23) > (In reply to ISHIKAWA, Chiaki from comment #22) > > I was wondering whether the POP3 download path properly checks for > > the 4GB limit of TB's Inbox before inserting the new message into Inbox: > > > > mail server -- (*　POP3 *)-> TB's Inbox > > > > I think we did check this before, and the code to check is in place and > > works (correct?) > > Yes, this exactly is bug 640371. If the check is not enough, please say so > in that bug. still need this bug report?

No idea. This bug needs retesting with current trunk where bug 640371 is fixed.

(In reply to :aceman from comment #25) > No idea. This bug needs retesting with current trunk where bug 640371 is > fixed. Is trunk == C-C source tree or should I test a different tree? If trunk == C-C, I am fairly confident this bug is no longer there. TIA

Yes trunk is the current c-c tree from which TB Daily builds are done. The version you normally get via hg. In that case, thanks for confirmation.