User-Agent: Mozilla/5.0 (X11; U; Linux i686; ja; rv:1.9.0.10) Gecko/2009042315 Firefox/3.0.10 Build Identifier: version 2.0.0.21 (20090302) I am reporting this problem which was origianlly experienced by a colleague of mine using thunderbird on Windows XP 32bit platform, but I could re-produce this on linux i86 32bit platform under controlled environment. (There is now a bug called 387502, which is similar. But it was marked "unconfirmed" and never followed up. It happened under windows with an earlier version. Mine is on a different platform with a recent version and I am attaching a test file for this.) Fact: It seems that there is a 4GB size limit of folder. Unforatunately, TB doesn't seem to properly keep this limit under control and allows the operations that make folders grow over this limit. Eventually, the user experiences problems (not being able to read messages, losing messages.) My action below is by copying a message into a folder, but my colleague's case was a large Sent folder. He had no way of knowing that sending a message (with an attachment that is largish) will go over the 4GB limit of folder, in this case, Sent folder, and was quite perplexed and as a result lost a day's worth of sent messages. I am attaching a file to re-create the problem. unpack Seed-for-big-folder And run the following command with the other script file (your directory portion of the pathname shold be changed) gawk -f /extra/ishikawa/download/create-dummy-mails.awk Seed-for-big-folder A large file (close to but less than 4GB ) is created. (The script creates a file consisting of slightly changed copies of a single largish message that contains an attachment. Each message is about 3.x MB.) Then follow the steps below. Seed-for-big-folder.gz Reproducible: Always Steps to Reproduce: 1. Place the "dummy-file" in your Mail directory (where Sent, Inbox, et al are located.) Also place "Seed-for-big-folder". 2. Start thunderbird and see it rebuilds the summary for this new folder files. It can take an hour on a slow computer (1GB, 512MB PC built in circa 2002.) especially for "dummy-file". 3. Notice that "dummy-file" holder it has 1342 messages. 4. Repeat this process and notice that TB doesn't show any warning. (repeat three times. I did this three times.) 4-a: Pick a message from Seed-for-big-folder, and move or copy it to dummy-file folder. 4-b: You may wanto to check that dummy-file grows over 4GB during this oepration. 4-c: Check the number of messages indeed increased by one. Quitting and restarting TB. 5. Now after three such operations, I find that The message counter of dummy-file showed 1345. But upon closer inspection, the messages shown for the 1343th, 1344th, and 1345th, are all garbled. 1343th seems to be more or less OK. 1344th showed a portion (near the header) of the second message in the original folder file. (I think lseek() or friends is wrapping the search position module 4GB). Ditto for 1345th file. (This shows the partial header portion of the THIRD message in the original folder file. Again search position is wrapped module 4GB) Actual Results: As mentioned, the addition of messages to grow a folder over 4GB succeeded without any warnings. But as a result, the messages thus added are not quite accessible any more. (If such messages are created for sending and this copying occurs during send operation and Sent grows over 4GB, the user loses precious messages!) Expected Results: Either I should - get the warning that the said copy/move operatoin grows the folder over 4GB and is kept from performing such operation. (I wonder though, what should happen when one sends out the message and copying takes place AFTER the message is sent anyway. Maybe we should create Sent-2 or some such folder and keeps the just sent folder there? But showing the WARNING/ERROR message is clearly required, and TB fails this.) 2. - be able to use a folder larger than 4GB (but this probably is a wish list.) But then again, TB needs to force 4GB limit everywhere and should not permit an operation that happens WITHIN TB to limit such problems. Also, since the 4GB limit seems to be very difficult to enformce from what I had read in bug 387502 report, I would suggest that on systems that permits large than 4GB file, TB should check for filesize >= 4GB in strategic places, and warn the user of potential problems after the fact. IMHO, the current failure of TB to enforce 4GB and failure to preserve messages is a fatal error. But more perplexing is the FAILURE to offer any warning/error message at all when that happens. We need to make two efforts (1) - to prevent and WARN the user when folder size is going to be over 4GB, and (2) - to detect that a folder has a filesize >= 4GB after the fact, and WARN the user to take remedical action.

Test script to create "dummy-file", a large (close to, but less than 4GB) mail folder from command line. It takes as its input as a file that contains a largish mozilla message. gawk -f create-dummy-mails.awk Seed-for-big-folder Seed-for-big-folder is attached separately. The script is adjusted for the first message in Seed-for-big-folder to create a large (< 4GB) folder file, which grows over 4GB.

use this as the first argument to the script attached separately after unpacking (bzip2 -d ). gawk -f create-dummy-mails.awk Seed-for-big-folder

I had to trim Seed-for-big-folder due to the upload limit size. It now has only two messages. So you should "copy" (instead of "move") the message in the said folder (mentioned in 4-(a) above) since there are only two messages now. TIA

If thunderbird has a checking code for the growing size of a folder like this, it may have a problem on a platform where C/C++ compiler doesn't support truly 64 bit size item. unsigned csize --- current size of the folder unsigned msize --- message size (to be added) unsigned nsize --- new size of the folder nsize = csize + msize; To avoid potential 32bit only arithmetic problem, we need to add the check for overflow of the addition, etc.. if ( (nsize < csize) || (nsize < mszie) ) { /* overflow. 2^32 bit is reached. */ warn the user and takes care of the 4GB limit. } The above check works even if there is no 64 bit arithemtic available on the platform. Well, if TB lacks an internal API that doesn't tell the folder (or the routine to handle the copy/move of a message), the size of the incoming message, then tough lack...

(In reply to comment #0) > (There is now a bug called 387502, which is similar. But it was marked > "unconfirmed" and > never followed up. It happened under windows with an earlier version. > Mine is on a different platform with a recent version and I am attaching a > test file for this.) You *should* have added your comments to bug 387502 itself if it is in fact the same bug. Differing versions and platforms almost never make it a separate bug. > We need to make two efforts > (1) - to prevent and WARN the user when folder size is going to > be over 4GB, and According to bug 387502, we do warn the user. Reading the code, we will throw a warning if there is less than 1 MB of free room left. > (2) - to detect that a folder has a filesize >= 4GB after the fact, > and WARN the user to take remedical action. That would be bug 387502, then.

(In reply to comment #0) > User-Agent: Mozilla/5.0 (X11; U; Linux i686; ja; rv:1.9.0.10) > Build Identifier: version 2.0.0.21 (20090302) > gawk -f /extra/ishikawa/download/create-dummy-mails.awk Seed-for-big-folder > A large file (close to but less than 4GB ) is created. > (The script creates a file consisting of slightly changed copies of a single > largish message that contains an attachment. Each message is about 3.x MB.) Mail offset in mail folder file is 32bits signed integer on Linux and Mac OS X. (4GB support, 32bits unsigned integer use, is still implemented on MS Win only.) So, if you externally create mail folder file greater than 2GB, I guess rebuild-index generates incorrect offset value(wraparound) for mails placed at larger than 2GB. It's same when "mail folder file greater than 2GB by MS Win" is mounted on Linux. It's current limitation/restriction. Does Tb 2 on Linux create local mail folder file greater than 2GB by mail copy of Tb between mail folders?

(In reply to comment #5) > (In reply to comment #0) > > (There is now a bug called 387502, which is similar. But it was marked > > "unconfirmed" and > > never followed up. It happened under windows with an earlier version. > > Mine is on a different platform with a recent version and I am attaching a > > test file for this.) > > You *should* have added your comments to bug 387502 itself if it is in fact the > same bug. Differing versions and platforms almost never make it a separate bug. > > > > We need to make two efforts > > (1) - to prevent and WARN the user when folder size is going to > > be over 4GB, and > > According to bug 387502, we do warn the user. Reading the code, we will throw a > warning if there is less than 1 MB of free room left. > I will switch t bug 387502 after this post. But from what I learned from the colleague who has a Sent folder reaching and going beyond 4GB, TB didn't seem to have thrown the warning/error at all. If it did, I think he noticed and took a remedial measure, but he didn't. My creating a large file externally is to try to re-create my colleague's condition. I suspect that the warning code is buggy somewhere since I can pleasantly create a slightly less than 2GB file and copy messages into it without any ill-effect at all to go beyond the 2GB limit on Linux. Please set the number of messages variable in the BEGIN section of the awk script by half, (that will create close to, but less than 2GB file) and try copying messsages into that folder (after placing it in the Mail subdirectory), TB happily (without any warning/error) accepts messages until it goes beyond 2GB. Then from what I read here, it is also a buggy behavior and TB should have issued warnings, etc.. > > (2) - to detect that a folder has a filesize >= 4GB after the fact, > > and WARN the user to take remedical action. > > That would be bug 387502, then. > > *** This bug has been marked as a duplicate of bug 387502 *** OK, I will follow up with 387502, but one reason I didn't follow it up was that the copying done by sending (or receiving presumabley) requires quite a different error recovery in terms of UI and thus I felt that my colleague's situation where Sent grew larger than TB can handle without HIS being aware of it requires special attention IMHO. Anyway, I will go to 387502 hereafter. Oh, to the final question in #6, TB creates larger than 2GB mail folder by copying messages happily without any error/warning from what I saw. > Reading the code, we will throw a > warning if there is less than 1 MB of free room left. There must be a subtle bug in there. (comparison of signed and unsigned, going over 2^32, etc..) TIA.

(In reply to comment #7) > Oh, to the final question in #6, TB creates larger than 2GB mail folder by > copying messages happily without any error/warning from what I saw. I saw it(larger than 4GB if MS Win) a few times several months ago during my test, but I could't see it these several months. I could see it again on MS Win by next test at last. (1) File-A: (2559) mails * (32KB per a mail) => File-Size=80MB, no File-A.msf File-B: (64-1) mails * (64MB per a mail) => File-Size=4GB-64MB, no File-B.msf (2) Restart Tb, click File-A to force Rebuild-Index. (3) Never touch File-B, so File-B.msf is not created. - emulation of "delete of msf" due to "outdated msf condition" - emulation of Save in Drafts/Sent, Junk move, Filter Copy/Move etc. without explicit mail folder open of Copy/Move target folder. (4) Create Message filter : If subject contains xxx, Copy to File-B. It copies all mails in File-A to File-B. (5) Run Filter on File-A => File size of File-B increased to 4,311,718,368 bytes(4.1GB). No warning of "exceeds 4GB". (6) Run Filter on Folder-A again. Warning of "exceeds 4GB", and copy was not executed.