User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3 Build Identifier: Versie 2.0.0.17 (20080914) Thunderbird currently has a maximum foldersize of 4GB, lets bring TB into 2008 and remove (or raise) this limit. I currently have 10GB of mail and this limit is really annoying. Workarounds for this 'feature' like making lots of different folders or deleting E-mails aren't solutions. Reproducible: Always Steps to Reproduce: 1. Receive lots of mail 2. Save mail 3. Have more than 4GB of mail Expected Results: Be able to handle more than 4GB of E-mails, other E-mailprograms (or even webbased E-mail) don't seem to have a problem with this.

Can you please give more information on this issue? Are you meaning the limit discussed here, http://kb.mozillazine.org/Limits_%28Thunderbird%29#Folders_and_messages?

Tyler: The issue should be clear, there is a 4gb/folder limit and TB warns if you reach this limit and doesn't allow folders with a size above 4gb. A simple "removing the limit" doesn't work, you have to check the used Filesystem. Fat32 doesn't support single files above 4gb.

@Matti, by >Can you please give more information on this issue I was meaning how the reporter wanted to fix this. Either by a UI option, or what be it. I understood exactly what the issue was. By asking him to read to article, I was simply pointing to the section: >unless the file system limits the maximum file size to a lower value. This >shouldn't be an issue unless you're using OS/2

If I'm not mistaken, "mailbox:" references are kept track of with an unsigned long integer, which is still 32 bit on a common-type machine. So, this may be a "natural" limit as well, unless those pointers can be extended.

I'm talking about the limit in the article Tyler mentions. Is the reason for this fat32? Then why not split the mailbox-files into multiple files with a maximum of 4GB per file? Or do we have to wait for a 64-bit TB?

Personally I would get nervous keeping such large folders and would probably reorganize them into a set of smaller folder, but I see that this may be a matter of personal preference and that there may be such use cases. The official Mozilla builds are all 32bit, which can run on 64bit machines without problems, but 64bit builds cannot run on 32bit machines. Also, the underlying file system may be a cause (there are probably many setups left using FAT or equivalent file systems, or connecting to a server with NFS v2), but that shouldn't be a reason to deny a larger folder size on systems which indeed would support it. Extending the pointers beyond 32bit values is the other relevant issue, but even then the question comes up if the currently used one-file mbox format would be still allow sufficient performance for, let's say, a 16GB mailbox, or if some other mechanism would become necessary.

Thank you. Now I really understand. Thank you Ron Parker

The limit of 0xFFF00000 bytes is defined here: http://mxr.mozilla.org/comm-central/source/mailnews/local/src/nsLocalMailFolder.cpp#3834 The name of the limit is kMaxFolderSize: http://mxr.mozilla.org/comm-central/ident?i=kMaxFolderSize

This bug is now is starting to affect Gmail users who wants to use Thunderbird, as Gmail stores all the messages in a folder called "All-mails" and many users (myself included) have more than 4gb of mail: http://getsatisfaction.com/mozilla_messaging/tags/bug_462665 https://bugzilla.mozilla.org/show_bug.cgi?id=532323

There are ways to not download All Mails - unfortunately , can't remember wher you need to set that option.

You can set it in account settings - synchronization & storage

(In reply to comment #11) > This bug is now is starting to affect Gmail users who wants to use Thunderbird, > as Gmail stores all the messages in a folder called "All-mails" and many users > (myself included) have more than 4gb of mail: As seen in bug 387502, "appending mail data to file" doesn't have 4GB limitation(64bit signed integer is used for seek). "4GB limit of mail folder"(MS Win, 2GB if Linux/Mac OS X) is caused by 32bit integer use for offset for local mail folder(data for .msf, key of Mail DB, for indexing of mail), 32bit signed integer : Linux, Mac OS X => Max 2GB 32bit unsigned integer : MS Win => Max 4GB and "2GB limit in file handling" is caused by 32 bit signed integer use in many codes for file handling. The "offset for local mail folder" corresponds to UID of IMAP, and, as seen in Bug 518918, 32bit unsigned integer looks to be used for it by Tb. (to afford to larger than 2G UID, use 64 bits signed integer to handle UID). It's irrelevant to "offline-store file size limitation", because "offset in offline-store file of IMAP" is not same as "offset for local mail folder", although Tb uses same Unix Mbox format for both "file for local mail folder" and "file for offline-store of IMAP". When I tested Bug 487992, Tb appended mail data to "offline-store file whose file size is greater than 4GB". And, AFAIR, the mail data was read normally by Tb. (Inreply t0 Comment #0) > Steps to Reproduce: > 1. Receive lots of mail > 2. Save mail > 3. Have more than 4GB of mail To mail1@lispeltuut.org(bug opener): What kind of problem occurred after above step 3? (case-1) offline use = off (case-2) offline use = on

Checked with Tb 3.0 release build on MS Win-XP. 1. A local mail folder file(unix mbox) has many mails. (say "F") File-size=4294963200 (4GB=4294967296) 2. Create IMAP folder "F", offline use=on 3. Copy 3 mails(175KB each) to "F" 4. Click "F" => 3 mails are displayed normally. File-size of F4 = 4295860173

Note: Next is not checked yet. Mail is displayed from offline-store larger than 4GB, or mail is re-dowloaded from server upon each mail display.

Bug 532323 was for IMAP offline-store specific problem. I should have comment to Bug 532323 for IMAP case.

Wada - I think you nailed the bug affecting Gmail users.

As "large file support" is not enabled yet for Linux/Mac OS X, fix of bug 362244 is required before implementation of enhancement by this bug for Linux/Mac OS X.

If bug 362244 will be fixed, limit of mbox file size can be enlarged to 4GB on Linux/Mac too, as already done on Win.

(In reply to comment #20) > If bug 362244 will be fixed, limit of mbox file size can be enlarged to 4GB on > Linux/Mac too, as already done on Win. Wada-san, Linux and Mac OS X can already handle over 4GB file via NSPR. Bug 362244 still occurs on FreeBSD and OpenBSD.

(In reply to comment #21) > Linux and Mac OS X can already handle over 4GB file via NSPR. Gotha! I've understood why "Platform:All" is set for Bug 387502 now. Thanks. Do you think next is required for Linux & Mac OS X too? Change offset value in MsgDb(.msf) of local mail folder from "signed 32bits integer" to "unsigned 32bits integer". Or fix of this bug(use 64bits signed or unsigned integer commonly) is sufficient?

(In reply to comment #22) > (In reply to comment #21) > > Linux and Mac OS X can already handle over 4GB file via NSPR. > > Gotha! I've understood why "Platform:All" is set for Bug 387502 now. Thanks. > Do you think next is required for Linux & Mac OS X too? > Change offset value in MsgDb(.msf) of local mail folder > from "signed 32bits integer" to "unsigned 32bits integer". > Or fix of this bug(use 64bits signed or unsigned integer commonly) is > sufficient? I think that it should be extended to 64bit. Most modern OSes handle 64-bit file offset in kernel/filesystem.

I just finished uploading a large amount of mail from Eudora to a Gmail account. This required first importing the Eudora mail to Thunderbird, where I did run into an error message re:4GB limit requiring me to split one of the larger folders in Eudora prior to importing. However, after uploading to Gmail then syncing back to Thunderbird, the All Mail folder from Gmail is 6.14GB. Yet, there was no error message and as far as I can tell it seems to be working fine in Thunderbird (ver 3.0). Can someone clarify what's happening here?

(In reply to comment #24) Magritte, for offline-store of IMAP case, see Whiteboard: and see Bug 532323. See also bug 537498.

(In reply to comment #24) > where I did run into an error message re:4GB limit Bug 387502 was fixed by Tb3.0b4, so error message is properly issued by Tb 3.0.

changed the title because bug 538512 was fixed.

A few GetSatisfaction topics to kick start this Bug http://getsatisfaction.com/mozilla_messaging/topics/sent_messages_not_going_to_sent_file?utm_content=topic_link&utm_medium=email&utm_source=reply_notification http://getsatisfaction.com/mozilla_messaging/topics/why_is_my_inbox_listed_as_full_when_its_not_and_preventing_me_from_getting_emails?utm_content=topic_link&utm_medium=email&utm_source=reply_notification http://getsatisfaction.com/mozilla_messaging/topics/the_folder_inbox_is_full_and_cant_hold_any_more_messages_to_make_room_for_more_messages_delete_any_old?utm_content=topic_link&utm_medium=email&utm_source=reply_notification http://getsatisfaction.com/mozilla_messaging/topics/why_my_inbox_shows_4gb_even_when_i_delete_all_messages?utm_content=topic_link&utm_medium=email&utm_source=reply_notification http://getsatisfaction.com/mozilla_messaging/topics/inbox_is_full?utm_content=topic_link&utm_medium=email&utm_source=reply_notification

Does all this commenting and other jargon mean the problem is unfixable? It is causing me havoc, which I can't endure long. Let us know when to expect being able to get back to legitimate service!

We're working on a storage format that doesn't have this limitation, but it takes time.

Has this bug become a fatal, unfixable, software program failure? Since I first reported it about two months ago, I've received two program updates which obviously had nothing to do with this bug. I've received nothing to my inbox since 26 Jul, after having done exactly what I was told to overcome the problem, and I keep getting the same instructions to do the same procedure as before. Be straight forward and inform us that we need to begin using another e-mail software.

(In reply to comment #39) > Has this bug become a fatal, unfixable, software program failure? No, read the previous comment to yours that we are working on a format that does not have this limitation. Something like that does take time and will only be fixed in a new major release. It is something we can't fix in a minor release. Since I first reported it about two months ago, I've received two program > updates which obviously had nothing to do with this bug. I've received nothing > to my inbox since 26 Jul, after having done exactly what I was told to overcome > the problem, and I keep getting the same instructions to do the same procedure > as before. This sounds like a slightly different issue to this specific bug, I suggest you raise a separate bug describing exactly what your issues are.

Could anyone please explain which format is / will be chosen to alleviate this problem? AFAIK the "simplest" solution would be to use a MailDir format where each folder is a directory and each message is a file (i.e. much like an IMAP server's file system). Although I'm not sure if that's the "best" solution, since it would leave stuff like indexing up to the OS ... which may make TB act rather slow on some systems (even worse than presently). Maybe some "true" DB solution is in order, in which case the growth of the file needs to also be investigated - see this discussion: http://forums.mozillazine.org/viewtopic.php?f=30&t=2072105

This problem is not an enchancement request; this is a true bug in some scenarios. Today I tried to import a big Outlook profile (18 GB PST file on NTFS, messages split to 2 folders). TB reported everything to be OK (saying it has successfully imported ~6500 messages in one folder and ~2500 in another). The directories that TB created for imported mail (in profile directory on NTFS) have 2 huge files (one ~11GB, another ~5GB). But the number of messages that TB reports is ~2000 and ~1500. I checked that in each folder, only first N messages are available, and the last one is near the 4GB limit in the msf file. And right after the 0xFFFFFFFF address of the file, the data is totally garbled. The user that I hoped to move from Otlook to TB will stay with Outlook, as she is unsatisfied with this, and splitting filders isn't an option for her. And even if someone believes that making folders small is the way to go, please believe that this isn't always so for everyone. From the discussion above (and links) I understand the underlying reason. But anyway, even if Bug 402392 gets fixed, and MailDir will be an alternative, the 64-bit file addressing should be implemented on FSs that support this.

The patch in bug 402392 does remove the 4GB limit for berkeley mail stores as well, but it's not super well-tested because it was just something I did along the way; it's not the goal of the patch.

This bug has recently hit me on Windows 7 64-bit. In my case, the issue isn't that a larger file can't be written, but rather that the files are *so large* that they cause other software to fail. In particular, I keep my Gmail mail backed up on my Subversion repository. I consider this a perfectly valid scenario. My file 'MyProfile.default/ImapMail/imap.gmail.com/[Gmail].sbd/All Mail' has now hit 2.7 GB. This would normally be fine, but Subversion served over Apache using SSL cannot handle large requests, resulting in the '413 Request Entity Too Large' error: http://svn.haxx.se/users/archive-2008-01/0689.shtml. To resolve this on the Apache side requires SSLVerifyClient applied on the Virtual Host level rather than per-directory. I would rather have this solved on the Thunderbird side, where a mailbox file could be split across multiple files, rather than requiring that all of your mail is stored in a single file (and thus easily corruptable). This would solve a lot of the problems discussed in this thread, including mine. I'd love to submit a patch but my C knowledge is extremely rusty.

(In reply to Jevon from comment #48) > I would rather have this solved on the Thunderbird side, where a mailbox > file could be split across multiple files, rather than requiring that all of > your mail is stored in a single file (and thus easily corruptable). This is already being worked on in bug 402392 and bug 58308.

My sent folder reached 4GB limit and now it's gone. A new sent folder appeared with NO MAIL in it. I have a back up from 24 hours ago, but is there an easier way to get the sent folder mail back?

(In reply to walsh from comment #50) > My sent folder reached 4GB limit and now it's gone. A new sent folder > appeared with NO MAIL in it. I have a back up from 24 hours ago, but is > there an easier way to get the sent folder mail back? Have you ever compacted the folder? Is it really contain > 4GB of messages? Is the Sent folder on disk > 4GB? We've gotten rid of this limit in TB 12, but that hasn't come out yet.

Per bug 729896 walsh has filed in parallel, he's still using a 3.1.x version.

(In reply to walsh from comment #50) > My sent folder reached 4GB limit and now it's gone. A new sent folder > appeared with NO MAIL in it. I have a back up from 24 hours ago, but is > there an easier way to get the sent folder mail back? I have found somewhere on the web a way, how to get from this situation: 1. if you see at least some mails from the folder, you can create a new folder, where those mails will be copied, select all messages and copy (move) them. You have to be sure, that you do not exceed the 4G again. The other way is to use some good text editor which can edit 4G file and manually remove the last messages from the folder, to get to the working limit. I think that in case of exceeding the limit even the compacting does not work. Regards, zetka

Another alternative would be to use a free-standing MBox to EML converter - basically extracting all messages in TB's mbox file into individual EML files. Something like this might work: http://www.outlookimport.com/mbox-eml-converter-extract-emails-mailbox-thunderbird-eudora-entourage/ Then you can import those EML files back into TB into various folders (i.e. try not to reach the limit again). Not sure if your TB can do such out the box, if not look for the Import-Export-Tools addon. http://nic-nac-project.de/~kaosmos/mboximport-en.html On a side note, I've got an addon installed which displays each folder's mail count & size - and then as soon as it reaches >1GB I start splitting that folder (usually creating a subfolder with a date name and then all mails upto that date goes in there) ... like I'd have imagined an archive to have worked in the first place! The problem is with mail accounts such as gmail. Some of its folders contain "All Mail" - even if you move them to other folders. In this case I'm stuck with moving messages off the gmail account onto local. Thus I'm incapable of then reading those same messages on my phone. Thanks TB for still being stuck in the '70s.

Look, what happened to support for 2^64 byte size for files? As far as I know, it is still supported by the kernel and by the C libs. Why devise yet another kludge that will need maintenance and introduce yet more bugs. It seems to me that must be limitations in the mozilla SDK that prevent indexing into 2^64 sized files. What should be done is eliminate these limitations.

this was fixed by the patch in bug 402392, and will be in TB 12.

The tell tale sign that TB cannot handle a folder larger than 4GB is to look at the number under the TB tab called Order Received You have to first enable it by clicking on the tiny little square icon on the right hand extreme end of the message sort options such as Subject, Date, From, Recipient, ...etc. Click on that icon and enable (check mark) Order Received. If you have more than 4GB Inbox folder, the latest messages that came in after reaching 4GB will display a negative number in the Order Received column. If the number is not negative, then the warning message is a remnant after the bug was fixed, and someone forgot to remove it.

Are we sure this is fixed? I was trying to move all my 2012 mail into once single folder (sent & received) and the size of these mails are 5GB, Inbox 3.5 and sent 1.5. Once I moved the inbox, I can't anymore move the Sent folder messages into that folder, Thunderbird claims folder is full. So did I hit a 4gb mbox file limit? I thought this limit was lifted?

(In reply to Emre from comment #59) > Are we sure this is fixed? I was trying to move all my 2012 mail into once > single folder (sent & received) and the size of these mails are 5GB, Inbox > 3.5 and sent 1.5. Once I moved the inbox, I can't anymore move the Sent > folder messages into that folder, Thunderbird claims folder is full. > > So did I hit a 4gb mbox file limit? I thought this limit was lifted? I would say you have exposed a bug, but it is not clear if it is the same as the 4GB size limit. What I would do to test this is as follows: Check to see if you have latest thunderbird version. If not, update!!! exit thunderbird. cd to ~/.thunderbird/Profiles/YourProfileName/Mail/pop.gmail.com (if you are using imap, instead of pop, then pop.gmail.com with imap.whatever... now, take any of your folders such as trash: cat trash trash trash trash trash trash trash trash trash trash trash > trashx touch trashx.msf ls -l trashx if the size is still less than , ... say 6GB, then keep catenating to it the trash file several more times, until it well exceeds 4GB (such as 6GB would be a good test). Now restart thunderbird, and browse any other folder and try to move a message to the trashx folder. If it says it is full, the bug has NOT been fixed.

(In reply to JD from comment #58) (In reply to Emre from comment #59) (In reply to JD from comment #60) Are you looking symptom of bug 789679? See bug 789679 comment #30 and some my comments before it, please.

easy way to make a beyond-4GiB inbox in windows cmd shell: (MAKE A BACKUP!) I use the following makeinbox2GB.cmd file: ------------------------------------------ set targetexponent=10 set outd=\t del %outd%\inbox-* %outd%\inbox %outd%\inbox2G %outd%\z set ind=\Documents and Settings\Jim Michaels\Application Data\Thunderbird\Profiles\md4bzvt7.default\Mail\mail.renewalcomputerservices.com copy /b "%ind%\inbox"+"%ind%\drafts"+"%ind%\trash"+"%ind%\sent" %outd%\inbox copy /b %outd%\inbox /b %outd%\inbox2G /b set /a count+=1 set count=0 :lupe if "%count%"=="%targetexponent%" goto exitlupe copy /b %outd%\inbox2G+%outd%\inbox2G /b %outd%\z /b move /y %outd%\z %outd%\inbox2G set /a count+=1 dir %outd%\inbox2g goto lupe :exitlupe del %outd%\z dir %outd%\inbox2g Inbox is assumed to be 6MiB. if you want something larger than 2GiB, simply the 10 is essentially a power of 2 multiplier where the 10 is the target exponent. 7,072,238*2^10=7,241,971,712 mbox format is nice in that if you want to multiply your emails for testing purposes, concatenate itself (not sure how to do this with *nix) or concatenate 2 copies (should work with *nix and mac). the actual equation is originalInboxSize*2^targetexponent and can be calculated in ttcalc which is avaiable at http://ttcalc.sourceforge.net/

by the way, this is based on the copy command in the cmd shell which allows you to concatenate files together using + and you should also use /b all over the place in the copy command for forcing binary. just do copy /? to get help.

Status: Resolved/Fixed? You can bet it isn't. And I wonder if anything is really being done about this VERY significant bug besides nitpicking. Trust me, it's been there at least since TB v3 (possibly previously as well, I'm just stating *my* experience of it). I'm now at v17.0.3 running on Win XP Pro 32bit, Win 7 32bit, Win 7 64bit (3 different computers) and it's there still. All PC's are running on NTFS, so no file size limit exists OS-side. The problem is there in ALL my computers: as soon as a mailbox file gets close to 4GB (in my case it's always the Inbox), you get the folder full message and mail stops being downloaded. You have to either delete messages, or move some to another folder and compact; the file size drops, you hit Get Mail and everything works fine. Until it gets to 4GB again. I often come across this issue as my (main account) Inbox reaches 4GB with about a year-and-a-half's incoming mail. As I constantly use Quick Filter to search for emails and I need to be able to search for as long as period back as possible, I keep weeding out the unnecessary stuff from my Inbox to delay the moment when I have no alternative but to move some emails to a separate folder (say Inbox2011) under Local Folders. So there it is. I wish the developers sat down to address possibly the most long-standing bug in TB, instead of dealing with bells and whistles. After all, a mail client's primary job is to receive and send email.

Nikolas Kostakis: What is your used Filesystem ?

As I say in my message above, NTFS.

(In reply to Tyler Downer [:Tyler] from comment #3) Still having the same problem myself: TB Portable 17.0.4 on Win7 Pro 64 with the profile on a NTFS formatted (1TB) disc. I know it handles >4GB files as I've got several files in excess of 7GB (even some up to 20 GB for one single file). Just tried it doing a copy /b to combine a folder with itself. the folder was 2.35GB to begin with, the combined folder file is 4.7GB. Opened TB, waited for it to construct the new folder's index. Browsing in the folder works, opening messages works. I can even mark messages just fine. Though 4 problems: (1) Though no biggie at all: The Extra Folder Columns addon (as does the folder's properties dialog) displays the incorrect amount of messages as 2857 (the original folder contained 1631 so it should have been 3262). (2) Another minor issue: Extra Folder Columns (as does the folder's properties dialog) displays the size as 731MB. The original folder's size displays as 2414MB, so it should have displayed as 4828. This seems to be the 4GB limit - i.e. difference of 4097 MB. (3) Much more problematic: The number of messages displayed when opening the folder is as per EFC's / properties' listed 2857. (4) Worst issue: If trying to copy a single 0.7 KB message into that folder a dialog pops up stating: "The folder TestCombined is full, and can't hold any more messages. To make room for more messages, delete any old or unwanted mail and compact the folder". So no, this is still not fixed or it has since reverted back to its unfixed state. IMO, it is now serious time to consider making a full on effort to move away from MBox in preference of something like MailDir (http://en.wikipedia.org/wiki/Maildir) or perhaps MIX (http://en.wikipedia.org/wiki/MIX_%28Email%29). Here are the reasons I think this type of thing should be used instead of a flat MBox file: (1) 4GB limit would inherently be removed at the very base level. No workarounds to try and circumvent a format which wasn't designed to surpass this limit. (2) No more need for such long-winded compaction of folders. (3) Makes incremental backups much easier through any file-backup software. (4) Fragmentation of the message folder on disc is greatly avoided - would make for even faster access (especially using MIX due to its indexing). It would nearly avoid fragmentation entirely on most *NIX file systems, and most on MS file systems (well perhaps not FAT).

mbox has one nice feature, it is processable any any program that wants to process the emails. that is a big plus. so if like for examplke tb is broken, you can get your DATA BACK. can't do this with an undocumented format. and mbox is nice text-file format also. I don't think anyone is using FAT for their mailbox. but you will need some sort of reliable conversion tools if you are going to change formats. I think mbox is still usable. speed on MIX's part is a nice plus. if you have a ginormous inbox,you expect your email experience to be fast as possible. I discovered ways I could speed up my mbox processing for my mbox fixing tool simply by changing the way I did file I/O. got a 64x speed increase. and changed the processing time from 6.5 hours to 6 minutes. so take an exhausting look at your file i/o. right now I am using fopen,fgets, fprintf, fwrite on the mingw-w64 (gcc) compiler. mingw-w64 (gcc) gives me 64-bit and 32-bit executables (separate compilers for each). what I usually do is make a 32\ and 64\ directory and for windows I make shortcuts to the 64-bit exe and make the working directory the 64\ or 32\ so it sees the appropriate DLLs in that dir. i believe they also have compilers for linux too. voila, you now have 32+64-bit ff, tb, etc. and I think this same method is portable to linux and not sure about mac - someone knowledgeable about mac would have to answer on that.

ah,yes, and you can edit the mbox with any text editor like notepad++ if something is broken...

Myself often do login via a **** connection using ssh to my home computer, and do: $ less .mozilla/seamonkey/dgahgfs/foo.bar.com/Mail/Inbox to lookup my old emails. (Note: I don't need any fancy X11 client on my side, just whatever computer/os/just some terminal window). With transition to maildir I would be lost. And I even don't want to do the split into multiple files&subdirs! Do you know how many nested subdirs will that create on you filesystem for all emails you have (I have 63302)? Maildir is good for a server to avoid locking issues and gain speed, but for users storing the emails I don't see much advantage as long as the 4GB limit is gone. $ find .mozilla/seamonkey/dgahgfs/foo.bar.com/Mail/* -name \*.msf | while read f; do \ p=`echo $f | sed 's/.msf//'`; grep -c "^From -" $p; \ done | sum 63302 1 Homework: Do the above for a maldir. ;-) And yes, a number of times I used formail/procmail to postprocess my mbox folders or even vim to edit directly.

Why are many comments on issue around "4GB or 2GB limit" still added to this bug? Please note that this bug is never for all issues arround "4GB limit" nor meta bug for tracking of many relevant bugs. If "followup after backend change" is needed for problems such as bug 789679(4GB warning when Berkley Mbox), bug 793865(size display), bug 794303(compact), please open separate bug for them, with detailed description about phenomenon/problem, with sufficient data for problem analysis, in order to keep "one problem per a bug" at B.M.O. For suggestions on non-Berkley format(non Unix Mbox format): There is no need to add your suggestion to this bug. Work for supporting it is already started by bug 402392. See also bug 58308, which is similar to meta bug now. See also followup bugs for bug 402392. (In reply to irneb from comment #67) > Just tried it doing a copy /b to combine a folder with itself. the folder > was 2.35GB to begin with, the combined folder file is 4.7GB. Opened TB, > waited for it to construct the new folder's index. Browsing in the folder > works, opening messages works. I can even mark messages just fine. You are right. As I wrote in bug 794303 comment #0, "local mail folder file >4GB" is already supported by back end code of Tb, and even if "local mail folder file >4GB", "Repair Folder"(aka Rebuild-Index) works well and mails can be accessed normally. > Though 4 problems: > (1) Though no biggie at all: The Extra Folder Columns addon (as does the > folder's properties dialog) displays the incorrect amount of messages as > 2857 (the original folder contained 1631 so it should have been 3262). > (2) Another minor issue: Extra Folder Columns (as does the folder's > properties dialog) displays the size as 731MB. The original folder's size > displays as 2414MB, so it should have displayed as 4828. This seems to be > the 4GB limit - i.e. difference of 4097 MB. > (3) Much more problematic: The number of messages displayed when opening the > folder is as per EFC's / properties' listed 2857. > (4) Worst issue: If trying to copy a single 0.7 KB message into that folder > a dialog pops up stating: "The folder TestCombined is full, and can't hold > any more messages. To make room for more messages, delete any old or > unwanted mail and compact the folder". You are right. All is already know issue, and bug is already opened. Anyway, change to VERIFIED, because backend is already changed to support "larger than 4GB file".

Backend change was landed on Tb 12, so setting Thunderbird 12 in "Target Milestone:" field of this bug.

Over 4GB mailboxes are probably only supported if you manually switch the account to the new maildir-lite msg store from bug 402392. For the old mbox (the default) format, there are still some glitches that prevent >4GB folder usage so it is not allowed yet. Join bug 789679 to follow fixing of those.

umm, which comment number?link please?

Comment about what?

SO STRANGE It's funny. Since (I feel like) 10 years, eacht time when I or somebody else write about the bug, that TB cannot handle folders (mbox-format files) larger than 4 GByte, the following happens: A short discussion about filesystem limits arises, somebody tells that the problem is not dependent of the fs, somebody points out how to easily reproduce this error by simply copying files, somebody answers that folders > 4GB are bad for use and not necessary, then somebody finds out, that the bug report is a dupe of some other, and then the report is closed as "fixed". Funny - this happens now for years again and again and agein...

BTW: Somebody of the TB Team should declare at an easy to find place, that TB does have this 4 GB Limit. So everybody knows about this and everybody knows, that the TB Team knows this. And it would be nice, if a comment could be added, if and / or how the team will deal with this problem in the future. (Nothing vs. using another library for accessing large files vs. using another storage format vs. ..? )

(In reply to Knut Singer from comment #76) > SO STRANGE > It's funny. What is storange or funny? Problem is very simple since initial till end : Number of bits in a register is not Infiniy, it's finite. Therefore, maximum number what can be held in a register is finite. So, if a register is used for size, "maximum size" always exists. 32bits integer: -2^31 <= signed <= 2^31-1 0 <= unsigned <= 2^32-1 -2Gig <= signed <= 2Gig-1 0 <= unsigned <= 4Gig-1 64bits integer: -2^63 <= signed <= 2^63-1 0 <= unsigned <= 2^64-1 -8Exa <= signed <= 8Exa-1 0 <= unsigned <= 16Exa-1 (0) Mail data in a local file for an Mbox is accessed in Tb via; Account(server) + mbox/folder/file + Offset_in_file, and mail_size Call, Server + File + messageOffset, and messageSize A set of "Server + File" is "mil older" in Tb. Call the set of "Server + File" "msgFolder" here after. and, Tb assigns an unieque key in a File to a mail in a File. Call it "messageKey" here after. Summary: A mail is identified by : msgFolder + messageKey where : msgFolder == Server + File via messageKey, messageOffset and messageSize is obtained i.e. a message in Tb == Server + messageOfset + mesageSize (1) Initially, 32bits signed integer was used for messageOffset. "Negative value" was utilized for "error return code", "SEEK to backward" etc. So, limit of "Max 2GB" existed. (2) After OS support of "file size larger than 2GB", messageOffset is changed to 32bits unsigned interger in Tb on Win, so limit was changed to "Max 4GB" on Win, but it was Win only change, so "Max 2GB" remained on Linux and Mac OS X. (3) We were aware of critical problems when msgFolder size accidentally exceeded "Max 4GB" size limit, so "appending mail data over 4GB" was explicitly prohibited in msgFolder processing code. (4) By Tb 12, messageOffset is changed to 64bits signed interger in Backend code of Tb, then Tb was/is able to access mails in "File" larger than 4GB with no problem. - Tb can re-parse(repair folder) all mail data in "File > 4GB". - Tb can read any mail data in "File > 4GB". => This bug is changed to FIXED. However, 32bits signed or unsigned integer is still used in somewere of Tb's code for variable to hold Offset data which corresponds to messageOffset of msgFolder in Tb. => Change by (3) is still kept to avoid known critical problems. So, phenomenon of bug 789679 is still observed. => Some followup bugs are opened, for example, bug 794303 for Compact. There are several kinds of approach to resolve the "4GB limit" problem. (A) Extend actual limit by Tb's code to "Max of 64bits signed integer". Bug 789679, and other followup bugs. are example of this approach. We are trying to remove 32bits integer use which corresponds to messageOfset value(already 64bits integer in Backend code). Biggest problem in (A). "Removing 32bits integer use from anywhere in Tb's code" is far difficult than you think. Example of recently discovered issue: When messageOffset(64bits) exceeds 32bits unsigned integer value, messageKey(32bits unsigned integer) may wrap, or uniequness of messageKey may be lost. This is far critical than "file largr than 4GB can not be used". Second problem in (A). Even when "local message folder larger than 4GB" is normally used without any problem, following issues can't be resolved by (A). - It takes very long to Compact folder, because "huge HDD file copy" is involved in Compact.. - It takes long to open message folder due to huge .msf file caused by too many mails in too huge file. (B) Virtual Folder: New "Message Folder", "Message Database" in Tb. VirtualFolder == Set of "msgFolder" a mail in a VitualFolder is identified by; msgFolder# (=> Server# + File# => Account/Location column) + messageKey# (=> messageOffset and messageSize) messageOffset value is still limited to 32bits integer value by 32 bits integer use in many places, so each "File" size is limited to "Max 4GB". But max of total size of a VirtualFilder == N*(max of each File). If N is 32bits integer, max of VirtualFolder size==practically ∞. This is already achieved as "(Saved) Search Folder", "Unified Folder", View(All, Unread, Customized View), "Quick Search". (Quick Search is a "View" which is similar to a Customized View.) Biggest problem in VirtualFolder: Tracking of each msgFolder. An msgFolder(actual mail folder in Tb) can be used by multiple VirtualFolders. So, one mail isertion at an msgFolder will produce update of many VirtulFolders at same time. And, an VirtualFolder can have many msgFolders. So, "an mail insertion at a msgFolder" will produce re-indexing work of big index of many VirtualFolders at same time. Second big problem in (B). As known by current name of "(Saved) Search Folder", current VirtualFolder is a "search result", and "automatic refresh of Virtual Folder" is not executed upon each change in a msgFolder. So, bug of "Unified Folder is not updated!!!" is frequently, repeatedly opened. (C) Get rid of Berkley Mbox format(Unix Mbox format) == maildir support (C-1) Pluggable message store => already landed on Tb 12. (C-2) "one file per a message" support. Because "message size" is currently limited by 32 bits integer, file size never exceeds limit of 32bits integer. So, "Max 4GB file!!!" problem can't happen. Problem in (C-2) : Still in progress (D) SQL Mail Data Base, instead of MorkDB + maildir or Berkley. This was tried once, but it was given up by performance reason (response time). However, many items can be moved to SQL Data Base. - Static data such as column selection (currently saved in .msf) - Static data such as Synchronization setting(saved in .msf) - Account definition, Filter definition, ... I believe clever SQLite use should be reconsidered. After pluggable message store and maildir support completion, "structured mail data in SQL DB" like approach will be possible, unless digital signed mail/cyphered mail. - SQLite DB row for top level message header after UTF-8 conversion - SQLite DB row for message body part after UTF-8 conversion - SQLite DB rows for many/huge attachment parts - SQLite DB rows for nested mail parts(message/rfc822) I believe current developer's objective is "complete (C)", because (A) can't resolve problem of "Compact takes very long when huge file” in principle. To get rid of expensive Compact, best practice is "don't use expensive Compact". Anyway, here is bugzilla.mozilla.org to fix bug. Here is never discussion forum nor support forum. Please use newsgroup or forum. If you are talking about specific issue, read bug for the issue and add comment to the bug after read the bug well and understand the bug well, please. This bug is for "BackEnd support" and is ended as known by Status: field of this bug, and this bug is never for generic issue around "4GB file size" nor tracking bug of problems around "4GB file size".

Currently using TB 17.0.2 on Ubuntu (Latest ubuntu update). Ubuntu has not released a later version yet. My Sent folder size is $ ls -l Sent -rw------- 1 jd jd 4287923277 Mar 20 19:16 Sent Often when I am using my other (windows) laptop out at hotspots, I use the browser to read and send email. On the home machine, where I use Ubuntu, TB downloads from the mail server all of my undownloaded messages, including mail I sent while I was out at hotspots, into the Inbox. So, when I try to move messages I sent to the Sent folder, I get this message: The folder Sent is full, and can't hold any more messages. To make room for more messages, delete any old or unwanted mail and compact the folder. I hope this report helps. So, as far as I am concerned, the problem is very clearly and obviously NOT FIXED.

performance data for fopen,fwrite on old pentium 4 HT 2.8GHz (1 core 2 thread). one a 5400RPM disk, 0.014314GiB/s(GiBps) or 14.657931MiB/s(MiBps) or 0.114515Gibps or 117.263449Mibps 0.015370GB/s(GBps) or 15.369955MB/s(MBps) or 0.122960Gbps or 122.959638Mbps on a 7200RPM disk, 0.015217GiB/s(GiBps) or 15.582150MiB/s(MiBps) or 0.121736Gibps or 124.657199Mibps 0.016339GB/s(GBps) or 16.339068MB/s(MBps) or 0.130713Gbps or 130.712547Mbps use ttcalc or calculator to figure out speed. your mileage may vary with algorithm. this is what I get with my program worst case.

@WADA Hi, first thank you for your explanations. I don't want to discuss, but you asked me, whats so funny or strange about dealing with this this bug in bugzilla, so I want to try to explain this: You, and some other regulars of TB seem to KNOW about this very essential limitation, that will let run thousands of people into problems with TB. This becomes sharper, as since some years ago every new account is added with the feature of "synchronizing" by default and without explaining this to the user. So, I do not have any problem, if this limitation is well known and everybody is informed about this on TB site. But this does not happen. And so for most users who come across this limitation, this is of course a "bug". And then, if some of them know about bugzilla, and they write about this bug, they all come into these sad loops I described above. They are not informed, what is the problem (like you wrote to me), but all this stuff I mentioned above is told to them. And finally their bugreport is marked as "solved" and "Dupe of xxx" and the 4 GByte limit still exists. And this happens that way for years and years. That is very strange for people, who are not envolved in developing TB - or funny. It depends on the mood. PS: Sorry for my bad English

(In reply to JD from comment #79) > -rw------- 1 jd jd 4287923277 Mar 20 19:16 Sent 4287923277 == 3 GB + 1017 MB + 289 KB + 77 Bytes 4GB - 4287923277 == 6 MB + 734 KB + 947 Bytes > So, when I try to move messages I sent to the Sent folder, I get this > message: > The folder Sent is full, and can't hold any more messages. To make room for > more messages, delete any old or unwanted mail and compact the folder. "What occured" is explained by the message. Do as the message says. However, Tb currently has problem in Compact if folder after Compact is larger than 2GB. Do as follows. (1) Create second/third Sent folder, call Sent-2 and Sent-3. (make it sure that any is less than 2GB, to avoid unwanted problem) (2) Move 1/3 mails in Sent to Sent-2 (less than 2GB) Move 1/3 mails in Sent to Sent-3 (less than 2GB) => Remaing mails in Sent < 2GB (3) Compact Sent, wait end of Compact of Sent (4) You can use any of Sent, Sent-2, Sent-3, until file size increases to near 4GB again, although auto-compact may silently fail once file size exceeds 2GB again. Why never see any bug placed in Depends on: field and/or Blocks: field of bug before adding your comment? See bug 789679, which is already put in Depends on: field of this bug on 2012-09-14, for current file size limitation or issue or phenomenon around "4GB" : "4GB-4M", "4GB-4MB-1MB", "4GB-256", ... See bug 794303, which is already put in Depends on: field of this bug on 2012-09-25, for problem when nstmp exceeds 2GB.

(In reply to WADA from comment #82) > (In reply to JD from comment #79) > > -rw------- 1 jd jd 4287923277 Mar 20 19:16 Sent > > 4287923277 == 3 GB + 1017 MB + 289 KB + 77 Bytes > 4GB - 4287923277 == 6 MB + 734 KB + 947 Bytes > > > So, when I try to move messages I sent to the Sent folder, I get this > > message: > > The folder Sent is full, and can't hold any more messages. To make room for > > more messages, delete any old or unwanted mail and compact the folder. > > "What occured" is explained by the message. > Do as the message says. Wrong! My folders are totally compacted (and repeatedly). > > However, Tb currently has problem in Compact if folder after Compact is > larger than 2GB. This problem is MANY YEARS OLD!!! To close this issue as VERIFIED FIXED is at the very least: LAUGHABLE if not lamentable. So, fix the compaction issue and then let's see if it fixes the problem.

if this sent-1, sent-2, sent-3 is the operation I think it is (splitting sent intomultiple 2GiB folders), what you will see in TB without any changes is what I have gotten with my TB mail splitter. once tb is up, folders will appear (Sent-1, Sent-2, Sent-3,etc) one for each of the split folders. depending on how many of these you have, if you have a lot of them,it will of course take up a lot of visual space in TB - except for the original Sent box, it will only take up one file-looking mailbox thing. see here for a visual: https://a.fsdn.com/con/app/proj/tbmailsplit/screenshots/tbirdmailsplitGUI-tbirdshot.PNG

(In reply to Knut Singer from comment #81) > This becomes sharper, as since some years ago every new account is added > with the feature of "synchronizing" by default Please never discuss based on wrong assumption. (i) The "Synchronization" you call is IMAP only feature. But this bug is for msgFolder of "MorkDB + Berkley file" for POP3 account and pseudo account of "Local Folders". Why can the IMAP feature be relevant to this bug? (ii) "IMAP offline-store file used by auto-sync of IMAP" initially had problem of "4GB due to 32bits integer", but it was relatively quickly resolved by "changing to 64 bits integer" after release of Tb 3 which is first release of Tb shiped with "auto-sync is enabled by default". Why can the IMAP feature be relevant to this bug? > and without explaining this to the user. (iii) I know well about chaos what was produced by first release of Tb with "auto-sync enabled by default blindly". But that is absolutely irrelevant to this bug's problem on MorkDB + Berkley file for POP3/Local Folders. (Off Topic) > So, I do not have any problem, if this limitation is well known and > everybody is informed about this on TB site. > But this does not happen. Contributers in bugzilla.mozilla.org such as Developers, QA peoples, are all volunteers. If you think so, why you don't do it with your own hands? You　do not do anything by yourself in just asks for something to other people all the time? I can't think so.

(In reply to JD from comment #83) > > However, Tb currently has problem in Compact if folder after Compact is larger than 2GB. > This problem is MANY YEARS OLD!!! Oh, sorry for misleading expression. Tb *STILL* has problem in Compact if folder after Compact is larger than 2GB. I believe it's problem since initial of "larger than 2GB mbox file support" by Mozilla MailNews/Thunderbird. The problem can't be exposed unless mbox file size exceeds 2GB, and 4GB support was Win only for long time, and 4GB support on Linux/Mac is from Tb 12...

(In reply to Jim Michaels from comment #84) > if this sent-1, sent-2, sent-3 is the operation I think it is (splitting > sent intomultiple 2GiB folders), what you will see in TB without any changes > is what I have gotten with my TB mail splitter. > once tb is up, folders will appear (Sent-1, Sent-2, Sent-3,etc) one for each > of the split folders. depending on how many of these you have, if you have a > lot of them,it will of course take up a lot of visual space in TB - except > for the original Sent box, it will only take up one file-looking mailbox > thing. see here for a visual: > https://a.fsdn.com/con/app/proj/tbmailsplit/screenshots/tbirdmailsplitGUI- > tbirdshot.PNG Sorry, but this is a kludge at best, and not a solution. We might as well go back to one file per message, and fill the directory with bazzillions of files (see www.nongnu.org/ext2-doc/ext2.html for better info on this). The inode hash table for file (and dir) name entries is rumored to accomodate at least 1,200,000 dir entries, but probably more than that. I really would not want to have a dir full of millions of files. If mozilla cannot take the folder file size to accomodate 64 bits, then it is time for a whole new design and development.

(In reply to JD from comment #83) > (In reply to WADA from comment #82) > > (In reply to JD from comment #79) > > > -rw------- 1 jd jd 4287923277 Mar 20 19:16 Sent > > > > 4287923277 == 3 GB + 1017 MB + 289 KB + 77 Bytes > > 4GB - 4287923277 == 6 MB + 734 KB + 947 Bytes > > "What occured" is explained by the message. > > Do as the message says. > Wrong! > My folders are totally compacted (and repeatedly). No message was deleted since initial of the Sent folder? If nothing was deleted from the Sent folder, it's never Compacted, because there is no need to do Compact. If at least one message is deleted from the Sent folder, Compact is invoked by auto-compact or Compact or Compact Folders. If Compact is actually invoked for the Sent file, I think bug 794303 should occur by Compact if nstmp file size exceeds 2GB. You refer to www.nongnu.org/ext2-doc/ext2.html. Do you use Linux? If so, and if "no problem in Compact of folder near 4GB" in your environment, bug 794303 may be problem in Tb's per OS module or problem depends on OS's file I/O module. > My folders are totally compacted (and repeatedly). Was file size of the Sent(near 4GB) actually reduced by Compact of Sent folder when some mails are deleted from the Sent folder(file size should be reduced by Compact) and Compact of Sent folder is executed?

(In reply to Knut Singer from comment #81) > And finally their bugreport is marked as "solved" and "Dupe of xxx" and the 4 GByte limit still exists. I checked some dup'ed bugs, and I knew that some bugs were dup'ed to this bug simply because that symptom is 2GB/4GB limit relevant and/or that symptom is 32bits integer use relevant. It was perhaps caused by that this bug's summary was initially too-generic "Remove 4GB limit in Tb". (If such too-eneric summary, it should have been changed to meta bug for tracking...) However, request of "Remove any problem around 4GB size limit by single bug" is never helpful for problem analysis and bug fixing by developer at bugzilla.mozilla.org. Because "64bits integer use for messageOffset" is already done by this bug(bug 462665), "sorting out of issues around 4GB limit" is easier than before. And, "what is needed to remove the 4GB limit" is clearler than before by next followup bugs. > bug 789679 (folder full message by 4GB-4MB-1MB check) > bug 640371 (even after protection from "Over 4GB", "Over 4GB" occurs) > bug 794303 (Compact fails, if file size is greater than 2GB) > bug 793865 (32bits integer use what prohibits removing 4GB limitation) > bug 634544 (problem in POP3 download when file size exceeds limit) I have duped some bugs which was dup'ed to this bug to other bugs such as above followup bugs, and changed bug summary of this bug to reflect "this bug is for 64bits integer use in Backend for messageOffset" to summary.

Knut, JD, is the documentation at http://kb.mozillazine.org/Limits_%28Thunderbird%29#Folders_and_messages not sufficient. Can you update it or tell if there is anything missing? Thanks. WADA, I don't think the bug summary change is good. The offset is 64bit since long time.

(In reply to :aceman from comment #90) > The offset is 64bit since long time. Since long time from when? Far before date when this bug was changed to fixed(2012-02-24) in module used for Berkley Mbox file(never IMAP offline-use folder, never maildir)?

Looks like since bug 532323.

Hi, (In reply to WADA from comment #85) > (In reply to Knut Singer from comment #81) > > This becomes sharper, as since some years ago every new account is added > > with the feature of "synchronizing" by default > > Please never discuss based on wrong assumption. > > (i) The "Synchronization" you call is IMAP only feature. > But this bug is for msgFolder of "MorkDB + Berkley file" for POP3 > account and pseudo account of "Local Folders". > > Why can the IMAP feature be relevant to this bug? If "Synchronizing" is enabled (and poorly this is done by default when setting up an IMAP account (without asking or informing the user about this)), then not only headers are synchronized, but the complete mails. So even with an IMAP account the user runs into problems, when the inbox folder reaches 4 GByte and he didn't know about this complet synchronizing function. My advice simply is, unless this 4 GB limitition is not solved, NOT to enable synchronisation of IMAP accounts by default. > Contributers in bugzilla.mozilla.org such as Developers, QA peoples, are all > volunteers. > If you think so, why you don't do it with your own hands? > You　do not do anything by yourself in just asks for something to other > people all the time? I can't think so. I respect the work of the volunteers in gratitude - really. For myself I cannot do something at the code "with my own hands" as I'm not able to program things like this. I really can't - I think like most other users. And I even did not critizise the limitation of 4 GB of mbox-files. What I critizised is the fact, how this limitation is dealt with in bugzilla. Always, year for year, users who run into this limitation, have to make through the same procedure, as I described in comment 76. THAT was my point of critic. BTW: Why - as long this limitiation will exist - make it an official, well communicated limitation of TB. Then all the bug-reports about this are unneccessary. Instead make some useful programming to inform the user about folders that will become too large in advance (like when reaching the quotas of the IMAP-Folders) AND do not set the "synchronising" feature for IMAP accounts as default.

Sorry, the above last paragraph should be: BTW: Why *not* ...

(In reply to Knut Singer from comment #93) > My advice simply is, unless this 4 GB limitition is not solved, NOT to > enable synchronisation of IMAP accounts by default. "Size limitation problem due to 32bits integer use for offset value in IMAP offline-store" is already resolved by bug 532323(Target Milestoe:Thunderbird 3.1a1, so landed on Tb 3.1) Why "2GB/4GB limit of IMAP offline-store problem what was exposed by Tb 3.0 but is already resolved in Tb 3.1" be relevant to this bug? > make some useful programming to inform the user about folders that will become too large in advance (like when reaching the quotas of the IMAP-Folders) (snip) Message of "Folder full, Delete mails, Compact folder, please" is the "informing to user by Tb about 'reached size limitation'" in local mail folder case. What is reason why you still repeat writing complaints about already resolved problem in this bug? I agree with you that the message is better improved. Open separate bug for message if you think change of message is mandatory, please.

(In reply to :aceman from comment #92) > Looks like since bug 532323. Thanks for pointing. (In reply to Knut Singer from comment #93) > What I critizised is the fact, how this limitation is dealt with in bugzilla. To Knut Singer: Changes by "Fixing of this bug on local mail folder in Tb 12", which was actually done in patch for bug 402392(pluggable message store support), is similar one to changes by bug 532323 on IMAP offline-store file in Tb 3.1. Quick history. "2GB/4GB limit of IMAP offline-store file" was, as you know, unfortunately exposed widely by "auto-sync enabled by default in Tb 3.0" and by "6GB or 10GB [Gmail]/All Mail" of Gmail IMAP", then "2GB/4GB limit of IMAP offline-store file" was removed by bug 532323 and fix was landed on Tb 3.1. But "2GB/4GB limit of local mail folder" couldn't be touched because "removing the limit in local mail folder" is very very difficult as seen in followup bug of 789679. In tough work for development of "pluggable message store file" which is prerequisite of "maildir support", "removal of 4GB limit due to 32bits integer for offset of local folder in Backend" was fortunately achieved by similar(but bigger) changes to bug 532323 on IMAP offline-store file. However, it could be in "Backend only". See, read, and understand well followup bug 789679 and bugs set in dependency of that bug, please. As you know, "Remove 4GB limit in local mail folder from *anywhere* of Tb" is not achieved yet. It may be nearly impossible because available time of volunteers is very limited, even though any bug of software can be resolved in principal. Writing your complaints repeatedly in *BUG* report at bugzilla.mozilla.org is very easy, and no one can prohibit or forbid it. However, it never helps developer to resolve actual bug in Tb. Currently known big issues even after "<4GB support in Backend" are: - Compact doesn't look tolerant with file larger than 2GB - messageKey which is 32bits unsigned integer may overflow, if following happens. when file size < 4GB-4MB-1MB, mails are downloaded via POP3, and file size exceeds 4GB by a downloaded mail, then more than 256 mails are downloaded additionally by the single POP3 download. Because Tb looks to have logic like "if messageOffset exceeds 0xFFFFFF00, use messageKey=highest messageKey + 1", messageKey value may over flow by the more than 256 mails. This is surely rare, but there is no guarantee that it never occurs. - There are "32bits integer use hard to transfer to 64bits integer". Unless these big problems will be resolved, "file size greater than 4GB" will surely produce critical problem of folder corruption, data loss etc. These are reasons why size limitation at "4GB-4MB-1MB" is enabled even after "<4GB file" is supported by Backend. These are currently processed in followup bugs. Please help developers to resolve problem in such followup bugs. For documentation. There are some good places for helpful documentation. Mozilla Wiki, MozillaZine Knowledge Base, ... Please write helpful documents at such places.

(In reply to Jim Michaels from comment #84) > if this sent-1, sent-2, sent-3 is the operation I think it is > (splitting sent intomultiple 2GiB folders), > what you will see in TB without any changes is what I have gotten with my TB mail splitter. You are correct. Procedure I described is "split to less than 2GB folders", and is almost same as what Tb mail spliter does do. > readme.txt of Tb mail splitter > http://sourceforge.net/projects/tbmailsplit/files/2.15/readme.txt/download "Why Tb mail splitter was mandatory" was; 1) During "Max 2GB by 32bits signed integer" era, if once "Over 2GB" happens, there was no way to recover data over 2GB. Bug 640371 is a cause of accidental "Over 2GB". Even if some healthy mails are copied and deleted(flagged EXPUNGED), Compact fails due to "Over 2GB". 2) Tb mail splitter was perhaps implemented to resolve above problem. By the splitter, each file size is kept less tha 2GB, so any mail can be accessed normally. 3) Even after limit was changed to "Max 4GB by 32bits unsigned integer", once problem of "over 4GB" happens, there was no way to recover data over 4GB. Bug 640371 is a cause of "Over 4GB". Compact fails due to "Over 4GB". 4) So, Tb mail splitter was still neeeded for recovery from Over 4GB. After Tb 12, "manual spltting to less than 2GB" is possible even when "Over 4GB" happens. 5) By Tb 12, "Max 4GB" in Backend was removed by "64bits signed integer use". So, even if "Over 4GB" happens, "data over 4GB" is accessible and data loss due to "Over 4GB" won't occur. "Move mails to other folder" is now possible even if "Over 4GB" data. 6) However, bug 789679 occurs because Tb has still "Max 4GB" limitation to avoid critical problems due to "Over 4GB". So, "reducing file size far smaller than 4GB" is needed. Improvements in Tb 12 or newer in this case is: Before Tb 12: Tb mail splitter is needed to recover from Over 4GB. After Tb 12: Manual mail move/manual Compact by Tb is possible. And, Tb looks to still have problem of bug 794303 in Compact. So, manual keeping of "flle size after Cmpact < 2GB" is needed before Compact. This is procedure I described. Needless to say, even after Tb 12, Tb mail splitter is very useful in recovery from "Over 4GB", "Folder full due to near 4GB", and "Compact failure due to larger than 2GB". I can't understand why can someone return response such as "but this is a kludge at best" to your advice or recomendation or help...

(In reply to Jim Michaels from comment #84) > it will only take up one file-looking mailbox thing. see here for a visual: > https://a.fsdn.com/con/app/proj/tbmailsplit/screenshots/tbirdmailsplitGUI-tbirdshot.PNG You looks to send "2GB mails per a day" or "2GB mails per several days"... Because you use Tb mail splitter, you looks able to use batch type operatin while Tb is not running. If so, following by batch may help your handling of many 2GB archived Sent folders. (1) Sent => Sent-xx-yy-zz by Tb mail　ｓｐlitter (2) By batch(small Script), Move ...\Mail\<Server>\Sent-xx-yy-zz to ...\Mail\<Server>\Sent-Arcive.sbd/yyyy.sbd/mm.sbd/xx-yy-zz with creating ( ____ == null, no file extension ) ...\Mail\<Server>\Sent-Arcive____ ...\Mail\<Server>\Sent-Arcive.sbd\yyyy____ ...\Mail\<Server>\Sent-Arcive.sbd\yyyy.sbd/mm___ You can see them like; Sent-Archive yyyy mm xx-yy-zz <= less than 2GB, archived sent mails (3) In Tb 12 or newer, Sent-xx-yy-zz splitted by Tb mail splitter can be "less than 4GB-4MB-1MB", if nothing will be added to Sent-xx-yy-zz or Sent-Archive/yyyy/mm/xx-yy-zz, and if there is no need to do Compact of Sent-xx-yy-zz or Sent-Arcive/yyyy/mm/xx-yy-zz. (4) If Tb 12 or newer only, following may be easier for "archive big Sent folder" in your case. Before "Over 4GB" happens by bug 640371, and before warning of "Folder full" is shown, Move ...\Sent and \Sent.msf to ...\Sent-Archive.sbd\yyyy-mm-dd and yyyy-mm-dd.msf then create null file of ...\Sent

(In reply to JD from comment #87) > We might as well go back to one file per message, and > fill the directory with bazzillions of files > (see www.nongnu.org/ext2-doc/ext2.html for better info on this). > The inode hash table for file (and dir) name entries is rumored > to accomodate at least 1,200,000 dir entries, but probably more > than that. > I really would not want to have a dir full of millions of files. Yes, a one of biggest problems in "one file per a mail" is "possible severe performance related issue when very large number of entries are held in a directory". Even when "merely 10000 files per a directory", getting directory data of all mails" is usually takes sufficiently long. If "0 / 0.msf" to "9999 / 9999.msf" is created where "N" is null file under Tb's mail directory, it's perhaps miserable :-) If static mail folder(no frequent append of mails, no frequent delete of many mails, no need of frequent Compact), current "MorkDB + Berkley Mbox file" is not so bad, except "relatively long time in folder open due to big .msf file". And, as seen in Tb 12 and newer, "MorkDB + 4GB Berkley Mbox file" is sufficiently fast in many situations, and is torelant with accidental "Over 4GB". Problem often seen is rather "slowness in mail folder open when very many mails are held in a mail folder". If "Backup of big mail folder on file server"(big in both file size and number of mails), "MorkDB + one huge Berkley Mbox file" may be better than "MorkDB + huge number of small files in a directory", because cost of "file open close per a file" is not negligible. If so, "backup of big mbox on DVD" is practical, and is possible merely by adding support of "read only .msf file", and if "unmount of Mbox" is supported by Tb, DVD can be unmounted without termiation of Tb. This is one of reasons why I tested Tb's behaviour around 4GB and reported bug 640371 and bug 794303. In pricipal, to support "Berkley Mbox file far larger than 4GB" by Tb, (A) "simply remove code who rejects file larger than 4GB-4MB-1MB" and (B) "simply remove 32bits integer use for offset at everywhere" is suffienet, because "local file for Mbox greater than 4GB" is already fully supported in Backend code of Tb by enhancement for "pluggable message store". (A) was very simple, however, (B), prerequisite of (A), was far harder than I guessed. See, read, and understand followup bugs, please. > If mozilla cannot take the folder file size to accomodate 64 bits, > then it is time for a whole new design and development. You don't read comment #78? You always add comment to bug without reading previous comments? Although "one file per mail" is not enabled by default yet because it's still in progress, "one file per mail" is already usable by developer's great efforts. Try it, and report bug if you find problems in it, and help developers to complete "one file per a mail" support, please. Because; - Apple used Berkley Mbox(multiple mails in a file) in Apple Mail V1. (Import from Berkley Mbox file was officially supported.) - Then, Apple quickly changed to "one file per mail" in Appl Mail V2. (Import from Berkley Mbox file was officially removed by V2. ) (Apple's offcial way to import mails in Berkley Mbox file was: ) ( Upload mails to IMAP server, and access by Apple V2 via IMAP.) - But, I don't know critical perfornce problem due to "one file per a mail" in Apple Mail. I guess that PC mailer user who deeply loves "1,000,000 mails or far more mails per an Mbox" is very rare. If my guess is right, "critical performance issue due to one file per a mail" will occur on very limited users only and is edge case. Please test Tb's new "one file per a mail" feature.

(In reply to JD from comment #83) > To close this issue as VERIFIED FIXED is at the very least: LAUGHABLE if not lamentable. However, following proposition　is always true, except when "somewhere"==="anyhhere" ;-) "Remove 4GB limit (at somewhere) in Tb" != "Remove 4GB limit at anywhere in Tb" Because status=FIXED for original bug summary of "Remove the 4GB mailbox limit" is perhaps funny for some peoples, I changed bug summary("at somewhere" == "in Backend") to reflect actual changes done when this bug's status was set as FIXED. Proposition of "Remove 4GB limit at anywhere in Tb" is too huge to resolve by single bug report in bugzilla.mozilla.org. "Remove 4GB limit at anywhere in Tb" is now being tried in followup bug 789679 and some bugs. Please help developers to resolve Tb's bug, instead of merely adding comments of complaints only to bug in bugzilla.mozilla.org.

my averaging has shown to me that 3,415,696,385 bytes of emails (multiplied from 18 emails) has the following stats: Sat 03/23/2013 0:13:36.50||E:\t\inbox_split\tt|>copy ..\inbox2* i2 ..\Inbox2 Overwrite i2? (Yes/No/All): y ..\Inbox2-0000000002_Date2013-01-08_29bf95e0-5a0d-11e2-a39b-00111118fbdf ..\Inbox2-0000000003_Date2013-01-08_3090cc5c-5a0d-11e2-a39b-00111118fbdf ..\Inbox2-0000000004_Date2013-01-08_3090cc5d-5a0d-11e2-a39b-00111118fbdf ..\Inbox2-0000000005_Date2013-01-08_3c9b87e1-5a0d-11e2-a39b-00111118fbdf ..\Inbox2-0000000006_Date2013-01-08_42b78bf1-5a0d-11e2-a39b-00111118fbdf ..\Inbox2-0000000007_Date2013-01-08_48dab70c-5a0d-11e2-a39b-00111118fbdf ..\Inbox2-0000000008_Date2013-01-08_4ed7bc97-5a0d-11e2-a39b-00111118fbdf ..\Inbox2-0000000009_Date2013-01-08_54e0addf-5a0d-11e2-a39b-00111118fbdf ..\Inbox2-0000000010_Date2013-01-08_54e0ade0-5a0d-11e2-a39b-00111118fbdf ..\Inbox2-0000000011_Date2013-01-08_60d391ea-5a0d-11e2-a39b-00111118fbdf ..\Inbox2-0000000012_Date2013-01-08_60d391eb-5a0d-11e2-a39b-00111118fbdf ..\Inbox2-0000000013_Date2013-01-08_6ce5747a-5a0d-11e2-a39b-00111118fbdf ..\Inbox2-0000000014_Date2013-01-08_72ff1631-5a0d-11e2-a39b-00111118fbdf ..\Inbox2-0000000015_Date2013-01-08_79296857-5a0d-11e2-a39b-00111118fbdf ..\Inbox2-0000000016_Date2013-01-08_7f4c9372-5a0d-11e2-a39b-00111118fbdf ..\Inbox2-0000000017_Date2013-01-08_857baa4a-5a0d-11e2-a39b-00111118fbdf ..\Inbox2-0000000018_Date2013-01-08_857baa4b-5a0d-11e2-a39b-00111118fbdf ..\Inbox2-0000000019_Date2013-01-08_919e3d49-5a0d-11e2-a39b-00111118fbdf 1 file(s) copied. Sat 03/23/2013 0:16:33.48||E:\t\inbox_split\tt|>dir i2 Volume in drive E is caviar_e Volume Serial Number is 68D5-651A Directory of E:\t\inbox_split\tt 03/23/2013 12:16 AM 3,415,696,385 i2 1 File(s) 3,415,696,385 bytes 0 Dir(s) 270,145,454,080 bytes free Sat 03/23/2013 0:17:17.00||E:\t\inbox_split\tt|>prtime -exec E:\prj\tbmailsplit\tbmailsplit-2.15\32\tbmailsplit --count i2 \; operation: count "i2". operation: count "i2". please wait for count (this can take up to 3 minutes)... please wait for count (this can take up to 3 minutes)... 33792 emails 33792 emails elapsed time: 0 days, 00:03:03.969 Sat 03/23/2013 0:20:43.06||E:\t\inbox_split\tt|> notice that the copy op took 3 minutes on my old XP box. so did the count operation. let's try a split. first, let's break out ttcalc. 3,415,696,385 bytes/33792 emails=3,415,696,385/33792=101,080.030,332,623bytes/email average. so to get 1.9e9 instead of 2GiB chunks which leaves me with 247,483,648 bytes of slop, I want to split them on 1.9e9/(3,415,696,385/33792)=18,796.986 emails or 18796 emails. unless you are attaching CD or DVD ISOs or scanned images (300*300*11*8.5=8,415,000 bytes - there goes your 4MB chunks theory, people do this sort of thing unknowingly and wonder why their email is so slow!) in the email, you probably won't have a problem with this. by the way, the equivalent of the copy op I did to merge the files in linux/mac/*nix is cat. if the files are numbered in sequence, the OS will sort them alphanumerically, so they must have zero-padding. that means having a pre-allocated number of digits! I suggest you use an alphabetic code instead of digits and have a function to generate the string from an int64_t or uint64_t. of course, it could just be plain hex also,but alphanumeric using 32 or 64 cross-platform-filesystem-compatible character sets would have the file length signature shorter by 1/4. since everything is line-based, getting the file positions where the split points are can speed up splitting, since you don't have to start over from the beginning over and over. this is in regards to MBOX. plus, MBOX is compressable, should you want to do a backup/restore of your mail folders (a nice option if you want to backup into DVDs, and 7-zip commandline can archive into volumes, but it's not good at absolute filepaths). so let's split: Sat 03/23/2013 0:17:17.00||E:\t\inbox_split\tt|>prtime -exec E:\prj\tbmailsplit\tbmailsplit-2.15\32\tbmailsplit --count i2 \; operation: count "i2". operation: count "i2". please wait for count (this can take up to 3 minutes)... please wait for count (this can take up to 3 minutes)... 33792 emails 33792 emails elapsed time: 0 days, 00:03:03.969 Sat 03/23/2013 0:20:43.06||E:\t\inbox_split\tt|>prtime -exec E:\prj\tbmailsplit\tbmailsplit-2.15\32\tbmailsplit --split i2 18796 \; operation: split "i2" into 18796 pieces. operation: split "i2" into 18796 pieces. please wait for count (this can take up to 3 minutes)... please wait for count (this can take up to 3 minutes)... 33792 emails 33792 emails please wait for splitting, this can take up to 6 minutes... chunk size is recalculated at 2 emails, therefore actual number of splits/chunks will be 16897. please wait for splitting, this can take up to 6 minutes... chunk size is recalculated at 2 emails, therefore actual number of splits/chunks will be 16897. ...doing split number 16897 of 18796 The Thunderbird .msf summary file "i2.msf" doesn't seem to exist, but that's OK. done. elapsed time: 0 days, 00:12:00.875 Sat 03/23/2013 0:46:31.04||E:\t\inbox_split\tt| I am not using the technique of remembering the file positions. though I should.

heh, the chunk size of those splits was awful small... I had them reversed. the parameter of the tbmailsplit program is number of splits you want, not chunk size. so I ended up with 18786 files, not 3. my algorithm does not use the file position method. first the emails are counted. that's the 1st 3 minutes. if you already know the count, you are off to a good start. from there you can calculate chunksize and split size, whatever. chunkSize = (emailCount / numSplits) + ((emailCount % numSplits)>0?1:0); if you are going to split chunks, you need to know about the off-by-1 errors you can get into doing that. this is from code I wrote and applies to splitting in general.

replacement for above equations: chunkSizeRecalc = (emailCount / numSplits) + ((emailCount % numSplits)>0?1:0); numSplitsRecalc = (emailCount/chunkSizeRecalc) + ((emailCount/chunkSizeRecalc)>0?1:0);

this is a kludge at best.

Jim, do you have a test setup where you could test a development build of Thunderbird on test messages to find out if there are any problems at >4GB?

hmm. I have my business mail on tb. I could easily move out the mail folders and stop the internet so I wouldn't get any new messages, but my config is still in there. how would I do this without affecting my current setup? I can provide you with my code for double.cmd, a script I use to multiply the size of my inbox. you just get out your calculator or ttcalc, paste in the size of your file from the cmd shell, and multiply it by 2^poweryouwant. I use 10 for power for inbox sizes of 7MB. -----makeinboxpowerof2.cmd rem handle help switch if /i @%1@ equ @@ goto help if /i @%1@ equ @/help@ goto help if /i @%1@ equ @/h@ goto help if /i @%1@ equ @/?@ goto help if /i @%1@ equ @-help@ goto help if /i @%1@ equ @-h@ goto help if /i @%1@ equ @-?@ goto help if /i @%1@ equ @--help@ goto help if /i @%1@ equ @--h@ goto help if /i @%1@ equ @--?@ goto help if @%2@==@@ goto err if @%3@==@@ goto err rem set targetexponent=12 set targetexponent=%3 rem set outdir=\t set outdir=%2 del %outdir%\inbox %outdir%\inbox8G %outdir%\z rem set indir=\Documents and Settings\Jim Michaels\Application Data\Thunderbird\Profiles\md4bzvt7.default\Mail\mail.renewalcomputerservices.com set indir=%1 copy /b "%indir%\inbox"+"%indir%\drafts"+"%indir%\trash"+"%indir%\sent" %outdir%\inbox copy /b %outdir%\inbox /b %outdir%\inbox8G /b set /a count+=1 set count=0 :lupe if "%count%"=="%targetexponent%" goto exitlupe copy /b %outdir%\inbox8G+%outdir%\inbox8G /b %outdir%\z /b move /y %outdir%\z %outdir%\inbox8G set /a count+=1 dir %outdir%\inbox8g goto lupe :exitlupe del %outdir%\z dir %outdir%\inbox8g goto end :err @echo ERROR: there need to be 3 arguments. see --help goto end :help @echo makeinboxpowerof2.cmd - take inbox and create outdir\inbox8G the size of (the size of inbox)*2^^integerTargetExponent @echo usage: makeinboxpowerof2.cmd "inboxdirNoEndingBackslash" "outdirNoEndingBackslash" integerTargetExponent @echo suggested value for integerTargetExponent is between 10 to 12 for an inbox of size 7MB. @echo do a dir of your inbox, and copy+paste into ttcalc and use ttcalc to calculate the resulting size like this: 7,607,491*2^^10 @echo it is highly suggested you use double quotes around inboxdirNoEndingBackslash and outdirNoEndingBackslash. @echo goto end :end this can build you a mulltiGB set file by making a copy of and multiplying the size of the inbox you already have.

You can create a new profile (thunderbird -P) and then create an account in there and generate the big folders with your script. You should still backup your working profile to safe place. And then you must make sure to run the proper TB version on the proper profile. Or can you use another user (login) on your machine? That would be much safer.

how would I handle mixing of programs? would I be using an installer, or is this standalone? what is the ftp URL you want me to use? I was basically concerned about my settings if I upgraded the program. when I need to go back to the production version if I need to. I hope the profile is what handles this, yes?

You will use a zipped package of Thunderbird that you just extract into a separate folder. It will not overwrite your main TB installation. You run this separate TB and create a new profile for it with separate settings and mail data. Here are the steps: https://support.mozillamessaging.com/en-US/kb/using-multiple-profiles Just never forget to call this dev version with the -ProfileManager command line argument.

nightly? what FTP address from the mozilla FTP archives at ftp://ftp.mozilla.org?

No, we have a special build. Please join bug 793865 and we tell you the link when it is created.

if it's any help, fsetpos and fgetpos can handle 64-bit positions. fseek and ftell are 32-bit.

a comment on 1-file per email: I have discovered through testing that when you get to 16k files in a directory, you start getting really slow "dir" commands, file searches, and accesses to that directory, that appear like lockups,because the OS is busy processing the $MFT. not sure what linux/mac does.