User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:6.0) Gecko/20100101 Firefox/6.0 Build ID: 20110811165603 Steps to reproduce: Upgraded Thunderbird from 3.1.x to 6.0. I'd go back to 3.1.x, but cannot find it on the site. And it would probably be futile anyway, as the .msf files are gone. I've been a very long time user, and not seen anything this catastrophic since the earliest days. THIS IS TERRIBLE!!! Actual results: All my .msf files disappeared, or were changed to zero length. * Folder message counts no longer display. * Searches no longer work. * Recent folder highlighting no longer works. * Message filters seem to work for some things, but not others (leaving the messages behind). I can get the .msf to update properly by clicking on each individual folder. Not practical, as I have many thousands of folders. See screen shots of a small number of my folders, and the related files. Expected results: If there's some conversion going on, it should be handled promptly and internally, without user intervention. Cease distributing Thunderbird 6.0!

Here's the .msf files for these folders. As you can see, the only correctly sized .msf is the mp-c.zz folder currently clicked upon (and the enclosing empty apple folder). Note that acd-bugs.all has no .msf at all! Please *STOP* distributing Thunderbird 6.0 until this is fixed.

Probably worth noting that I've got 7.59 GB of email sorted into several thousand folders, in many cases going back 20+ years. Are you running into some kind of pointer sized or script timing problems? Loss of today's email would be very annoying, but the older folders are all backed up.

You're the only person who has reported this, which makes me suspect it's something to do with your setup (e.g., a virus checker zeroing out the files). In any case, a simple way of causing a rebuild of the .msf files is to do a search on the account (edit menu, find, search messages) and pick the account as the search scope, and do whatever search you want. That will cause a reparse of all the folders in the account. But it sounds like some program is interfering with Thunderbird's ability to save .msf files.

several thousand folders might end up running into limitations on the number of open file handles (that's true on linux, not true on windows, not sure about the mac). But we shouldn't be opening each file.

Already tried search messages before reporting the bug, which did reparse in older versions. Now, doesn't even replace the missing ones, let alone update the zero length ones. In 6.0, it only searches those with good .msf. I can search for a known From name, and it won't find most of them -- only those folders I've clicked on once today, none of the subfolders. No, I don't even run a virus checker. And this is a regression from 3.1.12. All those .msf files existed mere hours ago. Running 6.0 deleted them.

(In reply to David :Bienvenu from comment #4) Tried the search from the account screen again, this time with !@#$%^&*&^%$#! -- guaranteed not to find anything anywhere. Still didn't refresh. Here's a copy of a shorter subfolder file list; still has mostly missing .msf files. I've not yet clicked on a subfolder here (in IRTF), so it hasn't generated the zero length .msf files. I have 3-5 level deep subfolders (IRTF is the same level as Inbox), and 6 accounts plus local. The previous file example was apple, which is a subfolder of Developer, which is the same level as Inbox. Note that it updated all the .sbd dates earlier today. It just didn't touch them during search, and didn't handle missing .msf files in subfolders. Whenever I've clicked on a subfolder elsewhere, it generates the zero length .msf files, and takes quite awhile to do it! It displays the account screen while building zero length .msf files.

Are those files the same names as the the folder names, or are they munged names? They look like real names (or, rather, they don't look like our munged names, which are names we use when the file name contains illegal chars). I really don't know what's going on - you could generate a MSGDB log and from that we could see which db's it tries to open and what errors it encounters - https://wiki.mozilla.org/MailNews:Logging Doing a search should show us trying to open db's.

OK. I've made the sh file, as directed. I've killed off panacea and the FolderTree, and all the .msf files: ~ bill$ find ~/Library/Thunderbird/Profiles/*.default/ -name *.msf -exec rm -f {} \; ~ bill$ sh ~/MSGDB I've run it three times, then run the whole series twice. Utterly reproducible. Never finds anybody. The interesting thing I've discovered is: (1) the first run makes 4K .msf files only for any empty mail files. (2) the second run makes zero length .msf files for most files missing one (but sometimes skips some), then deletes the zero length .msf files on quit. (3) the third run makes the zero length .msf files again, but this time keeps them all on quit. I didn't know that earlier today, as I'd only quit and restarted once. (4) every .sbd file seems to be modified on start, and again on quit. I have no idea whether this information is useful.

I'll not bother with the 3rd run after search and after close. Hopefully, those log files tell you something? (That 1st is a lot longer than the nearly identical 2nd and 3rd.)

Is there anything more I need to test before switching back to 3.1.12?

Tried to go back to 3.1.12. Didn't work. Starting from rm .msf as above, 3.1.12 also only adds .msf to empty mail files. Search doesn't regenerate .msf files, nor does it find anything. What 3.1.12 doesn't do is zero out existing .msf files, even after 3 runs. So, my mail has been good for a decade of Thunderbird (and Netscape before it), because I already had good .msf files. I'm well and truly screwed. And I'm telling every news organization that will listen. Don't upgrade to 6.0!

Now, I've also reported that the .msf isn't regenerated on search (bug 682721) or on POP (bug 682725). This bug is focused on the evilness of replacing with zero length .msf, and/or deleting .msf during search and close. Please *STOP* distributing Thunderbird 6.0 until this is fixed!!!

bug 682731 filed for search not rebuilding the missing summary files. I don't know what caused your summary files to become invalid/missing in the first place, which is this bug, I suppose.

kinda agree with david - msf files aren't known to just disappear by something caused in Thunderbird.

(In reply to Wayne Mery (:wsmwk) from comment #26) > kinda agree with david - msf files aren't known to just disappear by > something caused in Thunderbird. Yet they were fine for years, then disappeared immediately after I upgraded from TB 3.1.12 to TB 6.0. I noticed on my very first run! I did a second run trying search to rebuild. When that didn't work, I filed this bug. My guess is that they were replaced by the mass of zero length .msf (see "2nd run IRTF before search" or "3rd run IRTF before search"), then deleted after the search or the quit (compare with other IRTF attachments). But why did they get replaced by the zero length .msf in the first place? Any recent change to that code? I'm guessing somebody changed from C to a script, and it didn't have correct permissions? Or you're using json or something with an integer overflow problem? Reminder, there are several thousand folders 5 levels deep, and nearly 8 GB of data. Both are possible integer overflow possibilities. It wouldn't be the first time. I'd already reported the gloda integer overflow Bug 550764, and have never turned gloda back on.... On July 6, when I upgraded to 5.0, it didn't crash on the administrative account (which has no email files), but crashed immediately on startup on this user account. It didn't get far enough in startup to even touch a mail file (I checked). I quickly reverted to 3.1.11 without problem. Before that, my previous TB crash was July 24, 2009. A good long time!

> but crashed immediately on startup on this user account. Please post the crash report ID see http://support.mozillamessaging.com/en-US/kb/Mozilla-Crash-Reporter#w_viewing-crash-reports > Upgraded Thunderbird from 3.1.x to 6.0. so If I understand correctly, you attempted version 5 once or twice, reverted to version 3.1.11, then jumped to version 6 (skipping 5). Is that correct? have you tried reproducing, with a test profile, jumping from version 3.1.x to 6.0? > I'd already reported the gloda integer overflow Bug 550764, and have never turned gloda back on I don't see that 550764 (which is no doubt a fixed duplicate) had anything to do with int overflow. (=>critical, blocker is for bugs which block development/testing - this issue isn't that pervasive)

Do you run a lot of extensions ?

(In reply to Ludovic Hirlimann [:Usul] from comment #29) > Do you run a lot of extensions ? None.

(In reply to Wayne Mery (:wsmwk) from comment #28) > > but crashed immediately on startup on this user account. > > Please post the crash report ID > see > http://support.mozillamessaging.com/en-US/kb/Mozilla-Crash- > Reporter#w_viewing-crash-reports > The 5.0 startup never got far enough to do Mozilla crash report. The Apple CrashReporter did send a report to Apple, the ID is thunderbird-bin_7335ED35-7902-51EB-8A38-CEEB9D4B914B (presuming you get and archive the Apple traces somewhere). The dumps on this machine have long since rolled over or been removed. > > Upgraded Thunderbird from 3.1.x to 6.0. > > so If I understand correctly, you attempted version 5 once or twice, > reverted to version 3.1.11, then jumped to version 6 (skipping 5). Is that > correct? > Repeating myself in excruciating detail: Downloaded and attempted 5.0 once on the empty administrative account (to get rid of the "this was downloaded from the Internet" warning), and then once on the user account (it crashed on startup). Clicked OK to send the crash report to Apple. Threw away the administrative Library/Thunderbird folder (and Caches and Preferences). Reverted to 3.1.7 (the last install file I've got), then updated back to 3.1.11 via Help Check for Updates. Restored the user Thunderbird main folder files from backup (leaving the Mail subfolder in place untouched). Checked the Caches and Preferences, but the last modified date hadn't been changed. Posted a Macintouch item to warn folks away from 5.0. They stuck it under Eudora, but no matter.... Updated to 3.1.12 via Help Check for Updates (on admin). Worked fine. Noticed no problems on the user account. Waited 5 days until weekend. Downloaded 6.0. Ran once on administrative as usual. Ran once on user. Noticed that the unseen message count (in parenthesis) wasn't showing. Quit first run. Drilled down into my file folders, noticed most .msf files were missing. Gone! Nothing! Nada! Ran again on user. * Did search to update the .msf files (as I remembered we used to do long ago), figuring this was just a normal part of the update process. Nothing. * Tried clicking on Inbox. Noticed that it took a very long time (oddly, the account screen appeared, then switched back to the usual list). Some .msf files showed up, but most were zero length. * Tried clicking on several other folders. Same result. * Tried downloading messages. No improvement, but the file dates were updated in the usual folders. * Created some new folders under Inbox. Moved 15MB of earlier Inbox messages into them, cutting Inbox down to 2 MB, thinking that it might be a length problem. Tried search on Inbox and subfolders. No joy. * Tried search on various subfolders, thinking it might be a scripting timeout problem on the large Inbox.sbd files and associated subfolders. No joy. * Took a screen shot showing that most of the unseen message counts were missing. * Quit second run. Posted this bug with screenshot. Took another screen shot of the files. Posted. Spent hours trying to figure out differences in subfolders. Posted many more screenshots. > have you tried reproducing, with a test profile, jumping from version 3.1.x > to 6.0? > No. Spent rather a lot of time reproducing the problem (repeatedly) with my usual profile, as you can see from all the attachments. Please walk through the attachments, comparing file lists visually. I could get the profile from backup, make a new one, and reproduce it, but why bother? I've already spent most of a weekend reproducing it above. Apparently, you don't think this is a real problem??? > > I'd already reported the gloda integer overflow Bug 550764, and have never turned gloda back on > > I don't see that 550764 (which is no doubt a fixed duplicate) had anything > to do with int overflow. > You might try actually reading Bug 550764. It's not marked fixed duplicate. Somebody marked it "incomplete", because I wouldn't spend 48 useless hours going to a backup and running through the upgrade steps to see whether it had been fixed in 3.1.3. I'd argue that was a misuse of the incomplete marking -- but what do I know? :-( It was confirmed as an integer overflow at 2 GB. Whether you've fixed that, I have no idea. I'm not willing to run gloda to find out.... Why, you ask? Nobody has ever fixed Bug 550770. In 2 years! Of course, you might ask why I haven't contributed any code to this project since circa 1999....

(In reply to william.allen.simpson from comment #31) > (In reply to Wayne Mery (:wsmwk) from comment #28) > > > Upgraded Thunderbird from 3.1.x to 6.0. > > > > so If I understand correctly, you attempted version 5 once or twice, > > reverted to version 3.1.11, then jumped to version 6 (skipping 5). Is that > > correct? > > > Repeating myself in excruciating detail: my question wasn't a value judgement on your work ... I'm only interested in a 1-2 sentence response as to whether my summary statement is correct, or not correct. I think your restatement means my statement is correct, but I'm not sure. (less wordes is more in this case) > > have you tried reproducing, with a test profile, jumping from version 3.1.x > > to 6.0? > > > No. Spent rather a lot of time reproducing the problem (repeatedly) with my > usual profile, as you can see from all the attachments. Please walk through > the attachments, comparing file lists visually. > > I could get the profile from backup, make a new one, and reproduce it, but > why bother? I've already spent most of a weekend reproducing it above. > Apparently, you don't think this is a real problem??? I make no such judgement. I'm not sure it's the best path to helping understand the issue, but it is one of potential path. > > > I'd already reported the gloda integer overflow Bug 550764, and have never turned gloda back on > > > > I don't see that 550764 (which is no doubt a fixed duplicate) had anything > > to do with int overflow. > > > You might try actually reading Bug 550764. It's not marked fixed duplicate. > Somebody marked it "incomplete", because I wouldn't spend 48 useless hours > going to a backup and running through the upgrade steps to see whether it > had been fixed in 3.1.3. I'd argue that was a misuse of the incomplete > marking -- but what do I know? :-( Not going to invest substantially more time in your bug is your choice - no one expects endless hours of time. But please understand that when a reporter totally stops responding in a bug where the reporter is mostly likely the only person able to reproduce (i.e. no triager has been able to or it's not likely to be able to), or the issue is still not well understood, then it's not reasonable to keep it open. > It was confirmed as an integer overflow at 2 GB. I don't see that bug 550764 comment 0 or bug 550764 comment 9 demonstrate that it has anything to do with 2 GB. where is the statement to that effect? (let alone confirmation from a developer.) > Why, you ask? Nobody has ever fixed Bug 550770. In 2 years! well, that bug is also a thorn in my side too. But it's not a good basis for indicting the state of all other bugs. > Of course, you might ask why I haven't contributed any code to this project > since circa 1999.... whatever you choose is fine. In summary, I'm simply trying to help advance the bug past Uncomfirmed. Please don't read more into it that that.

(In reply to Wayne Mery (:wsmwk) from comment #32) > > It was confirmed as an integer overflow at 2 GB. > I don't see that bug 550764 comment 0 or bug 550764 comment 9 demonstrate > that it has anything to do with 2 GB. > where is the statement to that effect? > (let alone confirmation from a developer.) > Interesting non-denial denial. I'm pretty sure that several of the bugs I was reporting or CC'd at the time mentioned a 2GB limit that caused problems for gloda, and a 4GB limit that caused utter failure and infinite loops. A quick search for all local storage limits yields: Bug 462665. So, I'm guessing this is an indexing problem. But hey, what do I know? Perhaps somebody stuck in code that checks the phase of the moon. We know already that somebody checked in code that stopped regenerating missing .msf files on search, a long-standing feature going back to at least 2001-2002. (David and I go back at least that for together, so I trust he knows *something* even though we don't always agree.) I don't know when that stopped working, but googling shows it was still suggested as a solution in 2008. We already know that somebody checked in code that stomps through the entire tree and regenerates 4K .msf files on zero length mboxes often found on subfolders. That might be a good place to start looking. Heck, why doesn't it check for other missing mboxes at the same time? But until Bug 682731 code makes it into the 3.1.x release cycle, I cannot even regenerate my msf's. Therefore, I cannot test updating to 6.0.x to generate any more logs. So, you're stuck with the copious logs and snapshots I've already given you. If you cannot confirm from them, there's nothing more I can do.... I've already lost more than $10K income over this, so I'm plenty **** off.

sorry you've lost $$. drivers team will need to decide if bug 682731 makes it back to v3.1.x. For now, the earliest it will appear is v8. as for gloda, the folder limits still exist, but iirc any overflow issues where indexing would simply stop working are resolved.

again, sorry this has been a bad experience for you. Bug 68273 is never going to hit the 3.x branch. So if this can be reproduced on a current version, i.e. v10 or newer, please reopen the bug report and we'll work through it.

OS X 10.6, same problem for months now since at least TB 17, now TB 24, several 100 folders and subfolders, local folder 6 GB. Msf files disappear several times a week to several times a day, no obvious reason. After many years of happy TB use I guess I will have to switch to apple mail if there is no solution soon. Pleas help!

Found the reason: Too many folders. Apparantly, TB can't handle very many folders. After I removed my archives folder (including yearly copies of large parts of my folder structure) form the TB profile .msf files have not been deleted any more.