User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.3a) Gecko/20021212 Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.3a) Gecko/20021212 Open a newsgroup with *lots* of messages (in the 10thousands), and let Mozilla download all headers (not only 500). Look at taskmanager, ans watch Mozilla consuming lots of memory until it finally loaded all messages. If the newsgroup was downloaded previously, just watch the memory footprint of Mozilla skyrocket when opening the newsgroup. It seems to load the complete .msf file into memory. Now odo the very same with NW4.X. You'll notice that Netscape hardly needs *any* additional memory, regardless how much headers the newsgroup holds. Reproducible: Always Steps to Reproduce: 1. Create a new account on news://support-forums.novell.com 2. Subscribe to the group novell.community.chat. Currently, this group holds 300.000 messages 3. Let Mozilla download all headers in this group. Go on a long vacation, don't expect yor PC to ever come back from this task. It'll start trashing memory and finally hang at around 70.000 downloaded headers (depending on the memory installed in your workstation, 70.000 was the max I could achieve with 256MB). Actual Results: Mozilla will use virtually infinite amounts of memory, and finally crash and burn a W2K PC with 512MB of physical memory and no other apps running. This is at least 25 times the memory footprint of NS4.8, without ever completing the task succesfully, and without any additional functionality Mozilla offers over Netscape in News. Expected Results: The same as Netscape 4.8. NS4.X opens, downloads and displays above example group without ever using more than 20MB of memory.

Maybe related to bug 145441.

I'm not convinced that this is related, it sounds like a different issue to me. But bug 191491 (which was marked - IMHO incorrectly - as duplicate of bug 145441), definitely is a duplicate of this one. The symptoms match exactly.

This 'feature' is really boring. Additionally it seems to have the effect that loading the message list of huge newsgroups takes way too much time. My mozilla regularly uses 300 megs of RAM when syncing newsgroups for offline reading.

I was using Moz 1.2,1.3,1.4,1.5,1.6 for downloading and reading newsgroups. One problem remains - when downloading newsgroups for offline use, Mozilla eats memory like crazy - now it's 1 Gb. News folder now is 4 Gb in size. RAM usage is increasing each time. So 1GB isn't the last stop, i think... My PC has 384 Mb RAM and WinXP as OS.

Was there ANY Mozilla release that did not have that problem? As far as I know, Netscape 4.x did not have that issue.

The problem is obviously Mozilla's internal database (Mork) which needs to load the complete .msf files before being able to access their contents. Can't we just integrate/update Netscape 4.8's corresponding storage algorithms?

(In reply to comment #5) > Was there ANY Mozilla release that did not have that problem? As far as I know, > Netscape 4.x did not have that issue. No. All Mozilla versions, as well as all Thunderbirds show this behaviour. This is a major showstopper for any *serious* newsreader usage of the two. Unfortunately, this 100% and easily reproducible Bug isn't even confirmed after almost two years. So I doubt this will ever change.

Or we need news-specific storage methods.

(In reply to comment #0) > User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.3a) > Gecko/20021212 > Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.3a) > Gecko/20021212 > > Open a newsgroup with *lots* of messages (in the 10thousands), and let Mozilla > download all headers (not only 500). Look at taskmanager, ans watch Mozilla > consuming lots of memory until it finally loaded all messages. If the newsgroup > was downloaded previously, just watch the memory footprint of Mozilla skyrocket > when opening the newsgroup. It seems to load the complete .msf file into > memory. > Now odo the very same with NW4.X. You'll notice that Netscape hardly needs > *any* > additional memory, regardless how much headers the newsgroup holds. > > Reproducible: Always > > Steps to Reproduce: > 1. Create a new account on news://support-forums.novell.com > 2. Subscribe to the group novell.community.chat. Currently, this group holds > 300.000 messages > 3. Let Mozilla download all headers in this group. now over 580k messages. Using TB 1.5 - started at 26,451k and ended at 449,944k

Memory usage for, 3 newsgroups containing hundreds of thousands of messages each, meant that Thunderbird 2 was starting up with 130MB of memory usage on Windows XP. Unsubscribing from these newsgroups and restarting dropped it down to 28MB. There's got to be a solution to this.

Two problems exist: 1. We hold a reference to all msf's loaded so far in a cache 2. Each msf is fully loaded in memory. #2 will be hopefully fixed by bug 11050, but what to do about #1? We probably need to see what happens with the Moz2 memory stuff.

Accessed novell.community.chat and hit 144MB, cycled through several newsgroups/servers and now I'm at 120MB. That's a far cry from the 450MB of comment 9 (In reply to comment #11) > Two problems exist: > > 1. We hold a reference to all msf's loaded so far in a cache > 2. Each msf is fully loaded in memory. > > #2 will be hopefully fixed by bug 11050, but what to do about #1? We probably > need to see what happens with the Moz2 memory stuff. bienvenu, Joshua, do you agree comment 10 would likely have been addressed by the "close unused db" fixes done in the past few months? If so, given my test, I think we can demote this but to minor or normal

Thunderbird version 2.0.0.21 (20090504) compiled on FreeBSD7/AMD64 from FreeBSD Ports. Being subscribed to some 25 gmane newsgroups (but actually opening only 2-3 of them to read messages) my Thunderbird usually shows in top(1) as 1060MB size, 880MB resident… which seems pretty "major" to me, but maybe the changes you refer to are not yet in 2.0.0.21? That value does NOT increase opening (for the first time in the session) new newsgroups, it seems they get loaded during the initial scan-for-new-messages and kept for good.

(In reply to comment #13) > ... maybe the changes you refer to are not yet in 2.0.0.21? correct. Sorry Lapo, I failed to state I tested with 3.0b3pre 20090515 nightly build. These fixes will not appear in version 2 (1.8 branch). So you will want to test nightly build if you are able. Don't need to uninstall Thunderbird 2, just backup your profile and install the nightly in it's own directory. ftp://ftp.mozilla.org/pub/thunderbird/nightly/latest-trunk/

there's nothing can do about one giant newsgroup given Mork's all-in-memory implementation - but we can close db's when the user isn't using the folder, as you say, Wayne.

Obviously this is a long-standing issue, dating back to 2002. But I never noticed it until Thunderbird 3.1.2 (which I tried in September 2010). I can accept a few hundred MB, or even 1 GB of memory, but I cannot accept the >>30 minutes. Note, however, in 2.0.0.14 I only see 80 MB. Thunderbird 2.0.0.14 works OK for a newsgroup with 600,000 messages, only a few hundred of which are unread, and it's deleting messages older than 30 days -- 80 MB of "Real memory" on my Mac. Something very bad happened between Thunderbird 2.0.0.14 and 3.1.2 -- when clicking on this newsgroup in 3.1.2, CPU goes to 100%, memory remains at 89 MB, with no network usage. So I suspect 3.1.2 has an infinite loop somewhere; probably related to sorting messages (see my description in bug 597688, which has been marked a duplicate of this one). In the newsgroup preferences there is an option to "delete message older than 30 days" -- can't that be used to reduce the size of the DB? Indeed it looks like 2.0.0.14 did just that (when I discovered it a few weeks ago before upgrading, its speed went up considerably).

Bug# 595145 was mine and while there are some similarities I haven't noticed this as primarily a memory issue... who knows. The issue that I see is a minimal amount of network traffic but a huge amount of processor activity by the Thunderbird (3.1.7 currently) process. Thunderbird becomes almost completely worthless for thirty or more seconds but it always comes back (my box is 64-bit and I have 8 GB RAM so maybe I'm past the memory problem by brute force). Either way this single bug has me on the brink of changing because it seems (not tested... just seems) that Thunderbird decides to do its refresh of the newsgroup messages at the exact time I decide to use Thunderbird (not always, but regularly). My workaround is to go away and come back in a couple minutes when all is well for a while longer. It is likely that the original reporter and I have the same forums involved so duplicating this should still be pretty straight-forward. It may be worth mentioning that my ~/.thunderbird directory is over 1 GiB in size so I suspect this has something to do with the disk more than anything else, perhaps involving the CPU as it thrashes the disk.

The purpose of this attachment is to show memory usage for my Thunderbird setup over several hours. As you can see the memory fluctuates periodically and I believe those fluctuations, though not extreme, correspond to the time when Thunderbird takes over one of my CPUs/cores. What this also shows is that Thunderbird is taking an average of 3.6% of my total CPU time (over several hours) through its high utilization during those times of high activity (ps's %CPU shows total cpu time divided by total runtime). The math probably makes more sense than my description... 99% * 30 seconds averages out to 3.6% over about 13.75 minutes, which is close to my 13-minute refresh interval.

I'm not sure it is really the same bug but it looks like close. I am subscribed to another large nntp forum : news://server.amis.tv:119/nzn.fr.delphi, that currently holds around to 132800 posts. Periodically Thunderbird halts to responds, whatever I am doing in it, and restart a few time later. It is extremely boring when I am writing some message. During these halts, the task manager show that Thunderbird is using 100% time of one core. The amount of ram eaten is stable, around 150 Mb (from 148.5 to 151.5 Mb) and do not show large variations. That problem was also discussed in forum mentionned above, under the subject "§§ Thunderbird n'aime pas server.amis.tv.Delphi ? §§" starting on 2010 november 27th. It was reported that many Thunderbird users has the same problem. One of them even reported that he leaved Thunderbird because of this and he now uses SeaMonkey that does not have these halts, despite it is based on the same sources than Thunderbird.

Pascal could you get a stack trace when this happens (see https://developer.mozilla.org/en/how_to_get_a_stacktrace_with_windbg) ?

As mentioned before I do not think my bug matches this one (not a high memory issue... more like Pascal's it's a high processor utilization issue) but today I found another clue. When I run something like `find ./ -mmin -4 | xargs ls -ld` from my user's .thunderbird directory I see that one file per subscribed forum is updated every time Thunderbird polls and finds either some or no changes. EVEN IF there are no changes every file is changed, at least timestamp-wise. Thinking this may be just a 'touch' of some sort I backed up a file and then compared it to the original after a change and found the new file, even though the forum had no new posts. The lines added to the file, despite a lack of new posts to the forum, follow: <quote> @$${3D0F0{@ [1:^9F(^A9^709A)] @$$}3D0F0}@ @$${3D0F1{@ @$$}3D0F1}@ @$${3D0F2{@ @$$}3D0F2}@ @$${3D0F4{@ [1:^9F(^A9=0)] @$$}3D0F4}@ </quote> I have no idea what these lines are doing but they happen every polling interval of Thunderbird (fifteen minutes currently) and appear to mean almost nothing. I see more data added when something shows up in the forum to which this file corresponds. Can this be changed/fixed? Why doesn't Thunderbird just compare what it receives as far as forum counts via NNTP to the contents of the (in my case) ../.thunderbird/u9a8njvw.default/News/newsrc-forums.novell.com file?

For the second time in recent months I am unable to download one large newsgroup, because *two* applications of the news server show up in task manager when I click to download it, and they utilize 100% of the CPU. It only happens with this one ng on that server and it is the largest one. The last time I had to re-subscribe and enter all my filters again which is quite a chore on that group, but I would really like to avoid this again. Are there any solutions anywhere to this problem with SeaMonkey 2.0.10?