Richard Neill Reporter
	Description • 17 years ago

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.5) Gecko/20061201 Firefox/2.0.0.5 (Ubuntu-feisty)
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.5) Gecko/20061201 Firefox/2.0.0.5 (Ubuntu-feisty)

In normal usage over about a week (a few hours/day, accumulating about 200 tabs), Firefox slowly grows to fill *lots* of memory. At some point, it will suddenly crash, without warning. I have the memory monitor taskbar applet running constantly in KDE, and I can see FF using all my RAM, but I still have about half of my 4 GB of swap unused. Other apps run fine.

What I think is happening is that firefox is trying to allocate RAM over the 4GB/process limit, and the kernel then kills it (either the oomkiller, or it segfaults). At any rate, there's no easy way to predict when FF is about to crash, nor is there any error message.

I'm currently using Ubuntu Feisty on x86, with 2GB physical RAM, and 4GB of swap. I'm upgrading to a 64-bit machine soon - and one of the advantages will be to allow FF to address 16GB of swap without crashing. 

I have the following extensions installed: Adblock+FiltersetG, Webdeveloper, ImageZoom, TabMixPlus, HTML Validator. The only plugins installed are Java, Flash, and mplayer.

I'm becoming rather too well acquainted with the session restore feature!


Reproducible: Always

Steps to Reproduce:
1.
2.
3.



There are several things going wrong here, I think.

1)*Why* does FF need so much memory? 

2)If FF is about to try to allocate RAM over the 32-bit limit, a warning or error message would be nice!

3)I use the web too much ;-)   Maybe FF is trying to tell me something!

If there are any tests I can do that would provide useful information to fix this, please let me know.

	WADA:World Anti-bad-Duping Agency
	Comment 1 • 17 years ago

Even though Mozilla family is one of the biggest memory eaters, and many resource leak bugs are still open, not so many infinite&huge memory consumption problems(for both virtual & real) are reported. And severe memory leak by extension(add-on) really existed in the past. Read Bug 320915 Comment #12 thru Bug 320915 Comment #26 for memory leak by Adblock in the past.

(Q1) Does your problem re-produced with newly created profile?
(When leak by Adblock in Bug 320915, leak occurred even when -safe-mode or)
(after disabling of Adblock. So test with new profile is required.        )

> Firefox slowly grows to fill *lots* of memory

(Q2) Virtual memory size? Real meomory size? Or both?
(Q3) What is increase of size(both virtual and real) per operations such as some tabs open&close, per hour, per day?

(Q4) Is there any need to keep open Firefox for a week?
Startup time of Mozilla family is improved very much than before, so termination of Firefox per a day is recently not so big issue. And relatively large memory consumption(virtual and/or real) usually doesn't become very big issue such as killed process due to 4GB limit, although thrashing may occur if all available real memory is exhausted by Firefox and Firefox requests more real momory.
      

	Jesse Ruderman
	Comment 2 • 17 years ago

> accumulating about 200 tabs

Do you mean having 200 tabs open at once?

	Richard Neill Reporter
	Comment 3 • 17 years ago

Yes, I do mean about that. Nevertheless, that's still around 20MB per tab, which seems rather excessive! [I'd have thought that 200kB for the page source, flushed to disk cache, and 20kB for in-memory structures would be more than enough].

	timeless
	Comment 4 • 17 years ago

we actually keep back pages too, 20mb per tab is perfectly reasonable.

you didn't fill in expected results. this bug is invalid.

	Jesse Ruderman
	Comment 5 • 17 years ago

> we actually keep back pages too

We have a small limit of bfcached pages across the entire browser (not per tab), so that won't add much to the 200 tabs.

> you didn't fill in expected results. this bug is invalid.

Sure he did.  Comment 0 suggested:
1. Use less memory per page.
2. Don't segfault after allocation fails.
3. Don't allow the user to load a new page or open a new tab or window when almost OOM.

This bug is invalid, but only because it's too broad.  3 would be a perfectly reasonable request on its own, while 1 and 2 are each hundreds of separate bugs (footprint/mlk for 1, and oom crash for 2).

One thing that's likely to help with #1 (when many tabs are open) is keeping fewer decoded images in memory (bug 327280, bug 386377).

	Robert Bradbury
	Comment 6 • 17 years ago

This is yet another of Mozillia/Firefox "assuming there is an infinite amount of memory" problems.  I documented this in several bugs over a year ago.  The fundamental problem is the the CPP code in the source does not have proper code do "catch" memory allocation failures.  As a result when memory allocations fail random errors are thrown.  SEGVs are a logical consequence.

The fundamental problem is that there is Q&A testing in a memory limited environment.  I do not know if you can create such under Windows, but it is easy to do under Linux using "ulimit".  And if you ulimit Firefox down (so that it will not start), and then increase its memory allowance on a 1-2K basis (just to see where it will fail) it will fail quite regularly.  The "fuzzy" region appears to be between 80-120K -- depending a lot ont the size of ones bookmarks file, the plugins installed, etc.

	Richard Neill Reporter
	Comment 7 • 17 years ago

The other problem is that FF does not seem to release memory when tabs and windows are closed. Starting with 200 tabs open, and going down to 50, the virtual memory and resident size stay approx constant, even slightly increasing. Does ff ever actually release memory? It seems to me that everywhere there ought to be a free(), there's actually an implicit leak().

	Robert Bradbury
	Comment 8 • 17 years ago

Richard, please be very very careful when discussing this problem to clarify which OS you are discussing.   That is because the behavior of memory allocation may be very OS specific.

I understand the problem under Linux, perhaps the MAC, because they are based on UNIX and the system calls which were defined under it.

The way the memory allocation scheme works -- once you have allocated it you are largely stuck with it.  You allocate a page full of images to display, but then if you stick a history or bookmark record after it you've got all that memory.  Unless you can clear and free *all* memory references in the virtual memory the memory will remain allocated.  It is possible to shrink ones total use of virtual memory, but it in practice it remains very very hard to do so [1]

In practice this would not be a problem if the memory management library (glibc) and Firefox and Linux (in terms of its page swapping functions) played more nicely together.  I.e. if the VM system under Linux really worked, excessive memory consumption by Firefox would not be a problem.  But my current opinion is that it does not and thus the excessive memory consumption directly impacts on Firefox performance.

Until the Firefox designers decide to implement "OS specific" memory allocation strategies users are screwed.

1. Under Linux, I have looked at the glibc memory manager, and it will try to shrink the total OS memory usage if the tail end of the memory space is free.  But the probability of it being free in a complex memory management situation is very very low.

	Richard Neill Reporter
	Comment 9 • 17 years ago

Robert, thanks for your explanation. If I understand you correctly, that means that, on Linux, the following behaviour is expected:

Start Firefox.

Open 200 tabs      #memory usage is now, say, 1GB

Close all but one. #memory usage is still about 1GB, since it cannot be 
                   #effectively freed, and returned to the OS.


The question is, if I now open another 200 tabs, is firefox 

 (i)going to re-use the same memory that had earlier been allocated to firefox  

 (ii)going to need a second 1GB of RAM?



Case (i) is just a nuisance, although it does suggest that per-tab memory usage should be lower.

Case (ii)would be a real problem, since the memory is not being reused even by fx.


The solutions seem to be:

* run 64-bit Fx, with a huge amount of swap. Then, firefox can allocate as much memory as it wants to, and the OS can just page it out.

* get the in-memory structures for each tab to be smaller. (eg save the parsed-html to disk cache only, and re-render it each time the user switches tab)

* If this is a kernel bug, pass it on to lkml.

	Robert Bradbury
	Comment 10 • 17 years ago

Richard, there are a complex set of issues here.  And I will try to deal with ?some of them.

> Close all but one. #memory usage is still about 1GB, since it cannot be 
>                   #effectively freed, and returned to the OS.

Yep, most probably because the history records have claimed the "high" memory space.  Though it could be java or javascript -- depends on what your are browsing.  So long as one has "claimed" the high memory space it is essentially impossible to give it back to the OS.

It requires a fundamentally different architecture with distinct memory spaces (pages, history, bookmarks, etc.) to fix this.


> The question is, if I now open another 200 tabs, is firefox 
 (i)going to re-use the same memory that had earlier been allocated to firefox  

Yes, ignoring any potential memory leak issues, it will be reused.  The glibc 
memory allocator is pretty good about that.  

 (ii)going to need a second 1GB of RAM?

I routinely run in 1.5GB of ram with hundreds of tabs open.  O.5-1.0GB are very problematic.


> Case (i) is just a nuisance, although it does suggest that per-tab memory usage should be lower.

Agreed but I am not sure how much this is a Firefox problem vs. an X-windows problem.

> Case (ii)would be a real problem, since the memory is not being reused even by
fx.

No, its definitely being reused.  So even if you have 1GB, if you are closing tabs, windows, etc. for the most part it is getting recycled.  A separate question is whether Linux pages on personal workstations efficiently.  And that   I've got serous problems with.  What you may be dealing with is paging fragmentation.  Long term use of firefox is going to produce what could be called a "fragmented heap".  If the OS is not properly designed to deal with this (and I do not believe Linux is), overall performance will suffer.

The solutions seem to be:

> * run 64-bit Fx, with a huge amount of swap. Then, firefox can allocate as much memory as it wants to, and the OS can just page it out.

Yep, *if* the Linux paging strategy were tuned to deal with personal workstations (where paging is "on demand") -- and I have yet to figure out if or how to do that.

> * get the in-memory structures for each tab to be smaller. (eg save the
parsed-html to disk cache only, and re-render it each time the user switches tab)

Yep, lots of luck on this.  This was more like the old Netscape architecture which seems to have been given up in the current framework.  I do not understand the current mentality but it seems to involve "memory is free".

> * If this is a kernel bug, pass it on to lkml.

It is not technically a kernel bug.  The glibc and kernel interfaces are quite clear and are being used properly (which is what results in growing virtual memory usage).

What is a potential lkml topic is the fact that that Linux does not currrently deal with an "out of memory" and "fragmented heap" problem.

I believe this is not currently recognized as "significant" by lkml managers.

	Peter Weilbacher
	Comment 11 • 17 years ago

This should be fixed at least since Firefox 3 beta 4, now that the new allocation scheme using jemalloc is active? For more details, see <http://blog.pavlov.net/2008/03/11/firefox-3-memory-usage/>.

I did a quick test with Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9pre) Gecko/2008033004 Minefield/3.0pre and opened 120 tabs and closed them down again.
I watched the RES column in top that is supposed to be a good measure of the actual memory usage: after start it used 38m, when all of them were loaded it used 214m, after closing them all and going to about:blank it used only 95m. Then I opened 50 (more graphic intensive) pages in tabs, it used 151m, and closed them down again. It used 107m after going to about:blank on the last one. Clearing all private data freed another 10m, switching to a new window I arrived at 82m. Note that all this was with a fresh profile and without any installed plugins. Doing something like this a few more times, it arrived back at around 70 to 80m after closing all tabs. So I don't think there is a problem any more.

As others have said before, this bug is much too broad to get any real developer attention, so I'm resolving as WFM. If you find specific pages where memory use increases a lot and does not decrease again, please file a new bug.