Closed Bug 327418 Opened 19 years ago Closed 9 years ago

Memory use does not go down when memory cache is enabled

Categories

(Core :: Networking: Cache, defect, P1)

Product:
Core
Component:
Networking: Cache
Platform:
x86
Windows XP
Type:
defect

Tracking

()

Status:
RESOLVED WORKSFORME
Milestone:
mozilla1.9alpha1

People

(Reporter: 9pt4x6002, Unassigned)

References

(
URL
)

Details

(Keywords: memory-leak, Whiteboard: [MemShrink:P3][necko-backlog]) 

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1

When the memory cache is enabled, memory use does not significantly decrease after closing tabs.  If one disables memory cache, memory usage decreases as expected upon closing tabs containing memory intensive sites.  The value of browser.sessionhistory.max_total_viewers seems to make no difference.

Reproducible: Always

Steps to Reproduce:
1. make sure browser.cache.memory.enable is true.
2. open the following URLs in new tabs:
http://forums.fark.com/cgi/fark/comments.pl?IDLink=1899697
http://forums.fark.com/cgi/fark/comments.pl?IDLink=1899563
http://forums.fark.com/cgi/fark/comments.pl?IDLink=1899543
http://forums.fark.com/cgi/fark/comments.pl?IDLink=1899499
http://forums.fark.com/cgi/fark/comments.pl?IDLink=1899318
http://forums.fark.com/cgi/fark/comments.pl?IDLink=1897678
http://forums.fark.com/cgi/fark/comments.pl?IDLink=1897627
3. watch memory usage increase to ~400MB
4. close those tabs

Actual Results:  
Memory usage drops to ~200MB.  about:cache shows the memory cache using 39MB out of a maximum storage size of 5MB.

Expected Results:  
Memory usage drops down to ~60MB (approximately the same as memory footprint upon starting the browser).

If you set browser.cache.memory.enable to false and repeat the steps above, memory usage will drop as expected.  Because of bug 184304, certain sites will not load with the memory cache disabled.  For now, I am using proxomitron to filter out no-cache header directives.
When I try the set of steps above, memory use decreases to well below 100 MB in both Firefox 1.5.0.1 and SeaMonkey 1.0 on Windows XP SP 2. I have the memory cache set to the default values in both browsers. about:cache shows the memory cache using 28 MB out of the maximum capacity of 31 MB. Did you try a new profile? Oh, well, let's see if others can reproduce...
Thanks for the bug report.
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Target Milestone: --- → mozilla1.9alpha
Keywords: mlk
I assume this _is_ happening with a clean profile (no extensions) right?
i just tested with a new profile and it's still occurring but my previous figures are off (guess my extensions use more memory than i realized).  with memory cache enabled, memory usage drops to around 100MB after closing tabs.  with memory cache disabled, it drops down to around 20MB after closing tabs.  apologies, i should have remembered to do that when submitting the bug.
Thanks for the extra info.  It sounds like to me like we must have a memory leak associated with decoded image objects.  Perhaps they are referencing other objects that are not accounted for in the memory cache size totals.
Severity: normal → critical
Some figures using an optimized trunk build on Linux (w/ a 10000k memory cache):

                   VmSize   VmRSS   VmData
a) initial state    59616   34040    24072 
b) w/ 7 fark tabs  103508   77104    67300
c) initial state    93744   71664    57700

The initial state is just showing a small HTML page.  I loaded that as my home page, and then I loaded the 7 fark.com tabs.  Then, I closed the 7 fark.com tabs, which restored the initial configuration.

I did a tracemalloc diff between point (a) and (c), and the results were:

                        ---- Base ----   ---- Incr ----   ----- Difference ----
Type                    Count    Bytes   Count    Bytes   Count    Bytes %Total
TOTAL                   82714  8179177   88976 10446302    6262  2267125 100.00
void*                   60792  6482060   65487  8608628    4695  2126568  93.80
X-unclassified            514    42893     518    98613       4    55720   2.46
gtk-unclassified         7329   384438    8064   415166     735    30728   1.36
nsImageGTK                 30     3840     125    16000      95    12160   0.54
nsStandardURL             591   106380     630   113400      39     7020   0.31
xpti-unclassified         479   518512     485   524656       6     6144   0.27
gfxImageFrame              30     1560     125     6500      95     4940   0.22
imgRequest                 23     2392      56     5824      33     3432   0.15
nsChildWindow              14     3304      28     6608      14     3304   0.15
nsGlobalWindow              2      496      14     3472      12     2976   0.13
nsWeakReference           147     1764     386     4632     239     2868   0.13
nsProperties               32     1536      85     4080      53     2544   0.11
PRLock                    289    25432     305    26840      16     1408   0.06
imgContainerGIF            13     1092      26     2184      13     1092   0.05
nsSupportsCStringImpl      23      552      56     1344      33      792   0.03
nsXPCComponents            14      728      26     1352      12      624   0.03
nsObsoleteACStringThunk    331    19080     369    19688      38     608   0.03
nsLocalFile               254    30480     259    31080       5      600   0.03
nsPrincipal                 4      464       9     1044       5      580   0.03
OTHER                   11803   552174   11923   555191     120     3017   0.13

If I'm reading this correctly, then it is saying that there was just 2267125 bytes or ~2M worth of live object growth between point (a) and (c).  Since the memory cache is capped at 10M, and because the amount of memory in use according to the memory cache is close to 10M, I suspect that the additional 8M or so of bloat is actually attributable to X pixmap objects stored on the X server.  (The memory cache is told about the size of those, I believe.)

So, I'm tempted to blame heap fragmentation, but it would be nice if there were some way to test for that.
I repeated my test with the memory cache disabled, and I did not observe much of a difference.  Perhaps the reported behavior is particular to Firefox on Windows.
On Windows, I observed pretty much the same behavior as on Linux.  The only difference in allocated memory between the run with the memory cache enabled and the memory cache disabled is the memory held by the memory cache.
interesting.  i was able to reproduce this a number of times with my regular profile, and the one time i tried it with a fresh profile.

also, not related to this specific testcase, but before disabling the memory cache, firefox would tend to grow in footprint to at least a few hundred mb after using it for regular browsing by the end of the day.  since disabling the memory cache, usage always drops back down to normal after closing tabs.
Gecko/20060130 SeaMonkey/1.0 , W2k/SP4 with 512MB RAM, Matrox G450 AGP, almost no other applications running

I'm not sure which numbers from TaskManager are important (if any), but anyway. I unintenionaly ran almost three tests, but each one with completely new clear profile, so I rounded it to tens of thousands (it doesn't matter I think). browser.cache.memory.enable=true, just navigating from the start page to this bug and doing test :

                   MemoryUs   VMSize   GDIObj
a) initial state    23000     13000     150 
b) w/ 7 fark tabs  320000    310000     420
c) initial state    70000     60000     170

about:cache shows storage in use 21500, inactive ~19000.
those figures are a lot closer to what i'm seeing.  at least someone else can reproduce it.
This time another machine and the trunk build:
Gecko/20060217 SeaMonkey/1.5a , WinXP Pro / SP2, 512MB RAM, Matrox G550 AGP

                   MemoryUs   VMSize   GDIObj
a) initial state    25000     15000     170 
b) w/ 7 fark tabs  324000    313000     450
c) initial state    55000     44000     170

about:cache shows storage in use 21127, inactive ~18475.

Much nicer but probably still a little off. Do we have something better than TaskManager to use?
Priority: -- → P1
(In reply to comment #6)
> I did a tracemalloc diff between point (a) and (c), and the results were:

You could also use diffbloatdump.pl for a more detailed diff.  See http://lxr.mozilla.org/mozilla/source/tools/trace-malloc/diffbloatdump.pl#40
(In reply to comment #6)
>                    VmSize   VmRSS   VmData
> a) initial state    59616   34040    24072 
> b) w/ 7 fark tabs  103508   77104    67300
> c) initial state    93744   71664    57700

Darin Fisher, what size of Vm will be used when next steps are added?
  d) w/ 7 fark tabs of different images from ones for step b).
  e) initial state
  f) w/ 7 fark tabs of different images from ones for step b) and step d).
  g) initial state
If VmSize of b) == d) == f) or difference is not so large, and if c) == e) ==  g) or difference is not so large, it can be guessed "memory is allocated, and not freeded, but reused".
And it can be guessed a variation of phenomena of Bug 215491(single large image case).
Then it can be guessed issue of Bug 130157 because Bug 215491 was closed as DUP of Bug 130157.
I know this problem for about several months. Especially when opening many new tabs (watch some sites as www.fotocommunity.com) this can GET a real problem for memory usage.

I tried the mentioned workaround with setting memory-cache off. With your example site memory usage starts at about 40M, grows to 140 MB (there i aborted and closed all tabs) and then dcreases to about 100 MB. Opening 2 new tabs with bugzilla led to fast growing to about 130 MB.

read the following use case to see that this bug is really annoying.


use case:
ok. maybe I am an extreme user but just visit www.fotocommunity.com, choose a channel, open each picture site (30) and then open each single picture in a new tab (this makes sure that you can watch all pictures without the need to wait for them being loaded - so to say: manual prefetching). this will open about 300 tabs. Your memory usage grows to about 900 MB. When closing all of them after a while, memory usage doesn't fall back but stays at about 800 - 900 MB and Windows freezes (with 1,2GB RAM) for some time because of swapping.
The only way to get memory back to normal usage is to close all instances of Firefox.
(In reply to comment #15)
I went to www.fotocommunity.com and opened over 400 tabs of pictures, all open at once. VM Size went to almost 900 MB. After closing all the pictures, VM Size is back down to under 200 MB, which is around what it was before I started. This is using a recent SeaMonkey 1.0.x nightly build on Windows XP SP2 on a computer with 1 GB of RAM. I noticed quite a bit of disk swapping, especially after closing the tabs and trying to use SeaMonkey again, but SeaMonkey stayed fairly responsive during the whole procedure. Perhaps you're using an extension that causes the RAM not to be released? Can you try reproducing the problem with a new profile? Can you try reproducing with a recent nightly build (any branch or trunk) of Firefox or SeaMonkey?
Hi, I haven't tested the same site but my experience after a couple of hours of browsing today using a nightly seamonkey build
'Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9a1) Gecko/20060515 SeaMonkey/1.5a'
showed Mem Usage of 405MB and VM Size = 1208 MB

In my view there was something recently that made the memory leaks much worse (or the browser more stable).
I had never seen my seamonkey grow above 700MB VMSIZE (it usually crashed soon after 600M).

P.S. I am not using any Extensions (except Live HTTP Headers, which doesn't work with this profile).
(In reply to comment #17)
> In my view there was something recently that made the memory leaks much worse...

If there's a new problem, please report it in a new bug report. This bug report is about an older problem.
-> reassign to default owner
Assignee: darin.moz → nobody
Status: ASSIGNED → NEW
dbaron, is this something you can answer?

(In reply to WADA from comment #14)
> (In reply to comment #6)
> >                    VmSize   VmRSS   VmData
> > a) initial state    59616   34040    24072 
> > b) w/ 7 fark tabs  103508   77104    67300
> > c) initial state    93744   71664    57700
> 
> Darin Fisher, what size of Vm will be used when next steps are added?
>   d) w/ 7 fark tabs of different images from ones for step b).
>   e) initial state
>   f) w/ 7 fark tabs of different images from ones for step b) and step d).
>   g) initial state
> If VmSize of b) == d) == f) or difference is not so large, and if c) == e)
> ==  g) or difference is not so large, it can be guessed "memory is
> allocated, and not freeded, but reused".
> And it can be guessed a variation of phenomena of Bug 215491(single large
> image case).
> Then it can be guessed issue of Bug 130157 because Bug 215491 was closed as
> DUP of Bug 130157.


Note scratch's (the reporter's) email address fails. I don't know about Jiri's.
Flags: needinfo?(dbaron)
I still can reproduce this issue.
Test environment: Win XP SP3, all third party add-ons disabled

1. open www.fotocommunity.de --> mem consumption of firefox ~100mb
2. Open xx pictures, each picture in a new tab --> >600mb
3. Close all tabs but 1 --> 175mb
repeat 1-3 several times --> mem consumption constantly grows

It looks to me, as per tab used and closed, ~1mb remains in memory.
(In reply to WADA from comment #14)
> (In reply to comment #6)
> >                    VmSize   VmRSS   VmData
> > a) initial state    59616   34040    24072 
> > b) w/ 7 fark tabs  103508   77104    67300
> > c) initial state    93744   71664    57700
> 
> Darin Fisher, what size of Vm will be used when next steps are added?
>   d) w/ 7 fark tabs of different images from ones for step b).
>   e) initial state
>   f) w/ 7 fark tabs of different images from ones for step b) and step d).
>   g) initial state
> If VmSize of b) == d) == f) or difference is not so large, and if c) == e)
> ==  g) or difference is not so large, it can be guessed "memory is
> allocated, and not freeded, but reused".

I agree that if b==d==f and c==e==g, then we'd be observing behavior that's far less serious than if that isn't (roughly) the case.  It's relatively normal for there to be some fragmentation in memory allocation that prevents blocks from being able to be returned to the system, though there are things we can do to improve that in cases where it's actually causing serious problems, if it is.

That said, I think the MemShrink folks would be able to better evaluate where this bug report should go.
Flags: needinfo?(dbaron)
Whiteboard: [MemShrink]
Honza, Michal, perhaps something to be aware of when rewriting the cache?

I don't think improvements to the old cache are a high priority for us since we're getting rid of it.
Whiteboard: [MemShrink] → [MemShrink:P3]
In the new cache code there is a single limit for any data that the cache backend keeps in memory (metadata, indexing data, content data) that when overlapped makes oldest entries go away from memory to consume only up to the limit again.

I'll go though this bug and see if it is still about concern or not.
Whiteboard: [MemShrink:P3] → [MemShrink:P3][necko-backlog]
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.