B2G has high heap-unclassified. At startup, it's ~13mb (~27%) in the main process and ~4.8mb (~20%) in the homescreen process. We need to get a better handle on where this memory is going.

> At startup, it's ~13mb (~27%) in the main process and ~4.8mb (~20%) in the homescreen process. Per attachment 668326 [details].

Next week I'll start working again on the integrated DMD (bug 717853) which should help greatly here. BTW, bug 798510 is going to increase the amount of heap-unclassified :P

> BTW, bug 798510 is going to increase the amount of heap-unclassified :P The stack is non-heap memory. It should increase only the fraction heap-unclassified / explicit. :)

> Next week I'll start working again on the integrated DMD (bug 717853) which should help greatly here. I just hope that's not too slow to run on our devices. If only I had a fast ARM device (with lots of memory) that B2G worked on, but I do not believe SGSII builds work these days.

I was able to bring one back to life, Frankenstein-style, a week or two ago. FWIW.

> I just hope that's not too slow to run on our devices. Even if it is, it'll give better information on the desktop builds.

Here's NewDMD output for a Linux64 desktop build of b2g, with hwaccel off, having just started and unlocked it. The output is less intuitive but more comprehensive than OldDMD -- it's a tree of stack traces. To navigate, either: - navigate around using the "u05" labels (for vim users, the '*' and '#' searches are ideal for this), or - just read it top to bottom, it's not that long. In short, there are a few moderate-sized things (e.g. 100--200 KiB) and lots of small things. Quite a few of the moderate-sized things are cairo and gtk stuff which we'll never get memory reporting for, AFAICT.

bz, I had trouble with the CSS cases working out what class members weren't being measured. Any help with the stacks containing the following functions would be much appreciated: - nsCSSFrameConstructor::ConstructRootFrame x 2 - nsCSSValue::StartImageLoad - nsUnicharStreamLoader::OnStopRequest x 2 - GetToken Thank you!

> - nsCSSFrameConstructor::ConstructRootFrame x 2 The first one looks like it's coming from adding an entry to the hashtable that maps tags to lists of rules that match those tags. It's a bit odd that this is showing up. RuleCascadeData::SizeOfIncludingThis calls mRuleHash.SizeOfExcludingThis(), which calls PL_DHashTableSizeOfExcludingThis on mTagTable. So presumably we're not calling SizeOfExcludingThis on the relevant rule processor at all? Which means we're not calling it on the relevant style set? I'm really not sure what's going on there. :( The second one is the same thing, but space in the array in the hashtable entry, not the space for the hashtable itself. > - nsCSSValue::StartImageLoad This is the mURI of a css::URLValue. There's a comment in css::URLValue::SizeOfIncludingThis about maybe adding it sometime. It's a refcounted nsIURI object, so a bit hard to do right, possibly. :( Though in this case, I guess having the URLValue claim it owns it would make sense. Why we have 180KB of data: URIs for presumably background images in CSS is a separate question. ;) > - nsUnicharStreamLoader::OnStopRequest x 2 The first one is memory for a selector. Generally we do have memory reporting for those, looks like. If we're not getting the info for this one, that suggests it's an entire stylesheet we don't memory-report? Hard to tell what that sheet is based on the stack; that's just general stylesheet-loading code. :( All I know is that it's a stylesheet loaded from a jar: URI. Maybe XBL resource sheets or something? Do we report UA stylesheets? The second is the same thing, but for rulesets, not selectors. Probably from the same stylesheet that's not being reported. > - GetToken This is string allocation under set_backgroundImage. I would bet money it's allocation of the string value of the css::URLValue. And further, I would bet that this is inline style, unless there's a script that's going through doing .backgroundImage sets on rules in stylesheets involved. If so, there are comments in nsAttrValue::SizeOfExcludingThis about not counting container->mValue.mCSSStyleRule because of double-counting.... I have no idea why these scripts are setting 500KB-some worth of background URIs, by the way. I guess it's more data: URI games. :(

> I'm really not sure what's going on there. Maybe XBL rule processors? Do we report those?

Is there an unexpected amount of XUL/XBL stuff being allocated here? The only XUL/XBL used explicitly in b2g is a <xul:window> in the main process. (ISTR that the <video> controls are implemented with XBL though, so it will sneak in eventually.)

> Is there an unexpected amount of XUL/XBL stuff being allocated here? We don't have direct allocations of XUL/XBL. We have the following: 1) Stylesheets loaded from jar: URIs that we are not accounting for somehow. 2) Rule processors that we are not accounting for that hang off a document parsed by the XML parser.

Oh, and said XML parser is NOT using nsXULContentSink, so it's not parsing a document with the XUL MIME type. For what it's worth.

bz says we use the XML parser for XHTML files (if they're .xhtml). There are some .xhtml error pages floating around, but I'd definitely like to find out what's instantiating the XML parser.

There's a good chance that comment 12 item 1 is chrome://browser/content/content.css, loaded via the stylesheet service. As far as item 2, a better stack shows it's coming through this line in nsStyleSet::FileRules: (*aCollectorFunc)(mRuleProcessors[eAgentSheet], aData); which is odd, since style sets report mRuleProcessors[eAgentSheet]....

(In reply to Boris Zbarsky (:bz) from comment #15) > There's a good chance that comment 12 item 1 is > chrome://browser/content/content.css, loaded via the stylesheet service. I added a reporter for nsStyleSheetService and it only counted 50KB worth of memory.

OK. Let's add measurements for bug 799796 and see what happens.

clee just hit a really nasty bug where his device got into a state where it could no longer launch new OS processes. The reason ended up being that the b2g process on his phone was using almost 50MB more than I've seen any b2g process use before. We were able to grab an about:memory dump, but it basically said "I don't know where that 50MB is going" :/. Good to know though; hopefully the work here will clarify that. Will file a new bug for runaway b2g memory usage.

Comment on attachment 672213 [details] about:memory dump with tremendous amount of unclassified C heap (Not a patch. Muscle memory.)

> Good to know though; hopefully the work here will clarify that. In general, I wouldn't get your hopes up. At least, in our experience, high heap-unclassified often points to something which is normally tiny. There are hundreds or thousands of these allocation sites, and we can't hope to write memory reporters for all of them. If we can get STR, then Nick can run it through DMD and see what's going on.

If I'm able to reproduce this, I'll post them.

This heap-unclassified is very frustrating, because we're starting to feel qualitatively better from all the wins, and then dark matter undoes all that :(. There's not a lot that jumps out at me. We have a lot of ~1MB anonymous maps, many of which are mostly committed. There's a *ginormous* ~30MB mapping, which is more than half full. I'd like to know what that is. However, I don't have any evidence it's part of the heap-unclassified. One (reaching ...) hypothesis might be that we're accidentally using bionic heap as well. I wonder if we can poke at /dev/memory and try to find jemalloc headers? Will still work on STR, but I mostly only see this on my dogfood phone (which I'm trying not to destroy). Will try simulating dogfooding conditions on a phone I'm happy to destroy.

heap-unclassified refers to "jemalloc heap unclassified". If we were using a different heap, it wouldn't show up as heap-unclassified.

> We have a lot of ~1MB anonymous maps, many of which are mostly committed. I believe the JS engine and jemalloc both map memory in 1mb chunks. I wonder what the 30mb chunk is.

I would like to know that myself. Given that it seems 10-15MB of heap-unclassified is "expected", this looks like our huckleberry (one of, at least).

We have been discussing this fellow 4f500000-51200000 rw-p 00000000 00:00 0 Size: 29696 kB Rss: 17872 kB Pss: 17872 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 17872 kB Referenced: 17872 kB Anonymous: 17872 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Locked: 0 kB Let me see if that's present before things get into a bad state. Should also be able to break on large mmap()s to see who's doing this.

I'm not able to re-conjure this fellow on otoro or (rebooted) unagi with some basic testing. There are a considerable number of 2MB+ anonymous mappings that have a nontrivial % of committed pages. This merits some further investigation, since per jlebar gecko isn't directly allocating them, AFAIK. Some of this might be necko disk cache though; I don't know how that works. dhylands, I'm thinking of a simple process to check this; let me know if you have cycles to poke at it. What we can do is, instrument bionic mmap() with a "poor man's breakpoint" that's hit when conditions are met. For example, void* mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset) { if ((-1 == fd || (flags & MAP_ANONYMOUS)) && length >= 1 << 21) { write(1, "bp\n", 3); } We can break on that in gdb and then audit the callers.

So far have found - bug 805689 - we're making a >2MB allocation from the js engine when the keyboard is loaded with mmap instrumentation. Not seeing the Great White Whale yet though, still poking around.

I also see another >2MB when loading the browser, under #6 0x411d1b30 in allocateFromArena (cx=<value optimized out>, thingKind=js::gc::FINALIZE_OBJECT2_BACKGROUND) at /home/cjones/mozilla/inbound/js/src/jsgc.cpp:1612 #7 js::gc::ArenaLists::refillFreeList (cx=<value optimized out>, thingKind=js::gc::FINALIZE_OBJECT2_BACKGROUND) at /home/cjones/mozilla/inbound/js/src/jsgc.cpp:1857 #8 0x411ff3ca in NewGCThing<JSObject> (cx=0x48080600, clasp=0x41689b6c, type_=0x4b25a0a0, parent=<value optimized out>, kind=js::gc::FINALIZE_OBJECT2_BACKGROUND) at /home/cjones/mozilla/inbound/js/src/jsgcinlines.h:457 Are these super-large allocations or normal growths of the arenas?

- > 2MB allocation under JSScript::makeAnalysis in browser - same for SMS - nsJSContext::GarbageCollectNow in b2g process when loading dialer app - in SMS app; must be the giant JS telephone number lib #14 0x41263ebe in js::Compressor::init (this=0x0) at /home/cjones/mozilla/inbound/js/src/jsutil.cpp:76 #15 0x4123f636 in internalCompress (arg=0x42721a20) at /home/cjones/mozilla/inbound/js/src/jsscript.cpp:951 - also loading SMS app #16 0x409ddd82 in EnsureStringLength<nsString> (aChannel=<value optimized out>, aData=0x4e196000 "/**\n * @license\n * Copyright (C) 2010 The Libphonenumber Authors\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this file except in compliance with the Licens"..., aLength=192108, aHintCharset=..., aDocument=0x48f2b000, aString=...) at ../../../dist/include/nsReadableUtils.h:358 - NewGCThing<JSObject> when loading SMS app - sms again #16 0x409ddd82 in EnsureStringLength<nsString> (aChannel=<value optimized out>, aData=0x4e27f000 "// Copyright 2012 The Closure Library Authors. All Rights Reserved.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the Lic"..., aLength=212565, aHintCharset=..., aDocument=0x48f2b000, aString=...) at ../../../dist/include/nsReadableUtils.h:358 - sms again #19 0x411db938 in js::types::TypeCompartment::sweep - (i'm going to stop listing all the SMS things that show up :( ) So didn't catch anything unexpectedly significant in round 1. There are a lot of 3MB+ mappings that remain unexplained, will start earlier in startup next time.

> I also see another >2MB when loading the browser, under > > #6 0x411d1b30 in allocateFromArena (cx=<value optimized out>, > thingKind=js::gc::FINALIZE_OBJECT2_BACKGROUND) at > /home/cjones/mozilla/inbound/js/src/jsgc.cpp:1612 > #7 js::gc::ArenaLists::refillFreeList (cx=<value optimized out>, > thingKind=js::gc::FINALIZE_OBJECT2_BACKGROUND) at > /home/cjones/mozilla/inbound/js/src/jsgc.cpp:1857 > #8 0x411ff3ca in NewGCThing<JSObject> (cx=0x48080600, clasp=0x41689b6c, > type_=0x4b25a0a0, parent=<value optimized out>, > kind=js::gc::FINALIZE_OBJECT2_BACKGROUND) at > /home/cjones/mozilla/inbound/js/src/jsgcinlines.h:457 > > Are these super-large allocations or normal growths of the arenas? That seems odd. You've omitted some stack frames, but AFAICT that ends up in AllocChunk() which always maps 1MB pages. So I don't see why you'd be getting a >2MB allocation. > - > 2MB allocation under JSScript::makeAnalysis in browser That description is vague so it's hard to interpret, but that function uses |analysisLifoAlloc| which bug 804891 directly affects. > - in SMS app; must be the giant JS telephone number lib > #14 0x41263ebe in js::Compressor::init (this=0x0) at /home/cjones/mozilla/inbound/js/src/jsutil.cpp:76 > #15 0x4123f636 in internalCompress (arg=0x42721a20) at /home/cjones/mozilla/inbound/js/src/jsscript.cpp:951 That's source compression, so there must be a big JS source file.

When the GC allocates a chunk on Linux, it wants it to be 1MB-aligned. So we allocate 2MB and then munmap the borders. Kind of gross. I think we discussed using posix_memalign, but there were some issues involving jemalloc. I don't remember though. Maybe we aren't allowed to decommit pages out of the chunk if it comes from jemalloc? It seems like we should be able to, though.

To emphasize: These large mappings should not contribute to "heap-unclassified", because that number is "total number of bytes in the jemalloc heap" minus "total number of bytes covered by memory reporters". That doesn't mean the large mappings aren't a problem, although I kind of suspect they aren't. > So we allocate 2MB and then munmap the borders. Kind of gross. Gross, but not problematic in terms of RSS. IIRC jemalloc also wants 1mb-aligned pages in many cases (chunk_alloc_mmap). That's probably why you're seeing that too. malign also uses chunk_alloc_mmap for large allocations. Different versions of jemalloc employ different strategies for getting these aligned chunks. The code we currently have, which we backported from the latest jemalloc, has a heuristic where if the last time it called mmap it got an aligned mapping, this time it will try to mmap just 1mb and hope it's aligned. If it's not aligned, it falls back to the same "allocate 2mb and unmap the edges" strategy that the JS engine apparently is using.

(In reply to Justin Lebar [:jlebar] from comment #34) > To emphasize: These large mappings should not contribute to > "heap-unclassified", because that number is "total number of bytes in the > jemalloc heap" minus "total number of bytes covered by memory reporters". > If some code malloc(30MB)s, jemalloc can't hand that back in 1MB chunks. If there's no memory reporter for that allocation, it will go unreported. Large mappings are still suspect. Unless I misunderstand something.

> If some code malloc(30MB)s, jemalloc can't hand that back in 1MB chunks. If > there's no memory reporter for that allocation, it will go unreported. If by "unreported" you mean "will fall under heap-unclassified", then yes.

DMD tells me that the b2g-desktop-64 main process has 1.3 MiB of stacks similar to this one: calloc (/home/njn/moz/mi2/xpcom/glue/standalone/nsGlueLinkingDlopen.cpp:67) _render_glyph_outline (/home/njn/moz/mi2/gfx/cairo/cairo/src/cairo-ft-font.c:1395) _cairo_ft_scaled_glyph_init (/home/njn/moz/mi2/gfx/cairo/cairo/src/cairo-ft-font.c:2292) _cairo_scaled_glyph_lookup (/home/njn/moz/mi2/gfx/cairo/cairo/src/cairo-scaled-font.c:2836) _composite_glyphs (/home/njn/moz/mi2/gfx/cairo/cairo/src/cairo-image-surface.c:3969) _clip_and_composite (/home/njn/moz/mi2/gfx/cairo/cairo/src/cairo-image-surface.c:2307) _cairo_image_surface_glyphs (/home/njn/moz/mi2/gfx/cairo/cairo/src/cairo-image-surface.c:4079) _cairo_surface_show_text_glyphs (/home/njn/moz/mi2/gfx/cairo/cairo/src/cairo-surface.c:2763) _cairo_gstate_show_text_glyphs (/home/njn/moz/mi2/gfx/cairo/cairo/src/cairo-gstate.c:1998) _moz_cairo_show_glyphs (/home/njn/moz/mi2/gfx/cairo/cairo/src/cairo.c:3523) GlyphBuffer::Flush(_cairo*, gfxFont::DrawMode, bool, gfxTextObjectPaint*, gfxMatrix const&, bool) (/home/njn/moz/mi2/gfx/thebes/gfxFont.cpp:1550) gfxFont::Draw(gfxTextRun*, unsigned int, unsigned int, gfxContext*, gfxFont::DrawMode, gfxPoint*, gfxFont::Spacing*, gfxTextObjectPaint*) (/home/njn/moz/mi2/gfx/thebes/gfxFont.cpp:1907) gfxTextRun::DrawGlyphs(gfxFont*, gfxContext*, gfxFont::DrawMode, gfxPoint*, gfxTextObjectPaint*, unsigned int, unsigned int, gfxTextRun::PropertyProvider*, unsigned int, unsigned int) (/home/njn/moz/mi2/gfx/thebes/gfxFont.cpp:4746) gfxTextRun::Draw(gfxContext*, gfxPoint, gfxFont::DrawMode, unsigned int, unsigned int, gfxTextRun::PropertyProvider*, double*, gfxTextObjectPaint*, gfxTextRun::DrawCallbacks*) (/home/njn/moz/mi2/gfx/thebes/gfxFont.cpp:4948) DrawTextRun(gfxTextRun*, gfxContext*, gfxPoint const&, unsigned int, unsigned int, PropertyProvider*, unsigned int, double*, nsTextFrame::DrawPathCallbacks*) (/home/njn/moz/mi2/layout/generic/nsTextFrameThebes.cpp:5915) nsTextFrame::DrawTextRun(gfxContext*, gfxPoint const&, unsigned int, unsigned int, PropertyProvider&, unsigned int, double&, bool, nsTextFrame::DrawPathCallbacks*) (/home/njn/moz/mi2/layout/generic/nsTextFrameThebes.cpp:5930) nsTextFrame::DrawText(gfxContext*, gfxRect const&, gfxPoint const&, gfxPoint const&, unsigned int, unsigned int, PropertyProvider&, nsTextPaintStyle const&, unsi gned int, nsCharClipDisplayItem::ClipEdges const&, double&, bool, unsigned int const*, nsTextFrame::DrawPathCallbacks*) (/home/njn/moz/mi2/layout/generic/nsTextFrameThebes.cpp:6067) nsTextFrame::PaintOneShadow(unsigned int, unsigned int, nsCSSShadowItem*, PropertyProvider*, nsRect const&, gfxPoint const&, gfxPoint const&, gfxContext*, unsign ed int const&, nsCharClipDisplayItem::ClipEdges const&, int, gfxRect&) (/home/njn/moz/mi2/layout/generic/nsTextFrameThebes.cpp:5377) nsTextFrame::PaintText(nsRenderingContext*, nsPoint, nsRect const&, nsCharClipDisplayItem const&, nsTextFrame::DrawPathCallbacks*) (/home/njn/moz/mi2/layout/generic/nsTextFrameThebes.cpp:5876) nsDisplayText::Paint(nsDisplayListBuilder*, nsRenderingContext*) (/home/njn/moz/mi2/layout/generic/nsTextFrameThebes.cpp:4572) mozilla::FrameLayerBuilder::DrawThebesLayer(mozilla::layers::ThebesLayer*, gfxContext*, nsIntRegion const&, nsIntRegion const&, void*) (/home/njn/moz/mi2/layout/base/FrameLayerBuilder.cpp:3256) mozilla::layers::BasicThebesLayer::PaintThebes(gfxContext*, mozilla::layers::Layer*, void (*)(mozilla::layers::ThebesLayer*, gfxContext*, nsIntRegion const&, nsIntRegion const&, void*), void*, mozilla::layers::ReadbackProcessor*) (/home/njn/moz/mi2/gfx/layers/basic/BasicThebesLayer.cpp:139) Cairo is completely lacking in memory reporters.

b2g-desktop-64 also has 900 KiB under stacks like this one: moz_malloc (/home/njn/moz/mi2/memory/mozalloc/mozalloc.cpp:64) PL_DHashAllocTable (/home/njn/moz/mi2/bdmd64/xpcom/build/pldhash.cpp:84) ChangeTable(PLDHashTable*, int) (/home/njn/moz/mi2/bdmd64/xpcom/build/pldhash.cpp:526) PL_DHashTableOperate (/home/njn/moz/mi2/bdmd64/xpcom/build/pldhash.cpp:612) AppendRuleToTagTable(PLDHashTable*, nsIAtom*, RuleValue const&) (/home/njn/moz/mi2/layout/style/nsCSSRuleProcessor.cpp:610) RuleHash::AppendRule(RuleSelectorPair const&) (/home/njn/moz/mi2/layout/style/nsCSSRuleProcessor.cpp:652) AddRule(RuleSelectorPair*, RuleCascadeData*) (/home/njn/moz/mi2/layout/style/nsCSSRuleProcessor.cpp:2849) nsCSSRuleProcessor::RefreshRuleCascade(nsPresContext*) (/home/njn/moz/mi2/layout/style/nsCSSRuleProcessor.cpp:3204) nsCSSRuleProcessor::GetRuleCascade(nsPresContext*) (/home/njn/moz/mi2/layout/style/nsCSSRuleProcessor.cpp:3145) nsCSSRuleProcessor::RulesMatching(AnonBoxRuleProcessorData*) (/home/njn/moz/mi2/layout/style/nsCSSRuleProcessor.cpp:2355) _ZL17EnumRulesMatchingI24AnonBoxRuleProcessorDataEbP21nsIStyleRuleProcessorPv (/home/njn/moz/mi2/layout/style/nsStyleSet.cpp:479) nsStyleSet::FileRules(bool (*)(nsIStyleRuleProcessor*, void*), void*, nsIContent*, nsRuleWalker*) (/home/njn/moz/mi2/layout/style/nsStyleSet.cpp:778) nsStyleSet::ResolveAnonymousBoxStyle(nsIAtom*, nsStyleContext*) (/home/njn/moz/mi2/layout/style/nsStyleSet.cpp:1207) nsCSSFrameConstructor::ConstructRootFrame(nsIFrame**) (/home/njn/moz/mi2/layout/base/nsCSSFrameConstructor.cpp:2557) PresShell::Initialize(int, int) (/home/njn/moz/mi2/layout/base/nsPresShell.cpp:1689) nsContentSink::StartLayout(bool) (/home/njn/moz/mi2/content/base/src/nsContentSink.cpp:1181) nsXMLContentSink::MaybeStartLayout(bool) (/home/njn/moz/mi2/content/xml/document/src/nsXMLContentSink.cpp:899) nsXMLContentSink::HandleStartElement(unsigned short const*, unsigned short const**, unsigned int, int, unsigned int, bool) (/home/njn/moz/mi2/content/xml/documen t/src/nsXMLContentSink.cpp:1065) nsXMLContentSink::HandleStartElement(unsigned short const*, unsigned short const**, unsigned int, int, unsigned int) (/home/njn/moz/mi2/content/xml/document/src/ nsXMLContentSink.cpp:953) non-virtual thunk to nsXMLContentSink::HandleStartElement(unsigned short const*, unsigned short const**, unsigned int, int, unsigned int) (/home/njn/moz/mi2/cont ent/xml/document/src/nsXMLContentSink.cpp:955) nsExpatDriver::HandleStartElement(unsigned short const*, unsigned short const**) (/home/njn/moz/mi2/parser/htmlparser/src/nsExpatDriver.cpp:385)

Removing the MemShrink tag because this is a meta-bug and we have tags on all the dependent bugs.

(Not working on this.)

B2G is no longer a thing.