This is a tracking bug for the various about:memory reporters that need to be added.

Here is one testcase that about:memory currently doesn't account for: 1. open any website (e.g. http://google.com) 2. execute the following bookmarklet: javascript:if(undefined == document.body.asdf){document.body.asdf = []};var a=document.body.asdf;var i,j,tmp;for(i=0;i<10000;i++){tmp=[];for(j=0;j<10000;j++){tmp[j]="a"} a.push(tmp)};alert("done");void(0); This simply attaches a huge nested array to the DOM tree (yeah, i know, bad practice) and gobbles up about 1GB of memory which doesn't show up in about:memory. If you don't get the "done" alert the script possibly failed due to an OOM abort, you'll have to close the tab and try again with lower numbers for the loop counts.

JSScripts should be tracked. With 10 gmail tabs open I saw them account for 17% of the live heap. Here's one call stack from Massif: o1o> 17.03% (70,660,318B) 0x67648A4: JSScript::NewScript(JSContext*, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigne d int, unsigned int, unsigned int, unsigned short, unsigned short, JSVersion) (jsutil.h:239) | o2o> 16.64% (69,033,931B) 0x67675CE: JSScript::NewScriptFromCG(JSContext*, JSCodeGenerator*) (jsscript.cpp:1260) | | o3o> 15.73% (65,262,175B) 0x66DBFCE: js_EmitFunctionScript (jsemit.cpp:3678) | | | o4o> 15.67% (65,014,983B) 0x66D9BE7: js_EmitTree (jsemit.cpp:4640) | | | | o5o> 08.47% (35,117,714B) 0x66D7723: js_EmitTree (jsemit.cpp:6176) | | | | | o6o> 08.46% (35,109,464B) 0x66D585A: js_EmitTree (jsemit.cpp:5984) | | | | | | o7o> 08.13% (33,744,883B) 0x66D8622: js_EmitTree (jsemit.cpp:5892) | | | | | | | o8o> 08.07% (33,496,774B) 0x66D636D: js_EmitTree (jsemit.cpp:5451)

JSObject slots should be tracked, ie. in allocSlots(), growSlots() and shrinkSlots(). With 10 gmail tabs open I saw them account for 7% of the live heap.

Also various strings are important, eg: o1o> 11.67% (48,410,287B): nsStringBuffer::Alloc() (nsSubstring.cpp:209) | o2o> 10.55% (43,754,212B): nsXMLHttpRequest::ConvertBodyToText() (nsXMLHttpRequest.cpp:762) | | o3o> 10.55% (43,754,212B): nsIXMLHttpRequest_GetResponseText() (dom_quickstubs.cpp:26975) | | o4o> 10.06% (41,739,008B): js_NativeGet() (jscntxtinlines.h:326) | | | o5o> 10.06% (41,739,008B): InlineGetProp() (StubCalls.cpp:1868) | | | | o6o> 10.06% (41,739,008B): js::mjit::stubs::GetProp() (StubCalls.cpp:1894) Not sure how best to record them -- lump all strings together, or try to be smarter than that.

Here's the raw data I'm getting these ideas from, BTW.

The Hunspell spell-check accounts for 2.7MB on my machine, it looks like it's a constant amount. See bug 653817 comment 7.

js::PropertyTable::init() and js::PropertyTable::change() together account for 4.3% in the gmail example.

My testcase from comment #2 still shows up under heap-unclassified. Considering that it is trivial to use up memory like that and many poorly written javascript libraries attach all their state to dom objects i would say it would be useful to add instrumentation for that case.

(In reply to comment #2) > Here is one testcase that about:memory currently doesn't account for: > > 1. open any website (e.g. http://google.com) > 2. execute the following bookmarklet: > javascript:if(undefined == document.body.asdf){document.body.asdf = []};var > a=document.body.asdf;var > i,j,tmp;for(i=0;i<10000;i++){tmp=[];for(j=0;j<10000;j++){tmp[j]="a"} > a.push(tmp)};alert("done");void(0); Oh, bug 571249 will fix that. (Well, the follow-up bug 664647 will do so.) I just tried your JS snippet in a build with the js/object-slots reporter enabled (plus per-compartment reporters from bug 661474): 1,624,547,222 B (100.0%) -- explicit ├──1,494,174,525 B (91.97%) -- js │ ├──1,314,183,902 B (80.90%) -- compartment(http://www.google.com.au/) │ │ ├──1,310,986,032 B (80.70%) -- object-slots │ │ ├──────1,436,610 B (00.09%) -- scripts │ │ ├────────983,040 B (00.06%) -- mjit-code │ │ ├────────572,112 B (00.04%) -- tjit-data │ │ │ ├──444,000 B (00.03%) -- allocators-reserve │ │ │ └──128,112 B (00.01%) -- allocators-main │ │ ├────────131,072 B (00.01%) -- tjit-code │ │ ├─────────69,676 B (00.00%) -- mjit-data │ │ └──────────5,360 B (00.00%) -- string-chars So that's good! :)

I've changed this back to Core/General; although the numbers show up in about:memory, the reporters themselves are mostly in Core, and about:memory doesn't need changing for them to show up once they are added.

Oh, sorry for changing the component without reading this.

I'm changing this bug's title and repurposing it slightly. Instead of being a simple tracking bug for new reporter's, it's a bug for getting the "heap-unclassified" number (which is currently typically around 35--50%) down as low as feasible. Obviously, bugs for new reporters will still block this bug. All of the things identified above have been spun-off into separate bugs (some of which have been fixed) except for the nsXMLHttpRequest strings one in comment 5. A couple more stack traces of interest from the Massif data: o1o> 05.00% (20,748,856B) nsCSSExpandedDataBlock::Compress() (mozalloc.h:229) | o2o> 04.99% (20,688,600B) (anonymous namespace)::CSSParserImpl::ParseDeclarationBlock() (Declaration.h:124) | | o3o> 04.91% (20,371,668B) (anonymous namespace)::CSSParserImpl::ParseRuleSet() (nsCSSParser.cpp:2612) | | | o4o> 04.91% (20,349,128B) nsCSSParser::Parse() (nsCSSParser.cpp:963) | | | | o5o> 04.91% (20,349,128B) mozilla::css::Loader::ParseSheet() (Loader.cpp:1585) | | | | o6o> 04.86% (20,156,400B) mozilla::css::Loader::LoadInlineStyle() (Loader.cpp:1807) | | | | | o7o> 04.86% (20,156,400B) nsStyleLinkElement::DoUpdateStyleSheet() (nsStyleLinkElement.cpp:298) | | | | | o8o> 04.76% (19,743,840B) nsStyleLinkElement::UpdateStyleSheetInternal() (nsStyleLinkElement.cpp:209) o1o> 03.81% (15,797,388B) nsTextFragment::SetTo() (nsMemory.h:68) | o2o> 03.81% (15,797,388B) nsGenericDOMDataNode::SetTextInternal() (nsGenericDOMDataNode.cpp:350) | o3o> 02.50% (10,366,215B) nsHtml5TreeOperation::AppendText() (nsHtml5TreeOperation.cpp:197) | | o4o> 02.50% (10,366,215B) nsHtml5TreeOperation::Perform() (nsHtml5TreeOperation.cpp:531) | | o5o> 02.48% (10,300,277B) nsHtml5TreeOpExecutor::RunFlushLoop() (nsHtml5TreeOpExecutor.cpp:489) | | | o6o> 02.45% (10,160,236B) nsHtml5ExecutorFlusher::Run() (nsHtml5StreamParser.cpp:153) o1o> 02.92% (12,120,960B) ChangeTable (pldhash.c:563) | o2o> 02.91% (12,068,864B) PL_DHashTableOperate (pldhash.c:649) | | o3o> 01.02% (4,233,984B) AddSelector() (nsCSSRuleProcessor.cpp:2608) | | | o4o> 01.02% (4,233,984B) nsCSSRuleProcessor::RefreshRuleCascade() (nsCSSRuleProcessor.cpp:2715) | | | | | o3o> 00.68% (2,839,040B) RuleHash::AppendRuleToTable() (nsCSSRuleProcessor.cpp:528) | | | o4o> 00.68% (2,822,656B) RuleHash::AppendRule() (nsCSSRuleProcessor.cpp:560) | | | | | o3o> 00.33% (1,367,040B) mozilla::FramePropertyTable::Set() (nsTHashtable.h:188) | | | o4o> 00.32% (1,329,152B) nsIFrame::GetOverflowAreasProperty() (FramePropertyTable.h:237) | | | | | | | o4o> 00.01% (37,888B) in 6 places, all below massif's threshold (00.10%) | | | | | o3o> 00.24% (1,009,152B) AppendRuleToTagTable() (nsCSSRuleProcessor.cpp:540) | | | o4o> 00.22% (897,024B) RuleHash::AppendRule() (nsCSSRuleProcessor.cpp:565) | | | | | o3o> 00.21% (888,832B) nsDocument::AddToIdTable(mozilla::dom::Element*, nsIAtom*) (nsTHashtable.h:188) o1o> 02.87% (11,917,512B) nsCSSSelectorList::AddSelector() (mozalloc.h:229) | o2o> 02.87% (11,907,936B) (anonymous namespace)::CSSParserImpl::ParseSelectorGroup() (nsCSSParser.cpp:3644) o1o> 01.63% (6,742,008B) (anonymous namespace)::CSSParserImpl::ParseRuleSet() (mozalloc.h:229) | o2o> 01.62% (6,713,136B) nsCSSParser::Parse() (nsCSSParser.cpp:963) | | o3o> 01.62% (6,713,136B) mozilla::css::Loader::ParseSheet() (Loader.cpp:1585) | | o4o> 01.59% (6,613,920B) mozilla::css::Loader::LoadInlineStyle() (Loader.cpp:1807) | | | o5o> 01.59% (6,613,920B) nsStyleLinkElement::DoUpdateStyleSheet() (nsStyleLinkElement.cpp:298) | | | o6o> 01.56% (6,488,640B) nsStyleLinkElement::UpdateStyleSheetInternal() (nsStyleLinkElement.cpp:209) See the attachment for full details. Looks like we really need some CSS memory reporters! There were a couple of >1% stack traces for CSS things that I haven't copied in here either.

It was suggested at today's MemShrink meeting that it might be worth someone digging into memory dumps or something else to push this along. But the relevant information is actually all there in Massif's output. The hard part of interpreting that output is knowing how the allocation traces Massif reports match up with existing memory reporters. But a similar problem would occur for anyone who dug through memory dumps. I think a better way forward would be to identify sites for which our heap-unclassified number is unusually high. Comment 2 and bug 669005 are both good examples (and both are now fixed). I know Massif is a pain to run on Firefox, so I'm happy to run it on any such sites that people can find. Or if you just know of any remaining decent-sized chunks of memory that aren't covered, please speak up. (Bug 669117 was identified this way by khuey, IIRC.)

If we want a snappy name for this bug, I propose "Dark Matter".

On Linux64, if I load just about:memory?verbose and http://blogs.technet.com/b/dmelanchthon/archive/2011/07/19/featurematrix-outlook-client-und-exchange-server-reloaded.aspx, I get a "heap-unclassified" of ~50%. The next highest bucket is "js" with 32%. Full output is attached. That page includes lots of tables.

That actually suggests an interesting approach to tracking down additional darkmatter: automatically check the heap-unclassified numbers for a bunch of webpages, and look at sites that have unusually large numbers to see what is weird about them. Relatedly, in https://bugzilla.mozilla.org/show_bug.cgi?id=670689#c23 bz says: "And by the way, I'm guessing that in this case the remaining unclassified stuff is also somewhere in the js engine. Might be worth it to look up where."

(In reply to comment #17) > That actually suggests an interesting approach to tracking down additional > darkmatter: automatically check the heap-unclassified numbers for a bunch of > webpages, and look at sites that have unusually large numbers to see what is > weird about them. Yes. See comment 14 :)

Nice catch with Bug 675132. Is it possible to use Massif (which I assume is recording the amount requesting of every malloc?) to see how much space is being wasted by allocations-slightly-more-than-powers-of-two? Or that something you already set up to ferret this out? The cycle collector allocates memory in chunks, and I could imagine it accidentally wasting a bit of space in the same way, so it could probably be happening all over the place.

(In reply to comment #19) > Is it possible to use Massif (which I assume is > recording the amount requesting of every malloc?) to see how much space is > being wasted by allocations-slightly-more-than-powers-of-two? Nope. Massif replaces the heap allocator with its own version, and the amount of rounding-up is a characteristic of the heap allocator. Well, I guess I could try to get Massif to round up the same as jemalloc, but there's an easier path forward -- see bug 675136 where I've found that jemalloc rounding could be accounting for something like 17% of "heap-allocated", i.e. roughly half of "heap-unclassified".

Just to clarify about Massif: it makes a half-hearted attempt to account for extra space needed by the allocator -- it adds 8 bytes to the amount recorded for each heap block, on the assumption that the average overhead is that size. You can change it with the --heap-admin option, but it's always a constant. So it's pretty hopeless. Also, that overhead-per-block is recorded separately from the requested-amount-per-block and it's not even shown in those allocation trees that I cut and paste into bugs all the time. So I haven't even considered jemalloc's rounding until just today. tl;dr: Massif is hopeless for measuring extra space caused by jemalloc's rounding and book-keeping.

Yeah, I meant doing some kind of post-processing step in Massif to round up the same as jemalloc, but looks like you cut out the middle man.

I was thinking some more about how to get *serious* about hunting down all the heap-unclassified bytes. You'd add instrumentation code to maintain a data structure that records 1 bit of information for every malloc'd byte. A 0 means "not reported", a 1 means "reported". You'd also add instrumentation code to record, for every heap block, the stack trace at its allocation point. Then, you'd modify every heap memory reporter so that when it is queried, it sets the "reported" bits of all the heap bytes it counts. Once about:memory was loaded, you'd iterate over all the heap blocks. Any heap block fully reported would be ignored. Any heap block partially or not-at-all reported would be recorded. You'd aggregate repeated stack traces for recorded heap blocks and present the stack traces in order so that the ones responsible for the most unreported bytes are shown first. Also, while the memory reporters are doing their thing, you'd complain about any byte on the heap whose "reported" bit was set more than once -- such bytes are double-counted. You could print out its stack trace and the name of the reporter that was second to count it. Tracking one-bit-per-byte metadata and per-heap-block stack traces is something that Valgrind excels at, BTW. A Valgrind tool that implemented this analysis would have to use its own heap allocator rather than jemalloc (that's just how Valgrind works), so it would have to be careful about the rounding up of request sizes, but it could definitely be made to work. Memory reporters would use client requests to tell the tool which bytes have been accounted for. A client request would also be used to tell the tool when all the memory reporters have been consulted. There's a fairly obvious candidate for who might implement this :P

I spun the Valgrind tool idea off as bug 676724.

Bug 291643 should be blocking darkmatter as XML produces lots of heap-unclassified

I've set a specific goal of 10% dark matter for "typical" cases, so that this bug has an end goal. We can argue about the meaning of "typical" when we're getting close. (With Gmail and this bug open, I currently see 22.93% dark matter.)

A lot of bugs are of the form "this totally non-typical page generates lots of heap-unclassified." It seems to me that bugs of that form are still worth tracking (perhaps in a separate bug), since it's hard to understand what we're doing wrong on those pages if about:memory fails us.

Something that bug 699951 made me think of is that there's one situation that's hard to measure with DMD, because it's so slow -- long sessions with lots of browsing. (I've left DMD running overnight before, but that doesn't involve opening/closing lots of sites.)

Nicholas, what about Bug 646575?

(In reply to Phoenix from comment #29) > Nicholas, what about Bug 646575? What about it?

Sholdn't it be added to this list too?

It's unclear to me how bug 646575 relates to this bug. Can you explain?

Nicholas, do you know if there are some memory reporters for the memory use of plugin-container.exe (I mean especially about Flash/Java/Silverlight)? Does it count explicitly in 'Other Measurements' in about:memory?

(In reply to Loic from comment #33) > Nicholas, do you know if there are some memory reporters for the memory use > of plugin-container.exe (I mean especially about Flash/Java/Silverlight)? > Does it count explicitly in 'Other Measurements' in about:memory? We currently do not measure plugin-container.exe at all.

Bug 648415 is open for plugin-container measurement, but no progress has been made on it.

FWIW, my "heap-unclassified" is regularly under 10% on my Linux64 box, running a trunk build. It was 6.4% at the end of the day a couple of days ago. And it's currently 10.9% on my Mac, which is on FF19 beta.

Install Stylish and this "issue" would be easily resolved :D

> Install Stylish and this "issue" would be easily resolved :D I don't understand what you mean by 'this "issue"'. Is heap-unclassified high with Stylish installed?

(In reply to Nicholas Nethercote [:njn] from comment #38) > Is heap-unclassified high with Stylish installed? Right, you'll get something like 16% from beginning and growing to ~30% on usage

I'm just curious -- I recognize that this meta-bug is mostly tracking heap-unclassified data in Firefox, and that there's a separate meta-bug for B2G. Is there a comparable bug for Thunderbird? (On my machine at the moment, heap-unclassified is at 33%, which makes it tricky to help diagnose the memory leak I seem to be seeing...)

> Is there a comparable bug for Thunderbird? Nope. You might want to talk to jcranmer, who was working on some Thunderbird memory reporters just the other day. Alternatively, DMD might help you: https://wiki.mozilla.org/Performance/MemShrink/DMD

(In reply to Nicholas Nethercote [:njn] from comment #41) > > Is there a comparable bug for Thunderbird? > > Nope. You might want to talk to jcranmer, who was working on some > Thunderbird memory reporters just the other day. Alternatively, DMD might > help you: https://wiki.mozilla.org/Performance/MemShrink/DMD I tried using DMD on Thunderbird to diagnose bug 844937. All it shows is that Thunderbird desperately needs memory reporters for mailnews/ code, which is the work jcranmer has started in bug 480843.