I ran DMD on MemBench today (http://gregor-wagner.com/tmp/mem). Here's something that came up at the top. Unreported: 57 blocks in block group 2 of 14,611 6,168,576 bytes (6,079,132 requested / 89,444 slop) 0.37% of the heap (0.93% cumulative); 2.33% of unreported (5.90% cumulative) Allocated at malloc (/home/njn/moz/mi4/memory/build/replace_malloc.c:151) 0x417170 moz_malloc (/home/njn/moz/mi4/memory/mozalloc/mozalloc.cpp:64) 0x7f33815d00a5 nsStringBuffer::Alloc(unsigned long) (/home/njn/moz/mi4/xpcom/string/src/nsSubstring.cpp:177) 0x7f337d3016b7 nsAString_internal::MutatePrep(unsigned int, unsigned short**, unsigned int*) (/home/njn/moz/mi4/xpcom/string/src/nsTSubstring.cpp:130) 0x7f337d301d93 nsAString_internal::SetCapacity(unsigned int, mozilla::fallible_t const&) (/home/njn/moz/mi4/xpcom/string/src/nsTSubstring.cpp:553) 0x7f337d303079 nsAString_internal::SetCapacity(unsigned int) (/home/njn/moz/mi4/xpcom/string/src/nsTSubstring.cpp:532) 0x7f337d302fbf nsAString_internal::SetLength(unsigned int) (/home/njn/moz/mi4/xpcom/string/src/nsTSubstring.cpp:583) 0x7f337d30314f bool EnsureStringLength<nsString>(nsString&, unsigned int) (/home/njn/moz/mi4/dmdd64/content/base/src/../../../dist/include/nsReadableUtils.h:359) 0x7f337b975a6b nsScriptLoader::ConvertToUTF16(nsIChannel*, unsigned char const*, unsigned int, nsAString_internal const&, nsIDocument*, nsString&) (/home/njn/moz/mi4/content/base/src/nsScr iptLoader.cpp:1070) 0x7f337b975798 nsScriptLoader::PrepareLoadedRequest(nsScriptLoadRequest*, nsIStreamLoader*, tag_nsresult, unsigned int, unsigned char const*) (/home/njn/moz/mi4/content/base/src/nsScriptLo ader.cpp:1184) 0x7f337b976240 nsScriptLoader::OnStreamComplete(nsIStreamLoader*, nsISupports*, tag_nsresult, unsigned int, unsigned char const*) (/home/njn/moz/mi4/content/base/src/nsScriptLoader.cpp:109 5) 0x7f337b975b74 nsStreamLoader::OnStopRequest(nsIRequest*, nsISupports*, tag_nsresult) (/home/njn/moz/mi4/netwerk/base/src/nsStreamLoader.cpp:100) 0x7f337afc9003 nsHTTPCompressConv::OnStopRequest(nsIRequest*, nsISupports*, tag_nsresult) (/home/njn/moz/mi4/netwerk/streamconv/converters/nsHTTPCompressConv.cpp:94) 0x7f337b00946c nsStreamListenerTee::OnStopRequest(nsIRequest*, nsISupports*, tag_nsresult) (/home/njn/moz/mi4/netwerk/base/src/nsStreamListenerTee.cpp:52) 0x7f337afc78d7 mozilla::net::nsHttpChannel::OnStopRequest(nsIRequest*, nsISupports*, tag_nsresult) (/home/njn/moz/mi4/netwerk/protocol/http/nsHttpChannel.cpp:5027) 0x7f337b101c56 non-virtual thunk to mozilla::net::nsHttpChannel::OnStopRequest(nsIRequest*, nsISupports*, tag_nsresult) (/home/njn/moz/mi4/netwerk/protocol/http/nsHttpChannel.cpp:5063) 0x7 f337b1022dd nsInputStreamPump::OnStateStop() (/home/njn/moz/mi4/netwerk/base/src/nsInputStreamPump.cpp:553) 0x7f337af78213 nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*) (/home/njn/moz/mi4/netwerk/base/src/nsInputStreamPump.cpp:374) 0x7f337af77717 non-virtual thunk to nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*) (/home/njn/moz/mi4/netwerk/base/src/nsInputStreamPump.cpp:397) 0x7f337af7830f nsInputStreamReadyEvent::Run() (/home/njn/moz/mi4/xpcom/io/nsStreamUtils.cpp:82) 0x7f337d28ee70 nsThread::ProcessNextEvent(bool, bool*) (/home/njn/moz/mi4/xpcom/threads/nsThread.cpp:628) 0x7f337d2bbf86 NS_ProcessNextEvent_P(nsIThread*, bool) (/home/njn/moz/mi4/dmdd64/xpcom/build/nsThreadUtils.cpp:237) 0x7f337d22d639 mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) (/home/njn/moz/mi4/ipc/glue/MessagePump.cpp:82) 0x7f337cef1c2d MessageLoop::RunInternal() (/home/njn/moz/mi4/ipc/chromium/src/base/message_loop.cc:216) 0x7f337d329206 I think this block might disappear once everything's finished loading, but it can take up plenty of space while things are loading. I couldn't work out where the allocated blocks ended up being stored. nsStreamLoader::mContext is involved, but I couldn't get further than that.

The exact stack above is showing the allocation of the Unicode buffer for the script text. This is stored in nsScriptLoadRequest::mScriptText. It should go away when the nsScriptLoadRequest does (so typically after the script is evaluated, I would think). Multiple MiB is a bit odd unless there are lots of large scripts involved....

There were 57 of them, which is 108,220 bytes per block on average, which doesn't seem unreasonable. MemBench opens 150 sites in quick succession, and some of them were still loading stuff when I ran DMD. So the question is whether it's worth measuring this transient stuff. If I wanted to write a memory reporter for this, how do I get to the nsScriptLoadRequests? I.e. what data structure are they stored in?

There are several arrays that hold these things inside nsScriptLoader: 281 nsTArray<nsRefPtr<nsScriptLoadRequest> > mNonAsyncExternalScriptInsertedRequests; 282 nsTArray<nsRefPtr<nsScriptLoadRequest> > mAsyncRequests; 283 nsTArray<nsRefPtr<nsScriptLoadRequest> > mDeferRequests; 284 nsTArray<nsRefPtr<nsScriptLoadRequest> > mXSLTRequests; You can get to the nsScriptLoader from the document; there is precisely on nsScriptLoader per document.

Thanks, bz! Getting from an allocation stack trace to the stored location often requires someone who understands the code well.

You're welcome. Thank you again for measuring this stuff! ;)

Does calling EnsureStringLength(foo, x) and then foo.SetLength(y) realloc to a smaller jemalloc bucket if y fits in a smaller bucket than x? See nsScriptLoader::ConvertToUTF16 and bug 819868.

(In reply to Boris Zbarsky (:bz) from comment #3) > There are several arrays that hold these things inside nsScriptLoader: > > 281 nsTArray<nsRefPtr<nsScriptLoadRequest> > > mNonAsyncExternalScriptInsertedRequests; > 282 nsTArray<nsRefPtr<nsScriptLoadRequest> > mAsyncRequests; > 283 nsTArray<nsRefPtr<nsScriptLoadRequest> > mDeferRequests; > 284 nsTArray<nsRefPtr<nsScriptLoadRequest> > mXSLTRequests; > > You can get to the nsScriptLoader from the document; there is precisely on > nsScriptLoader per document. I'm part way through this. The above members contribute tiny amounts. It turns out that this is the culprit: nsTArray<PreloadInfo> mPreloads;

> Does calling EnsureStringLength(foo, x) and then foo.SetLength(y) realloc to > a smaller jemalloc bucket if y fits in a smaller bucket than x? AFAICT it doesn't. (Tracing through the code I ended up in nsTSubstring_CharT::MutatePrep and it returns early if |capacity <= curCapacity|.) But if you want to know definitively I suggest tracing through with a debugger, since you'll know exactly which SetLength() calls are the relevant ones.

> nsTArray<PreloadInfo> mPreloads; Ah. So I wonder whether we end up preloading stuff but not using it or something... Or whether we just preload it all in parallel and then have it in memory until we can run it in series.

I haven't seen this come up again. Doesn't seem important.