Andrew McCreight [:mccr8] Reporter
	Description • 11 years ago

Recent tree closures (bug 932781 and bug 937997) have indicated we don't really know what is going on with memory usage in Mochitests.  There are a variety of problems, from absurdly memory intensive tests (bug 933072 and some other one in bc) to actual leaks either in the test harness or in the actual code (such as bug 932867 and bug 937997).  None of these things have been particularly difficult to fix, we just haven't measured them.

In bug 937997, shu and froydnj did some cool stuff to check mochitest-bc memory usage at various points, via about:memory and DMD.  I think we should expand that across all mochitests.  I bet we could find more leakiness, places more memory reporters would be useful, more tests that use immense amounts of memory, generally improving our stability in both the browser and TBPL.

I'd like to land code so somebody can easily collect about:memory or DMD reports at many checkpoints, though it would be off by default.  We also need some way to visualize or analyze a set of measurements taken over time.  If it is useful enough, we could have some kind of arewemochitestslimyet.com-ish thing to track regressions.  areweslimyet is great for the stuff it tests, but Mochitests run a much larger percentage of our code.

	Nathan Froyd [:froydnj]
	Comment 1 • 11 years ago

An about:memory dump a third of the way through mochitest-bc contains this:

4,009 (100.0%) -- observer-service
└──4,009 (100.0%) -- referent
   ├──2,428 (60.56%) -- weak
   │  ├──1,763 (43.98%) ── dead
   │  └────665 (16.59%) ── alive
   └──1,581 (39.44%) ── strong

1,839 (100.0%) -- observer-service-suspect
├────799 (43.45%) ── referent(topic=formsubmit)
├────486 (26.43%) ── referent(topic=xpcom-shutdown)
├────289 (15.72%) ── referent(topic=dom-window-destroyed)
└────265 (14.41%) ── referent(topic=memory-pressure)

A similar dump from ~150 tests earlier has:

1,888 (100.0%) -- observer-service
└──1,888 (100.0%) -- referent
   ├────953 (50.48%) ── strong
   └────935 (49.52%) -- weak
        ├──497 (26.32%) ── dead
        └──438 (23.20%) ── alive

1,094 (100.0%) -- observer-service-suspect
├────363 (33.18%) ── referent(topic=xpcom-shutdown)
├────337 (30.80%) ── referent(topic=formsubmit)
├────225 (20.57%) ── referent(topic=memory-pressure)
└────169 (15.45%) ── referent(topic=dom-window-destroyed)

Pretty sure those numbers shouldn't be changing that much over the course of the test.  Working on hunting those down.

	Ed Morley [:emorley]
	Comment 2 • 10 years ago

Nathan - big thanks for your efforts digging into this - especially since it seems like at least one of the issues identified has resulted in fixes to real-world use cases \o/ (Bug 961793 fixed bug 964386)

	Andrew McCreight [:mccr8] Reporter
	Comment 3 • 10 years ago

In bug 990355, we're OOMing during worker chrome tests because test_fileSubWorker.xul spins up 8 workers at the same time, and thanks to GGC, each worker consumes 25MB of contiguous space for a nursery.

	Andrew McCreight [:mccr8] Reporter
	Comment 4 • 10 years ago

Excuse me, 16MB per worker.