(No longer employed by Mozilla) Aaron Klotz Reporter
	Description • 8 years ago

When I sat down this morning to start working, I noticed that my Nightly from September 22 was extremely unresponsive.

I managed to get a profile and from what I see here, most of the time is being spent in GC. Notice that the average event lag has dropped to ~46 seconds!

See the profile in URL field.

	Jon Coppeard (:jonco) (PTO until 14th September)
	Comment 1 • 8 years ago

(In reply to Aaron Klotz [:aklotz] from comment #0)
The stack trace indicates this is coming from nsJSEnvironmentObserver::Observe so it's performing full non-incremental GCs in response to a memory pressure event.  Was your system running low on memory?  I think this can be triggered by the OS or by an allocation failure internally.

	(No longer employed by Mozilla) Aaron Klotz Reporter
	Comment 2 • 8 years ago

Private working set was hovering at around 1.5GB.

	(No longer employed by Mozilla) Aaron Klotz Reporter
	Comment 3 • 8 years ago

Thanks Jon. I'm going to change components since this looks more like a problem with our memory pressure detection than with GC itself.

Gabriele, I'm wondering if this could be fallout from bug 1301667. Any thoughts?

	(No longer employed by Mozilla) Aaron Klotz Reporter
	Comment 4 • 8 years ago

Interesting: Once I finally closed this misbehaving Firefox process, I also experienced a shutdown crash:

https://crash-stats.mozilla.com/report/index/9c9277d9-9da5-4a99-af26-0b17f2160928

It looks to me like the memory pressure observer was still firing even though the observer was nulled out!

	(No longer employed by Mozilla) Aaron Klotz Reporter
	Comment 5 • 8 years ago

Further notice that the available virtual memory at the time of the crash is around 318.5MB... could we be constantly cycling in and out of a memory pressure state? ie, GC increases free memory above 250MB, but then something happens right away to drop it under, GC fires again, and we just keep ping-ponging back and forth...

	Gabriele Svelto [:gsvelto] Assignee
	Comment 6 • 8 years ago

(In reply to Aaron Klotz [:aklotz] from comment #5)
> Further notice that the available virtual memory at the time of the crash is
> around 318.5MB... could we be constantly cycling in and out of a memory
> pressure state? ie, GC increases free memory above 250MB, but then something
> happens right away to drop it under, GC fires again, and we just keep
> ping-ponging back and forth...

Yes, that's possible and it's perfectly possible that my patch in bug 1301667 pushed the boundary just so that in your scenario memory usage ends up hovering around it. We had a similar problem in Firefox OS (bug 1234176) which I fixed by implementing a floating trigger with an exponential back-off mechanism. That was strictly FxOS-specific but the same logic could be used in the available memory tracker on Windows. I'll try to cook up a patch tomorrow so you can test it and see if it helps.

	Gabriele Svelto [:gsvelto] Assignee
	Comment 7 • 8 years ago

Sorry for the delay; I've been busy with bug 1264367 and couldn't find some time for this. I'll look into it ASAP.

	Gabriele Svelto [:gsvelto] Assignee
	Comment 8 • 8 years ago

Bug 1307018 is showing the same symptoms. To play it safe I'll tune back the parameters I've changed to the previous values they had since the symptoms here are pretty bad. I'll replace them with a floating trigger as soon as I have some spare time to work on it.

	Gabriele Svelto [:gsvelto] Assignee
	Comment 9 • 8 years ago

Seeing the issues here and in bug 1307018 I'd like to play it safe and revert the thresholds to their previous values.

I'll open a new bug to implement a generic floating trigger like the one I made for B2G in bug 1234176. The idea is to have two levels of memory pressure (hard and soft) and an exponential back-off mechanism to switch between the two so that we don't enter a cycle where we keep pointlessly jumping above and below a fixed threshold.

	Pulsebot
	Comment 10 • 8 years ago

Pushed by gsvelto@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/7d77126c45ee
Revert the low memory thresholds to their previous values r=njn

	Phil Ringnalda (:philor)
	Comment 11 • 8 years ago
bugherder

https://hg.mozilla.org/mozilla-central/rev/7d77126c45ee