User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36 OPR/58.0.3135.79

Steps to reproduce:

As recommended in bug 675539 I register separate bugreport about low-memory detection on Linux.

https://bugzilla.mozilla.org/show_bug.cgi?id=675539#c200

Possible way to detect low-memory on Linux is PSI https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/Documentation/accounting/psi.txt which should be available since Linux 4.20.

You could also check google's solution
https://bugs.chromium.org/p/chromium/issues/detail?id=463606#c23

My suggestion? If a system has no swap, you check for some cutoff of free RAM (and possibly cache?) and start discarding tabs as you approach it.

If swap is on, I would be curious to see the result of measuring pageout/pagein rates of the firefox process to indicate when to discard tabs. ps has a flag for showing major page faults (paging to disk) per process so presumably there is a POSIX call for this information.

My thought is, on something like a 4GB system if you have firfox plus some other stuff going on you can easily have 600-800MB of "junk" get swapped out (least recently used after all... I suppose it's the loaded but inactive daemons, various chunks of gnome, etc..) resulting in plenty of RAM for the working set. So measuring total system page rate will not provide useful info for when to discard tabs. Your two scenarios where you'd get firefox pageout; 1) You start some other large application, so it would begin to page firefox. Cuttings tabs then would presumably increase performance of both firefox and the other application. 2) firefox RAM usage has crept up enough to start paging, an appropriate time to cut RAM usage.

Cheers!
--Henry

I realized that the major page fault is actually measuring page ins not page outs.. so it would not stop firefox from being swapped out to begin with. Nevertheless the most problematic swap behavior is thrash (one too many tabs, and performance drops off a cliff as the active working set exceeds available RAM..) You would have probably a moment of slowdown as firefox starts to thrash, then the tabs would be cleared and it'd speed back up. Not ideal but much better than having that "one too many tabs" bring the system to it's knees thrashing.

Cheers!
--Henry

New link to documentation: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/Documentation/accounting/psi.rst

Also https://serverfault.com/questions/437138/which-metric-should-i-use-to-determine-when-a-server-is-low-on-memory (PSI info, plus others)

Also there's the mechanism used by Webkit:
https://trac.webkit.org/browser/webkit/trunk/Source/WebKit/UIProcess/linux/MemoryPressureMonitor.cpp
https://trac.webkit.org/browser/webkit/trunk/Source/WTF/wtf/linux/CurrentProcessMemoryStatus.cpp

Kris, I'm assigning this bug to you because I've been told that you will help integrate Linux's memory pressure events with Firefox tab unloading. Thanks!

...MemAvailable versus MemTotal.

• If memory drops below 5% availability, we are in a memory pressure scenario
• If MemAvailable is not large enough to accommodate a content process, we are in a memory pressure scenario
• If we are in a memory pressure scenario, notify the observers from the main thread.

The value I decided to use to represent a content process was based on observation and should be adjusted if it is not representative of what we consider a "typical" content process.

This WIP needs the following:

• Break out memory pressure monitoring onto its own thread, to avoid getting queued up behind offthread frees.
• Finish wiring up nsIObserver and use it to shut down the timer, the thread, and meminfo calls. In its current state it will continue polling indefinitely until timers shut down.
• Make the polling interval adjustable based on the available memory; we do not need to poll as frequently in extremely high memory scenarios.
• Observe user-interaction-active and user-interaction-inactive, and use these to start/stop memory polling. We don't need to poll when the user is inactive.
• Create a new notification specific to observed memory pressure. This will let us be more proactive in unloading tabs.
• Set a crash annotation when system memory is low.

In its current state you can easily test it by making a large allocation (or adjusting any of the memory thresholds). It will fire off a memory-pressure notification in these scenarios.

Tab unloading is a nice-to-have for our Fission Release experiment, but doesn't need to block it. I will move this bug from Fission Milestone M8 to MVP.

This patch makes the browser.low_commit_space_threshold_mb pref available to linux and also adds a percent value pref.

This bug is a soft blocker for Fission MVP. We'd like to fix it before our Release channel rollout, but we won't delay the rollout waiting for it.

Please consider looking into using already done glib feature https://www.hadess.net/2019/12/gmemorymonitor-low-memory-monitor-2nd.html for this task

Please consider looking into using already done glib feature https://www.hadess.net/2019/12/gmemorymonitor-low-memory-monitor-2nd.html for this task

Already addressed in the linked bug 1579198. It's far too new, it requires a far too new glib, far too new kernel, and so on. We have to support much older systems too.

(In reply to Gian-Carlo Pascutto [:gcp] from comment #13)

Already addressed in the linked bug 1579198. It's far too new, it requires a far too new glib, far too new kernel, and so on. We have to support much older systems too.

Stuff which linked bug addressed was user-space daemon as I understood. I wanted to refer directly to the GMemoryMonitor inside Glib.

Well, since we talking about new feature, I believe recent enough glib isn't the biggest issue.

I'm aware that Firefox has not so many resources to re-implement already existing upstream solutions in basic Linux libraries (in compare to companies as Google, which can afford to waste many resources on unimportant things), so that's why I'm reminding there is existing solution for described issue.

Stuff which linked bug addressed was user-space daemon as I understood.

If you look closely you'll see it's the same project from the same author. The code was moved into glib instead of a user-space deamon, but it doesn't change the fundamental problem from our side that we can't rely on it being present yet. Maybe in a few years after Debian/Red Hat/Ubuntu all have cycled their LTS releases :-/

Well, since we talking about new feature, I believe recent enough glib isn't the biggest issue.

Firefox updates independently of distros, so yeah, requiring a newer glib or kernel is actually pretty much a showstopper. You can be running Ubuntu 18.04 (kernel 4.15, glib 2.56) and you'll be on Firefox 91.0.2 now and will get this feature in Firefox 94.

Vendoring glib (or enough of it to support the code in question) into the Firefox tree is certainly more work than the relevant part of the patches here (and wouldn't help pre-5.3 kernels) the low memory detection is like 50 LOC, the rest is hooking it up into Firefox.

Could be runtime detection based on GLib version and not using this feature on old distributions solution?

Ubuntu 20.04, Debian 11 and the latest Fedoras have at least glib 2.64, we could presumably use weak symbols.

(In reply to David Heidelberg [:okias] from comment #14)

I'm aware that Firefox has not so many resources to re-implement already existing upstream solutions in basic Linux libraries (in compare to companies as Google, which can afford to waste many resources on unimportant things), so that's why I'm reminding there is existing solution for described issue.

I had already looked into that but it's not really a good fit for what we're trying to do here - besides having limited support outside of bleeding-edge distros. That monitor uses pressure stall information which is essentially a QOS measure. We've discussed using similar information on other platforms too to avoid swapping - because that's desirable from a user POV - but that's something we'd have to try at a later stage based on the relative success of the tab unloading initiative and the data we'll gather from it.

What we're trying to achieve here is avoid OOM scenarios for which we have proper data based on available memory measures (and which already works on other Linux-based platforms with triggers based on available memory such as Android). Using PSI for the same purpose would be an entirely new effort as we have no measurements that would allow us to correlate PSI events to OOM crashes and thus set the right thresholds to avoid them.

Clearing Fission Milestone because low-memory detection doesn't need to block shipping Fission MVP on Linux.

This is going to suspend hidden tabs too, isn't it? The "Auto Tab Discard" extension currently does so.

Backed out for causing bustage on AvailableMemoryWatcher.cpp and other failures.

• backout: https://hg.mozilla.org/integration/autoland/rev/8924882f9ca9c40e3fc7c83d03fbbdd0eda707bc

• push: https://treeherder.mozilla.org/jobs?repo=autoland&revision=44b7971a0893a7a8498e36e746b3297cedf98a6e

• failure logs:

  • /builds/worker/checkouts/gecko/xpcom/base/AvailableMemoryWatcherLinux.cpp:217:44: error: use of undeclared identifier 'gIsGtest'
  • TEST-UNEXPECTED-FAIL | browser/modules/test/browser/browser_TabUnloader.js | Uncaught exception - [Exception... "ServiceManager::GetService returned failure code:" nsresult: "0x80570016 (NS_ERROR_XPC_GS_RETURNED_FAILURE)" location: "JS frame :: resource:///modules/TabUnloader.jsm :: init :: line 164" data: no]
  • gecko/xpcom/base/AvailableMemoryWatcher.cpp:49: undefined reference to `mozilla::CreateAvailableMemoryWatcher()'

Backed out changeset 4b8b155c97a5 (Bug 1532955) for causing build bustage in TestAvailableMemoryWatcherLinux.cpp CLOSED TREE

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&collapsedPushes=842382&selectedTaskRun=G--QZYCyQmG3XLKpoWrApA.0&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=4b8b155c97a59940d010e7e46f4d09f6dda02a66

Log: https://treeherder.mozilla.org/logviewer?job_id=359951700&repo=autoland&lineNumber=72082

Backout: https://hg.mozilla.org/integration/autoland/rev/9b6a11144423c7fc3f0ccfb172dd85ec7092b2fe

https://hg.mozilla.org/mozilla-central/rev/267c8b31a363

Since the status are different for nightly and release, what's the status for beta?
For more information, please visit auto_nag documentation.

We can use this to restrict tab unloading to beta and earlier for QA testing.

Comment on attachment 9257734 [details]
Bug 1532955 - Introduce a pref to turn tab unloading on/off.

Beta/Release Uplift Approval Request

• User impact if declined: Tab unloading continues to release
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This is simply a pref that turns off the tab unloading portion of the memory monitor in 96 to give us more time for QA testing.
• String changes made/needed:

We shipped Fx96 today already. Is this something we'd want to ship in a dot release still? This patch probably should have gone into a new bug rather than being attached to one that was closed a month ago :-\

Comment on attachment 9257734 [details]
Bug 1532955 - Introduce a pref to turn tab unloading on/off.

At this point, we should move this patch to a new bug if it's something we want to go forward with.

My thoughts about this feature are that it seems stable and I haven't seen any crash trends with it. It appears to work in testing, and I've designed the memory monitor to silently fail in the event that it can't behave normally to prevent any issues on untested distros. I will check with QA to see if we want to go forward with a dot release that turns off the feature, or if we can let this version of tab unloading stay on release.

To update, we've decided to leave tab unloading enabled on release. It seems generally stable and telemetry indicates that it should be working. We don't receive OOM crashes in Linux so we can't prove it is reducing our OOM crash rate but we can use information from other platforms to indicate that it should be helping.

On firefox 96.0, how does one disable tab unloading behaviour? Set browser.low_commit_space_threshold_percent to 0?(In reply to Kris Wright :KrisWright from comment #34)

To update, we've decided to leave tab unloading enabled on release. It seems generally stable and telemetry indicates that it should be working. We don't receive OOM crashes in Linux so we can't prove it is reducing our OOM crash rate but we can use information from other platforms to indicate that it should be helping.

On firefox 96.0, how does one disable tab unloading behaviour? Set browser.low_commit_space_threshold_percent to 0?

I'm convinced I noticed a sudden crash of firefox, after setting browser.low_commit_space_threshold_percent to 0.

(In reply to vidwer+mozbugzilla from comment #36)

On firefox 96.0, how does one disable tab unloading behaviour? Set browser.low_commit_space_threshold_percent to 0?(In reply to Kris Wright :KrisWright from comment #34)

To update, we've decided to leave tab unloading enabled on release. It seems generally stable and telemetry indicates that it should be working. We don't receive OOM crashes in Linux so we can't prove it is reducing our OOM crash rate but we can use information from other platforms to indicate that it should be helping.

On firefox 96.0, how does one disable tab unloading behaviour? Set browser.low_commit_space_threshold_percent to 0?

Setting browser.tabs.unloadOnLowMemory to false disables tab unloading.

Are you experiencing problems with tab unloading or have any feedback about it?

(In reply to Haik Aftandilian [:haik] from comment #38)

(In reply to vidwer+mozbugzilla from comment #36)

On firefox 96.0, how does one disable tab unloading behaviour? Set browser.low_commit_space_threshold_percent to 0?(In reply to Kris Wright :KrisWright from comment #34)

To update, we've decided to leave tab unloading enabled on release. It seems generally stable and telemetry indicates that it should be working. We don't receive OOM crashes in Linux so we can't prove it is reducing our OOM crash rate but we can use information from other platforms to indicate that it should be helping.

On firefox 96.0, how does one disable tab unloading behaviour? Set browser.low_commit_space_threshold_percent to 0?

Setting browser.tabs.unloadOnLowMemory to false disables tab unloading.

Are you experiencing problems with tab unloading or have any feedback about it?

On Ubuntu, after having upgraded from 95.0.1 to 96.0 I started seeing spontaneous browser crashes.

@vidwer, do you have any crash reports listed in about:crashes that you can share?

(In reply to Haik Aftandilian [:haik] from comment #40)

@vidwer, do you have any crash reports listed in about:crashes that you can share?

Not yet. For the time being I reverted to firefox 95.0.1.