Joel Maher ( :jmaher ) (UTC -8) Assignee
	Description • 10 years ago

in looking at a linux private bytes regression (https://bugzilla.mozilla.org/show_bug.cgi?id=1073662#c71), it raises a question of what we should and shouldn't be measuring.

On linux we collect:
Main RSS - from the browser: http://hg.mozilla.org/build/talos/file/2bdbc49134c4/talos/pageloader/chrome/memory.js
Private Bytes - http://hg.mozilla.org/build/talos/file/2bdbc49134c4/talos/cmanager_linux.py#l74
XRes - http://hg.mozilla.org/build/talos/file/2bdbc49134c4/talos/cmanager_linux.py#l124

It could be that one or more of those are not very useful for actually tracking regressions.


On OSX, we track:
Private Bytes - (process list + filter): http://hg.mozilla.org/build/talos/file/2bdbc49134c4/talos/cmanager_mac.py#l33
Main RSS - from the browser: http://hg.mozilla.org/build/talos/file/2bdbc49134c4/talos/pageloader/chrome/memory.js


I suspect some of this is subject to concern as well.


On Windows we collect:
Main_RSS - (from the browser, just like osx and linux)
Private Bytes - http://hg.mozilla.org/build/talos/file/2bdbc49134c4/talos/cmanager_win32.py
% Process Time - http://hg.mozilla.org/build/talos/file/2bdbc49134c4/talos/cmanager_win32.py
Modified Page List Bytes - http://hg.mozilla.org/build/talos/file/2bdbc49134c4/talos/cmanager_win32.py

Please weigh in here on what we should remove from the list, change, or add.

	Emanuel Hoogeveen [:ehoogeveen]
	Comment 1 • 10 years ago

(In reply to Joel Maher (:jmaher) from comment #0)
> On linux we collect:
> Private Bytes -
> http://hg.mozilla.org/build/talos/file/2bdbc49134c4/talos/cmanager_linux.
> py#l74

This tallies up all the mappings flagged as writable. The problem with this is that it doesn't take into account pages that have been MADV_DONTNEED - these mappings technically still have the writable flag, but we know Linux drops the physical pages.

(In reply to Joel Maher (:jmaher) from comment #0)
> On OSX, we track:
> Private Bytes - (process list + filter):
> http://hg.mozilla.org/build/talos/file/2bdbc49134c4/talos/cmanager_mac.py#l33

From GetProcessData (a bit farther up), this seems to actually be the Virtual Memory Size. I don't know how much sense it makes to track this, since we seem to be pretty stable around 3.8 GiB.

	Joel Maher ( :jmaher ) (UTC -8) Assignee
	Comment 2 • 10 years ago

is there a way to adjust our private bytes collection for linux to ignore MADV_DONTNEED?


It sounds like there is a good argument for disabling private bytes for osx.

	Mike Hommey [:glandium]
	Comment 3 • 10 years ago

(In reply to Joel Maher (:jmaher) from comment #2)
> is there a way to adjust our private bytes collection for linux to ignore
> MADV_DONTNEED?

Parse /proc/pid/smaps instead of /proc/pid/maps, it has more info.

	Joel Maher ( :jmaher ) (UTC -8) Assignee
	Comment 4 • 10 years ago

it looks like we had a couple more regression emails (showed up this morning) related to private bytes on linux32, I should take care of this sooner than later.

;glandium, is the format of /proc/pic/smaps the same as maps?  Would the same code in http://hg.mozilla.org/build/talos/file/2bdbc49134c4/talos/cmanager_linux.py#l74 work on /proc/pids/smaps?

If so, I can change that easily and disable the collection for osx10.8!

	Emanuel Hoogeveen [:ehoogeveen]
	Comment 5 • 10 years ago

(In reply to Joel Maher (:jmaher) from comment #4)
> is the format of /proc/pic/smaps the same as maps?  Would the
> same code in
> http://hg.mozilla.org/build/talos/file/2bdbc49134c4/talos/cmanager_linux.
> py#l74 work on /proc/pids/smaps?

From [1], it's quite different. I'm not really sure what the calculation could look at to get what we want - perhaps the 'Locked' size of each mapping? The example puts that number at 0, so I'm not sure if that would work in practice.

Also, note that it says "The /proc/[pid]/smaps file is present only if the CONFIG_PROC_PAGE_MONITOR kernel configuration option is enabled."

[1] http://man7.org/linux/man-pages/man5/proc.5.html

	Mike Hommey [:glandium]
	Comment 6 • 10 years ago

Locked is for mlock()ed pages. Rss is what you probably want. Which would then only differ to the RSS bytes we already capture by the fact that they only apply to writable mappings.

	Joel Maher ( :jmaher ) (UTC -8) Assignee
	Comment 7 • 10 years ago

hmm, this is sounding more and more like the rss we collect from the browser is going to be the most valuable for linux memory collection.  At the very least, lets turn off osx private bytes.

	Joel Maher ( :jmaher ) (UTC -8) Assignee
	Comment 8 • 10 years ago

anyone else can jump in here and veto this.

	William Lachance (:wlach)
	Comment 9 • 10 years ago

Comment on attachment 8524591 [details] [diff] [review]
disable private bytes on osx (1.0)

The code change looks fine, I don't know enough about the state of memory measurement on Linux/Mac to say whether this is the right thing though.

	Joel Maher ( :jmaher ) (UTC -8) Assignee
	Comment 10 • 10 years ago

https://hg.mozilla.org/build/talos/rev/0f02ba39f027