Before enabling automatic tab unloading on Nightly, we should land telemetry probes that allow us to evaluate 1) how much tab unloading interferes with users' browsing and 2) how effective it is at reducing OOM crashes. We might not need any new probes for #2.

As a starting point, the Fission team came up with the following ideas for what to measure.

• Tab unload count
  o How often do we unload tabs?
• Tab reload count
  o How often do we reload unloaded tabs?
• Time from unload to reload starting
  o How long have unloaded tabs been unloaded before we reload them?
  o This will tell us whether our unloading heuristics is unloading too aggressively.
  o 30 seconds would be bad. 30 hours would be good.
• Time from reload to page loaded
  o Do we see any performance overhead for reloading unloaded tabs compared to reloading already loaded page?
  o For this metric, Randell Jesup recommends the Page Load event over FCP (First Contentful Paint) or TTFI (Time To First Interactive).

With the new design of Bug 1701368, I'm thinking of the following measurements to evaluate TabUnloader:

1. TAB_UNLOAD_TO_RELOAD histogram
   This will cover "Time from unload to reload starting" and "Tab reload count" above. We can add a timestamp to a tab when it is unloaded, and then get a duration when the same tab is reloaded.

2. memory_watcher category in the event ping
   This will cover "Tab unload count" above. When we detect the memory is no longer low, we can record the number of tabs we unloaded and the number of memory-pressure events we dispatched since the low-memory situation started. It would be also useful to record how long the low-memory situation lasted.

Based on the discussion in Slack, we will skip "Time from reload to page loaded" for now. It seems that we have not collected "reloading already loaded page", so implementing both "reloading unloaded tabs" and "reloading already loaded page" will take some time. The other metrics are more important than this one for the purpose to evaluate TabUnloader.

In addition to the two metrics mentioned in comment 2, I realized we need two more metrics to simply represent the Tab unload/reload count to make a comparison.

3. browser.engagement.tab_unload_count scalar
   Incremented by 1 when TabUnloader successfully unloaded a tab.
4. browser.engagement.tab_reload_count scalar
   Incremented by 1 when an unloaded tab was reloaded.

Here are the questions that can be answered with the telemetry above:

• How many times did we unload a tab and reload an unloaded tab in the process?
• What is the percentage of reloaded-after-unloaded tabs to unloaded tabs?
  • 10% would be ok, but 50% would be too many.
• If an unloaded tab is reloaded by the user, how long has it been unloaded?
  • 30 hours would be good, but 30 seconds would be too short.
• When the memory situation moves back to normal, what have we done since the low-memory started?
  • If no tab was unloaded and no memory-pressure was sent, low memory was caused by some external factor and it was gone on its own without our efforts.
  • If many tabs were unloaded and no memory-pressure was sent, unloading a tab worked well.
  • If few tabs were unloaded and memory-pressure were sent, unloading a tab did not help a lot, and memory-pressure was helpful.
• How often did the memory situation move back to normal in the same process, and how long was each low-memory situation?
  • If it's short and too often, probably we were on the border between low-memory and high-memory, switching back and forth.

Comment on attachment 9231415 [details]
data-review-request-bug-1715858

PRELIMINARY NOTES:
Oh wow, this is a most excellently-filled-out form. Future folks will benefit greatly from how you've broken down what meaning to ascribe changes in each metric. Lovely!

DATA COLLECTION REVIEW RESPONSE:

Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes.

Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, Toshihito Kikuchi is responsible.

Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

Is the data collection request for default-on or default-off?

Default on for all channels.

Does the instrumentation include the addition of any new identifiers?

No.

Is the data collection covered by the existing Firefox privacy notice?

Yes.

Does the data collection use a third-party collection tool?

No.

Result: datareview+

The new histogram TAB_UNLOAD_TO_RELOAD records how long a tab had been
unloaded until it was reload by a user. With this data, we can evaluate
the selection logic to choose a tab to unload. For example, if many of
unloaded tabs are reloaded within 30 seconds or so, we unload a wrong tab.

This patch adds two scalars:

• browser.engagement.tab_unload_count: Incremented by 1 when TabUnloader
  successfully unloaded a tab.
• browser.engagement.tab_reload_count: Incremented by 1 when an unloaded
  tab was reloaded.

The value of browser.engagement.tab_reload_count will be the same as
the count of the histogram TAB_UNLOAD_TO_RELOAD data, but we still need
this scalar to compare it to browser.engagement.tab_unload_count, which
tells us the ratio of reload-after-unload tabs to all unloaded tabs.

Depends on D120019

The new event is recoreded when we detect the system's memory is no longer low.
Each event object contains three numbers:

1. how many times a tab was unloaded during the low-memory situation
2. how many memory-pressure events were dispatched during the low-memory
   situation
3. how long we were in the low-memory situation in seconds

These need to be collected as the event ping because the memory situation
may repeatedly be switched between low-memory and high-memory. We want to
collect it every time the memory situation gets back to normal. If we collect
it as the main ping (like Histogram), all numbers are summed up and we cannot
evaluate each low-memory period.

Depends on D120020

https://hg.mozilla.org/mozilla-central/rev/6a6b0113e4b0
https://hg.mozilla.org/mozilla-central/rev/e44a25753c4e
https://hg.mozilla.org/mozilla-central/rev/8daa945f97b4