We should look into a method to snapshot the current state of Firefox while a user is running into perf issues for diagnosis by Mozilla engineers. Here's my initial put sent via email: "In an ideal world, when a user runs into performance problems, they could press a key combination when a hang is happening in order to provide us with all of the information we need to diagnose. Apple has something similar (see [1]), and it was absolutely crucial in investigating perf issues. 'The stackshot daemon is used to capture stack traces for each thread on the system; this includes both user space and kernel stacks. The resulting view of the system is internally consistent. Stack pages that are paged out are not captured--this caveat does not apply to kernel space stacks, which are wired. The stack snapshot is triggered upon pressing a special key chord; this is currently Control-Option-Command-Shift-Period' <...> [1] https://developer.apple.com/library/mac/#documentation/Darwin/Reference/Manpages/man1/stackshot.1.html"

The main driver for this is to find the root cause for performance regressions between Firefox versions, many times found once the build is released to beta users: * In FF10, GC/CC pauses were investigated for at least 2-3 weeks ahead of release due to a reported perf regression without ever finding the root cause. * In FF11, general "slowness" was reported prior to release. Cheng think he may have narrowed it down to a page load issue. Without any further leads or the ability to easily diagnose an external user's perf issues, we may not fully understand the issue ahead of release. The end goal would be to tell all Firefox users to "press key-key-key to create a snapshot of Firefox when you run into hangs, slowness, high memory usage, etc." and then have them send the resulting output through SUMO to Mozilla engineers.

This is the goal of the 'Gecko Profiler', about:jank. We're still working on a release targeting developers before it gets picked up part of snappy and is used in any automated way (this is likely far away with current resources). Currently, on mac, you can switch to 'nightly-profiling' branch and install the 'Gecko Profiler' extension, and surf normally. Once you detect a hang you hit Command+Shift+O and you upload the profile and file a bug. This is similar to Shark's WTF mode but is much easier to use and share. See bug 722188 for an example of a bug I found this the profiler during day-day to surfing.

(In reply to Benoit Girard (:BenWa) from comment #2) > This is the goal of the 'Gecko Profiler', about:jank. We're still working on > a release targeting developers before it gets picked up part of snappy and > is used in any automated way (this is likely far away with current > resources). > > Currently, on mac, you can switch to 'nightly-profiling' branch and install > the 'Gecko Profiler' extension, and surf normally. Once you detect a hang > you hit Command+Shift+O and you upload the profile and file a bug. This is > similar to Shark's WTF mode but is much easier to use and share. > > See bug 722188 for an example of a bug I found this the profiler during > day-day to surfing. It's awesome to hear that we already had an effort underway, and that it's been proven to help diagnose perf issues! What data would be needed to include Gecko Profiler work in the Snappy effort and get more eyes on productizing it? Right now our telemetry data isn't reliable due to bug 731662 and bug 707320, and the only other channel for perf feedback is Twitter and input (which Cheng notes has had an uptick in mentions of slowness over the past couple of releases). The one remaining piece to the puzzle would of course be to come up with a submission/triage process for captured profiles.

So currently Snappy is working on a short term solution: bug 712109. Unlike the profiling addons that require user input, this tool will collect data in the background.

(In reply to Benoit Girard (:BenWa) from comment #4) > So currently Snappy is working on a short term solution: bug 712109. Unlike > the profiling addons that require user input, this tool will collect data in > the background. I took a look at the attached chromehang info in bug 712109 ( https://bugzilla.mozilla.org/attachment.cgi?id=593497) but it's unclear to me if the data collected will help identify areas with perf regressions, or root cause of the issues themselves. Would you happen to know?

(In reply to Alex Keybl [:akeybl] from comment #5) > (In reply to Benoit Girard (:BenWa) from comment #4) > > So currently Snappy is working on a short term solution: bug 712109. Unlike > > the profiling addons that require user input, this tool will collect data in > > the background. > > I took a look at the attached chromehang info in bug 712109 ( > https://bugzilla.mozilla.org/attachment.cgi?id=593497) but it's unclear to > me if the data collected will help identify areas with perf regressions, or > root cause of the issues themselves. Would you happen to know? It'll help to prioritize embarrassing hangs(we'll start with 30s and work our way down). It'll find root causes(assuming they can be seen from the stack). I suspect this bug is a dupe of bug 732522.