Memory usage spikes are detected by Mozmill endurance tests between Oct 5 and Oct 6 nightlies, on all OSs. The averages heighten as follows: Linux: 89 MB -> 111 MB (22MB) Mac: 176 MB -> 202 MB (26MB) Windows: 78 MB -> 116 MB (38MB) The above high averages are still maintained. Report 2011-10-05 (low values): http://mozmill-release.brasstacks.mozilla.com/#/endurance/reports?branch=10.0&platform=All&from=2011-10-05&to=2011-10-05 Report 2011-10-06 (high values): http://mozmill-release.brasstacks.mozilla.com/#/endurance/reports?branch=10.0&platform=All&from=2011-10-06&to=2011-10-06 Changeset: http://hg.mozilla.org/mozilla-central/rev/8c82de08425d

I don't think that's the right changeset. Can you please give us the link to the pushlog for all checked-in patches on that day?

Pushlog between changesets: http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=70e4de45a0d0&tochange=8c82de08425d This includes bug 414946, which was highlighted as a major change that we should track.

bug 414946 only changed Mac, so there's no way it would change memory usage on Linux and Windows.

It could be http://hg.mozilla.org/mozilla-central/rev/5e8ac8219e2d, although that change should have *reduced* our memory usage. I'll bisect this using the instructions at [1]. [1] https://wiki.mozilla.org/QA/Automation_Services/Projects/Endurance_Tests/Documentation

Thanks Justin, I'm just running a bisect now. Will report what I find.

I'm unable to reproduce this locally (Mac OS X 10.7.1) with a smaller number of iterations (10). In fact, I see the memory _decrease_ with the later changeset... I'm now running with 100 iterations to see if this makes a difference.

I was able to reproduce on my Linux box and bisected down to [1], which is surprising, but not inconceivable. I ran the default number of iterations (one?), though. Dave, if you're running many iterations, maybe you could check whether that changeset causes the problem for you? [1] http://hg.mozilla.org/mozilla-central/rev/5e8ac8219e2d

I'm having no luck reproducing this locally. 100 iterations was still reporting less memory on the later changeset: 10 iterations: 70e4de45a0d0: avg. resident: 166MB - http://mozmill-archive.brasstacks.mozilla.com/#/endurance/report/33b7a45edd451740c5fdd9aa4e2932a1 8c82de08425d: avg. resident: 145MB - http://mozmill-archive.brasstacks.mozilla.com/#/endurance/report/33b7a45edd451740c5fdd9aa4e2a99a7 100 iterations: 70e4de45a0d0: avg. resident: 156MB - http://mozmill-archive.brasstacks.mozilla.com/#/endurance/report/33b7a45edd451740c5fdd9aa4e5adcbd 8c82de08425d: avg. resident: 148MB - http://mozmill-archive.brasstacks.mozilla.com/#/endurance/report/33b7a45edd451740c5fdd9aa4e40c3b2 Justin: Can you try running with >1 iterations to see if you can replicate this issue? The command line should be something like: ./testrun_endurance.py --iterations=10 --entities=10 --report=http://mozmill-archive.brasstacks.mozilla.com/db/ Nightly.app

I'll run that command, sure. FWIW, in my tests, I saw a much bigger difference in the max reported RSS, rather than the average. You may also want to try on a 10.6 machine; this regression may only be apparent on platforms where we use jemalloc, which is everywhere except Mac 10.5 and 10.7.

Before [1], after [2]. I don't see a significant difference in RSS here. Strange... What exactly does the number of iterations specify? The page says "Restart: No"; does it not restart the browser between each iteration? Is there a way to get the raw data (i.e., the RSS for each iteration)? [1] http://mozmill-archive.brasstacks.mozilla.com/#/endurance/report/33b7a45edd451740c5fdd9aa4e5f7445 [2] http://mozmill-archive.brasstacks.mozilla.com/#/endurance/report/33b7a45edd451740c5fdd9aa4e65e231

Sorry, I was actually running these tests on Mac OS X 10.6.8. Iterations is the number of times the test snippet will be repeated. By default the browser should be restarted between each _test_. I'm not sure why yours is disabled, as you would need to pass --no-restart on the command line to do that. Is your mozmill-automation up to date? There was a while we restarts were disabled by default. To see the raw report data (including all iterations) use the following: http://mozmill-archive.brasstacks.mozilla.com/db/33b7a45edd451740c5fdd9aa4e5f7445 http://mozmill-archive.brasstacks.mozilla.com/db/33b7a45edd451740c5fdd9aa4e65e231

Dave, there's a lot of noise in the numbers I see when I look at [1] sorted by build. Maybe what we're seeing is bimodality in the data; it looks like the memory max is either ~107 or ~137. [1] http://mozmill-release.brasstacks.mozilla.com/#/endurance/reports?branch=10.0&platform=All&from=2011-10-06&to=2011-10-11

There's a lot less variation when you filter by platform. The values in the summary table are Explicit Memory though, and this issue relates to Resident. We don't currently have a suitable view in the Mozmill Dashboard, so we're using a Google Spreadsheet https://docs.google.com/spreadsheet/ccc?key=0AsTwehGIbXUkdE5CSEFqTTI1Tm9HY0RWOF80dDhsVWc#gid=8 You can see that all platforms are consistently reporting an increase in this metric since 20111006.

Ah, I see. There's still strong bimodality on Linux builds, but that's a separate issue. What's the plan for reproducing this?

The issue with the Linux builds is likely due to some of those running Firefox 64bit, which is not indicated in the dashboard reports at present. As for next steps, that's a great question and I'm open to suggestions. Henrik: What would you suggest? Would you object to me putting a custom build on qa-horus and attempting to replicate the issue there? Any other suggestions?

Feel free to setup a test environment to build Firefox, if you can't reproduce this behavior locally.

Otherwise use tinderbox builds to narrow down the range even more before doing a hg bisect: http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-debug/ http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-debug/ http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32-debug/

Okay, I think I've finally worked this out. I was unable to replicate the issue even running on the same box (in fact, memory usage drops between the two builds). It turns out this coincides with introducing the system graphics details to the endurance test reports. So the increase is likely related to a change in the endurance testing shared module and not Firefox. I'm happy to close this issue as INVALID. Thanks Mihaela for opening it, and please now use the latest results as the new baseline.