Last "no leak": http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1284070845.1284072931.27301.gz OS X 10.5 comm-central-trunk debug test mochitest-other on 2010/09/09 15:20:45 rev:8e3d1110643b moz:49291ab908cc "All" 'L-FAIL' from http://tinderbox.mozilla.org/showbuilds.cgi?tree=SeaMonkey&maxdate=1284099946&hours=24&legend=0&norules=1 http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1284080444.1284082943.8226.gz&fulltext=1 OS X 10.5 comm-central-trunk debug test mochitest-other on 2010/09/09 18:00:44 rev:0d8a88a129ca moz:fc0b88873756 to http://tinderbox.mozilla.org/showbuilds.cgi?tree=SeaMonkey&maxdate=1286519146&hours=24&legend=0&norules=1 OS X 10.5 comm-central-trunk debug test mochitest-other on 2010/10/07 07:39:05 rev:afb7f32df77e moz:a0e92be82b27 which gives a 1-month regression timeframe :-( "First" leak: http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1286474001.1286475697.7539.gz&fulltext=1 OS X 10.5 comm-central-trunk debug test mochitest-other on 2010/10/07 10:53:21 rev:afb7f32df77e moz:4d2eff8cc112 { TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | leaked 1270371 bytes during test execution TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | leaked 1473 instances of AtomImpl with size 20 bytes each (29460 bytes total) TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | leaked 1 instance of BackstagePass with size 24 bytes TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | leaked 13 instances of CSSImportRuleImpl with size 44 bytes each (572 bytes total) TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | leaked 70 instances of CSSImportantRule with size 16 bytes each (1120 bytes total) TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | leaked 40 instances of CSSNameSpaceRuleImpl with size 40 bytes each (1600 bytes total) } http://build.mozillamessaging.com/tinderboxpushlog/leak-analysis/?id=1286474001.1286475697.7539.gz&tree=SeaMonkey { leaked 4 DOMWINDOW(s) } Eventually, all perma-orange have been fixed. Only this leak keeps this suite orange. NB: The regression timeframe is for MacOSX, which maybe wasn't the best choice. Let me check Linux and Windows history too.

***** Linux: Last good: http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1284075375.1284078205.18251.gz Linux comm-central-trunk debug test mochitest-other on 2010/09/09 16:36:15 rev:8e3d1110643b moz:743e2ea36e51 then 1-month "broken leak report" period. First leak: http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1286471528.1286474098.31494.gz Linux comm-central-trunk debug test mochitest-other on 2010/10/07 10:12:08 rev:afb7f32df77e moz:db10eec8f8ab ***** Windows: http://tinderbox.mozilla.org/showbuilds.cgi?tree=SeaMonkey&maxdate=1284099946&hours=24&legend=0&norules=1 Still good: http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1284067548.1284070599.15871.gz WINNT 5.2 comm-central-trunk debug test mochitest-other on 2010/09/09 14:25:48 rev:15d43aabbb79 moz:9c5a7e114cda First leak: http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1284081906.1284085313.18439.gz&fulltext=1 WINNT 5.2 comm-central-trunk debug test mochitest-other on 2010/09/09 18:25:06 rev:958cb6ba12fb moz:23b8229b4cef { TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | leaked 342055 bytes during test execution TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | leaked 141 instances of AtomImpl with size 20 bytes each (2820 bytes total) TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | leaked 1 instance of BackstagePass with size 24 bytes TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | leaked 2 instances of CSSImportRuleImpl with size 48 bytes each (96 bytes total) TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | leaked 32 instances of CSSImportantRule with size 16 bytes each (512 bytes total) TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | leaked 11 instances of CSSNameSpaceRuleImpl with size 44 bytes each (484 bytes total) } http://build.mozillamessaging.com/tinderboxpushlog/leak-analysis/?id=1284081906.1284085313.18439.gz&tree=SeaMonkey { leaked 6 DOMWINDOW(s) } just before its 1-month "broken leak report" period, which (without checking further) ended with http://tinderbox.mozilla.org/showbuilds.cgi?tree=SeaMonkey&maxdate=1286519146&hours=24&legend=0&norules=1 http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1286486143.1286489693.11150.gz&fulltext=1 WINNT 5.2 comm-central-trunk debug test mochitest-other on 2010/10/07 14:15:43 rev:8d9a2e529de2 moz:1354806a3b6c { TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | leaked 585339 bytes during test execution TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | leaked 280 instances of AtomImpl with size 20 bytes each (5600 bytes total) TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | leaked 1 instance of BackstagePass with size 24 bytes TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | leaked 2 instances of BodyRule with size 16 bytes each (32 bytes total) TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | leaked 3 instances of CSSImportRuleImpl with size 44 bytes each (132 bytes total) TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | leaked 49 instances of CSSImportantRule with size 16 bytes each (784 bytes total) } Based on Windows case, the regression timeframe would be http://hg.mozilla.org/comm-central/pushloghtml?fromchange=15d43aabbb79&tochange=958cb6ba12fb 1 mozmill fix, = unrelated. http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=9c5a7e114cda&tochange=23b8229b4cef 1 push of 8 changesets, by Robert Strong: would one of bug 316890, bug 546593 or bug 594694.

Bug 546593 changed app update code and several mochitest chrome tests. If these leak in SeaMonkey and not Firefox then there is likely something new that is different about SeaMonkey since there is nothing Firefox specific about any of those tests. I would be willing to disable all of the mochitest chrome tests for SeaMonkey and you could monitor whether you are still leaking over a couple of months since this doesn't appear to happen all that often on SeaMonkey. Bug 316890 changed the updater and xpcshell tests for the updater so that can be ruled out. The remainder of the patches I pushed were ride alongs and you would need to contact the author's of those other patches for details. Is this happening with every single run? If not, then just because it started after a specific push doesn't in anyway mean that the push is to blame.

Removing dependency on bug 316890 since that changed a separate binary for executing updates which only runs during xpcshell tests so there is no possiblle way it could be responsible.

(In reply to comment #2) > I would be willing to disable all of the mochitest chrome tests > for SeaMonkey and you could monitor whether you are still leaking over a couple Finding out which test(s) leaks should be the first step, indeed. (helpwanted) I am currently trying to use MoMe Try to compile a logrefcnt SM build I could run on my Windows 2000... > of months since this doesn't appear to happen all that often on SeaMonkey. I'm not sure I understand that. > Is this happening with every single run? If not, then just because it started Yes: every run, every platform. (as far as I checked) > after a specific push doesn't in anyway mean that the push is to blame. Either we were lucky with the Windows regression timeframe, or it is misleading us: narrowing down which test(s) triggers this should hopefully tell us...

(In reply to comment #4) > > of months since this doesn't appear to happen all that often on SeaMonkey. > > I'm not sure I understand that. I was going by the lack of a bug report for this specific issue and the lack of comments in this bug in that if it were happening often there would typically be a record of the fact. > > Is this happening with every single run? If not, then just because it started > > Yes: every run, every platform. (as far as I checked) Can you provide links to a few of the logs previous to the following push? The SeaMonkey tinderboxen showed no builds when trying to go back in time for me. http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=9c5a7e114cda&tochange=23b8229b4cef

(In reply to comment #5) > I was going by the lack of a bug report for this specific issue and the lack of I don't get that either: this is the bug report for this leak. > comments in this bug in that if it were happening often there would typically > be a record of the fact. Well, I thought every bug was supposed to be permanent by default, unless marked as random-orange or the like. In addition, I hoped "Only this leak keeps this suite orange" would be explicit enough that this was permanent. > Can you provide links to a few of the logs previous to the following push? The > SeaMonkey tinderboxen showed no builds when trying to go back in time for me. What kind of logs are you looking for exactly? Odd, comment 0 and comment 1 have all the useful links I think: you should get what you are looking for by starting at http://tinderbox.mozilla.org/showbuilds.cgi?tree=SeaMonkey&maxdate=1284099946&hours=24&legend=0&norules=1 and going backward as necessary. (I just checked again: waterfall and build logs work for me...)

Another point of fact... Bug 546593 and all mochitest app update tests only run via mochitest-chrome so this couldn't be due to bug 546593 since the leak occurs in browser-chrome tests. Removing dependency.

Also notice the range for the first Seamonkey cb-sea-linux-tbox mochitest-other browser-chrome leaks No Leak: http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1284075375.1284078205.18251.gz http://hg.mozilla.org/comm-central/rev/8e3d1110643b http://hg.mozilla.org/mozilla-central/rev/743e2ea36e51 Leak: http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1284084294.1284087391.27086.gz http://hg.mozilla.org/comm-central/rev/0d8a88a129ca http://hg.mozilla.org/mozilla-central/rev/ce92aa354849 Comm-Central range: http://hg.mozilla.org/comm-central/pushloghtml?fromchange=8e3d1110643b&tochange=0d8a88a129ca Mozilla-Central range: http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=743e2ea36e51&tochange=ce92aa354849

And bug 594694 was actually just a string change, which should be unrelated. This leaves us with no suspect :-/ ***** I eventually succeeded at creating the Opt build I wanted with MoMe Try. Results are not very obvious: No leak with: caps docshell : browser_bug349769.js, browser_bug388121-1.js, browser_bug388121-2.js layout modules netwerk testing Leak with (any of): content : 2 docshell : 7 - 3 = 4 others dom : 2 I didn't checked in details: suite toolkit I noticed some reported errors on some tests but that didn't seem directly related. (I didn't check whether one of SeaMonkey extensions could somehow trigger this.) Helpwanted to figure it out further.

(In reply to comment #8) > Leak: There is no leak to see there: this build reports 'L-FAIL'. > Mozilla-Central range: And I did star it with "Bug 595024 - Add prompts for indexedDB permissions (port bug 591516 app parts)" That said, I would be pleased if that would be the underlying cause: but we lack an explanation atm.

(In reply to comment #10) > (In reply to comment #8) > > > Leak: > > There is no leak to see there: this build reports 'L-FAIL'. Sorry about that... long night. Looks like the browser-chrome no leak build before the L-FAIL build that you starred happened after the changesets in the range you listed in comment #1 had landed though. http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1284075375.1284078205.18251.gz

Took a quick look at the logs and it appears that similar leaks are happening for both the browser-chrome test run and the mochitest-a11y test run. Perhaps they have the same root cause. browser-chrome: TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | leaked 1485 instances of AtomImpl with size 20 bytes each (29700 bytes total) TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | leaked 1 instance of BackstagePass with size 24 bytes TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | leaked 12 instances of CSSImportRuleImpl with size 44 bytes each (528 bytes total) TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | leaked 75 instances of CSSImportantRule with size 16 bytes each (1200 bytes total) TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | leaked 42 instances of CSSNameSpaceRuleImpl with size 40 bytes each (1680 bytes total) mochitest-a11y: TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | leaked 282 instances of AtomImpl with size 20 bytes each (5640 bytes total) TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | leaked 1 instance of BackstagePass with size 24 bytes TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | leaked 2 instances of BodyRule with size 16 bytes each (32 bytes total) TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | leaked 3 instances of CSSImportRuleImpl with size 44 bytes each (132 bytes total) TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | leaked 49 instances of CSSImportantRule with size 16 bytes each (784 bytes total)

(In reply to comment #12) > Took a quick look at the logs and it appears that similar leaks are happening > for both the browser-chrome test run and the mochitest-a11y test run. Perhaps > they have the same root cause. I didn't talk about m-a11y here because SeaMonkey still has perma-failures in that suite (and because it's not run on MacOSX), but other than that I noticed this too and confirm the suspicion ;-) (If I remember correctly, a11y harness is +/- like browser-chrome harness: if not related to SeaMonkey extensions, it could be related to the harness? Or just some SeaMonkey code obviously (like "doorhanger" or whatever)...)

I chose a test at random to try and narrow what is triggering the leak: http://mxr.mozilla.org/mozilla-central/source/content/html/document/test/browser_bug592641.js { 24 function load1Soon() { 25 ctx.tab1Browser.removeEventListener("load", load1Soon, true); 26 // onload is fired in OnStopDecode, so let's use executeSoon() to make sure 27 // that any other OnStopDecode event handlers get the chance to fire first. 28 executeSoon(load1Done); 29 } } If I replace line 28 by { gBrowser.removeTab(ctx.tab1); finish(); } then the test doesn't leak. If I do this in load1Done(), the leak reappears. I tried some executeSoon()/setTimeout() too, which didn't help. Loading "about:blank" doesn't help either. (Sadly, already quite a long time spent on this (and a few bug filed, fwiw) but not yet a clue on what the issue actually is. I'll just let it rest for a while...)

(In reply to comment #14) > If I replace line 28 by > { > gBrowser.removeTab(ctx.tab1); > finish(); > } > then the test doesn't leak. Ah, as a last thought, I wonder if that could be related to SeaMonkey session-restore somehow?? (Iirc, SeaMonkey already had a bug +/- related to that not so long ago.)

(In reply to comment #15) > Ah, as a last thought, I wonder if that could be related to SeaMonkey > session-restore somehow?? > (Iirc, SeaMonkey already had a bug +/- related to that not so long ago.) You mean the one where bfcache removeTab doesn't work from inside a load event handler, so we just do a regular remove in that case?

Bug still there in SM 2.9 to 2.12a1.