Closed
Bug 794420
Opened 12 years ago
Closed 12 years ago
Win debug mochitest-other runtime increased from ~90mins to 115mins in the last 2 weeks
Categories
(Testing :: Mochitest, defect)
Tracking
(firefox18+ fixed)
RESOLVED
FIXED
mozilla19
People
(Reporter: emorley, Assigned: bholley)
References
Details
(Keywords: regression, Whiteboard: [qa-])
Attachments
(2 files, 3 obsolete files)
(deleted),
text/csv
|
Details | |
(deleted),
patch
|
khuey
:
review+
bajaj
:
approval-mozilla-aurora+
|
Details | Diff | Splinter Review |
WinXp debug m-oth runs are now taking 115-120 mins, whereas two weeks ago they were more like 88-90 mins.
Reporter | ||
Comment 1•12 years ago
|
||
Last ~89 min WinXP run was:
https://tbpl.mozilla.org/?jobname=Rev3%20WINNT%205.1%20mozilla-central%20debug%20test%20mochitest-other&rev=710d9d21f533
-> https://tbpl.mozilla.org/php/getParsedLog.php?id=15241338&tree=Firefox
Then increased to ~95 mins in one of these pushes:
https://tbpl.mozilla.org/?jobname=mochitest-other&rev=fcd9acafa3ff
or
https://tbpl.mozilla.org/?jobname=mochitest-other&rev=f7c89de3ab43
Then increased to a whopping 112 mins in this push:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=mochitest-other&rev=633082d3e546
The first range might just be noise, but the latter seems excessive.
bholley, do you have any ideas? :-)
(Only seems to affect Windows, not
Blocks: 792036
Summary: Investigate WinXP debug mochitest-other runtime increase from ~90mins to 115mins in last 2 weeks → Investigate Win debug mochitest-other runtime increase from ~90mins to 115mins in last 2 weeks
Reporter | ||
Updated•12 years ago
|
Summary: Investigate Win debug mochitest-other runtime increase from ~90mins to 115mins in last 2 weeks → Win debug mochitest-other runtime increased from ~90mins to 115mins in the last 2 weeks
Assignee | ||
Comment 2•12 years ago
|
||
Seems pretty probably that it's related to the mega SpecialPowers testsuite fixup I pushed. I see two ways to narrow the problem down:
1 - Do some pushes and figure out which of those patches caused the regression. My money would be on dc84f6a237d7, a8583bee3c37, or 1f7c4fae49c0, so those are probably the only ones that we should check to start.
2 - Determine if one test (or a few tests) got a lot slower, or if the entire test suite just got slower across the board.
Ed, which do you think is easier? The first, I'd imagine?
Reporter | ||
Comment 3•12 years ago
|
||
I've just looked at the WinXP debug m-oth logs of your push and the one prior, and get:
Prior:
mochitest-chrome: 23 mins, 26 secs
mochitest-browser-chrome: 1 hrs, 1 mins, 58 secs
mochitest-ally: 5 mins, 35 secs
mochitest-ipcplugins: 1 mins, 14 secs
After the SpecialPowers fixup:
mochitest-chrome: 39 mins, 13 secs
mochitest-browser-chrome: 1 hrs, 42 secs
mochitest-ally: 5 mins, 40 secs
mochitest-ipcplugins: 1 mins, 8 secs
Will attach the test runtimes for mochitest-chrome.
Reporter | ||
Comment 4•12 years ago
|
||
Reporter | ||
Comment 5•12 years ago
|
||
Reporter | ||
Comment 6•12 years ago
|
||
Attachment #664904 -
Attachment is obsolete: true
Attachment #664905 -
Attachment is obsolete: true
Assignee | ||
Comment 7•12 years ago
|
||
Wow, awesome analysis Ed! Looks like the blame lies squarely with mochitest-chrome, which is lucky for our purposes, because most of the patches didn't touch mochitest-chrome at all. In fact, I'm pretty sure this was the only changeset that would have an affect on mochitest-chrome:
https://hg.mozilla.org/integration/mozilla-inbound/rev/1460d44ec8de
Assignee | ||
Comment 9•12 years ago
|
||
I've given up trying to reproduce this in a VM for the moment. Bisecting with some try pushes to test my theory about which commit we're dealing with:
https://tbpl.mozilla.org/?tree=Try&rev=00574bdb9666
https://tbpl.mozilla.org/?tree=Try&rev=2a2fd94ade0e
https://tbpl.mozilla.org/?tree=Try&rev=c2f895f02fed
Assignee | ||
Comment 10•12 years ago
|
||
Ok, the culprit patch was the one I predicted in comment 7. Subdividing that patch:
https://tbpl.mozilla.org/?tree=Try&rev=c4bcd76495b5
https://tbpl.mozilla.org/?tree=Try&rev=5bf9ad25c813
https://tbpl.mozilla.org/?tree=Try&rev=8fe5a715792b
Assignee | ||
Comment 11•12 years ago
|
||
Ok, so it looks like attaching the raw Components object is the culprit. That means either we're calling that function way too many times, or we're iterating over a gigantic number of scopes in FindInJSObjectScope. I added some logging that will tell us:
https://tbpl.mozilla.org/?tree=Try&rev=51f5b40ba655
Assignee | ||
Comment 12•12 years ago
|
||
Hm, looks like the issue _might_ be that we're iterating over a ton of scopes. I've fixed that over in my patches in bug 797821, so let's see if that fixes it:
https://tbpl.mozilla.org/?tree=Try&rev=6dfd704c8abb
Reporter | ||
Comment 13•12 years ago
|
||
Just wanted to say thank you for your persistence with this! :-)
Assignee | ||
Comment 14•12 years ago
|
||
Ahah! I think I regressed bug 722428.
Assignee | ||
Comment 15•12 years ago
|
||
Assignee | ||
Comment 16•12 years ago
|
||
Bingo. The hard edge here causes us to leak every window ever, a la bug 722428.
Kyle, did you ever figure out the pathology of this here? Why do we hold onto
the SpecialPowers object associated with the window after the window is gone?
It would be great if there was some way to test this...
Attachment #669730 -
Flags: review?(khuey)
No, I did not.
Comment 18•12 years ago
|
||
Sweet, now I have another hail-mary scapegoat for bug 792215! Which leaks 20% of nsGlobalWindows, only on WinXP mochitest-chrome. I'll see if that helps.
Assignee | ||
Comment 19•12 years ago
|
||
Attachment #669730 -
Attachment is obsolete: true
Attachment #669730 -
Flags: review?(khuey)
Attachment #669739 -
Flags: review?(khuey)
Comment on attachment 669739 [details] [diff] [review]
Remove hard edge from SpecialPowers to window.Components. v2
Review of attachment 669739 [details] [diff] [review]:
-----------------------------------------------------------------
I would be curious to know why Windows is more affected than the other platforms.
Attachment #669739 -
Flags: review?(khuey) → review+
Assignee | ||
Comment 21•12 years ago
|
||
Assignee | ||
Comment 22•12 years ago
|
||
Reporter | ||
Updated•12 years ago
|
Reporter | ||
Comment 23•12 years ago
|
||
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla19
Comment 24•12 years ago
|
||
Can this be uplifted to aurora as FF18 is affected and this may help us building faster :) ?
Assignee | ||
Comment 25•12 years ago
|
||
Comment on attachment 669739 [details] [diff] [review]
Remove hard edge from SpecialPowers to window.Components. v2
[Approval Request Comment]
Bug caused by (feature/regressing bug #): bug 792036
User impact if declined: Mochitest-chrome runs take 20 minutes longer on win dbg.
Testing completed (on m-c, etc.): m-c
Risk to taking this patch (and alternatives if risky): extremely low risk. test only.
String or UUID changes made by this patch: none
Attachment #669739 -
Flags: approval-mozilla-aurora?
Comment 26•12 years ago
|
||
Comment on attachment 669739 [details] [diff] [review]
Remove hard edge from SpecialPowers to window.Components. v2
Approving on aurora as the patch is low risk and check comment 24
Attachment #669739 -
Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
Reporter | ||
Comment 27•12 years ago
|
||
Updated•12 years ago
|
Whiteboard: [qa-]
You need to log in
before you can comment on or make changes to this bug.
Description
•