Closed Bug 926264 Opened 11 years ago Closed 9 years ago

Intermittent Jetpack command timed out: 1800 seconds elapsed, attempting to kill

Categories

(Add-on SDK Graveyard :: General, defect, P1)

Product:
Add-on SDK Graveyard
Component:
General
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME
Milestone:
mozilla30

People

(Reporter: philor, Unassigned)

References

Details

(Keywords: intermittent-failure)

Attachments

(4 files)

https://github.com/mozilla/addon-sdk/pull/1378
(deleted), text/x-github-pull-request
mossop
: review+
Details
https://github.com/mozilla/addon-sdk/pull/1380
(deleted), text/x-github-pull-request
mossop
: review+
Details
https://github.com/mozilla/addon-sdk/pull/1382
(deleted), text/x-github-pull-request
mossop
: review+
Details
Link to Github pull-request: https://github.com/mozilla/addon-sdk/pull/1512
(deleted), text/x-github-pull-request
jsantell
: review+
Details
https://tbpl.mozilla.org/php/getParsedLog.php?id=29064345&tree=Mozilla-Inbound Windows 7 32-bit mozilla-inbound pgo test jetpack on 2013-10-13 15:07:51 PDT for push 20ffeada8eca slave: t-w732-ix-025 Testing private-browsing-supported... Using binary at 'firefox/firefox.exe'. Using profile at 'c:\users\cltbld\appdata\local\temp\tmpatcuuq.mozrunner'. command timed out: 7200 seconds elapsed, attempting to kill program finished with exit code 1 elapsedTime=7200.347000
Priority: -- → P2
This seems to have spiked in popularity recently. Dave, Irakli, any ideas who could take a look at this?
Flags: needinfo?(rFobic)
Flags: needinfo?(dtownsend+bugmail)
Priority: P2 → P1
As best as I can tell this bug is a catch-all for a bunch of different things. The summary isn't accurate for the current set of failures. It looks like everything since 20th is the same failure, everything before looks different. Figuring out what changed then would be good, doesn't look like it is SDK code.
Flags: needinfo?(dtownsend+bugmail)
It looks like there is a TEST-PASS occurring after the TEST-END, and it looks like there is a race condition such that a pass could happen after done() is called. I"m guessing that may be the issue here.
Assignee: nobody → evold
Flags: needinfo?(rFobic)
Comment on attachment 8369242 [details] https://github.com/mozilla/addon-sdk/pull/1378 We should land this to mercurial as well to see what effect it has
Attachment #8369242 - Flags: review?(dtownsend+bugmail) → review+
Commits pushed to master at https://github.com/mozilla/addon-sdk https://github.com/mozilla/addon-sdk/commit/1880e4e581f47f4bd162446c283e1f05fb71a432 Bug 926264 - Intermittent Jetpack command timed out: 7200 seconds elapsed, attempting to kill starting up private-browsing-supported https://github.com/mozilla/addon-sdk/commit/4e2ffb53a64717015aff5a5bd51c1120934efc40 Merge pull request #1378 from erikvold/926264 Bug 926264 - Intermittent Jetpack command timed out: 7200 seconds elapsed r=@Mossop
(In reply to Dave Townsend (:Mossop) from comment #117) > Comment on attachment 8369242 [details] > https://github.com/mozilla/addon-sdk/pull/1378 > > We should land this to mercurial as well to see what effect it has https://hg.mozilla.org/integration/fx-team/rev/9626709959f2
Wups, I got out of the habit of looking at the log - comment 135 is after you landed on fx-team, but, it's not in the same place, it's in places/main.testVisitCount.
Is there at least an open bug which nobody will ever fix on having some sort of reasonable internal harness handling of hangs, instead of taking 161 minutes to have buildbot output the exact same failure message for every single hang in every single test?
(In reply to Phil Ringnalda (:philor) from comment #150) > Is there at least an open bug which nobody will ever fix on having some sort > of reasonable internal harness handling of hangs, instead of taking 161 > minutes to have buildbot output the exact same failure message for every > single hang in every single test? You're welcome to file one, I have no idea how we could implement it though. The problem is that the test run completes, as far as the test harness is concerned. But then something stops the app from shutting down. I have no idea what that is.
https://hg.mozilla.org/mozilla-central/rev/9626709959f2
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla30
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
It looks like we try to exit twice for testaddons, and this may be messing up our resultFile.
Comment on attachment 8370466 [details] https://github.com/mozilla/addon-sdk/pull/1380 Good catch, let's get this in
Attachment #8370466 - Flags: review?(dtownsend+bugmail) → review+
OS: Windows 7 → All
Hardware: x86 → All
(In reply to TBPL Robot from comment #229) > KWierso > https://tbpl.mozilla.org/php/getParsedLog.php?id=34162280&tree=Fx-Team > Ubuntu VM 12.04 x64 fx-team debug test jetpack on 2014-02-05 11:30:15 > revision: 77095ca6e53b > slave: tst-linux64-ec2-331 > > command timed out: 7200 seconds elapsed, attempting to kill Well it looks like my last commit didn't resolve the issue..
Assignee: evold → nobody
https://hg.mozilla.org/mozilla-central/rev/d1170c5a635a
Assignee: nobody → evold
Status: REOPENED → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → FIXED
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
It seems like the failing testaddons alows occur when done() is called inside of a window close listener, so I added a setTimeout before exit() is called, and I added some additional logs here.
Attachment #8371667 - Flags: review?(dtownsend+bugmail)
(In reply to Erik Vold [:erikvold] [:ztatic] from comment #269) > Created attachment 8371667 [details] > https://github.com/mozilla/addon-sdk/pull/1382 > > It seems like the failing testaddons alows occur when done() is called > inside of a window close listener, so I added a setTimeout before exit() is > called, and I added some additional logs here. "the failing testaddons all occur when"*
Comment on attachment 8371667 [details] https://github.com/mozilla/addon-sdk/pull/1382 A couple of changes I'd like to see here first
Attachment #8371667 - Flags: review?(dtownsend+bugmail) → review-
Depends on: 969053
Comment on attachment 8371667 [details] https://github.com/mozilla/addon-sdk/pull/1382 I made the requested changes I think.
Attachment #8371667 - Flags: review- → review?(dtownsend+bugmail)
Comment on attachment 8371667 [details] https://github.com/mozilla/addon-sdk/pull/1382 Perfect, thanks
Attachment #8371667 - Flags: review?(dtownsend+bugmail) → review+
Commits pushed to master at https://github.com/mozilla/addon-sdk https://github.com/mozilla/addon-sdk/commit/3117486febbdf7f1e19f832380a7ba456e699591 Bug 926264 using a setTimeout before exiting tests and adding some additional logs https://github.com/mozilla/addon-sdk/commit/68dc28c2b9fce0778d7c7e43e36b1a23a1b4730b Merge pull request #1382 from erikvold/926264p3 Bug 926264: using a setTimeout before exiting tests and adding some additional logs. r=Mossop
Thank you for all your work on this, but since the massive Linux64-only increase in instances looks to have started on the afternoon of January 28th, at the time when we switched from one sort of Amazon instance to another sort for Linux64 only, I'd recommend putting off any more work on it until the effects of that switch have been verified - at least you'll know that you have to figure out why SSD rather than spinning metal, or whatever the change in processor was, would be the cause.
Depends on: 969590
Bug 967816 comment 5 explains the two-step graph, one set of slaves getting busted January 28th, the others February 2nd.
(In reply to TBPL Robot from comment #355) > philor > https://tbpl.mozilla.org/php/getParsedLog.php?id=34323015&tree=Fx-Team > Ubuntu VM 12.04 x64 fx-team debug test jetpack on 2014-02-07 14:46:33 > revision: aa65dca3dbff > slave: tst-linux64-spot-303 > > command timed out: 7200 seconds elapsed, attempting to kill This log suggests that the call to mozrunner to kill the firefox process is failing. This might be helped by bug 897370 if we can ever get the mozbase guys to review the patches in the dependant bugs.
Depends on: 897370
https://hg.mozilla.org/mozilla-central/rev/4de8fc843d5e
Status: REOPENED → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → FIXED
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Depends on: 969971
No longer depends on: 969971
Blocks: 966070
No longer blocks: 966070
Blocks: 945981
On the test slave I ran this until it hung. The firefox window was gone but the process was still there so something in firefox isn't exiting correctly.
Commit pushed to master at https://github.com/mozilla/addon-sdk https://github.com/mozilla/addon-sdk/commit/9cc3db3af7e67840ee356c810442f87c023387f3 Revert "Bug 926264 using a setTimeout before exiting tests and adding some additional logs" This reverts commit 3117486febbdf7f1e19f832380a7ba456e699591.
I don't have time to investigate here any further and I don't want to block someone else.
Assignee: evold → nobody
Depends on: 1020458
Blocks: 1020473
Summary: Intermittent Jetpack command timed out: 7200 seconds elapsed, attempting to kill starting up private-browsing-supported → Intermittent Jetpack command timed out: 7200 seconds elapsed, attempting to kill
Updating summary now that the timeout was dropped.
Summary: Intermittent Jetpack command timed out: 7200 seconds elapsed, attempting to kill → Intermittent Jetpack command timed out: 1800 seconds elapsed, attempting to kill
Attachment #8439659 - Flags: review?(jsantell) → review+
Commits pushed to master at https://github.com/mozilla/addon-sdk https://github.com/mozilla/addon-sdk/commit/4db3254f04c9935e0382ff5c4735e5d748f40393 Bug 926264 use aAllowDelayedLoad=true for loadFrameScript calls https://github.com/mozilla/addon-sdk/commit/ede480dbb479b199231c5f9ad715060ad2541059 Merge pull request #1512 from erikvold/926264v5 Bug 926264 use aAllowDelayedLoad=true for loadFrameScript calls r=@jsantell
This was a feature of the cfx based harness which is no longer used on the main trees.
No longer blocks: 1020473
Inactive; closing (see bug 1180138).
Status: REOPENED → RESOLVED
Closed: 11 years ago → 9 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: