Closed Bug 838325 Opened 12 years ago Closed 12 years ago

Intermittent clone mozharness failed with "command timed out: 1320 seconds without output, attempting to kill"

Categories

(Release Engineering :: Applications: MozharnessCore, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: philor, Assigned: mozilla)

References

()

Details

(Keywords: intermittent-failure)

Attachments

(1 file, 1 obsolete file)

+++ This bug was initially created as a clone of Bug #817513 +++ Brought to you by bug 817513 comment 82, and the need to have cloning mozharness do the same thing that cloning tools does, set RETRY on any failure. https://tbpl.mozilla.org/php/getParsedLog.php?id=19456211&tree=Mozilla-Inbound Rev4 MacOSX Snow Leopard 10.6 mozilla-inbound debug test marionette on 2013-02-05 08:37:41 slave: talos-r4-snow-067 command timed out: 1320 seconds without output, attempting to kill
This can't retry in the standard sense, since there's no tools repo to use retry.py. We are setting retry, but only if we match one of these regexes: http://hg.mozilla.org/build/buildbotcustom/file/21b0f0b5e41c/status/errors.py#l12 I think we need to increase the timeout.
Attached patch increase ScriptFactory clone timeout (obsolete) (deleted) — Splinter Review
Assignee: nobody → aki
Attachment #710543 - Flags: review?(armenzg)
It can't retry, but it can RETRY, exactly like bug 817513 does for cloning tools, with log_eval_func=rc_eval_func({0: SUCCESS, None: RETRY}). Sucks to be hg.m.o when we come RETRYing, but if it doesn't like it, it shouldn't time us out.
Comment on attachment 710543 [details] [diff] [review] increase ScriptFactory clone timeout I really don't think a 1 hr timeout is the right thing to do here.
Comment on attachment 710543 [details] [diff] [review] increase ScriptFactory clone timeout I agree with edmorley on this one. If it hasn't checked out in 22 minutes... In fact, I would make the timeout smaller and RETRY sooner. We should RETRY the job (not the step) the way that philor says. If would be great if we could use hgtool.py support (which can recover better) but that would probably be more work (tech debt).
Attachment #710543 - Flags: review?(armenzg) → review-
Attached patch retry (deleted) — Splinter Review
Attachment #710543 - Attachment is obsolete: true
Attachment #710818 - Flags: review?(armenzg)
Comment on attachment 710818 [details] [diff] [review] retry Review of attachment 710818 [details] [diff] [review]: ----------------------------------------------------------------- This should do. Thanks.
Attachment #710818 - Flags: review?(armenzg) → review+
Comment 10 is just trying to mess with your head - it's actually a build/tools/ clone, and one that retried.
In production.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Blocks: 793022
Product: mozilla.org → Release Engineering
Component: General Automation → Mozharness
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: