Closed Bug 748940 Opened 13 years ago Closed 13 years ago

HG outage Apr 25, 2012

Categories

(Developer Services :: General, task)

x86_64
Linux
task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: rail, Assigned: bkero)

References

Details

(Whiteboard: [buildduty][outage])

We're seeing some 500 errors trying to clone from hg.m.o: ======== Started clone build tools failed (results: 5, elapsed: 0 secs) (at 2012-04-25 13:10:09.913842) ========= hg clone http://hg.mozilla.org/build/tools tools ..... abort: HTTP Error 500: Internal Server Error program finished with exit code 255 it was very slow ~1h ago. Also there was an email from According to monitor 'hg.mozilla.org', the http service on 'hg.mozilla.org' has not been working as specified since 2012-04-25 13:12:13. Message: Timeout while connecting ..... Regards, - the WatchMouse Team
Whiteboard: [buildduty][outage]
This happens from time to time, and the short term solution is to restart the http daemon on the web heads. Service was restored for this within 5 minutes of the report reaching me on IRC. I'm uncertain of the underlying cause of this and will need to dig deeper, perhaps into the hgweb code itself in order to determine a root cause. Since service is restored, I'm going to bring this down to normal importance and use it as a bug to track determining the root cause of failure.
Assignee: server-ops → bkero
Severity: critical → normal
bkero: trees remain closed because of hg lossage. Details in bug#748939. Not sure if these are same hg issues, so not sure if I should close as DUP or raise to blocker.
joduinn: this seems to be related to a missing file related to http://build.mozilla.org, the hg 404 was a red herring. More details as they're revealed in #build.
(In reply to Ben Kero [:bkero] from comment #3) > joduinn: this seems to be related to a missing file related to > http://build.mozilla.org, the hg 404 was a red herring. More details as > they're revealed in #build. bkero: ah, good to know. Can you (or whomever knows what happened) update this bug when problem understood. Meanwhile, I'm happy to see trees open again, so this can stay normal, and its fine to DUP/CLOSE the other blocker bug#748939.
Component: Server Operations → Server Operations: Developer Services
QA Contact: phong → shyam
404 errors were not related to HG, see bug 748996 for the details.
Assignee: bkero → server-ops
Status: NEW → RESOLVED
Closed: 13 years ago
Component: Server Operations: Developer Services → Server Operations
QA Contact: shyam → phong
Resolution: --- → FIXED
Assignee: server-ops → bkero
Component: Server Operations → Server Operations: Developer Services
QA Contact: phong → shyam
Component: Server Operations: Developer Services → General
Product: mozilla.org → Developer Services
You need to log in before you can comment on or make changes to this bug.