This looks to have started failing from within a few hours ago (roughly around 7pm pacific August 3rd) markh says: I tried updating both version-control-tools and cinnabar, but that didn't help either maybe it was the deploy - almost exactly 3 hours ago: gps> and hg.mo, mozreview, autoland, and vcs sync are now all on hg 4.2.2. finally. am I'm fairly sure my push that worked was before that heycam says: can't push to mozreview from git or hg It fails when pushing to an existing bug with mozreview or new bug without. Here's the output I get: $ git mozreview push Traceback (most recent call last): File "~/.mozbuild/version-control-tools/git/cinnabar-debug-push.py", line 117, in <module> sys.exit(main(sys.argv[1:])) File "~/.mozbuild/version-control-tools/git/cinnabar-debug-push.py", line 85, in main pushed = push(repo, store, {commit: (None, False)}, heads, ()) File "~/git-cinnabar/cinnabar/hg/__init__.py", line 674, in push reply = repo.unbundle(cg, repo_heads, '') File "/usr/local/lib/python2.7/site-packages/mercurial/wireproto.py", line 443, in unbundle stream = self._calltwowaystream('unbundle', cg, heads=heads) File "/usr/local/lib/python2.7/site-packages/mercurial/httppeer.py", line 359, in _calltwowaystream return self._callstream(cmd, data=fp_, headers=headers, **args) File "/usr/local/lib/python2.7/site-packages/mercurial/httppeer.py", line 230, in _callstream resp = self.urlopener.open(req) File "/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 435, in open response = meth(req, response) File "/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 548, in http_response 'http', request, response, code, msg, hdrs) File "/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 467, in error result = self._call_chain(*args) File "/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 407, in _call_chain result = func(*args) File "/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 909, in http_error_401 url, req, headers) File "/usr/local/lib/python2.7/site-packages/mercurial/url.py", line 405, in http_error_auth_reqed self, auth_header, host, req, headers) File "/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 887, in http_error_auth_reqed return self.retry_http_basic_auth(host, req, realm) File "/usr/local/lib/python2.7/site-packages/mercurial/url.py", line 416, in retry_http_basic_auth return self.parent.open(req) File "/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 435, in open response = meth(req, response) File "/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 548, in http_response 'http', request, response, code, msg, hdrs) File "/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 473, in error return self._call_chain(*args) File "/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 407, in _call_chain result = func(*args) File "/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 556, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) urllib2.HTTPError: HTTP Error 500: Internal Server Error abort: error performing cinnabar push; please report this bug

What I got from hg is: > pushing to ssh://reviewboard-hg.mozilla.org/autoreview > searching for appropriate review repository > redirecting push to ssh://reviewboard-hg.mozilla.org/gecko > searching for changes > remote: abort: abandoned transaction found! > remote: (run 'hg recover' to clean up transaction) > abort: stream ended unexpectedly (got 0 bytes, expected 4)

I ran `hg recover` on the server. I see pushes coming through now. Still unsure how it got in a bad state. Will likely have to wait until morning to debug further.

What I know are gandalf@aviary.pl and ehung@mozilla.com were the last people to push to the gecko repo before the first error (each around 03:20 UTC). rchien@mozilla.com was the first to encounter an HTTP 500 at 03:22:22. cliu@mozilla.com established a series of SSH connections starting at 03:21:55 through 03:34:25. They appear to be push attempts. But they all appeared to fail. (The logs don't tell me why.) I'm pretty confident saying that cliu@mozilla.com was the first person to encounter the problem. I just don't know if it was (inadvertently) caused by her client or whether she was the unlucky first victim. liuche: I don't suppose you have your terminal output from your first mozreview push attempt, do you?

Okay! Looking through my history, it looks like I started to push some commits to reviewboard, and then cancelled the push because I wanted to rebase. Mea culpa, it looks like. First push (with ctrl-c): > pushing to ssh://reviewboard-hg.mozilla.org/autoreview > searching for appropriate review repository > redirecting push to ssh://reviewboard-hg.mozilla.org/gecko > running checkstyle… success! > searching for changes > ^Cinterrupted! > remote: adding changesets > remote: adding manifests > remote: adding file changes > remote: Killed by signal 2. > remote: Killed by signal 2. Then I switched to central, pull -u, switched back to my bookmark (didn't bother to rebase after all), and then tried to push to reviewboard again. And this is where my subsequent failed push happened: > pushing to ssh://reviewboard-hg.mozilla.org/autoreview > searching for appropriate review repository > redirecting push to ssh://reviewboard-hg.mozilla.org/gecko > running checkstyle… success! > searching for changes > remote: abort: abandoned transaction found! > remote: (run ‘hg recover’ to clean up transaction) > abort: stream ended unexpectedly (got 0 bytes, expected 4) At this point I gave up and went home, but things worked the next morning so I forgot about it. I promise to never ctrl-C in the middle of a push ever again.

Ctrl-C on client side isn't suppose to affect server like this, I believe...

Correct: you should be able to abort on the client and the server should fail gracefully. After all, a network hiccup causing the client to lose a connection to the server is essentially the same as aborting the push via client-side process termination.

This happens to try today. > Creating temporary commit for remote... > pushing to ssh://hg.mozilla.org/try > searching for changes > remote: abort: abandoned transaction found! > remote: (run 'hg recover' to clean up transaction) > temporary commit removed, repository restored > abort: stream ended unexpectedly (got 0 bytes, expected 4)

This feels like a regression from upgrading servers from 4.1 to 4.2 (or another change made around that time). I have no clue what's causing it because I can't reproduce it locally via ^C. But I'm looking.

https://bz.mercurial-scm.org/show_bug.cgi?id=5658 is what I think is happening.

I've submitted some patches upstream. Our likely path forward is a custom build of 4.2 with the patches backported. That will hold us over until we upgrade servers to 4.3, which is likely a few weeks away.

Why not simply downgrade to 4.1? Are we already relying on the server being 4.2?

We can't run a stock 4.1 because the CVE patches aren't there. I'd rather run 4.2 with a minimal 4.3 backport than 4.1 with all the security patches backported.

None of the CVEs fixed in 4.3 matter on the server side. And I don't see any other CVE addressed in 4.2.

We perform working directory checkouts on the server in some cases. I'm paranoid and don't want vulnerable code *anywhere* on the server.

Comment on attachment 8897647 [details] ansible: install a custom Mercurial 4.2 with interrupt backports (bug 1387324); https://reviewboard.mozilla.org/r/168912/#review174318

Upstream accepted my patches to 4.3. I was waiting for that to occur before deploying this. So I'll deploy this sometime today, after I get in the office.

Doing this now'ish.

Pushed by gszorc@mozilla.com: https://hg.mozilla.org/hgcustom/version-control-tools/rev/dfebe316dc8d ansible: install a custom Mercurial 4.2 with interrupt backports ; r=glob