noted during FF 13.0b5. First round of win32 repacks failed due to 500 error caused by network issues in scl3 - so these would not have recovered. First pass of beta-push-to-mirrors script also failed due to intermitent hg.m.o failure. Rebuild succeeeded.

Sounds like this is a general problem with hgtool.py or our retry code.

What retry code? hgtool.py doesn't retry, makeHgtoolStep makes a RetryingShellCommand. push-to-mirrors.py doesn't use hgtool.py (and it wouldn't buy it anything), it just uses tools/lib/python/util/hg.py, and the only thing in there that retries is, for some reason, cleanOutgoingRevs. And "'clone scripts' step does not retry" would be the magical unicorn pony of bug 712205, always having tools checked out everywhere (though even after that, we probably do need to have _the_ retry code of having hg.py retry everything instead of using a RetryingShellCommand that does scripts no good).

To clarify, the clone I'm speaking of is the step 6 for the any of the "release-mozilla-beta-win32_repack_n/6" builders. The command sequence, with failure, is: (view as text) hg clone http://hg.mozilla.org/build/tools scripts in dir /builds/slave/rel-m-beta-psh-mrrrs/. (timeout 1200 secs) watching logfiles {} argv: ['hg', 'clone', 'http://hg.mozilla.org/build/tools', 'scripts'] environment: BUILDBOTCUSTOM=http://hg.mozilla.org/build/buildbotcustom BUILDBOT_CONFIGS=http://hg.mozilla.org/build/buildbot-configs CLOBBERER_URL=http://build.mozilla.org/clobberer/always_clobber.php CVS_RSH=ssh G_BROKEN_FILENAMES=1 HISTSIZE=1000 HOME=/home/cltbld HOSTNAME=linux-ix-slave23.build.mozilla.org INPUTRC=/etc/inputrc LANG=en_US.UTF-8 LOGNAME=cltbld MAIL=/var/spool/mail/cltbld PATH=/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/cltbld/bin PWD=/builds/slave/rel-m-beta-psh-mrrrs SHELL=/bin/bash SHLVL=1 SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass TERM=linux USER=cltbld _=/tools/python/bin/python using PTY: False abort: HTTP Error 500: Internal Server Error program finished with exit code 255 elapsedTime=433.046163

Yeah, that's pure bug 712205 - that step shouldn't fail because it shouldn't exist.

Either one of bugs 712205, 764077 would fix this one.

(In reply to Chris AtLee [:catlee] from comment #5) > Either one of bugs 712205, 764077 would fix this one. One of these is fixed now!