Sure, there's tons of hg failures from various issues flying around, but what there isn't is any explanation of why it makes sense that only on Android tests, every three or four pushes we have most, but not all, of the `hg clone http://hg.mozilla.org/build/tools tools` buildsteps failing a la https://tbpl.mozilla.org/php/getParsedLog.php?id=8556923&tree=Mozilla-Inbound with requesting all changes adding changesets adding manifests adding file changes added 2117 changesets with 4561 changes to 940 files updating to branch default 465 files updated, 0 files merged, 0 files removed, 0 files unresolved abort: 00changelog.i@: ambiguous identifier! program finished with exit code 255 On a different network? With a different version of hg on the client? The non-Android buildstep doesn't admit failure and doesn't mark the job as red when it fails, and desktop Firefox really is failing just as often?

The bad part is, for the last six pushes on mozilla-inbound (the only place there really are pushes, at least that are doing Android tests), every single push has had this turn about half its Android tests red. The good part is, that doesn't appear to actually cause any problem - my memory of failing to clone build/tools is that it messes up the reboot step, but apparently Android doesn't use it for rebooting, and it apparently doesn't use it for cloning from mirrors since it doesn't do that either, so... does it actually have any reason for cloning build/tools, other than habit? The bad part is, because they are red, and because we have tons of incomprehensible Android red for which the only reasonable solution is "retrigger it, and if it turns green ignore the red," even though these runs run every test, pass every test, are perfect in every way except the color on tbpl, people keep on retriggering them, which will get expensive if we run half the tests twice, and a quarter three times, and an eighth four times.

This is looking much better since cshields made a tweak (ie none at all on the current tip of inbound, rev fbf07f7e8c93).

fix for this was increasing fetch_maxchunksize in varnish (/etc/sysconfig/varnish) on dm-vcview04

Another 18 on mozilla-inbound at 18:35, and a second set of 4 so far, at 18:55.

And a batch on Try at 19:16, so if it ever really went away, it's baaaack.

Still is still hitting pretty frequently across trees

varnish was the culprit here and as of about an hour ago, is being bypassed. These errors should go away.