Automation is currently seeing lots of failures like those in https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-beta-linux64-add-on-devel/1491563674/mozilla-beta-linux64-add-on-devel-bm74-build1-build65.txt.gz. I'm still not sure why the Mercurial operations are failing and I'm committed to getting to the root cause. But I've identified a mitigation. Currently, some mozharness jobs that perform a Firefox VCS clone aren't using the --upstream feature of robustcheckout. This means that the Mercurial repo in the shared store may initially be populated with e.g. mozilla-release - a repo thousands of changesets behind beta, aurora, central, etc. When the machine processes an e.g. beta job, it has to `hg pull` thousands of changesets for the beta repo. This takes extra time and adds significant load to the server. --upstream with the mozilla-unified repo causes the initial clone to be seeded with a superset of changesets from all the repos. This means that any subsequent `hg pull` should be minimal. A minimal pull maximizes the changes that it will succeed. We should add --upstream https://hg.mozilla.org/mozilla-unified to all of the jobs and clobber all the builders to force a mozilla-unified clone to be present.

Comment on attachment 8855890 [details] Bug 1354625 - Use mozilla-unified for upstream for all Firefox builds; https://reviewboard.mozilla.org/r/127766/#review130472

Comment on attachment 8855889 [details] Bug 1354625 - Use mozilla-unified for single_local try builds upstream; https://reviewboard.mozilla.org/r/127764/#review130468

Pushed by gszorc@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/c92bc314c18b Use mozilla-unified for single_local try builds upstream; r=aki https://hg.mozilla.org/integration/autoland/rev/508dca13e963 Use mozilla-unified for upstream for all Firefox builds; r=aki

https://hg.mozilla.org/releases/mozilla-beta/rev/e7ed0129efda https://hg.mozilla.org/releases/mozilla-beta/rev/65bd52f60c9f

https://hg.mozilla.org/releases/mozilla-aurora/rev/5a334ecbdb96 https://hg.mozilla.org/releases/mozilla-aurora/rev/f9d279dfaec2

https://hg.mozilla.org/releases/mozilla-esr52/rev/0dfab14fffa1 https://hg.mozilla.org/releases/mozilla-esr52/rev/f4802fea742c

A spot checked some logs and this seems to have the desired effect! I love paving over problems. That leaves bug 1291926 for me to address the root cause...

https://hg.mozilla.org/mozilla-central/rev/c92bc314c18b https://hg.mozilla.org/mozilla-central/rev/508dca13e963