Closed
Bug 851398
Opened 12 years ago
Closed 8 years ago
Avoid rm -rf; hg clone; hg up pattern and purge instead
Categories
(Release Engineering :: General, defect)
Release Engineering
General
Tracking
(Not tracked)
RESOLVED
WORKSFORME
People
(Reporter: gps, Unassigned)
References
(Blocks 1 open bug)
Details
(Whiteboard: [buildfaster:P1][capacity])
+++ This bug was initially created as a clone of Bug #851270 +++ I was poking through more of the build logs and noticed a common pattern: rm -rf foo hg clone http://hg.mozilla.org/foo foo hg up -r We're doing this for build-tools, mozharness, and main source repositories. This pattern is silly for a few reasons: 1) It is more filesystem intensive than hg purge --all && hg pull -u -r 2) All the excessive cloning creates extra load on the Mercurial server. We're shooting ourselves in both feet here. We should systematically replace all occurrences of the rm -rf; hg clone; hg up pattern with essentially the logic: if -e $dest: status=`hg purge --all -a` if status != 0: # Only clobber if purge was not successful! rm -rf $dest if ! -e $dest: hg clone -u $rev http://... $dest else: hg pull hg up -C -r $rev Maybe the actual logic needs to be slightly more complicated to ensure repo URLs are the same, etc. But, you get my gist. Enacting these changes will cumulatively shave minutes off of build times and should significantly reduce load on hg.mozilla.org by eliminating thousands of clones per day.
Comment 1•12 years ago
|
||
Additional point: when we *do* do a clone, we should instead download and unbundle a bundle if one is available (http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/), then `hg up` to get the latest. Bundles are static files on an FTP server, and are quicker in general. And once you've grabbed a bundle, you can use it again (until you want a new starting bundle): you never need to talk to the server for those changesets.
Comment 2•12 years ago
|
||
(In reply to Richard Newman [:rnewman] from comment #1) > Additional point: when we *do* do a clone, we should instead download and > unbundle a bundle if one is available > (http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/)... hgtool does this already, the bundles at that location are updated by RelEng to feed it.
Assignee | ||
Updated•11 years ago
|
Product: mozilla.org → Release Engineering
Comment 3•11 years ago
|
||
As I wrote in another bug, be aware that unbundle is actually much slower than cloning. (In reply to Gregory Szorc [:gps] from comment #0) > hg clone -u $rev http://... $dest Also note this is super slow as well (giving a revision when cloning). It's much faster to clone the repository entirely, and strip it if necessary. Or, for try, to clone *another* repository (like mozilla-central), and then pull -r $rev.
Reporter | ||
Comment 4•11 years ago
|
||
(In reply to Mike Hommey [:glandium] from comment #3) > (In reply to Gregory Szorc [:gps] from comment #0) > > hg clone -u $rev http://... $dest > > Also note this is super slow as well (giving a revision when cloning). It's > much faster to clone the repository entirely, and strip it if necessary. Or, > for try, to clone *another* repository (like mozilla-central), and then pull > -r $rev. You are confusing -r <rev> with -u <rev>. -u will clone the entire repo and update to the changeset specified. -r will only pull down ancestors of the specified changeset. I confirmed that -u doesn't bleed into -r by looking at the source code.
Comment 5•11 years ago
|
||
We only use pull/clone -u when fetching from try, otherwise we pull/clone the entire repo and update to the revision we want. Is the slowness from unbundling due to network, or something else?
Comment 6•11 years ago
|
||
(In reply to Chris AtLee [:catlee] from comment #5) > Is the slowness from unbundling due to network, or something else? From bunzipping. See bug 958286 comment 6
Comment 7•11 years ago
|
||
ipsec issues resolved without needing to implement this, no longer blocks
No longer blocks: 957502
Updated•10 years ago
|
Blocks: try-tracker
Comment 11•8 years ago
|
||
robustcheckout has fixed this nearly everywhere.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → WORKSFORME
Assignee | ||
Updated•6 years ago
|
Component: General Automation → General
You need to log in
before you can comment on or make changes to this bug.
Description
•