Closed Bug 987267 Opened 11 years ago Closed 10 years ago

use gzip compression for hg bundles

Categories

(Release Engineering :: General, defect)

x86_64
Windows 8.1
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: taras.mozilla, Assigned: rail)

References

Details

Attachments

(1 file, 1 obsolete file)

hg operations are gonna take as long or longer than builds once we implement caching on windows. According to glandium's stats, hg unbundling only happens on ~3% of the builds, but it does lead to some surprisingly-slow build times. Simply zipping up the whole repo ends in a 3x speed up, switching to gzip bundles is a 1.4x speedup. Lets swap to gzip bundles asap(or just hg clone --uncompressed on windows which would be fastest) ++ mkdir mozilla-central ++ cd mozilla-central ++ hg init . ++ hg unbundle ../mozilla-central.hg adding changesets adding manifests adding file changes added 174831 changesets with 983111 changes to 153611 files (run 'hg update' to get a working copy) real 31m59.826s user 0m0.015s sys 0m0.015s ++ cd mozilla-central ++ hg init . ++ hg unbundle ../mozilla-central.gz.bundle adding changesets adding manifests adding file changes added 174831 changesets with 983111 changes to 153611 files (run 'hg update' to get a working copy) real 22m59.116s user 0m0.015s sys 0m0.030s ++ rm -fR mozilla-central ++ unzip -q mozilla-central.zip real 11m19.762s user 0m0.015s sys 0m0.000s 1.1G mozilla-central.zip 914M mozilla-central.7z 790M mozilla-central.gz.bundle 631M mozilla-central.hg
Blocks: 940789
Attached patch gzip.bundle.diff (obsolete) (deleted) — Splinter Review
Switching to gzip is easy. We'll need to regenerate the bundles (see https://wiki.mozilla.org/ReleaseEngineering/Merge_Duty/Steps#Create_new_hg_bundles) zipping .hg would require changes in hg-tool.py
Attachment #8395825 - Flags: review?(catlee)
Attached patch gzip.bundle.diff (deleted) — Splinter Review
err, wrong position
Attachment #8395825 - Attachment is obsolete: true
Attachment #8395825 - Flags: review?(catlee)
Attachment #8395828 - Flags: review?(catlee)
Attachment #8395828 - Flags: review?(catlee) → review+
Regenerated the bundles.
(In reply to Rail Aliiev [:rail] from comment #4) > Regenerated the bundles. <3 Any reason to keep this open? We can pursue other options in another bug another time.
Assignee: nobody → rail
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Summary: hg bundles suck → use gzip compression for hg bundles
23 minutes to do an unbundle from a gzipped bundle and 11:20 to unzip?! My computer can do a gzip unbundle in 8 minutes. And 4 of those should go away with bundle2 (I'd like to add cache data into bundles so clients don't need to regenerate it).
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: