Closed
Bug 461395
Opened 16 years ago
Closed 16 years ago
add support for PARALLEL_DIRS to build system, ||ize content/
Categories
(Firefox Build System :: General, defect)
Firefox Build System
General
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: ted, Assigned: ted)
References
(Blocks 1 open bug)
Details
Attachments
(1 file)
(deleted),
patch
|
benjamin
:
review+
|
Details | Diff | Splinter Review |
Currently, the directories in DIRS get built sequentially. If you have a lot of directories, like the content/ top-level directory, this means that even for a no-op build, you have to sequentially recurse into each one only to figure out that you have nothing to do. At least for content, (and probably for a lot of other places) we could in fact build these in parallel without any harm. This patch adds support for a PARALLEL_DIRS var to the build system. The directories in this var will be built in parallel before any directories in DIRS. The normal export/libs passes apply, so all PARALLEL_DIRS will be built export, then all DIRS, then all PARALLEL_DIRS will be built libs, then DIRS. On my dual-core Windows machine, a no-op -j2 build with this patch in content/ (no code changes, nothing to build) saves about 33% of the build time (19 seconds without the patch, 12 seconds with). It doesn't appear to win anything on a clobber build, since all the time is spent compiling, but I'm hoping that bug 428532 + this will help that. I'll post more numbers as I get them. The approach of this patch does require us to manually opt-in directories into this approach, but I think that's the only way to do it without breaking things.
Assignee | ||
Comment 1•16 years ago
|
||
Here are some numbers for a stock build, build with this patch, and with this patch + bug 428532. This patch alone helps a no-op build pretty well, but doesn't do much for a clobber. However, this patch + the other patch helps quite a bit in the clobber case, and lets us go far more parallel. I was able to crank up to -j8 and still see wins, even on only a dual core system. I guess msys I/O is just that bad! all times on Windows XP, Core 2 Duo, 2Gb ram ====================================== rebuild (nothing to do): (time make -C ../obj-firefox-opt-libxul/content/) no patches: -j1 real 0m19.297s -j2 real 0m19.078s -j4 real 0m18.922s with parallel dirs patch: (||izing almost everything under content/) -j1 real 0m18.000s -j2 real 0m12.047s -j4 real 0m11.907s -j8 (just for kicks) real 0m11.562s with parallel dirs patch + single compiler invocation: -j1 real 0m18.781s -j2 real 0m12.797s -j4 real 0m12.187s -j8 real 0m12.953s ====================================== full build (make -C ../obj-firefox-opt-libxul/content/ clean && time make -C ../obj-firefox-opt-libxul/content/) no patches: -j1 real 6m36.766s -j2 real 4m6.859s -j4 real 4m8.438s with parallel dirs patch: (||izing almost everything under content/) -j1 real 6m33.157s -j2 real 3m56.000s -j4 real 4m0.641s with parallel dirs patch + single compiler invocation: -j1 real 4m23.703s -j2 real 3m40.438s -j4 real 3m5.125s -j8 (appears to be the sweet spot for my setup) real 2m40.656s -j10 real 2m44.812s
Assignee | ||
Comment 2•16 years ago
|
||
Linux results, not directly comparable because it's different hardware etc, but still interesting. Looks like this patch helps a bit, especially in the no-op case, but the bug 428532 patch either breaks even or performs worse than the normal case. I guess gcc doesn't do so well at compiling multiple files at once. all times on Ubuntu 8.04 x86-64, Core 2 Quad, 4Gb ram ====================================== rebuild (nothing to do): (time make -C ../obj-firefox-opt-libxul/content/) no patches: -j1 real 0m2.706s -j2 real 0m2.664s -j4 real 0m2.682s -j8 real 0m2.740s -j16 real 0m2.668s with parallel dirs patch: (||izing almost everything under content/) -j1 real 0m2.697s -j2 real 0m1.531s -j4 real 0m1.077s -j8 real 0m0.986s -j16 real 0m0.971s with parallel dirs patch + single compiler invocation: -j1 real 0m2.706s -j2 real 0m1.528s -j4 real 0m1.027s -j8 real 0m0.970s -j16 real 0m0.979s ====================================== full build (make -C ../obj-firefox-opt-libxul/content/ clean && time make -C ../obj-firefox-opt-libxul/content/) no patches: -j1 real 5m8.063s -j2 real 2m45.116s -j4 real 1m43.837s -j8 real 1m44.881s with parallel dirs patch: (||izing almost everything under content/) -j1 real 5m10.261s -j2 real 2m36.112s -j4 real 1m28.544s -j8 real 1m28.391s with parallel dirs patch + single compiler invocation: -j1 real 5m9.812s -j2 real 2m37.949s -j4 real 1m45.383s -j8 real 1m39.458s -j10 real 1m37.789s
Assignee | ||
Comment 3•16 years ago
|
||
Comment on attachment 344511 [details] [diff] [review] add PARALLEL_DIRS support, use it in content This patch is a net win on both Windows and Linux. We should get this in. The patch in bug 428532 needs some more work, but it's a separate issue. I need to test this on OS X still, but I don't see it being wildly different than Linux.
Assignee | ||
Comment 4•16 years ago
|
||
Yeah, looks like OS X is comparable to Linux. This patch is still good, the other patch will need some work (and probably should only be enabled on Windows.) all times on Macbook Pro, Core 2 Duo, 2Gb ram ====================================== rebuild (nothing to do): (time make -C ../obj-firefox-opt-libxul/content/) no patches: -j1 real 0m4.713s -j2 real 0m4.671s -j4 real 0m4.644s with parallel dirs patch: (||izing almost everything under content/) -j1 real 0m4.765s -j2 real 0m2.909s -j4 real 0m2.781s -j8 (just for kicks) real 0m2.895s with parallel dirs patch + single compiler invocation: -j1 real 0m3.935s -j2 real 0m2.381s -j4 real 0m2.422s -j8 real 0m2.436s ====================================== full build (make -C ../obj-firefox-opt-libxul/content/ clean && time make -C ../obj-firefox-opt-libxul/content/) no patches: -j1 real 6m22.019s -j2 real 3m45.578s -j4 real 3m45.685s with parallel dirs patch: (||izing almost everything under content/) -j1 real 6m29.834s -j2 real 3m36.530s -j4 real 3m37.213s with parallel dirs patch + single compiler invocation: -j1 real 6m28.274s -j2 real 3m44.641s -j4 real 3m49.319s
Updated•16 years ago
|
Attachment #344511 -
Flags: review+
Assignee | ||
Comment 5•16 years ago
|
||
Pushed: http://hg.mozilla.org/mozilla-central/rev/80406c10f795
Status: NEW → RESOLVED
Closed: 16 years ago
Resolution: --- → FIXED
Updated•7 years ago
|
Product: Core → Firefox Build System
You need to log in
before you can comment on or make changes to this bug.
Description
•