User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:2.0b6pre) Gecko/20100903 Firefox/4.0b6pre Build Identifier: Mozilla/5.0 (Windows NT 6.1; rv:2.0b6pre) Gecko/20100903 Firefox/4.0b6pre The way startup cache file gets written to disk causes severe fragmentation, that gets worse and worse over time. Reproducible: Always Steps to Reproduce: 1. delete startup cache (profile\startupCache\startupCache.4.little 2. restart Firefox 3. run analyze in some defragmentation program that shows file fragmentation Actual Results: Startup cache is severely fragmented, always 128 fragments when freshly created, and gets worse over time - 238 fragments (and that's for a 2,6MB file only) after 8 hours of usage and several Firefox restarts right now. Expected Results: The writing algo should be optimized to get at least some reasonable levels of fragmentation. System: Windows 7. Profile folder on a local, NTFS formated, fully defragmented, non-system disk with 400GB of contiguous free space. No system restore, shadow copies or any other unmovable files that could cause higher fragmentation.

Thanks for the excellent report.

Note (mostly) to self: Taras's patch in bug 594172 could help with this.

However, the main thing is to have a better timer for the initial write. Either extend the length of the timer, or (better) reset the timer when the user does some action.

This is a major issue on my system. I can run Firefox, close Firefox, open Diskeeper, see the startup cache is in 2000 or 3000 fragments. So I defrag. Reboot, run Firefox, close Firefox, open Diskeeper and guess what, it's back to being fragmented. Not as bad, but even so, it shouldn't be fragmented at all. Win7 Pro x64.

(In reply to comment #4) > This is a major issue on my system. I can run Firefox, close Firefox, open > Diskeeper, see the startup cache is in 2000 or 3000 fragments. So I defrag. > Reboot, run Firefox, close Firefox, open Diskeeper and guess what, it's back to > being fragmented. Not as bad, but even so, it shouldn't be fragmented at all. > Win7 Pro x64. what about other files in the profile directory? How much free space do you have on the drive with the startupcache?

Fragmentation on the other files in the profile directory are about normal when compared with other directories on the drive. I do defrag daily. As far as space, this is a RAID10 (4x250GB drives), 500GB total, 216GB free.

(In reply to comment #6) > Fragmentation on the other files in the profile directory are about normal when > compared with other directories on the drive. I do defrag daily. As far as > space, this is a RAID10 (4x250GB drives), 500GB total, 216GB free. You can try a test build from http://ftp.mozilla.org/pub/mozilla.org/firefox/tryserver-builds/tglek@mozilla.com-a754436412b3/tryserver-win32/firefox-4.0b5pre.en-US.win32.zip To see if it reduces your fragmentation. Delete your startup cache first.

Taras did some jar work to reduce fragmentation, next step is probably just to bump the timer timeout to something like 10 minutes.

Currently, the write algorithm buffers all the writes that happen during startup and writes them all out at once. This patch just extends that window to catch more writes. It does seem to help a bit, especially for the first-run fragmentation. I did not anticipate that each write with the zipWriter API would cause so much fragmentation, though, so it's worth thinking about whether we can get the zipwriter to allocate some fixed amount of space or other strategies to reduce this fragmentation.

It's a lot better with all the changes. Right now I get about 40 fragments when creating a new cache and it doesn't climb that much during usage.

Let's do this first, since the fragmentation of the first-run write will affect perf for all other runs as well. Long-term, we should occasionally / periodically defragment, perhaps by copying to a new file.

Comment on attachment 480204 [details] [diff] [review] extends buffer timer, plus some testing convenience >diff --git a/startupcache/StartupCache.cpp b/startupcache/StartupCache.cpp > // Wait for 10 seconds, then write out the cache. >- rv = mTimer->InitWithFuncCallback(StartupCache::WriteTimeout, this, 10000, >+ rv = mTimer->InitWithFuncCallback(StartupCache::WriteTimeout, this, 600000, > nsITimer::TYPE_ONE_SHOT); Stale comment. r=dwitte

Not sure how to mark this, but will also help Bug 601186 which is a Fennec blocker.

Ready to go.

I think Taras had some ideas on when we can copy the startupcache file to reduce fragmentation further.

Comment on attachment 481916 [details] [diff] [review] extends buffer timer, plus some testing convenience > // Wait for 10 seconds, then write out the cache. >- rv = mTimer->InitWithFuncCallback(StartupCache::WriteTimeout, this, 10000, >+ rv = mTimer->InitWithFuncCallback(StartupCache::WriteTimeout, this, 600000, > nsITimer::TYPE_ONE_SHOT); You might want to also change the comment to match the result: 10 seconds should be 10 minutes now.

the zipwriter fix reduced fragmentation by 10x, bunching up writes better in cache-on-thread bug should correct the rest.

Landed part of 586859 which takes care of abnormally high fragmentation.