User Agent: Mozilla/5.0 (Ubuntu; X11; Linux i686; rv:8.0) Gecko/20100101 Firefox/8.0 Build ID: 20111115184056 Steps to reproduce: To shave off some weight from the installer, we should avoid putting the compressed omni.jar in the installer but rather its contents and have the installer produce the jar/zip file. As a quick test I just downloaded the en-us windows installer of fx8.0.1, uncompessed the contents of the sfx in a folder and run 7z a -t7z -mx -m0=lzma lzma.7z core setup.exe (*) then I unpacked the contents of core/omni.jar in the directory core/omni.jar/, deleted the jar file and re-run the same 7z command. The size of the resulting files were 15.9 MB and 14.2MB respectively for the original and unpacked versions, i.e. the unpacked version is about 89-90% of the original. The downside is obviously the need, for the installer, to perform an additional task (i.e. compressing the contents of the omni.jar folder). Now, if we think this installer/size savings are worth it, there are two ways of tackling the problem. Either we roll our own solution (i.e. modify the build system to not produce the omni.jar file but to leave just its contents in a folder; modify the installer to take the contents of the folder and compress it in the omni.jar archive) or we contribute a 7zip filter that does all of this transparently (i.e. when a file A of a known archive type is inserted inside another archive B, the contents of A are decompressed before being archived in B; upon decompression of B, the contents of A are compressed again in A's original format. note: this is likely not a "lossless" operation, i.e. the original A and the decompressed A will likely be different, even though their contents will be the same) (*) I'm aware that the sfx of the installer is compressed using 7zip's manual stream binding voodoo; this is just a quick PoC

I once checked the install size when compressing an uncompressed omni.jar and it lessened the size. I don't think there is a bug for this. If this were to be done, the end result would have to be 100% deterministic since application update has to patch the omni.jar. Otherwise, we would have to include the entire omni.jar in every update which would significantly increase its size. The order of the files in the omni.jar are optimized as well so what ever method is used would also have to do this. We don't have a Linux or a Mac installer. For the above reasons, I don't think we will ever do this in the Windows installer or create a Mac and Linux installer to do this.

(In reply to Robert Strong [:rstrong] (do not email) from comment #1) > If this were to be done, the end result would have to be 100% deterministic > since application update has to patch the omni.jar. Otherwise, we would have > to include the entire omni.jar in every update which would significantly > increase its size. Not really. You would just send the files in omni.jar that needs to be modified; the installer would then unpack the local omni.jar, replace the files and repack everything. > The order of the files in the omni.jar are optimized as well so what ever > method is used would also have to do this. This is somewhat harder to do but I guess it could be done somehow (e.g. by providing a txt file with the file order, or by inheriting the order in which files are stored in the sfx).

Besides the other issues stated that weren't addressed and that the methods to address the two items above adding significant complexity for IMO relatively little value when the complexity and value are compared I still think having an uncompressed omni.jar would be a better solution. Also, a program to add the files to the omni.jar would need to be included with the installer which would lessen the size that is saved.

(In reply to Robert Strong [:rstrong] (do not email) from comment #3) > Besides the other issues stated that weren't addressed I see just an additional issue, i.e. the lack of the installer on mac and linux. I did not reply to that because then this solution simply does not apply. > and that the methods > to address the two items above adding significant complexity for IMO > relatively little value when the complexity and value are compared I still > think having an uncompressed omni.jar would be a better solution. Yes, but you would then end up with a 17MB omni.jar instead of the current 5MB one. I suspect this could cause massive regressions in cold startup times... > Also, a > program to add the files to the omni.jar would need to be included with the > installer which would lessen the size that is saved. Not really. All supported windows versions (because you said that the installer only exists on windows) have native support for reading/writing zip (i.e. jar) files.

I think we should ship an uncompressed omnijar and instead add some readahead magic(bug 613124) to make up for increased IO.

There's perhaps another way to reduce the omni.jar, files within .zip archives can be compressed using different compression algorithm, Deflate is the one most often associated to .zip files but newer ones are also available like Deflate64, PPMd, bzip2 and even LZMA (the same compression method used in .7z 7-Zip archives) see: http://en.wikipedia.org/wiki/Zip_%28file_format%29#Compression_methods If we add support for one or more of these newer/more efficient compression methods (moving from Deflate to Deflate64 is trivial) to the code that handles .jar files we would increase Firefox code size by a few KB but perhaps save hundreds of KB in the omni.jar file.

I did not mention that all Deflate compression engines are not equal some produce slightly smaller streams, Kzip for instance is known to often produce the smallest streams compared to standard zlib like implementations. http://advsys.net/ken/utils.htm

Made some quick tests using Kzip and 7-Zip, here is what we get on the 20 biggest files from omni.jar Uncompressed Deflate(curr) Deflate(Kzip) BZip2(7-Zip) LZMA(7-Zip) 417872 104116 100142 82677 87052 chrome/browser/content/browser/browser.js 402835 179225 167916 162932 139141 components/browser.xpt 327514 79259 75865 61496 64244 chrome/browser/content/browser/tabview.js 296594 62496 59735 52393 53468 chrome/browser/content/browser/orion.js 294632 76358 73104 58474 57611 jsloader/resource/gre/modules/XPIProvider.jsm 274467 58240 55691 45738 48601 modules/XPIProvider.jsm 207009 49763 47936 40894 42828 modules/HUDService.jsm 206908 57966 55308 44827 44393 jsloader/resource/gre/modules/HUDService.jsm 167721 44282 42659 35362 38021 components/nsSessionStore.js 165028 36234 34671 32981 31424 chrome/browser/content/browser/license.html 164805 36176 34613 32912 31364 chrome/toolkit/content/global/license.html 159186 26777 25619 20139 20764 res/dtd/htmlmathml-f.ent 149648 44713 42783 35671 35647 jsloader/resource/gre/components/nsSessionStore.js 147541 30845 29814 25497 27264 chrome/browser/content/browser/tabbrowser.xml 146340 41904 40183 33772 33181 jsloader/resource/gre/components/nsSearchService.js 131034 18448 17560 15365 15978 chrome/browser/content/browser/browser.xul 130960 30402 29186 25622 24408 jsloader/resource/gre/modules/PlacesUtils.jsm 127907 33878 32574 28852 29913 components/nsSearchService.js 119264 32015 30681 26643 25577 jsloader/resource/gre/components/nsUpdateService.js 118225 25017 24120 22149 22606 modules/PlacesUtils.jsm 110371 17683 16171 14524 13941 res/fonts/mathfont.properties 107082 24836 23950 21768 22268 components/nsUpdateService.js 102434 26972 25984 24903 24447 greprefs.js Kzip always improves Deflate compression, Bzip2 and LZMA could compress the files a bit further.

A bit easier to read this way.

(In reply to Frédéric Kayser from comment #6) > There's perhaps another way to reduce the omni.jar, files within .zip > archives can be compressed using different compression algorithm The problem is, mainly, that zip (and, thus, jar) does not support solid archiving. Try to compress the contents of omni.jar with the same algorithm but in solid and non-solid mode and you'll see quite dramatic differences. This is especially true when compressing many small (and rather similar) files.

We've achieved this same benefit though a different method in bug 1362377 by disabling compression on the omni.ja. It's still a zip file, it just doesn't use deflate anymore, so we can let the installer and the update packager use LZMA on the contents instead.