i'm seeing 77ms on cold startup on a 7200rpm hd to iterate over all manifest files and process them. on wince it's 400ms

Do you have a breakdown of how much time is directory listing and how much is reading?

http://people.mozilla.org/~tglek/manifests.txt is on the desktop. On Wince dir reading takes up more time due to lack of vfs cache, also messing about with urls is more expensive on wince, so there is cpu-related slowdown there.

Taras says combining manifests inside a single jar would help.

(In reply to comment #3) > Taras says combining manifests inside a single jar would help. er, combining all manifests into a single one, covering all jars. that'd have to be done as a packaging step.

This isn't just a Linux problem, right?

(In reply to comment #5) > This isn't just a Linux problem, right? no

Adding yet another option to JarMaker.py to write out to a single manifest file should be rather easy. http://mxr.mozilla.org/mozilla-central/source/config/JarMaker.py#173 is the code in question, don't have cycles to do that myself, though.

(In reply to comment #7) > Adding yet another option to JarMaker.py to write out to a single manifest file > should be rather easy. > > http://mxr.mozilla.org/mozilla-central/source/config/JarMaker.py#173 is the > code in question, don't have cycles to do that myself, though. That does look reasonable. I was originally thinking of doing this at build time to deal with not merging the locale manifest. Do you have a suggestion on how/where to best deal with locale manifest. I'm happy to do the grunt work here.

We could do some makefile foo in the locales Makefiles to disable the single manifest there.

How's this for a start? It doesn't handle locale manifests specially. On my Macbook reading the manifests is 25s of cold startup. With this patch, the single manifest is 15s. Questions: * The single manifest's basename should be what? * Re: comment 9, where do those locale Makefiles live?

How would Fennec's plan to ship multiple locales in a single package affect locale manifest handling? I guess it depends on how we build/package the locales.

Re the patch, the addition of --single-manifest should be conditional, perhaps some ifndef DISTINCT_MANIFEST MAKE_JARS_FLAGS += --single-manifest endif It should stay off for external customers, IMHO, so maybe it should be in autoconf.mk.in. More of a question for ted. For multi-locale builds, we need to think a bit, as we need that for regular l10n repacks. We had to define DISTINCT_MANIFEST in most of http://mxr.mozilla.org/mozilla-central/find?text=&kind=text&string=locales%2FMakefile.in, IIRC, even for the branding dirs there, too. And the equivalent on comm-central and mobile-browser, and possibly branches of central as fits.

I think if this makes things faster, why wouldn't everyone want it? Let's not try to please everyone unless someone specifically comes up with an issue.

The name should be chrome.manifest. We cannot do this unless we have a solution for either not merging the locale manifest, or repacking the locale manifest.

I believe the most effective way to deal with manifest slowness would be to solve bug 506841 via some fastload-type cache that would cache all of the manifests and not touch files in chrome dir unless some compreg scenario occurs.

(In reply to comment #10) > On my Macbook reading the manifests is 25s of cold startup. With this patch, > the single manifest is 15s. ms, not s, d'oh. (In reply to comment #15) > I believe the most effective way to deal with manifest slowness would be to > solve bug 506841 Then what's the status of this bug? Simply dependent on bug 506841 with no other work? Or wontfix per the suboptimal proposed solution in comment 4?

(In reply to comment #16) > (In reply to comment #10) > > On my Macbook reading the manifests is 25s of cold startup. With this patch, > > the single manifest is 15s. > > ms, not s, d'oh. > > (In reply to comment #15) > > I believe the most effective way to deal with manifest slowness would be to > > solve bug 506841 > > Then what's the status of this bug? Simply dependent on bug 506841 with no > other work? Or wontfix per the suboptimal proposed solution in comment 4? wontfixed this, should work on bug 511761 instead. My vision is to have the compreg/whatever(bug 511761) act as a guard for chrome cache(and everything else). Furthermore, a reworked/unified mmaped fastload cache would be a perfect place to dump the cached manifest data. Unfortunatelly reworking fasl will take quite a while, in the meantime i think we should cache manifests into a discrete file in profile dir.

We should cache manifests in startup cache

I noticed this same slowdown when profiling locally by the way on Windows. I get about 100ms of my startup time from parsing manifests. :(

bbondy, things have changed significantly since this was last discussed: we no longer enumerate any directories looking for manifests (which was one of the big slowdowns). Can you provide more details about what you're seeing?

bsmedberg, I was originally seeing this about twice as high as it actually was since ParseManifest is called recursively, but you can see from this stat that it's still pretty significant. nsComponentManagerImpl::RereadChromeManifests Calls: 1 Max: 0.036:36 Avg: 0.036:36 Min: 0.036:36 Total: 0.036:36 All of this time is taken up inside ManifestParser.cpp in the function ParseManifest.

those are milliseconds, so 36ms. Not huge but worth fixing.

Is it possible to break down how much of that is iowait and how much is us doing the dumb-but-necessary XPCOM registration for each manifest directive?

Ya sorry I'm still looking deeper, that was just an intermediate result.

I haven't looked at ways to optimize this yet, but the biggest thing that sticks out to me is: ~21ms is spent in nsComponentManagerImpl::ManifestXPT - FileLocation's constructor is ~3ms - FileLocation.GetData is ~4ms - data.Copy(buf, len); is ~2ms - xptiInterfaceInfoManager::GetSingleton()->RegisterBuffer(buf, len); ~12ms

Sounds like most of the time is just parsing the XPT file and producing the in-memory data. bsmedberg and I talked about making the xpidl compiler produce C structs that we could link directly into libxul instead of parsing XPT files, but it never actually went anywhere. Seems like a lot of work for 12ms, but I don't know what percentage of startup that is.

> but I don't know what percentage of startup that is. Only about 1% but it still helps the bottom line. I'll profile further to find out exactly where the problem is before implementing anything but that's a cool idea.

Be careful not to blow all the improvement on additional data relocations.