Steps to reproduce: 1) Install current branch build 2) Shut down mozilla. 3) Delete (or rename) the '...\Application Data\Mozilla' directory 4) Start mozilla. Note size of '...\Application Data\Mozilla\registry.dat' 5) Repeat step 4. 6) Repeat step 4. ... Actual results: 'registry.dat' grows by 45KB on each start of browser Expected results: 'registry.dat' should not need to grow in size, if nothing in environment has changed. (And if it did need to grow, obviously 45KB is way too much Reproducible 100% in current 07/15 branch build on win2k. (Haven't checked other platforms, or trunk, yet). Not entirely sure who owns this, so spreading the love among dveditz, dougt, ccarlen. Reassign as appropriate.

Are you seeing bug 109739?

*** This bug has been marked as a duplicate of 109739 ***

This is essentially the same problem as bug 109739, but there is a particular aspect to this bug report that makes it something that we can't just shrug our (Netscape) shoulders and say "well, end users won't see this". The 'registry.dat' file grows...: Mozilla Netscape ----------------------------------------------------------------- 1.1 07/15 trunk When a different When a different binary is started binary is started ----------------------------------------------------------------- 1.0 07/15 branch When a different EVERY time the same binary is started build is started This should really move to a bugscape bug for the commercial build. Hopefully we can find the cause of this happening only in the commercial build, and get that build to only add to registry.dat when a different build is started. [I'll note that the large file size doesn't appear to add to startup time, at least on win2k with a SCSI drive. (I'm less optimistic about Mac performance not being affected, although this bug still needs to be confirmed on Mac and Linux for the commercial branch).]

I opened http://bugscape.mcom.com/show_bug.cgi?id=17790 for the commercial only bug.

...and put the summary back. Duh.

jrgm, I've heard reports at mozillazine that this is happening for people using mozilla builds too.

Well, yeah. Bug 109739 covers it for the general case (i.e., someone who downloads new mozilla builds on a regular basis will have registry.dat grow huge over time). But that's (comparatively) not as bad as growing every time the browser is started. Or are you saying that someone can reproduce the above in a mozilla build. If so, we can reopen this bug, but as far as as I can see, the every time thing is only in commercial branch.

Mozilla builds happen on the branch, too -- why would it be a commercial-only problem? Are you blaming activation? 109739 was plugins, and that's common code.

er, never mind. And I had just read your chart, too. D'oh!

I think we should reopen this bug. http://www.mozillazine.org/talkback/read.php?f=2&i=2084&t=2084 reports it growing with every launch.

Reopening per Asa, 'registry.dat' growing with every launch

FYI, there is a long thread about this problem in netscape.netscape7.windows, a NS7 user is reporting that he discovered he had a 68MB registry.dat file after a few months using NS7PR1 !

To quote from the comment referenced by asa: "I have recently found out that my registry.dat file get 30k larger each time I switch from running mozilla.exe from the installed version to the debug version and vise-versa." Okay, that is column one from my table above, and that is a duplicate of bug 109739, no? I'll note again that I opened bugscape bug 17790 to cover the Netscape-specific aspect of this bug, which is that in the commercial build, starting the _same_ build will write the registry.dat _every_ time (i.e., not just when switching builds). But that 68MB number makes me feel ill :-(. It suggests that we must implement Pack (or equivalent), but I'd hate to have to call that on every startup.

When running different versions of the browser, there could be a version mis-match in the registry cache and cause the cache to invalidate. Because it has no Pack() method, it grows out of control. :(

--->Andrei's got this

I think we can dup this to bug 109739, given that bug is being really taken care of, which it seems to be as it got ADT1.

Can anybody please try today's Netscape build? I am using Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.1b) Gecko/20020719 Netscape/7.0b1+ and cannot repro the problem any more. I do have npcdt.dll in the Components folder, although it is not listed in the about:plugins page for some reason.

Yes, I also think that this bug should be duped to bug 109739. (The Netscape specific aspects are being dealt with in bugscape bug 17790). av: I believe that user agent string is for a trunk commercial build. The 'every time you launch' aspect of this bug is specific to the _branch_ commercial build. Refer to the table in comment #3. I can reproduce the comm. branch bug in today's builds on both a win98 and a win2k machine. Further comments in http://bugscape.mcom.com/show_bug.cgi?id=17790

On a side note, setting NSPR_LOG_MODULES=all:8 and running with -console, I'm surprised at how much a Netscape commercial build outputs. Does this add bloat or reduce performance when not used? I think this problem is caused by nsIModule-type "new" XPCOM plugins. npcdt.dll is one of these kinds of plugins (can be verified with depends.exe) and only ships with the Netscape commercial builds. Copying that DLL to my components folder and somehow getting it to "register", I was able to reproduce the problem in a Mozilla branch debug build. To verify a true XPCOM plugin is registered, set a breakpoint in the middle of |nsPluginHostImpl::LoadXPCOMPlugins|. Linux users may also be seeing this regularly as I think JRE 1.4 is one of these 'components'. First off let me say that I think there are *several* problems here. This patch attempts of the fix the problem of the growing registry on each start on Netscape commercial builds because of XPCOM style plugin/components (may help Linux w/JRE 1.4 too). The (ApplicationRegistry) registry grows because we detect plugins have changed each time we scan: // count plugins remained in cache, if there are some, that means some plugins were removed; // while counting, we should ignore unwanted plugins which are also present in cache PRUint32 cachecount = 0; for (nsPluginTag * cachetag = mCachedPlugins; cachetag; cachetag = cachetag->mNext) { if (!(cachetag->mFlags & NS_PLUGIN_FLAG_UNWANTED)) cachecount++; } // if there is something left in cache, some plugins got removed from the directory // and therefor their info did not get removed from the cache info list during directory scan; // flag this fact if (cachecount > 0) *aPluginsChanged = PR_TRUE; } When they change, we nuke the subtree and re-write the info. There is no Pack() method so it grows endless. We think plugins have changed because we are not marking or removing the cached plugin when it's a 'component' and loaded from the ApplicationComponentRegistry. The flag is not set in the loop above and cachecount is equal to my plugin/components. There may be several ways to fix this specific problem and this patch takes one stab at it by simply removing the cached tag for the XPCOM plugin. This list is cleaned up shortly thereafter anyway post scanning. We probably should not cache these plugins twice anyway in the registry but I thought just removing it may be least risky. Thoughts? Reviews?

(Re: NSPR logging: A PR_LOG call, in an optimized build with FORCE_PR_LOG set, but NSPR_LOG_MODULES not defined, adds three instructions (including a branch) to the line of execution. So, that's pretty minimal, but I'll check if there is a perceivable impact on startup or pageload (where it is called a lot) or on bloat (although unless FORCE_PR_LOG is widespread, I doubt that this has a major impact on code size). On the other hand, I think that having logging available has in the past helped to diagnose end user problems "in the field").

Peter, although I still cannot see the problem with npcdt plugin, I see now that this can happen. But instead of not caching XPCOM plugins why not just treat them as unwanted in the code you cited above? Something along the following lines: if (!*aPluginsChanged) { // count plugins remained in cache, if there are some, that means... // while counting, we should ignore unwanted plugins which are... PRUint32 cachecount = 0; for (nsPluginTag * cachetag = mCachedPlugins; cachetag; cachetag = cachetag->mNext) { if ((cachetag->mFlags & NS_PLUGIN_FLAG_UNWANTED) || !(cachetag->mFlags & NS_PLUGIN_FLAG_OLDSCHOOL)) continue; cachecount++; } I think the reason I want XPCOM plugins still be in our cache is that if such a plugin lives in the Plugins folder rather than in the Components folder we are going to be screwed again because every time we will think it has just been added. But given this patch is supposed to go to the branch only, I am fine with yours.

Andrei, You may need to delete components.reg and restart several times to get the CDT plugin to get picked up when |LoadXPCOMPlugins| is called. Use the branch. Do not have another copy of the CDT plugin/component in any plugins folder. You may have to use regxpcom as last resort. NS_PLUGIN_FLAG_OLDSCHOOL can not be used here because it is only valid after mEntryPoint has been set which is typically in |GetPluginFactory|. That's fine if we shouldn't remove the cache tag. We can just as well mark it as "unwanted" to avoid cachecount++. This patch does that and it also stops the appreg from growing on each startup. It still may grow at other times when plugins have changed. However, you do bring up a valid point if the same copy of the plugin is in both a plugins folder and a components folder. Currently, |LoadXPCOMPlugin| does not set |tag->mLastModifiedTime|, it is Zero. It may also cause us to always think new plugins have been installed. This may be yet another bug.

Comment on attachment 92191 [details] [diff] [review] patch v.2 I feel better, thanks, Peter, r=av

I was able to reproduce and test it eventually. The patch fixes things neatly. The problem with the registry growth has even more impact than it was originally thought (thanks jrgm for pointing this out): it grows on every about:plugins too which essentially means on every javascript:navigator.plugins.refresh. The patch fixes this too. I think we must check it in as it solves the original problem and is simple and safe. It will not interfere with more profound solution which dougt and serge are working on in bug 109937.

Sorry, 'bug 109937' should read 'bug 109739'.

Comment on attachment 92191 [details] [diff] [review] patch v.2 sr=dveditz

Adding adt1.0.1+ on behalf of the adt. Please check into the Mozilla branch after getting approval from drivers.

Comment on attachment 92191 [details] [diff] [review] patch v.2 a=asa (on behalf of drivers) for checkin to 1.1

a=chofmann for the branch. add the fixed1.0.1 keyword after checking in on the branch.

Formally marking fixed as the patch is on the trunk now.

marking as "mozilla1.0.1+" per Comment #29 From chris hofmann.

On the branch now. Marking fixed1.0.1

fix looks great, the registry has stopped growing. verified on today's build 0723. made sure that I had the CDT plugin present.

by ' the registry has stopped growing' , i meant, rigistry.dat does not increase if I repeatedly launch the same branch build (0723).

Thanks Shrirang! Verified

I'm not really sure if this has been "Fixed" maybe it's due to other changes, but I was one of the original multi-meg registry.dat file victims and just recently I checked and yes it was wan't growing, but it also wasn't optimized either. I guess with parsing out the pluginreg.dat file the registry.dat and the pluginreg.dat aren't cleaned out or something. Here's my example after already having cleaned out my registry.dat file back in August or September and using those build I had no problem with the file "growing", but here's what I found yesterday Steps to reproduce (at least on Windows Systems): - have mozilla open or in Quick Launch mode - go to the blah-blah\Application Data\Mozilla and rename the pluginreg.dat and the registry.dat files - close mozilla or exit out of quick launch - new files are created and here is what I've seen in size: old registry.dat 173k New 1.04kb old pluginreg.dat 7.67kb New 5.78kb Somewhere there is some cleanup that should be done, not really sure where though, thanks.

Reopened per Comment #36

The growth problem has been fixed. If you want to open a new bug on cleaning out the old cruft go ahead, but I doubt it'll get much action (in fact there may be such a bug, search for "registry packing" or similar). Effort by the profile folks is focusing on other problems such as portability, and may replace registry.dat entirely in the future.

Sorry about getting testy below, but here goes: Is this really fair to just dismiss this, since there could very well be users that have huge registry.dat files out there? This patch should really be marked, 'incomplete' or 'insufficient', cleaning this file should have been part of the solution, not spunoff for a future conceptualized rewrite. Oh well, I'll make sure to check my .dat files renegade files.

I just checked one of my other computers (one that I don't even use that much) and found a 1.1 meg registry.dat file, fun for the whole family!!!

Also, can someone guage the load and startup impact of such a large file? Just wondering.

re: startup impact, see comment #3 (and reconfirmed today). Despite the large size, this does not impact startup on win2k (and likely linux, although I'm not sure about mac [particularly os/9]). The app does not try to read this entire file; it just opens, maps, checks a header, seeks, reads, closes and unmaps (from what I can tell). I'd have preferred (in a perfect world) that this had been cleaned up. But that would be bug 10210 as dveditz points out.

I had posted a write up on my site about clearing this and just updated it include Netscape 7.01 based on this email that I got: "Re: Mozilla registry.dat files. A week ago I installed Netscape 7.01. It is based on Mozilla build 1.0.2, the latest stable release. I followed your instructions for renaming the registry.dat file, then restarting Netscape. Registry.dat file went from 2407KB to 25KB on restart, with no noticable change in settings or performance. Hopefully there will be a fix because I seriously like the browser. Good work on your finding!" Maybe Netscape should apply the patch soon or release 7.1 based on 1.2a, hope this helps. http://www.mrtech.com/news/messages/2466.html

um, and what if the users to whom you are recommending this procedure have more than one profile and/or have a profile that doesn't have the default profile name and/or that profile is not located in the default profile directory. How do they recover their existing profile(s)? [Actually, I'm going to defer to somebody from the installer group for an authoritative explanation about the above. But I thought you might like to know that deleting that file can have unintended consequences. You should warn your readers that they risk dataloss if they do this, and, at minimum, that they should back up their 'Application Data\Mozilla' before trying any of this.]

The registry.dat file stores the locations of all profiles used by the application as well as plugin information used by the application. People who regularly install and test are the ones most l likely to see this problem and the ones most able to be aware of the pitfalls of deleting this file. Mel's instructions seem to alert people to these issues. It would not be recommended for other users

verified-