When I put a large history.dat file (about 4 MB) in an empty profile folder and start it up, it takes over a full minute (with 100 CPU usage) before Firefox starts up. When I also put a compatibility.ini in it, it starts up quickly. I doubt if anyone will ever do this, but it is a (recent) regression, so I'm reporting it. Regression range for this is http://bonsai.mozilla.org/cvsquery.cgi?module=PhoenixTinderbox&date=explicit&mindate=1178037420&maxdate=1178040899 with two bugs in it, bug 378975 and bug 378038.

Oh well, my computer is cheating me about the compatibility.ini file. It has nothing to do with it I think. There must be another file in the profile except places.sqlite that causes the bug not to appear for the second time. The regression range is correct in any case.

Honestly, I can't see what either of those bugs would have to do with history import. But nothing else around there seems relevant either. I'd be happy to look into this if you can describe in a little more detail the way to set this up; I'm not that familiar with the profile business with Firefox... Also, can that history.dat compress enough to attach to this bug? If not, or if you don't want to make it public, can you mail it to me?

Well, I think I can reproduce this... but I have no idea what's going on here. :( For some reason, finishing up the places async IO, and shutting down that thread, takes forever, with 100% CPU during that time. But all the time seems to be spent in things like gtk event code and waiting on the X server... which is not likely to be an issue on Win XP. I'll attach a profile; it covers part of startup, from when things really should have appeared but hadn't to when I got bored (about 50 seconds all told). Benjamin, Seth, do you know what might be up here?

For what it's worth, I tried reverting the main change from bug 378975 (made InitialReflow sync again), and this problem did not go away.

So... this is very interesting... I'm seeing us hitting the history from the end of the load for resource://gre/res/hiddenWindow.html. The reflow at the end of that load does the scrollbar guessing thing, which gets the global history service, which does nsMorkHistoryImporter::ImportHistory... which takes a while, presumably. Not sure that would be enough to account for the 1-minute thing. This is all _before_ the EM restart. Not sure what happens after that, to be honest, but I bet that explains the regression range. That said, the profile I posted was started _after_ the history import is all done, and we're still chewing tons of CPU there. Now I bet we could change the scrollbar code to not look in the history for XUL documents or chrome documents or whatever... but I'm not sure why this is being a problem; doesn't Places import stuff on startup anyway?

> Benjamin, Seth, do you know what might be up here? sorry for the delay boris. As for perf problems with nsMorkHistoryImporter::ImportHistory() on startup, I haven't profiled it yet. as for after start up, so some thoughts: 1) do you have lots of livebookmarks? (see bug #329534 for example). That might explain the chewing of CPU after startup. 2) we have other bugs about perf problems with lots of history, for example see bug #223476.

> 1) do you have lots of livebookmarks? Seth, this is a profile that just contains a history.dat... And yes, I realize that there are existing performance problems with history. But why did they not appear here until we started getting the history service during the EM startup?

I can't reproduce the problem on Vista. It opens within 5 seconds. Although my XP laptop isn't slow at all, the other one is much faster. The bug was tested on both computers under exactly the same conditions.

Blocking, see also bug 329736

Maybe we should just back out bug 378975 or something... I really don't have the time to dig into this bug before 1.9, and given bug 380515 it seems there's more to things than just the Places history stuff? Of course that would regress Tp/Tp2... The other option is to not do this reflow that we're Of course what I really don't understand is why we're bothering to reflow the hidden window on non-Mac platforms.... and why we bother looking at the history for the hidden window on any platform. We really shouldn't do that, probably.

_need_ this for b1

a while back shawn and I were looking into bug #386739 about while migration from IE was taking so long, which is related. some thoughts we had: 1) for history, bookmark, import, run the import as a transaction [see runInBatchMode()] 2) we also noticed a big perf gain when we did: rv = mDBConn->ExecuteSimpleSQL( NS_LITERAL_CSTRING("PRAGMA synchronous = OFF")); NS_ENSURE_SUCCESS(rv, rv); moving off to m8

> 1) for history, bookmark, import, run the import as a transaction [see > runInBatchMode()] I think this is the right thing to do first, before b1. in migration.js, we have onMigratingMigrate() If we could do that work as the callback for nsINavHistoryBatchCallback, we would wrap all the history and bookmark import in a transaction, and this should make initial import (and manual import) much faster. note, initial profile import from IE is slow, due to importing history. we end up calling AddPageWithDetails() for every item in the IE history.

> in migration.js, we have onMigratingMigrate() that will not work, as I think we have to wait until after we call DoStartup() in nsIEProfileMigrator::Migrate() before we can instantiate the history or bookmark service.

looking at the mork importer code, importing history.dat (which this bug is about) already uses a transaction. but first run migration / import from ie (and other browsers), does not, which I'll spin off to another bug.

We've dropped the index on title, which will mitigate this a bit. Talked to seth the other day about turning off PRAGMA synchronous, or setting to NORMAL during import operations, which could provide up to 50% performance boost. The risk is that if the OS crashes or power is turned of while file operations are occurring, db corruption might occur.

I could still reproduce it in 9 May 2007 build, but I don't see this problem anymore with the latest trunk build. Tried it a couple of times but it appears to be fixed.

Resolving WFM as the reporter is saying that's the case. Please re-open if you're experiencing extended delays on first run when migrating from Fx2 to Fx3.

Bug 451915 - move Firefox/Places bugs to Firefox/Bookmarks and History. Remove all bugspam from this move by filtering for the string "places-to-b-and-h". In Thunderbird 3.0b, you do that as follows: Tools | Message Filters Make sure the correct account is selected. Click "New" Conditions: Body contains places-to-b-and-h Change the action to "Delete Message". Select "Manually Run" from the dropdown at the top. Click OK. Select the filter in the list, make sure "Inbox" is selected at the bottom, and click "Run Now". This should delete all the bugspam. You can then delete the filter. Gerv