Closed Bug 1319727 Opened 8 years ago Closed 5 years ago

Large places.sqlite causes huge delay when visiting unknown page from urlbar

Categories

(Toolkit :: Places, defect, P3)

50 Branch
x86_64
Windows 10
defect

Tracking

()

RESOLVED WORKSFORME

People

(Reporter: rinnkazul, Unassigned)

References

Details

(Keywords: perf, Whiteboard: [fxsearch])

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:50.0) Gecko/20100101 Firefox/50.0 Build ID: 20161104212021 Steps to reproduce: I have a large (but maintained by an occassional vacuum) places.sqlite database, as I've kept it since the days of old and it has ~720k rows and consumes ~330MB. I don't remember when this behavior changed, but basically if I type a page I've never visited before into the URL bar and hit enter, for some reason I have to wait for firefox to sift through the places.sqlite (likely looking for signs of me ever visiting such site before) and only after that it actually proceeds to visit that site. Since the profile is large and my processor is not very fast, this takes upwards to 15 seconds in some cases (!); even though the places.sqlite is on an SSD. Actual results: Firefox took over ten seconds to start trying to visit an unknown site after I typed its address into the URL bar and hit enter Expected results: Firefox should have instantly visited the site upon me hitting enter. The only "workaround" I currently have is to hit spacebar and then enter which for some reason causes firefox to instantly try to load the given site.
OS: Unspecified → Windows 10
Hardware: Unspecified → x86_64
Component: Untriaged → Places
Product: Firefox → Toolkit
I must note a so large db is not officially supported, we have expiration for a reason. Do you really need multi-year history? If you could do with limited history (maybe even just 1 or 2 years) it may help in general. That said, let's see what we can do, clearly having some sort of regression range (so knowing when the problem started) would make things much easier. After you type a page and press enter, we should not wait for a search, we should only be waiting for the first result. And Firefox 50 has indeed a fix that should largely improve the situation compared to 49. I wonder why you don't see that. Let's start from simple troubleshooting: is it slow in safe mode https://support.mozilla.org/kb/troubleshoot-firefox-issues-using-safe-mode ?
You know how it is. Users like to keep a track of things. I like knowing I visited a site previously and I sometimes dig very far in the history to find something I checked for example five years ago. At any rate, I tried safe mode and all that really did was make my firefox launch a bit faster. The slowness of "going to a page" remains. See the below gif example that I captured whilst in firefox safe mode. Assume I hit enter exactly just as I finished typing .com https://imgur.com/ZmXorYl You can see the delay is rather long.
I think the problem is that there is a long-running statement ongoing from the previous search, and the short running one here is serialized to that one. So we basically have to wait for a whole table scan. The first solution may be implementing an FTS index, so we can stop doing a linear scan, we plan this for 2017 already. The other solution is that we add support for sqlite3_interrupt to mozStorage, and forward that support to Sqlite.jsm. It may not be trivial though. First I'd add support only for read-only connections, cause it's too scary to interrupt random writes, this is easy. Second, we must ensure queries invoked after it are executed, that may require interesting thread synchronization strategies. Indeed "The sqlite3_interrupt(D) call is in effect until all currently running SQL statements on database connection D complete. Any new SQL statements that are started after the sqlite3_interrupt() call and before the running statements reaches zero are interrupted as if they had been running prior to the sqlite3_interrupt() call." I think we should also evaluate this, maybe even a partial support, it could be useful to be able to interrupt long running async queries.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Priority: -- → P3
Whiteboard: [fxsearch]
Depends on: 1320301
I cannot say for sure but I think this "bug" is partly (or mostly?) responsible for https://bugzilla.mozilla.org/show_bug.cgi?id=1318930 . If you aren't expecting a delay, you will assume that the enter key was simply not registered (and you will try hitting it again), at which point it will "eventually" work - but that's simply because the search query from the first enter will finish its scan of the places.sqlite . If you are not aware of what's happening, it will seem like an intermittent issue with the URLbar ignoring enter keypresses. It might be worth asking in that bug about the size of people's databases - because all of those seem to be upgrades and in that case I reckon they've been keeping their database for quite long. Pair it up with a not-so-speedy pc (but even on fast ones it still happens) or an overly-aggressive antivirus software (slowing down I/O to places.sqlite) and you've got this issue.
Priority: P3 → P2
Blocks: 1287418
My workaround is to click the "Go" button instead of pressing Enter. The "Go" button is much faster for some reason.
My workaround is to hit spacebar and then enter. For some reason it works for me.
(In reply to Masatoshi Kimura [:emk] from comment #5) > My workaround is to click the "Go" button instead of pressing Enter. The > "Go" button is much faster for some reason. That's so slow that I might as well wait for enter to be registered. (In reply to LittleVulpix from comment #6) > My workaround is to hit spacebar and then enter. For some reason it works > for me. That's weird, but even if it works it's not useful for many circumstances. Adding a space changes what you try to load. Can't use it when you edit the url, for one.
(In reply to Marco Bonardo [::mak] from comment #3) > I think the problem is that there is a long-running statement ongoing from > the previous search, and the short running one here is serialized to that > one. So we basically have to wait for a whole table scan. > Why do we need to wait at all? Why not just listen to enter asynchronously. And perform the appropriate action when it's pressed? The go button seems to work like that.
(In reply to avada from comment #8) > Why do we need to wait at all? Why not just listen to enter asynchronously. > And perform the appropriate action when it's pressed? The go button seems to > work like that. We need a way to interrupt a query abruptly, and that's why the dependency. Enter cannot just be handled as soon as pressed, cause you may type and press enter by memory (for example "mo"+Enter to go to mozilla), if a query is running we need to know whether it will change the result (or we could search for "mo" instead of visiting mozilla). There is space for improvements, but first of all we need the interrupt query support.
(In reply to Marco Bonardo [::mak] from comment #9) > (In reply to avada from comment #8) > > We need a way to interrupt a query abruptly, and that's why the dependency. > Enter cannot just be handled as soon as pressed, cause you may type and > press enter by memory (for example "mo"+Enter to go to mozilla), if a query > is running we need to know whether it will change the result (or we could > search for "mo" instead of visiting mozilla). > There is space for improvements, but first of all we need the interrupt > query support. This doesn't make any sense to me. The browser should do whatever is expected from what it shows. It's actually worse if the browser waits and does something other than to which you pressed enter. (Though usually this is not the case enter press is just swallowed) In other cases there's no need to wait at all. If the first word is a keyword then only what's in the urlbar that matters. (I'm guessing/hoping, that keywords are checked first)
Whiteboard: [fxsearch] → [fxsearch][qf]
Whiteboard: [fxsearch][qf] → [fxsearch]
Keywords: perf
Priority: P2 → P3

With Firefox ESR 68.3, is this still an issue?

Flags: needinfo?(rinnkazul)

I will test it this weekend and report back here.

Wow. It is actually resolved - firefox goes to a page it's never seen before as soon as I press enter. I compared it to my current version with exactly the same profile, and the difference is night and day.

I guess this bug can be marked as resolved then, good job!

Flags: needinfo?(rinnkazul)

thanks for reporting back.

Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.