The 'Forget this site' functionality still demonstrates the problems described above on the 66.0a1 (which is why I filed the duplicated mentioned above).

The deletion of the history older than 6 months happens rapidly itself.

But after that Firefox eats 11Gb RAM and 200%CPU for several hours.

It's possible to use firefox during cleanup process. (But it become very slowly.) It's event possible to close it. But the last time it crashed during close. I've send crash report.

For about 2 weeks now firefox nightly has had this problem. Any modification to the bookmarks is painfully slow. about:support doesn't count the number of bookmarks, so I don't know how many I have but 6 months ago it was 5k, I think.

Adding a bookmark takes ages and then the window does popup, it's empty for a while. I could record a video if requested, but there'd be nothing special to see. Should you require more information, I'd be happy to provide it

(In reply to Librem Grief from comment #74)

For about 2 weeks now firefox nightly has had this problem. Any modification to the bookmarks is painfully slow. about:support doesn't count the number of bookmarks, so I don't know how many I have but 6 months ago it was 5k, I think.

Hi, this is unlikely to be related directly to this bug. Please can you file a new bug and attach the about:support output from the places verification.

I knew about this bug (or just inefficiency) for a long time and today i found a workaround how to not sit with frozen browser for huge amount of time.

I was deleting 3.5k items.

As usual, everything froze and after some time the usual popup came on - Abort, continue, debug script.
I clicked debug and as it turns out, items were deleted from the list.

It looks like its not deleting taking time, but something else. Maybe some callbacks that should have been delegated from main thread to background jobs to not block the whole browser.

I must say this is one of those things that stops people from switching over, because this should not be the case. Especially not in 2019 after 8 years from the original bug report.

(In reply to pavelloz from comment #78)

It looks like [it’s] not deleting taking time, but something else. Maybe some callbacks that should have been delegated from main thread to background jobs to not block the whole browser.

See comment 28:

We also need to check what makes the end callback so slow. from the profile it looks like it's invalidateContainer trying to restore selection (particularly getNewRowForRemovedNode), and then the select event fires and causes oncommandupdate, that updates the commands on the selection. It's likely we can act at these points to largely improve the situation.

And see comments 64, 65.

When I select a large amount of history entries and press delete, Firefox hangs for a while, but when I select the same amount of history entries and press delete and directly deselect by selecting only one history entry, deleting is extremely fast.

(Emphasis and grammar edits mine.)

It’s restoring selection that seems to be slow.

Given the hint above, I have found that if after selecting all, hit delete, then esc a couple of times (which means that the highlighted list is apparently cleared, I can use the browser more or less normally (perhaps just a little slower) while the history continues to be deleted in batches - as I am typing this, in fact. What that means above the coding I cannot begin guess, but it is certainly not a locked browser now. The task was to delete some 36,000 entries - it has been going for about 9 minutes already.

Does that help?

= sorry, "about the coding"

At the moment we don't have the resources to spend on the investigation side.
I agree that one of the improvements involves the restoreSelection code path: https://searchfox.org/mozilla-central/search?q=restoreSelection&path=places
But is invalidateContainer being invoked too often, or is restoreSelection just slow? And if it's the former, what's the invalidateContainer stack, if it's an operation that is not expected to restore selection, can we temporarily disable that around that specific operation? and so on...
The code is mostly javascript, if you are proficient with that, it should be possible to follow the code and find improvements. The huge costs here are into verifying that those improvements don't cause unexpected bugs in other operations.

Is history being rewritten?

I think this is one of the oldest parts (if not the oldest) in Fx. I remember the same UI on version 3 or 4, which looks like its been forgotten, but its still an important feature to a lot of people.

Im asking, because i dont know if i should invest time to learn how to profile/mock data in Firefox to fix it, as im fairly proficient in javascript and i care about performance a lot.

Also, maybe thats a stupid question, but, which selection exactly it is restoring?

Simple scenario:

1. User selects 10 rows to delete.
2. User clicks delete

The end.

What restore selection is exactly trying to reselect if everything that user had selected, wanted to just delete and it shouldnt exist after the operation?

(In reply to pavelloz from comment #83)

Is history being rewritten?

We'd like to, finding resources is a problem, but there's the will. The question is "when", and likely it's still months far away.

(In reply to pavelloz from comment #84)

What restore selection is exactly trying to reselect if everything that user had selected, wanted to just delete and it shouldnt exist after the operation?

It depends, for example if the removal happens node by node, and we're not skipping reselection, after each removal the tree gets rebuilt and selection restored on the other items that have not been removed yet (and I suspect that's what happens). And at the end, we're likely to select something (probably the entry before the ones that were selected) but that's a single thing so it should be cheap. The bug is there, so explaining what happens is probably as expensive as fixing the bug.

Interesting way to trigger this bug: upgraded ubuntu from 19.04 to 19.10 (ff 72.0.1) with some few thousand entries in history.

After the first boot on 19.10, open firefox, open history. It will ask if you want to delete all history entries. No matter what you select, the Unresponsive Script popup (treeview.js:198 in this case) will show up and not response to anything (can barely move the mouse, no attempt to press spacebar/enter on the popup worked). I had to wait 17min just for the machine (core i3) to be able to switch to another physical TTY and i could not ever get a password prompt before the login attempt timedout. Eventually I got a OOM event that killed firefox for me.

I was very close to just unplug the computer during all this.

I will probably not be impacted by this bug anytime soon and it will slip my mind soon as I do not have much time for open source as I used to... But if someone can point me to the general direction of where the problem is I can spend some time to suggest a patch in the next week.

Although I cannot tell why it seems O(n^2), I have a workaround, maybe a potential fix.

When I open the history in the "Library", search for a domain, press "Ctrl+A" to select all, then press "delete" to delete the records.
It takes very long time as people mentioned above. Slow for 4000 items, and seems endless for 14000 items.

Then I open ~/.mozilla/firefox/xxx.default-xxx/places.sqlite, execute below sql statements. It finished the deletion in less than 1 second.

delete from moz_places where url like "http://example.net/%";
delete from moz_historyvisits where moz_historyvisits.place_id not in (
  SELECT id from moz_places
);

It seems the deletion doesn't cascade to other tables with foreign key reference so I'm not sure if I have missed some tables.

I guess the Firefox delete large among of history items very slowly is because it has to delete the items one by one (using statement like where id=?) because the user may not be selecting all matched result.
How about add a specific button "Delete all matched items" so this trick can be applied from the Firefox GUI?

(In reply to aabbcc1241 from comment #90)

How about add a specific button "Delete all matched items" so this trick can be applied from the Firefox GUI?

That would not be suitable as it doesn't address the UI side. The issue here, as has already been discussed, is getting the UI updated. As Marco has said in comment 82 and comment 85, the UI parts are the bits that really need the investigation here. My suspicion is that to fix this, we might also need to migrate our remove observers over to a new observer system so that we can batch notifications, but that's only a suspicion at this stage.

We should just throw away the history view and make a new one that is both useful to the users (the current one is not) AND fast.

I agree the suggest by Macro. As least for me, I don't care if the view is not updated in the middle of deletion.
Say it is deleting 1000 items, I won't mind if the UI display as if the 1000 items are still there until all of them are deleted.

8 years obnoxious bug
Why is it so hard to deal with this? (I haven't dared to look the code)

Why not just add another option do the UI: "Remove history older than:" and call directly the corresponding sqlite command without updating any UI componente.

Some discussion of this problem and bug on Hacker News:

https://news.ycombinator.com/item?id=24061787

(In reply to timbugzilla from comment #95)

Some discussion of this problem and bug on Hacker News:

https://news.ycombinator.com/item?id=24061787

This link shows a discussion about privacy and deleting cookies. It an entirely different subject.

The current issue is about slow updating the history database and UI component.

(In reply to sparcbr from comment #96)

This link shows a discussion about privacy and deleting cookies. It an entirely different subject.

They also discuss slow deleting from history and even link this bug.
https://news.ycombinator.com/item?id=24062006

A few quick checks: deleting from 50 (in 50s) to 300, it takes about 2 .5 s / 50 (in this machine) for the list to be shown as losing those items. But, immediately after that deletion the main FF process takes most of the cycles of one core for about 10 s / 50 (there is some but not a lot of disk activity). During this time switching tabs takes at least 1 s, maybe slower on occasions. From about 350 deletions, the second phase - busy main process - starts before the deletion from the list is shown. It would thus appear that even though the deletions have formally been completed, FF will become sluggish. Deleting a 1000 items it takes about 2 minutes for the history to lose those items (progressively) - that is longer than would have been expected, the main process CPU usage gets ragged in the overlap - mutual interference, competition? - and the switching delay increases to ~2 s. (This effect is certainly there at 500.) Repeated deletes of 50, for which you have to wait for the highlighted list to disappear, get slower and slower as the main process is involved.

Does this suggest anything? The question is what process occurs after the (apparent) deletion - sorting? Whatever that is, it appears not to be the actual deletion, but that thread competes with the deletion itself after a certain size. It seems to form a queue, and just goes on and on with (large enough) deletions occurring closely enough.

Initial list was about 13,000, still problematic by the time I reached 7,500 - these are nol really very big data sets. But, there is extra use of RAM during the second phase: 500 deletions required up to 2.3 GB more - that is 4.6 MB per item!, in at least 4 stages, with gradual recovery.

BTW: why are (101) and (102) tagged 'advocacy'?

They are all comments which are essentially "metoo" or comments designed to make us fix it sooner (e.g. "It is incredible that this problem still exists"). These types of comment are not helpful (as they spam everyone on the bug) and are against the etiquette: https://bugzilla.mozilla.org/page.cgi?id=etiquette.html. If they keep on happening, then we will likely mark this bug as restricted commenting.

We understand this is still an issue, and we recognise that it is a pain point for people who do hit it. However, it likely involves a significant amount of non-trivial work to fix. At the moment our focus is on improving other areas of Firefox, and we don't have the time available to dedicate to this.

Apologies. I had checked etiquette beforehand and could find no indication of what the obscure tag meant - it seems like the wrong word is being used (clarify that page with your elaboration?). Point taken, though, although I would have thought data such as (102) would be helpful, where only the last line seems offensive under the rules. Work-load and priorities understood; I assume then that no activity does not mean ignored or forgotten.
Thanks for the explanation. (Ironically, this reply is also technically empty ... )

In version 88 history deletes very fast. 288 rows deletes in 2-3 sec. Thanks.
In previous versions it could take time over 20 sec.

(In reply to kolio from comment #115)

In version 88 history deletes very fast. 288 rows deletes in 2-3 sec. Thanks.
In previous versions it could take time over 20 sec.

That's probably the effect of Daisuke's work with new history notifications, and in particular bug 1694818. That is part of the architecture work necessary to fix this, but it's just the beginning of the work.

500 in 20 s, 1000 in 65 s, 2000 in 3 min 25s, but no apparent extra RAM use or core saturation, and switching now OK (except window refresh in FF). Seems to be an improvement, indeed. Very glad to see work being done on this.

I found a work-around, and pin-pointed what causes the hanging.

If you have all the history entries highlighted during the deletion, then for example 3000 entries:

Delete will take this long for the first 300 and every after...
300 -> 15 seconds
300 -> 10 seconds
300 -> 5 seconds

There is this kind of pattern.

Work-around:

Click anywhere in the list of items to de-select all right after starting the delete request.

What happens: Firefox doesn't hang and starts deleting entries immediately.

Image explanation with work around steps:

https://i.imgur.com/nBafaYB.png

Note: I scrolled up a bit and saw another user has provided the same work-around 3 years ago: https://bugzilla.mozilla.org/show_bug.cgi?id=734643#c64

Coding Quick Fix?

- Add a "deselect all history items" UI command when deletion has begun

That does indeed appear to work - 1000 items in 15 s here, against 50 s with the highlight on.
Thank you.

Just tried with ww.google.com, 6078 entries. Highlighted all, clicked delete and then immediately clicked a couple times on the list. The screen initially was frozen for nearly a minute but then came back to life and went very quickly. Total time was about 85 seconds. Good work around until a builtin fix. Thanks

(In reply to bugbasher from comment #120)

I found a work-around, and pin-pointed what causes the hanging.

Smart, we are aware that the issue is due to re-selecting entries in the very long list, so this makes total sense, because you clear the selection there's nothing to re-select.

Why can't you just deselect the to-be-deleted entries automatically? Seems like that would be a very easy way to make this issue go away. They aren't going to exist after deletion anyway, so there is no reason to keep them selected.

Yes, it's possible that pre-calculating the final node to be selected before the removal starts, accumulating the removals in memory, and changing the selection before proceeding with removals, would improve things. It should be measured. If anyone is willing to modify the code and check the effect of that, you're welcome to. The code is more or less around this area https://searchfox.org/mozilla-central/source/browser/components/places/content/controller.js#930-947,990-1004

I've run up against this issue, found this thread, and wanted to point out that deletion can still take a long time even with the deselect workaround mentioned above (thanks for that by the way, it helps immensely). It took a few dozen seconds on my machine (macbook + firefox 91.0) to delete about 8000 items that were not selected, as opposed to a few dozen minutes for a similar number of selected items. Dozens of seconds is a lot better than dozens of minutes, especially when it means firefox is relatively more responsive in the meantime, but it's still much slower than it should be.

The biggest hang (the one users are circumventing by deselecting the nodes) seems to be caused by the eventListener at line 25 of browser/components/places/content/places-tree.js. In my case, this "while (true) {...}" hangs the entire browser. Problem is, I have no idea why this code exists in the first place and it's probably there for a reason.

But even without it, the functions PTV_nodeRemoved (this one is especially important) and PTV_invalidateContainer in browser/components/places/content/treeView.js probably also run for every single node that has been deleted. They're responsible for visually removing a node, rebuilding the view, selecting the next node, that kind of thing.

If these where called only at the end of the deletion, I think deleting a thousand entries could take mere milliseconds. The request is actually blazing fast, but these few eventListeners responsible for the visual side wake up hundreds of times for nothing.

Now I don't want to get your hopes high, I'm just reporting what I found, I'm not a FF dev and I don't know this code, but maybe I'll keep poking at it and see what I can find.

I got a 20k history row places.sqlite, and when I try to delete some history, the search is fast, but the delete operation speed is intolerably slow. Firefox hangs and every 3 minute delete 300. I terminate Firefox after 15minute delete only 900 rows, and there are 18000+ to delete.

By contrast, if I do the same operation by a tool name DB Browser for SQLite (open source on Github), it took a second for searching and deleting, then about another second to commit the change to the actual file.

I think either firefox commit to the sqlite after every row delete, or keep updating the "history window" by a query the database block the commit(so we can see the results drop 300 for a moment), or worst, it does both.

Why my comment140 mark as advocacy?
I post a comparison between Firefox and another SQLite software to Firefox places.sqlite by a real-world operation. it is more than a hours VS a few seconds.(Firefox is VERY slow and apperantly improvable)
I make conjectures of Why the speed is so slow.
I didn't push anyone to fix it soon, just don't understand the standard.

In the process of migrating remaining bugs to the new severity system, the severity for this bug cannot be automatically determined. Please retriage this bug using the new severity system.

Currently, deleting 2,000 history items causes 2,000 PlacesVisitRemoved events
to be fired. Each such event results in a call to invalidateContainer() in
treeView.js. This is an expensive function.

On a Macbook Pro 2015, it took 73 seconds to delete 2,000 history items.

This patch introduces a new event: PlacesBatching. A PlacesBatching(true) event
is fired before the PlacesVisitRemoved events, and then a PlacesBatching(false)
event is fired afterwards. treeView.js receives these batching
notifications and defers the call to invalidateContainer() until all
PlacesVisitRemoved events have finished. The end result is that
invalidateContainer() is executed only once instead of 2,000 times.

On a Macbook Pro 2015, thanks to this patch it took < 1 second to delete
2,000 history items, representing a performance increase of over 7,000%.

Depends on D158932

The following patch is waiting for review from a reviewer who resigned from the review:

:3ecdbelo, could you please find another reviewer?

For more information, please visit auto_nag documentation.

Changes to the patches are required, not a new reviewer.

Resigning in Phabricator means something like "I'm not able to review this patch (because it's out of my area of expertise or because I don't have time)", that's why the bot suggested finding a new reviewer.

Further investigation has revealed that repeatedly invoking
treeView.js invalidateContainer() is not the primary cause of the
poor performance when a user deletes a large number of
items from a history query.

Yes, one of the nsNavHistoryQueryResultNode observers is being
refreshed for every Page_removed event, resulting in
repeated calls to treeView.js invalidateContainer(). This
definitely has a performance impact.

However, even worse is that another of the nsNavHistoryQueryResultNode
observers is being incrementally updated for every Page_removed
event. This is causing the majority of the slowdown.

This patch provides a simple solution: when handling a PlacesEvent
containing 50 or more Page_removed events and no other events,
just refresh all the history observers once and return.

With this change, we are able to achieve the 7,000% speedup
when a user deletes 2,000 items from a history query.

Please file a bug blocking this one, and post the patch there, as I suggested.

Resetting assignee, since Francis worked on bug 1795070 that should have helped quite a bit with this issue, thank you a lot.
For now we'll keep this open to keep collecting feedback.

At present, deleting 2000 (without esc) took 30 s / 300 (when there was a list refresh) for the first 900, then accelerated a little; total 2.5 min. FF locked in the meantime still.
There has been a lot of discussion of event calls etc, but no mention of the oddly large memory usage. Is that not a factor?

Tried a few seconds ago on a not-so-recent mac book pro. Deleting ~2600 history entries took around 2 seconds. Pretty good :)

Please note the fix for bug 1795070 won't be released until around 13th December in Firefox 108. Please make sure you have that version before testing again/commenting.

108.0a1 (2022-11-10)/macOS 12.6.1 - deleted ~1000 records in ~3 seconds. TY Francis.

Marco, based on recent discussion, can this item be marked as closed?

I have just tested (109.0, 32-b, laptop) and 1000 went in about 2 s, and 2000 more in about 5 s, then 3 s - very hard to time!
109.0, 64-b (PC) took 30 s, 25 s and 25 s for 2000. Both Win 7 Pro, much improved for sure.
The problem is indeed substantially fixed, but the difference is a bit odd.

(In reply to Dr B W Darvell from comment #159)

I have just tested (109.0, 32-b, laptop) and 1000 went in about 2 s, and 2000 more in about 5 s, then 3 s - very hard to time!

This is likely the effect of a cache, it's normal.

(In reply to Daniel Nguyen from comment #158)

Marco, based on recent discussion, can this item be marked as closed?

Yes, bringing this on forever is not useful, it's better to move to new bugs in case someone wants to report specific cases.
Let's call this fixed by bug 1795070.

I'm happy that the situation improved a lot, when we started working on the new notifications system in Places, years ago, we knew it could be used to improve performance, this was a nice confirmation the direction was correct, Francis patch could leverage the system advantages.
There's a lot that can be done yet.