(In reply to Herman van Rink from comment #18)

Still strange that these need to be open just to create a list of recently
used folders ...

Yes, I thought there is also some other cache that stores folder properties so that they do not need to be loaded in full just to read their name and last access time. But I'm no expert in this area.

Meanwhile, I have fixed bug 809066 so that the Recent submenu is populated with all the folders only if you really open it, not when the folder picker (e.g. in the Move To context item) is built. That should improve things a lot. But sadly it probably want be backported do TB 60.

If you solved your problem with setting db.idle_limit then that is probably a good workaround for now, until then next release of TB.

Do you think there is anything more to be done in this bug?

I still see some delays as the memory usage is sometimes dropping well within the idle_limit. But not as far down as before so it's often a second or two(and now I know what yo look for)
I guess there are conditions in which caches are dropped for other reasons.

Finding a separate storage for these properties would still be a good optimization, but that should be a separate bug to keep it focused.
I've just opened bug 1519375 for that.

Lets close this one.

While I was playing with bug 809066 I did NOT see all the folders in 'Recent' list be opened and in the cache.

Can you please retry disabling the 'Quick Folder Move' addon whether it causes the slowness and opening/caching of many folders? AFAIK it allows some search facilities so maybe it loads more folders beforehand, than the base TB would do.

strange.
I've checked again and with that add-on disabled(and restarted)... I see the same delay.

I have about 1700 local folders. Message Filters: New or Edit is instant for me, even with "Quick Folder Move" installed.

In Comment 18 I learned that it's related to memory usage. If most of it is open in memory then it's instant. Do you have any modified config options?

Please run the performance profiler:

1. install profiler add-on into thunderbird - download the file from https://github.com/firefox-devtools/Gecko-Profiler-Addon/blob/master/gecko_profiler.xpi?raw=true and in Tools > add-ons click the gear to install add-on from file

2. Follow instructions at https://profiler.firefox.com/ (videos BASED ON FIREFOX at https://profiler.firefox.com/docs/#/./videos-intro )

3. Create a profiler URL and post it here.

You must be using Thunderbird beta from https://www.thunderbird.net/en-US/channel/ or current nightly build from https://archive.mozilla.org/pub/thunderbird/nightly/latest-comm-central/

Sad that the thunderbird-next PPA is no longer up to date.

I downloaded thunderbird-69.0b1.tar.bz2 and started it...

The 'New' button in the Message Filters dialog seems quick in my first few tries ... and opening the 'Copy To' context also opens directly ... only when opening the 'Recent' sub menu I see (?a shorter then before?) slowdown.
Good to see the result of #809066 being solved, thanks :aceman

Then it still takes almost 6 seconds to open the 'Recent' sub menu. All of which was spent in the addIfRecent function. Posted as https://perfht.ml/33bwcX6
Opening the same 'Recent' sub menu again a bit later is quick.
I did notice that when I open Recent via the 'Move To' menu it again takes these 6 seconds to load ... apparently that data is now shared??

Unfortunately as most add-ons are disabled because they're 'not compatible', testing the impact of/on the 'Quick Folder Move' add-on is not available for now.
Is there an easy way to override that without unzipping and adjusting the meta data?

PS: I had a bit of a hard time initially to find the profiller after installing the xpi.
I had to open View -> Toolbars -> Customize to add it to a toolbar. There it's double the height of the other icons, thus stretching the toolbar in height.

I hope this version makes it to (Ubuntu)stable soon :)

TB 69 will be released as beta only. Next stable release: TB 68 ESR planned for later this month. It should behave the same.

Thanks for the profiler output.
If all the time is used in addIfRecent(), I tried to change the algorithm there.
You can download and test build (of TB71) from there: https://queue.taskcluster.net/v1/task/J3FLB0WFSZeKFQD5yAOqBQ/runs/0/artifacts/public/build/target.tar.bz2
There is even output in the Error console of how long it took to build the Recent list.

But beware of the downgrading problem after you are done:
https://support.mozilla.org/en-US/kb/unable-launch-older-version-profile

You may want to test it on a copy of your profile.

Thanks ... I just tried that build.

Move to -> Recent: "Built recent list in:23895ms"

After this memory usage had fallen again in just a few minutes to ~ 1.5GB, peak was at 3.5gb+ when recent list was just finished.

Copy to -> Recent : "Built recent list in:20610ms"

Tools -> Message Filters -> New -> Choose folder -> Recent

Built recent list in:14604ms
When I close the message filters window, and directly open it again

Built recent list in:12ms

Apparently this is not cached in the same way, and just quick because most data is still in memory.

30 minutes later, opening message filters again... "Built recent list in:6596ms" (Memory usage was at ~3gb, so most was in memory)

Even after memory usage has fallen again the recent folder list for moveto/copyto opens quickly. So :)
It would still be nice if those caches could be combined into one though.

Thanks, so you say this experiment wasn't any improvement, it was actually worse, taking 23 seconds (compared to 6s in your comment 26) and taking 2GB of RAM temporarily. That is quite excessive if you only have 400 folders.

I'm afraid this workstation is far from a clean lab setting, but agree that it does not look good.
Do you have a link for me to the code changes?

The change of algorithm is here:
https://hg.mozilla.org/try-comm-central/rev/8fb9eedbfc193409f6ba9afe087ad13ca1f3a132

Instead of maintaining (often sorting the array and popping items) of the 15 most recent folders inside the function, I tried to just list all folders, sort by the recency and return the top 15.

So the path forward is bug 1519375 and we should close this one? Or make this dependent?

They are different approaches to the problem.
I would prefer to just fix the algorithm if it can be made fast, instead of maintaining some new folder cache.

Strictly looking at the title of this bug I'd say to continue with just bug 1519375.

The filter rules dialog was just a manifestation of the slow access to the recent folder list. And now that the recent list is moved to a lazy loaded sub menu it no longer affects this dialog.

If :aceman still sees an opportunity to optimize then we can either rename this or create a new bug with a title like 'Optimize code to generate recent folder list'.
But unless you can remove the need to load all folder data files from disk into memory I see only limited benefit in that. Having this list instantly available is very important when you use it for moving messages. I often use the quick-folder-move add-on which shows an autocomplete list of folders to move a message to(it prioritized recently used folders) (Development version on github

Herman, do you still see this performance issue?

(I use filters a lot and never see this)