User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0

Steps to reproduce:

This bug is a follow up bug from Bug 1502923 Comment 106.

Starting from 70.0a1 (2019-08-08) (32-bit) a ~4000 items CalDAv network calendar takes about 3mn to load... which is better than before but still way to much from end-users point of view :-)

There are gaps appearing between network requests (data gathering) which indicate delay due to Thunderbird processing between the requests, the aim of this bug shall be to remove progressively all such gaps that cause unnecessary loading delay in the measure of possible...

In a similar way, I have created a separate dedicated Bug 1581294 to assess performance for Google network calendar loading ~4000 items via lightning/calendar + Provider for Google Calendar (gdata-provider) for those that may be interested to follow up on those perf issues...

Please add "perf" key word.

FYI, few measure to evalutate performance progress...

One network CalDav calendar with about ~4000 items
ADSL - Unifinder All Events - No sorting column selected - No email - Dedicated new user profile - No Offline Support - No Reminders

Note about test 03 in previous comment, unifinder view remain empty almost till the end of calendar loading (till about network request 43)

I've asked before, but it seems to have been lost. Please save a performance recording using the developer tools and post it here for others to analyse.

Find attached network performance gathered via Task Manager on Windows for TB 68.1.1 during caldav calendar sync... huge gap between network requests...

More profiles will follow for more recent version of TB...

Does it really need 46 network requests to load such calendar?
See attached list of requests - Part 1

List of devtools network requests - Part 2

See also Bug 1543953...

Here is one full devtools > performance profile for loading one CaDAV calendar with ~4000 items in TB 70.0b2 (64-bit) - 1mn+ processing...

Few more test results... performance seems getting worse now again...

Note about Test 04

• Because unifinder remain currently empty/blank during initial loading, sorting or not by header column does not make difference with one calendar.

Note about Test 05

• Clear regression in term of performance as compared to gain obtained with 70.0b4 version.
• Though the unifinder is no longer blank/empty this time.

Find attached a TB 72.0a1 (2019-10-22) (64-bit) > DevTools > Performance profile recorded between 30s to 1mn during data sync (network calendar caldav with about ~4000items).

Hoping that would help dev team to find out root cause of poorer performance.

PS:
The buffer seems to be quickly filled in that version of TB so I cannot gather more info in the performance profile.

Daily versions use ical.js whereas beta versions use libical. It's not a straight comparison because there's known performance issues with ical.js which probably explain the difference. You can set the pref calendar.icaljs and restart to switch.

(In reply to Geoff Lankow (:darktrojan) from comment #18)

Daily versions use ical.js whereas beta versions use libical. It's not a straight comparison because there's known performance issues with ical.js which probably explain the difference. You can set the pref calendar.icaljs and restart to switch.

Shouldn't ical.js perform better than libical which it intends to replace?

At the meantime I have followed your advise and set calendar.icaljs to false in config editor options... here is the new result

Note about test 06

• Still, not much progress on performance improvement since 70.x branch...
• 1mn+ is still too long from an end-user point of view to load one calendar.

May be time to look into Bug 1543953, because reducing the number of network requests may drastically help reduce the time it take to load network calendar... each network request shall be able to load more than 100 items at a time especially on fast networks e.g 1GB connection. The best may be for TB to detect speed of network connection and adjust number of items it loads per network request consequently or let end-user choose the network request batch size...

Shouldn't ical.js perform better than libical which it intends to replace?

It should, but it doesn't, which is why the replacement hasn't happened.

Still, not much progress on performance improvement since 70.x branch...

None at all, since no perf bugs were finished during the 71 cycle. There are one or two things in the works, calendar performance hasn't been forgotten, but it's a complex problem that takes significant uninterrupted time to work on, and we all have a lot to do.

Actually that's not true, there were some bugs finished in that time, my search just didn't find them. I don't expect any of them had much bearing on this problem though.

(In reply to Geoff Lankow (:darktrojan) from comment #20)

Still, not much progress on performance improvement since 70.x branch...
None at all, since no perf bugs were finished during the 71 cycle. There are one or two things in the works, calendar performance hasn't been forgotten, but it's a complex problem that takes significant uninterrupted time to work on, and we all have a lot to do.

It is clear that calendar dev team is fixing bugs, for sure, but there must be a way to bring small performance improvements at each iteration somehow (at least the major ones) within the framework of work on the calendar feature... and other tasks you may have to do... Not all will be fixed at once (nor today), but step by step, iteration after iteration... it must be possible... breaking the performance issue into smaller parts and fixing each of them...

It is clear that performance issue are very hard to fix and time consuming (and boring to sort out) but they are a priority for end-users and therefore shall be for developers. I am myself trying to use my free time to help here... Performance must be integral part of the development cycle... not just something that need fixing afterwards when time is available... there should not be performance issue in the first place... those are long lasting overlooked issues that need fixing... better now than tomorrow :-)

I don't deny that you may not have the resources to do so directly... but there must be a way... other than waiting for summer 2020!
You are certainly better placed to know how to prioritise on the matter, and I really thank you for your efforts so far and global work on the calendar feature (much loved), it just need to continue somehow, please don't give up...
Is there a way you can get help somehow (temporarily)? Can we provide help somehow?

So far all my tests and reports are to raise awareness and help you fix the performance issues but if there is any way I can improve those to help you more, please tell.

There have been so much great progress so far... lets continue the efforts...

Looking forward to your one or two things in the works :-)

FYI and the record...

Regressions observed with Calendar CalDAV performance since 77.x branch and worthen in 78.x branch...

• 10mn+ to load one CalDAV calendar with 4000+ items - previously was 1mn+
• perf/not responding at startup as per bug 1642292

See Dev Tools > Performance report attached for 1st 40sec of calendar loading...

Also huge gaps still observed between network requests corresponding to TB processing time of the previous request... as observed before, the gap tends to increase over time...

As per bug 1642292 Comment 15, regressions could be related to the changes in bug 1546606.

FYI, additional perf statistics for the record...
There seems to be quite some improvements in TB 81.0b2 (64-bit) as delay seems cut in half now down to 7mn compare to 78.x :-)
Possibly due to other fix/optimisation in core and elsewhere...

Loading one network calendar calDAV with 4000+ items...

FYI, 20 seconds achieved at best in TB 81.0b4 (64-bit) to load one caldav network calendar with 4000+ items!
See below for reference and next comment...

(In reply to Richard Leger from Bug 1543953 comment 19)

Created Bug 1543953 attachment 9176520 [details]
tb.81.0b4.devtools.networking.caldav.one.calendar.4000items.png

(In reply to Richard Leger from comment #7)

I think it would be more efficient to do the following at TB startup:

2. Get the full calendar as .ics file via a GET request instead and parse it
   This take only 2seconds with my 4000 items calendar

Am I dreaming or I have been heard? What a surprise today! Yes, Yes, Yes... finally!

Just upgraded to TB 81.0b4 (64-bit) and testing loading my ~4000 items caldav network calendar... while measuring and monitoring I could see only one network request and I first thought... damn calendar loading is broken again?!

...but checking via DevTools > Networking...
...what a surprise!!!

The 50 multi-get request that have been hindering calendar loading performance for so long have now been replaced by one single GET request that load the entire .ics calendar at once!!! Oh my god! What a speed difference max 22 seconds top... (result obtained with new profile, no email, one caldav network calendar newly setup with ~4000 items)

• Was it intended or an unintended side effect of something else implemented, corrected or removed?
• Is that gonna be the new way?
• Would that perf improvement "stick" from now on?
  Can someone confirm?

No info in the release note:
https://www.thunderbird.net/en-US/thunderbird/81.0beta/releasenotes/?uri=/thunderbird/releasenotes/&locale=en-US&version=81.0&channel=beta&os=WINNT&buildid=20200914232954

Nor could I find any changelog in the source code that may explain such change:
https://hg.mozilla.org/releases/comm-beta/log/a8ea452b9f4cc059a4afebaf5d54af4e72e4b5c4
If someone could point out perhaps the changeset related to this change?

If that is intended that is great, great, great news to all TB calendar users out there!
Thank you...

I keep dreaming for now... hoping someone would confirm this reality!

I have yet to test difference on startup performance with my current default profile!

FYI, in addition of previous comment...

(In reply to Richard Leger from Bug 1543953 comment #20)

(In reply to Richard Leger from Bug 1543953 comment #19)

I have yet to test difference on startup performance with my current default profile!

I just re-tested and bad news, my existing already set calendar in my default profile (that I also use for emails) does not behave in the same manner. When I enabled it and run the sync, instead of one GET request as described in my previous comment, it still generate multi-GET requests to sync and therefore still affected by Bug 1642292.

This suggest that to get the new behaviour and related performance the calendar needs either to be deleted/re-created from scratch or that settings in my current already set calendars may need some adjustments to benefit from new behaviour... I would expect TB to take care of that upon upgrade :-)

Any clue?

(In reply to Richard Leger from comment #27)

2. Get the full calendar as .ics file via a GET request instead and parse it
   This take only 2seconds with my 4000 items calendar
   Just upgraded to TB 81.0b4 (64-bit) and testing loading my ~4000 items caldav network calendar... while measuring and monitoring I could see only one network request and I first thought... damn calendar loading is broken again?!
   The 50 multi-get request that have been hindering calendar loading performance for so long have now been replaced by one single GET request that load the entire .ics calendar at once!!! Oh my god! What a speed difference max 22 seconds top... (result obtained with new profile, no email, one caldav network calendar newly setup with ~4000 items)

False hope :-(
Explanation Bug 1543953 Comment 21
Back to square one...

FYI, some recent performance test results...

(In reply to Richard Leger from Bug 1642292 comment #100)

(In reply to Arthur K. [He/Him] from Bug 1642292 comment #95)

How is this behaving with 91.0b6?
CalDAV performance has improved, work still in progress...

#	PerfProfile	TB Version	LoadingTime(Disappearance)	DevTools > Networking	Add-ons	UnifinderView
11	https://share.firefox.dev/2XTgt0G	91.0.2 (64-bit)	~1m36	58 requests - 10.89MB Transferred - Finish: 2.48mn	TB Core Caldav (NoReminder/NoOfflineSupport)	Next 7 Days, NoHeaderColumnSorting
12	https://share.firefox.dev/3klxrw9	91.0.2 (64-bit)	~3m30	58 requests - 10.89MB Transferred - Finish: 3.19mn	TB Core Caldav (NoReminder/NoOfflineSupport)	All Events, SortByTitle

shall we duplicate this to bug 1543953? (reference bug 1543953 comment 54)

Hi Wayne,

(In reply to Wayne Mery (:wsmwk) from comment #31)

shall we duplicate this to bug 1543953? (reference bug 1543953 comment 54)

No, because those are two different issues...

As far as I can test in TB 106.0b5 (64-bit) on Windows 10, I no longer see any gaps between network requests so this bug has been resolved somehow...

The other bug 1543953 remain separate and open, I need to gather more data and update the report... soon...

Cheers,