I had seen this during the past few weeks, and it drives me crazy! :-) Today I spent a few minutes investigating, and here is the result. I opened a random bug (https://bugzilla.mozilla.org/show_bug.cgi?id=684592), it's very small with only 7 comments and 1 patch. While loading this page, the browser spends 17.2 seconds just waiting for any response from the network. And this is almost true for any but that I load. I don't have measurements on what the numbers used to look like before, but intuitively they seemed to be in the range of 1-2 seconds.

hrm, i've never seen load times that slow.. that bug loads in about 5 seconds on average for me, of which 1-2 seconds is waiting for network. what sort of speeds do you get if you are logged out of bugzilla?

It's usually a lot faster for me if I'm logged out.

(In reply to Ehsan Akhgari [:ehsan] from comment #2) > It's usually a lot faster for me if I'm logged out. are you able to quantify "a lot faster"? there's a lot of things that don't happen when you aren't logged in, so it should be faster... i'm wondering if you see a drop from 17 secs down to 5ish.

(In reply to Byron Jones ‹:glob› from comment #3) > (In reply to Ehsan Akhgari [:ehsan] from comment #2) > > It's usually a lot faster for me if I'm logged out. > > are you able to quantify "a lot faster"? there's a lot of things that don't > happen when you aren't logged in, so it should be faster... i'm wondering if > you see a drop from 17 secs down to 5ish. Yes, I see a lot of the bugs to load in the range of 5 second when I'm logged out. That's what I meant by "a lot faster". But I should also add that even that is a lot slower than what I used to see a while ago (maybe a month ago or so?)

ehsan, do you have many saved searches displaying in the bug footer (at the bottom of your bug pages)?

(In reply to Wayne Mery (:wsmwk) from comment #5) > ehsan, do you have many saved searches displaying in the bug footer (at the > bottom of your bug pages)? I have 31 of them! Are those the cause of the slowdown?

(In reply to Ehsan Akhgari [:ehsan] from comment #6) > (In reply to Wayne Mery (:wsmwk) from comment #5) > > ehsan, do you have many saved searches displaying in the bug footer (at the > > bottom of your bug pages)? > > I have 31 of them! Are those the cause of the slowdown? Out of curiosity, I just went to saved search preferences and unclicked the Show in Footer checkbox for all of them, and loading bus now seems to be faster! Of course, I may also be imagining things... ;-)

if bug 457116 comment 4 is still valid, then you are not imagining things. It seems I never filed a bug report on that issue. I'm not able to verify ATM, because my connection seems be in the toilet.

> Out of curiosity, I just went to saved search preferences and unclicked the > Show in Footer checkbox for all of them, and loading bus now seems to be > faster! Of course, I may also be imagining things... ;-) From a db/memory perspective it should not make a huge difference as we load in all queries into memory for each page load (probably shouldnt do that :) and loop over them to see which to display in the footer. So the rendering of the ones that do display in the footer can take time if you have *a lot* set that way. Ideally we should set it to not even load queries into memory for the footer if they are not set to dispay in the footer. I can look into that. dkl

hmm, I had not noticed search names were still in the html when set to not display. I don't recall if 3 years ago it was the same (I would think so, but but I don't remember checking the html). I do remember that it made a good improvement. And note, when you create a new saved search, it defaults to display.

IMO, saved searches have nothing to do here. I have 41 of them, and never had such problems before. IMO, the culprit could be the inline history extension. Disabling it for a few hours would give us a good indication.

(In reply to Frédéric Buclin from comment #11) > IMO, saved searches have nothing to do here. I have 41 of them, and never > had such problems before. IMO, the culprit could be the inline history > extension. Disabling it for a few hours would give us a good indication. It is a user preference which currently defaults to on. Could you try turning it off and then reloading your page to see if it makes a difference? dkl

(In reply to David Lawrence [:dkl] from comment #12) > It is a user preference which currently defaults to on. Could you try > turning it off and then reloading your page to see if it makes a difference? I tried with bug 38862 and its 235 comments, 15 flags, 32 attachments and 55 CC'ed users, and it takes 10-12 seconds while being logged out, ~13 seconds when logged in with inline history turned off, and 14-15 seconds when logged in with inline history turned on. Note that these results are pretty noisy and so differences may be even smaller than that. It appears very clearly that something is wrong even when being logged out. 10 seconds is far too much for a read-only bug report, especialla when you know all the hardware which is behind bmo.

(and I tried with both Fx 7.0.1 and Opera 11.51, in case the browser was part of the problem)

Yeah, I have not been able to see a meaningful association with either saved searches or inline history here as well. Can we add some logging code on the server side which measures how long it takes for various operations happening as part of a bug page load take? That should give us very good information on how to proceed.

Ping?

Sorry for the lack of action. We are working on different approaches to the slow loading issue. One change recently checked in that will show up with the next BMO update should help loading of show_bug.cgi a good deal for power users is bug 713144. dkl

bug 713144 has been in fixed since 01/04 which should help quite a bit. Also bug 715477 is going in this week which should also help. We will see how it goes. Some of the slowness lately has been network related and not much can be done on the application side for that. dkl

We made some major progress in this area. On my test machine, I divided the time by a factor of 2 (from 7.2 to 3.6 seconds, and I have another patch I'm working on which can save another 0.3 second in some cases). Let's hope a similar perf win will be noticeable on bmo.

(In reply to Frédéric Buclin from comment #20) > We made some major progress in this area. On my test machine, I divided the > time by a factor of 2 (from 7.2 to 3.6 seconds, and I have another patch I'm > working on which can save another 0.3 second in some cases). Let's hope a > similar perf win will be noticeable on bmo. Yay! Questions / observations: 1. BMO performance has gotten worse presumably since some point in time X (one year? two?), yet I don't see any of the recent performance bugs marked regression [1]. So a) are there bugs (software or hardware) that have not yet been found or documented? b) what performance reference tests are done for product bugzilla and BMO as patches check in or versions are rolled out so that performance regressions are not introduced? 2. there are several perf bugs that do not have the perf keyword [2] [1] perf/slow bugs created last two years: https://bugzilla.mozilla.org/buglist.cgi?type1-0-0=substring;list_id=2493068;field0-0-0=short_desc;type0-0-1=substring;field0-0-1=keywords;type1-0-1=allwordssubstr;resolution=---;resolution=FIXED;classification=Server%20Software;classification=Other;chfieldto=Now;chfield=[Bug%20creation];query_format=advanced;chfieldfrom=2y;value0-0-1=perf;type0-0-0=anywordssubstr;value0-0-0=slow;field1-0-0=short_desc;product=Bugzilla;product=bugzilla.mozilla.org;field1-0-1=short_desc [2] no perf kw https://bugzilla.mozilla.org/buglist.cgi?type1-0-0=notsubstring;list_id=2493111;field0-0-0=short_desc;type0-0-1=substring;field0-0-1=keywords;type1-0-1=allwordssubstr;resolution=---;resolution=FIXED;classification=Server%20Software;classification=Other;chfieldto=Now;chfield=[Bug%20creation];query_format=advanced;chfieldfrom=2y;value1-0-0=perf;type0-0-0=anywordssubstr;value0-0-0=slow;field1-0-0=keywords;product=Bugzilla;product=bugzilla.mozilla.org;field1-0-1=short_desc

(In reply to Wayne Mery (:wsmwk) from comment #21) > > Questions / observations: > 1. BMO performance has gotten worse presumably since some point in time X > (one year? two?), yet I don't see any of the recent performance bugs marked > regression [1]. So > a) are there bugs (software or hardware) that have not yet been found or > documented? That's a somewhat incomplete thought. It comes out better as these thoughts: a1) if none of the bugs cited thus far are regressions, then do we have more bugs yet to be found? a2) if none of the bugs are regressions, then are we doing some things significantly differently in BMO in recent times that has surfaced performance issues? a3) are some of the bugs regressions but not marked as such?

(In reply to Wayne Mery (:wsmwk) from comment #21) > a) are there bugs (software or hardware) that have not yet been found or > documented? If we knew which bugs haven't been discovered yet, we would have found them already. :-D > b) what performance reference tests are done for product bugzilla and BMO There are a few things which must be taken into consideration: - bmo is not the single Bugzilla installation in the world (fortunately!) so I'm not doing my testing on a copy of the bmo code + DB except in some very specific cases. I have a dev installation I'm playing with most of the time and the GCC Bugzilla code + DB (as I'm the maintainer of GCC Bugzilla). This gives different useful inputs. - bmo is full of local customizations and extensions, and some of them can have very bad (unknown) performances and are hard to debug unless you are specifically playing with the bmo DB. - the copy of the bmo DB I have in hands has been sanitized, meaning that attachments, groups, user prefs and settings have been deleted. So this heavily affects data as I'm pretty sure group settings play an important role. - the bmo configuration at Mozilla (mod_perl, DB slaves, load balancing, etc...) is very different from what we have when playing at home while doing debugging. This highly affects perf data. So a win of 3 seconds on my machine could result in a win of 0.3 second on Mozilla servers, depending on how things interact together. It's not rocket science. Ideally, to know if the problem comes from one of the bmo extensions would be to disable them all, and compare perf data. But this won't happen.

(In reply to Frédéric Buclin from comment #23) > So a win of 3 seconds on my machine could result in a win of 0.3 second on > Mozilla servers, depending on how things interact together. Hum, here is a good example of what I was saying here: My patch from bug 731055 (which is being backported to bmo in bug 731165) resulted in a perf win of 3 seconds (!) on my machine using my dev installation, but unfortunately only a win of 0.3 second with my snapshot of the bmo DB. Much less exciting! :( *BUT* another patch I'm currently playing with (no bug filed yet) has the exact opposite effect: it's responsible for a 0.3 second win on my dev installation (which is not bad but not really exciting), and is responsible for a 2.1 seconds win with my snapshot of the bmo DB! And I have no idea why. :)

(In reply to Frédéric Buclin from comment #23) > (In reply to Wayne Mery (:wsmwk) from comment #21) > > a) are there bugs (software or hardware) that have not yet been found or > > documented? > > If we knew which bugs haven't been discovered yet, we would have found them > already. :-D > > > > b) what performance reference tests are done for product bugzilla and BMO > > There are a few things which must be taken into consideration: > - bmo is not the single Bugzilla installation in the world (fortunately!) so > I'm not doing my testing on a copy of the bmo code + DB except in some very > specific cases. I have a dev installation I'm playing with most of the time > and the GCC Bugzilla code + DB (as I'm the maintainer of GCC Bugzilla). This > gives different useful inputs. indeed - which is why i asked if the bugzilla project (not just bmo) has suitably sized reference test dataset. > - bmo is full of local customizations and extensions, and some of them can > have very bad (unknown) performances and are hard to debug unless you are > specifically playing with the bmo DB. > > - the copy of the bmo DB I have in hands has been sanitized, meaning that > attachments, groups, user prefs and settings have been deleted. So this > heavily affects data as I'm pretty sure group settings play an important > role. :( but, BMO wouldn't be ruled by the same sanitation rules > - the bmo configuration at Mozilla (mod_perl, DB slaves, load balancing, > etc...) is very different from what we have when playing at home while doing > debugging. This highly affects perf data. naturlich anyway, my main point involves the thought ... how we find more perf regressions in some automated/controlled way, whether it be at bmo or bugzilla, and mark the bugs as such in bugzilla.

administrative changes also have an impact on the performance. rapid release tracking flags are implemented on bmo as normal bugzilla custom fields (with changes to how they are presented on the ui). bugzilla's custom field implementation involves adding each field as a column to the bugs table, which doesn't scale well beyond a handful of fields. we have over 60 custom fields, growing every 6 weeks, so the bugs table is extremely wide. there's a project underway to refactor this, however it's a complicated task to ensure compatibility with how the data is currently presented to our apis.

(In reply to Byron Jones ‹:glob› from comment #26) > bugzilla's custom field implementation involves adding each field as a > column to the bugs table, which doesn't scale well beyond a handful of > fields. we have over 60 custom fields, growing every 6 weeks, so the bugs > table is extremely wide. Yeah, that's bad. IMO, you should store everything in a separate table, e.g custom_flags, and let it behave the same way as normal flags. You wouldn't need any additional column to the bugs table. I did some profiling using the bmo DB I have in hands, and 4 seconds (35% of the load time) is wasted in the InlineHistory extension. This is very bad. :( I will give it a look.

(In reply to Frédéric Buclin from comment #27) > Yeah, that's bad. IMO, you should store everything in a separate table, e.g > custom_flags, and let it behave the same way as normal flags. You wouldn't > need any additional column to the bugs table. yup, that's the plan :)

Does show_bug.cgi load faster for you now?

Closing due to inactivity. Please reopen if loading bugs are still slower than usual. dkl