In response to: http://globau.wordpress.com/2012/10/09/bugzilla-mozilla-org-integration-best-practices/ TBPL uses bzapi in several places: 1) To fetch bug suggestions used in annotated summary generation. Where results are found these are stored in a psuedo-bugscache-come-search-results-cache, but purged after 3 hours. Where results aren't found, we just repeatedly make the same search each time we come across that failure in a log. Looking at 8th Oct, we made 4500 of these bzapi calls (which given that there are only 4000 oranges filed, of which only 1200 open, is a bit excessive). https://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/file/9c7f496beb97/php/inc/AnnotatedSummaryGenerator.php#l100 -> https://api-dev.bugzilla.mozilla.org/latest/bug?whiteboard=orange&summary= 2) To fetch bug comments prior to starring, in order to avoid duplicate bug comments (although the feature has been broken for ~1 year, but we're still fetching the comments! bug 748848): https://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/file/9c7f496beb97/php/submitBugzillaComment.php#l32 -> https://api-dev.bugzilla.mozilla.org/latest/bug/$bugid/comment 3) To submit the bug comments when starring: https://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/file/9c7f496beb97/php/submitBugzillaComment.php#l24 -> POST to https://api-dev.bugzilla.mozilla.org/latest/bug/$bugid/comment?username=$username&password=$password 4) In the client-side Data.js' .getBug(), which in a roundabout way is used by the UserInterface.js' _linkBugs() - which generates pretty (but pretty useless) onmouseover with bug state and summary for things like the pushlog, starred oranges in the annotated summary box, tree status message (except the latter seems broken). On the plus side these are cached locally. https://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/file/9c7f496beb97/js/Data.js#l28 https://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/file/9c7f496beb97/js/UserInterface.js#l752 -> https://api-dev.bugzilla.mozilla.org/latest/bug?id= Improvements we can make: a) In #1, #2 and #4 we don't specify the columns we need, which means we get the default of ~150 (!!) fields, thereby increasing search time, b.m.o load & time to transfer the results. Even in the case of #1, we only really need {bug number, summary, status, resolution, last changed}, and in the case of say #4 that can be reduced to {summary,status,resolution}. b) For #1 we should be using a keyword, not the whiteboard - bug 790571 and dependants. c) For #1 we currently break pretty much every guideline on the blog post. We should be using a proper bugscache of all filed oranges, rather than a list of search values and their results, that we can then just search directly rather than making many separate bzapi calls. We should then keep that bugscache up to date using 'last modified time' type searches, per the blog post. This would both avoid the need to purge our cache every 3 hours & mean we don't have results up to 3 hours out of date :-) d) For #2, the regex for this feature should be fixed (bug 748848) or else the block just commented out for now, seeing as it's broken anyway. Ideally we'd do duplicate detection in TBPL itself to avoid the expensive call for the many orange bugs that have hundreds of comments. e) For #4, we could just get rid of most of the onmouseovers (since they don't add much), or at least add a delay, so waving the cursor over the pushlog doesn't generate many bzapi calls for which the result wasn't even used. Not sure if #3 can be improved upon at all. I'll file bugs for those above that don't have them. Glob, if there are any other improvements you can think of that will help with load, I'm all ears :-)

Few observations: * Unsurprisingly, due to the 3 hourly bugscache purge, we search for most of the terms multiple times a day. * More delightfully, once the cache is purged, we then having multiple workers race to populate that entry, so see clusters of duplicate searches within a few seconds of each other. * Even more delightfully, we attempt to search for summary="1" & summary="Shutdown", which typically take 31 minutes and 90s respectively (next longest few are ~20-35s, then the rest are 3-15s each). Total cumulative time BzAPI spent waiting for b.m.o on 8th October for TBPL orange searches: ~8.5 hours ...and these times are from the b.m.o apache log so don't include the TBPL<->BzAPI step either! Makes me even more convinced we need a proper orange bug cache in the TBPL DB that is kept up to date using date-modified searches.

(In reply to Ed Morley [:edmorley UTC+1] from comment #1) > ...and these times are from the b.m.o apache log so don't include the > TBPL<->BzAPI step either! TBPL kills the annotated summary generation after 60 seconds, so I guess BzAPI must be the one merrily fetching results for the times that are longer than it. Also, trying this locally: https://api-dev.bugzilla.mozilla.org/latest/bug?whiteboard=orange&summary=1 ...is still pretty quick. Maybe I have my units wrong?

I've filed bug 834719 for switching to a proper bugscache (likely to be done as part of TBPL2). As far as this tracking bug goes, all of the work items have been identified and filed as dependants bugs (and all bar two fixed), so going to call this investigation bug fixed.