Before the recent bmo upgrade, bmo's quicksearch skipped full-text search for searches where I don't specify a field. This was good, because * Full-text searches are extremely slow, often to the point of making simple queries time out. (mkanat says adding RAM to the MySQL servers will help.) * Full-text hits are mostly unwanted, false positives, because of the size of our bug database. Please restore this customization so I don't have to prefix all my searches with "summary:" (and lose all the benefits of quicksearch). This is really slowing me down.

What customization? I'm not aware of any bmo-specific customization related to skipping full-text search. It may have just been a change in Bugzilla 3.6 itself.

Previously, searching comments could be disabled by a parameter because it always had terrible performance. At the moment, it has terrible performance on the bmo servers because they don't have enough RAM, and are actually swapping. (Normally the whole fulltext index is stored in RAM, but if it's stored in swap, that's considerably slower.)

Even with the slowness fixed I'll still want this fixed. Dealing with false positives, or typing out "summary:", slows me down a lot on almost every search.

It also causes many searches, such as https://bugzilla.mozilla.org/buglist.cgi?quicksearch=ALL+comp%3Aparser+crash, to return only 200 bugs. So it's useless for both "does this bug exist" searches and "find all such bugs" searches.

mkanat, considering Jesse is one of the top users of quicksearch, I think we should look into seeing what we can do here... Quicksearch is very powerful, and we don't want users to get so frustrated over using it and not finding what they want.

(In reply to comment #4) > It also causes many searches, such as > https://bugzilla.mozilla.org/buglist.cgi?quicksearch=ALL+comp%3Aparser+crash, > to return only 200 bugs. Yes, that's a bug. Would you file that separately?

reed: There are millions of Bugzilla users at various different organizations. So even though Jesse is an important user, I wouldn't change the defaults of the feature based around his needs. However, I think we could possibly add a user preference for "when I do a quicksearch, also search comments" that would solve his (and other people's) requests in this area.

Most likely we'd fix this for 4.0 and then Mozilla could backport it for 4.0.

Filed bug 581622 for the 200-bug limit.

Here is a patch that adds a new user preference called quicksearch_fulltext that defaults to off. If the user wants their quick searches to included the fulltext content then they can turn it on. Dave

Comment on attachment 510401 [details] [diff] [review] Patch to add quicksearch full text user preference (v1) Ah, I haven't heard enough input on this bug to be sure that the *default* value should be off. Are you solving a performance problem or a problem with the expected search results? If it's #1, this isn't the way to do it--there are other ways that we can discuss in other bugs. If it's #2, we'd need way more input to be sure that *logged-out users* should not be searching comments when they quicksearch. I actually usually find it quite useful that the comments are being searched (although only with some searches).

> Are you solving a performance problem or a problem with the expected search results? It really is both a performance problem and a result-quality problem. * Basic searches on bugzilla.mozilla.org take at least 12 seconds. If it's not the middle of the night, or if the search returns more than a few bugs, it takes even longer. This is far too long to keep a train of thought going. * Most "ALL" searches on bugzilla.mozilla.org time out due to this bug. This forces me to split my searches into one for open bugs and another for fixed bugs, and then I have to hope the bug I'm looking for isn't actually WFM. * It causes unwanted bugs to be returned more often than not. * When comments are included by default, I have to type "summary,whiteboard,component,product:" to get the behavior I usually want. When comments are not included by default, I only have to type "desc:" to include them.

Jesse: Thanks for your feedback. I would need to hear from a *very* broad cross-section of users that the *result quality* was a problem before making "off" the default. (I have other plans to handle the performance--as you may know, shaver has a very fast fulltext search for Bugzilla, so it is indeed possible to make that fast.) However, I would only have to hear from a relatively small cross-section of users that result quality was a problem in order to accept the implementation of the preference. So if a few more people here (across various organizations, not just Mozilla) give convincing data that shows that they get better result quality on their average searches without searching comments, then we could implement a preference. Otherwise, it seems like the best solution is to improve the relevance ordering and speed of quicksearch so that the best results show up at the top and less-good results show up lower.

Jesse's recent tweet notwithstanding, by what mechanism would you like to hear that feedback, Max? By all means, do whatever you think is in bugzilla's best interests as a matter of general principle, but not being able to modify this behaviour for myself (and, I suspect, for the mozilla bugzilla instance in general) is sadmaking. I don't want to put advocacy in bugs, but it's not clear to me how you're going to get this broad cross-section. Shaver's app is quite useful, to be sure - the principle reason I don't use it right now is because I am such a heavy quicksearch user, and he hasn't yet replicated the flexibility of the quicksearch. I'd caution you against drawing too many conclusions about the (incredible!) speed of his app though, since it relies on non-bugzilla data stores, iirc. Finally, I'll point out that for the use cases I (and most other drivers in bugzilla) employ quicksearch, relevance ordering is not really an aid. I'm more likely to sort explicitly by last activity or assignee or component in any event, so having false positives in the list is harmful regardless of their position. I apologize - this whole thing sounds more like advocacy than I want a bug comment to - where else should I bring these points?

I have been sad ever since quicksearch started searching comments by default. It was so easy to include comments when wanted before, and so hard to exclude them now. I suppose the '[' operator was a stab at limiting the damage, but it's broken (bug filed) and doesn't search as many fields. Please fix this. I don't care what default generic bugzilla or BMO choose as long as individual users can change their own behavior. If there's a patch can we get it in (to BMO, at least) and argue about the default value later? (In reply to comment #13) > Otherwise, it seems like the best solution is to improve the relevance ordering > and speed of quicksearch so that the best results show up at the top and > less-good results show up lower. trying to bury results in the list won't help me. I almost always specify the one I want for a particular purpose (ID order in mtgs, change date, prod/component, assignee, etc), and when I don't specify the order I'm usually looking for a count of bugs matching a criteria. The problem is scooping up unwanted bugs into the list in the first place.

Current state of our databases: 32 GB of RAM in each. The InnoDB data set is currently 24 GB. It needs 10 GB in buffers to handle user connections. Toss in MyISAM sort buffers, etc, and that puts the current DB memory needs at 38 GB. Which yes, is more than what's available. So yeah, if disabling the full-text search saves us memory use, then by all means.

er, disabling comment search I mean (by default)

I agree with the concerns here (I have stopped using quicksearch for all but flag-based searches), but don't really want to comment more until I understand the context for the original change. Any argument in support of reverting the change should be at least as strong as the argument for making it in the first place, and it would be useful to re-use the channel that gathered the initial wide user-base feedback. Max: can you link to archives or similar? Thanks, I want to be constructive here!

(In reply to comment #18) > Max: can you link to archives or similar? Thanks, I want to be constructive > here! Hey Shaver! Thanks for the feedback, and your points are well-taken. Here's what happened: * In previous versions of Bugzilla, Bugzilla always searched comments by default, however, it did it in a very slow fashion. Since it was slow, there was a parameter that allowed you to say "if somebody quicksearches for more than X words, don't search comments," that defaulted to 4. This was actually the behavior of the original quicksearch patch in bug 69793, so it has always been around. * It turned out that on an installation of any size, doing a LIKE search against the comments (which was what was being done) was so slow that searches would never complete unless the parameter was set to 0. * All large installations (with bmo being the first to go) eventually set this parameter to 0. * I discovered that if you changed the system to use the fulltext engine, that large installations *could* run searches against comments, with any number of terms. (This was a change from "could not run them at all" to "could run them".) I tested things out on our test databases on landfill and performance seemed pretty acceptable. * Since we no longer needed the parameter for the original purpose that it was intended for (to prevent killing the DB) we removed it and always made quicksearch search comments. * When bmo was originally upgraded to a version that searched comments by default, the database servers had significant issues with not having enough RAM. So I couldn't be sure that there was actually a performance problem with the new system. (I don't have any system with the size and hardware of bmo that I can test against myself, during development, either.) * Now that the bmo DB servers have been upgraded, I've seen that quicksearch is still slow. The ultimate solution to this performance problem will be to allow the use of external fulltext engines, which actually should be relatively easy once I have the time to do it. However, a fair number of people here have now commented that searching comments by default impairs their *result quality*, which is something that we didn't have data on before, since the "search comments if there are less than X words" parameter was set to 0 almost everywhere. I will start a thread on mozilla.dev.apps.bugzilla and see if people agree that either (a) a preference should be added for people to disable this (that defaults to off) or (b) if we should just stop searching comments entirely during quicksearch.

Oh, and for what it's worth, I *personally* would prefer that Quicksearch did not search comments by default. But it has been the actual default behavior of Bugzilla for so long that I need some people who were actually using it that way to say "yes, even when it did search comments back in the past, I didn't like that".

In an org that doesn't always put the best summaries, and where the same item has different names to different groups, we need to be able to quick search against the comments. I vote for (c) Add a parameter for either: (c.1) searching comments or not; or (c.2) search these fields [ ... ] (multi-select or CS list) when using the quicksearch. It would be more helpful to have quicksearch look at comment0 (description) for each bug rather than all comments, but that sounds more difficult than it is worth.

(In reply to comment #21) > I vote for (c) Which is Max's option (a). I read ahead all too often.

Okay, so the consensus from the discussion is that we should have this preference, but it should be *on* by default. Mozilla and other large installations can change that default to "off" using the normal Default Preferences interface (which I would recommend and appreciate, personally).

Comment on attachment 510401 [details] [diff] [review] Patch to add quicksearch full text user preference (v1) So dkl, you can check this in, but be sure to set "default" to "on" before checkin.

Comment on attachment 510401 [details] [diff] [review] Patch to add quicksearch full text user preference (v1) >+ "quicksearch_fulltext" => "Include fulltext when performing quick searches (slower)", And that needs to say "Include comments", not "Include fulltext"--the word "fulltext" means nothing to most Bugzilla users.

Committing to: bzr+ssh://dlawrence%40mozilla.com@bzr.mozilla.org/bugzilla/trunk/ modified Bugzilla/Install.pm modified Bugzilla/Search/Quicksearch.pm modified template/en/default/global/setting-descs.none.tmpl Committed revision 7721.