The mail.spotlight.enable pref needs UI, probably including some way to let users specify folders to exclude from indexing (e.g., spam). Sid, you said there was some common ui I might leverage here from the Vista search integration? Any thoughts?

Bryan, Can you give some guidance here on the right way for the pref UI to get added, such that users can: a) enable/disable Spotlight or Vista index of mail b) allow users to choose which folders do not get indexed NOTE: changing to Spotlight/Vista, since I think we can do the same thing on both platforms, aside from strings.

Bryan, see comment 1. Bugzilla didn't add you last time for some reason.

Bryan, I expect we can use the unified OS integration dialog we discussed a couple of months ago (bug 467116) for Spotlight too? We didn't discuss per-folder settings then, though.

(In reply to comment #1) > a) enable/disable Spotlight or Vista index of mail I think we'll want to use the same "OS Search Integration" enable / disable switch. > b) allow users to choose which folders do not get indexed For starters we shouldn't be indexing Spam mail by default. After that we could likely use the "Items for Offline Use" dialog to give us all the folders in all accounts. { to check this out, go to Account Settings -> Syncing & Disk Space, *Message Synchronization* (Advanced...) } I'm a little unsure where we should land this part though. (In reply to comment #3) > Bryan, I expect we can use the unified OS integration dialog we discussed a > couple of months ago (bug 467116) for Spotlight too? I believe so, yes.

I think I'm going to focus on more backend work, so reassigning to nobody.

Since we now have backend code that does OS search integration, I think we should make an explicit decision about what to do about it for Thunderbird. Turning search on by default for both Vista and OS X? Pro: users of those OSes get better system integration & user experience Con: their systems use extra CPU, memory, and disk space doubly indexing (Gloda + OS indexing). humph, Sid: do you know of other pros and cons we should be thinking about? Sid: I assume that the Windows search code also works on installs of XP that have the (backported) Vista search update from MS?

That should have read "for Thunderbird 3".

(In reply to comment #6) > Turning search on by default for both Vista and OS X? Just in case it isn't explicit enough, we shouldn't do this without pref UI to turn it off of course.

(In reply to comment #6) > > Turning search on by default for both Vista and OS X? While I think it can be done for OS X without trouble, we require a UAC prompt on Vista to enable indexing. Simply showing a UAC prompt without warning on first startup would be quite jarring, so we need to ask the user whether he wishes to enable it -- in fact, that's what we do right now. > > Sid: I assume that the Windows search code also works on installs of XP that > have the (backported) Vista search update from MS? It doesn't, actually -- the Windows Search MIME property handler that we use on Vista wasn't backported to XP.

I think we want to require a user decision before we enable this indexing, and I think we want to make sure we are posing the decision in terms of how the user searches for messages. example: "I want to search for my messages... ( ) From within Thunderbird. ( ) From within Thunderbird and using Spotlight (commonly accessed using the magnifying glass icon on your menu bar)/using Windows Search (you get to it somehow). This will require additional disk space and processing." This might want to be part of a bunch of annoying questions we need to add like whether we should store everything offline, whether the user pays for bandwidth and we should throttle autosync, etc. With my gloda-biased hat on (it is optional, I just like the way it looks): We really do not want these other indexers 'stealing' indexing time from gloda. If search sucks inside Thunderbird, the simplest explanation is Thunderbird sucks. If search sucks outside Thunderbird, the simplest explanation is the Operating System sucks. Since gloda backs off on its rate of indexing based on CPU use but the spotlight/vista indexers do not, it is going to look like Thunderbird sucks. With my rational hat on top of the other hat: Gloda should probably just sabotage the other indexers when gloda wants to index (important) things, disabling them. When gloda is done (hah!), they can do their thing.

> I think we want to require a user decision before we enable this indexing, and > I think we want to make sure we are posing the decision in terms of how the > user searches for messages. I don't think this makes any sense. I'll speak from the stand-point of OS X, but I think the same applies on Windows: search isn't something extra, it's something I expect. In the same way that the UI for actually doing searches isn't optional (the tb search box and Spotlight search box will be there no matter what you do), we have to provide good results. > This might want to be part of a bunch of annoying questions we need to add like > whether we should store everything offline, whether the user pays for bandwidth > and we should throttle autosync, etc. Having a pref/feature/extension for a bandwidth-lite version of tb is one way to couch this, for sure. But I still think it's not the general case. > With my gloda-biased hat on (it is optional, I just like the way it looks): We > really do not want these other indexers 'stealing' indexing time from gloda. > If search sucks inside Thunderbird, the simplest explanation is Thunderbird > sucks. If search sucks outside Thunderbird, the simplest explanation is the > Operating System sucks. Since gloda backs off on its rate of indexing based on > CPU use but the spotlight/vista indexers do not, it is going to look like > Thunderbird sucks. This isn't true imho. On Mac, if your tb results don't show-up in Spotlight, it isn't the OS that sucks, it's tb. Every Mac app works this way, so the absence is more than a little obvious. I'm never asked by Mail.app if I want to search a) in the app; b) in spotlight; c) in both. The answer is d) do both and don't ask me because they are different use cases, and I'll eventually want both. I search in tb (or Mail.app) when I know what I want is a message. I use Spotlight when I know what I want, but don't recall the context. I agree with you, though, about the need for gloda searches in tb to be good. We need that on, and they need to be good. > With my rational hat on top of the other hat: Gloda should probably just > sabotage the other indexers when gloda wants to index (important) things, > disabling them. When gloda is done (hah!), they can do their thing. I can't comment on how this should work wrt gloda, as I don't know the code. But I suspect there are cycles enough for both. There is certainly an expectation, especially on Mac, that both would be there. Welcome to Apple, where the apps just work, and yours better work too.

(In reply to comment #10) > > With my gloda-biased hat on (it is optional, I just like the way it looks): We > really do not want these other indexers 'stealing' indexing time from gloda. > If search sucks inside Thunderbird, the simplest explanation is Thunderbird > sucks. If search sucks outside Thunderbird, the simplest explanation is the > Operating System sucks. Since gloda backs off on its rate of indexing based on > CPU use but the spotlight/vista indexers do not, it is going to look like > Thunderbird sucks. The spotlight/Vista indexers currently index one message per second on idle (on Thunderbird's side) -- that isn't really stealing CPU cycles from anything. On the indexer side, at least the Vista indexer will back off when there's any use, and stop indexing altogether if there's heavy use. (In reply to comment #11) > This isn't true imho. On Mac, if your tb results don't show-up in Spotlight, > it isn't the OS that sucks, it's tb. Every Mac app works this way, so the > absence is more than a little obvious. I'm never asked by Mail.app if I want > to search a) in the app; b) in spotlight; c) in both. The difference there seems to be that I believe Mail.app uses Spotlight to show results -- the same goes for Windows (Live) Mail/Outlook and Windows Search. However, the user really shouldn't need to know about this -- he would expect fast, indexed searches in whatever UI the OS or application provides.

(In reply to comment #11) > I don't think this makes any sense. I'll speak from the stand-point of OS X, > but I think the same applies on Windows: search isn't something extra, it's > something I expect. In the same way that the UI for actually doing searches > isn't optional (the tb search box and Spotlight search box will be there no > matter what you do), we have to provide good results. My scenario of concern is this: 1) I install Thunderbird. I have lots of IMAP accounts. 2) Thunderbird downloads a copy of every message. (1x content overhead, 1x attachment overhead) 3) Gloda indexes every message. (~1x content overhead) 4) OS indexer indexes every message. (~1x content overhead, ?attachments?) 5) User is surprised and upset to be out several gigabytes of disk space, especially since none of 2 through 4 used to happen. Another motive is that we want to make sure users know that these things will show up in the OS search, whereas they didn't use to (if the users were thunderbird users). That doesn't have to be posed as a decision, though. I feel like people could get awkward surprises from a search that used to not include their email now including their email. Perhaps our "Welcome page" could have simple headings with easy actions to change these things, posed in terms of the user benefits, but with the 'costs' also mentioned: * Offline Access: Thunderbird is going to download copies of your messages in the background to your disk so you can access them more quickly. If you are concerned about network usage or disk space usage, _click here to change_. * Search: Thunderbird is going to index _all_ of your messages (_change_) for faster searching within Thunderbird and from the Operating System (_change_). Each index will require some disk space; _click here to select the folders to index to reduce the disk space required_. > I can't comment on how this should work wrt gloda, as I don't know the code. > But I suspect there are cycles enough for both. There is certainly an > expectation, especially on Mac, that both would be there. Welcome to Apple, > where the apps just work, and yours better work too. The use-case I am specifically thinking about here is the "I just installed Thunderbird to try it out", I have a lot of messages, and it would be nice if search was usable within a short period of time. We are going to improve gloda to try and be clever about what order it indexing things in, but it's still an uphill battle. I had forgotten that the Spotlight/Vista indexer went about its business so slowly, though. I retract any concern about it messing with gloda indexing.

(In reply to comment #13) > I had forgotten that the Spotlight/Vista indexer went about its business so > slowly, though. I retract any concern about it messing with gloda indexing. Okay, need to un-retract the concerns a little. The event-driven indexing still queues all messages notified on msgAdded and then drains the queue without any form of delay. Of lesser concern is the sweep indexing's _findNextHdrToIndex method that never yields while spinning on the header enumerator (and probing properties). This can be fairly CPU intensive on folders with a lot of messages. The mitigating factor is that this only becomes expensive once all the messages have been indexed (which would take a while at one message per second). However, both of those issues can be dealt with; the code in question is very clean and easily modified.

> My scenario of concern is this: > 1) I install Thunderbird. I have lots of IMAP accounts. I think right there you've outed yourself as a power user, and as such, extensions and various manual prefs are appropriate for you. I still submit that in the general case, by default, indexing should just happen. I also think that people use TB vs. Gmail so that they can get local things like search indexes, that if they really cared about space, they'd be leaving their mail (and mail client) in the cloud. > Another motive is that we want to make sure users know that these things will > show up in the OS search, whereas they didn't use to (if the users were > thunderbird users). That doesn't have to be posed as a decision, though. I > feel like people could get awkward surprises from a search that used to not > include their email now including their email. I don't think is will be awkward, and can be mitigated by proper release notes and feature info. If anything, this is something to say, "hey look what we do now!" > The use-case I am specifically thinking about here is the "I just installed > Thunderbird to try it out" I agree with this use case, but think that this would be better addressed via bug 467397.

Wow, sorry I've taken a long break from this bug. I'm looking at the question here from two possible concerns. Quoting from andrew's comment #10 I see "...additional disk space and processing" as the main interests behind our users. As far as processing goes I don't see this as a driving use case. Assuming we're somewhat smart and don't hog the users time up then saving processing time seems to only serve the fastidious. I am assuming that with a regular set of mail we don't behave badly, this assumption leaves the trouble case of users with low CPU power but lots of storage for mail. For low CPU and low storage I believe users will be concerned about the storage first and such we can help them turn off the OS search at that point. If we are overextending systems with low CPU but lots of storage I'm not really sure how we degrade from that gracefully. Hard drive space, compared to processing time, is a finite resource that I assume more users are concerned with immediately. If bug 482476 gets fixed then we'll have some options to help users save space and I imagine a good addition would be a link to OS Search options from that UI. I believe that would complete our "Where did my space go?" storyline for the user. So I'd recommend that we go with both search indexing options, this enables the best behaviour for our default case of users. For users with many IMAP accounts or very large mail archives I am going to assume that space will be a concern and they will use our space policy editor (w/ os search options added).

The Thunderbird drivers wish to release Thunderbird 3 as soon as possible. As a result, we feel that this bug shouldn't stand in the way of all the other good work getting into the hands of users sooner rather than later. Therefore we are retargeting it for 3.1. See http://ccgi.standard8.plus.com/blog/archives/242 for more details. The 3.1 release is expected to be a quick release soon after Thunderbird 3.

I suspect if this were the last bug standing for 3.1, we wouldn't hold the release for it, so setting to wanted-thunderbird+ and blocking-. Bryan, feel free to re-set if you disagree. As far as I can tell, the next step for this bug is some sort of UX mockup (ASCII art, perhaps?) that clarkbw can then give ui-review to.

I found this bug because I was slightly horrified when I came across literally thousands and thousands of mozmsg files littering my mail directories. I agree with comment 10 and comment 16 that there needs to be a warning like "This will require additional disk space and processing.", but I think it should more specifically alert the user to the fact that a separate file will be created for each and every message -- that's not something I would expect to happen without being asked or warned. Also (perhaps this should be a separate bug report?) the mozmsg files should be deleted when the Spotlight option is turned off -- most users will either never come across them or won't know how to delete thousands of files in a directory tree in one go (and also currently they wouldn't even know that there's a connection between these files and the Spotlight search they've turned off).