User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 Build Identifier: Hello, I know that there is ongoing work related to how the new 'Synchronization Storage' mechanism will work, with respect to polices relating to Space/Time (Bug 482476) and Volume (Bug 506024). I was told my request didn't quite fit either, so am spinning this off as a new request. Being that the *behavior* I want is already in place, this would simply be a matter of adding it to the GUI... I'd like to see a new sub-option for the 'Syncbronization & Storage' option - 'On-Demand' - that should be checked (along with the 'Keep messages for this account...' option - by default when a new IMAP account is created. See the following attachment for a simple graphical depiction of how this would look... Reproducible: Always

Confirming per discussion in bug 482476. I don't recall the name of the underlying preference right now, but I think it was a server-specific setting anyway (not sure if this can or should be extended to a folder property). Sorry for sending you from bug to bug, it's tricky to find the right granularity of those requests to be handled most efficiently.

Here we go, that's mail.server.default.autosync_offline_stores, thus should be covered by any %serverkey% construct. I'm making the title more descriptive.

Ok, thanks... So, just to be clear... You're saying as currently implemented, its a server-wide setting? I'd still like the ability to set this on a per folder basis, but I understand if its a lot of work for something that not many people would be interested in, it won't be a priority or might be marked WONTFIX. I have a *lot* of folders, and have a few I'd like to keep fully synced, but most not... Anyway, thanks for considering it...

That's probably a question David can answer better, but there is an attribute autoSyncOfflineStores defined in nsIImapIncomingServer.idl and therefore this setting should be available as mail.server.server#.autosync_offline_stores for each of your accounts. So, this would be available right away and only needs respective UI elements and wiring to be accessible through the Account Manager. As for the folder properties, that may be a different mechanism and I don't know if each attribute is easily accessible on a per-folder basis. On the other hand, offline synchronization enable/disable can be done individually for each folder, thus using a same granularity for download limits would be consistent.

This may need to block Tb3, since, as davida pointed out in bug 506024 comment 2, this could inadvertently hurt existing users with metered connections pretty badly.

As pointed out in one of the other bugs referenced in comment 0, we probably need some way to deal with this on Tb2 -> Tb3 upgrade so that folks who currently have metered connections don't get charged a bunch of money for not reading the release notes. This might mean profile-upgrade UI, since this is presumably impossible to detect automatically.

Ummm... I think my suggestion solves this problem nicely? In case I wasn't clear, I was saying this should be the new DEFAULT... Ie, instead of defaulting to FULL Sync, it now only defaults to 'Sync on Demand' (both options checked per my little graphic depiction attached above)...

But, of course, for upgrades, if the user has already set an account to full offline/sync mode, the 'Sync on Demand' checkbox should NOT be checked for that account. Although, if this new option I'm asking for can be made to work on a per folder basis, then for any folders that are NOT set to offline/sync mode, the new 'On-Demand' option could/should be checked by default.

To make this more intuitive probably option during account setup - full sync/on demand sync is more elegant. So user can barely miss this option.

Ok with me, but the DEFAULT (pre-selected) choice should be 'Sync On Demand' - this prevents ugly surprises for those with lots of IMAP accounts and/or very large mailstores (like me and many many others)...

So... any comment from the devs on whether or not this can be made a per folder option, just like the current offline option? Meaning - add another column in the 'Tools' > 'Account Settings' > 'Synchronization & Storage' > 'Advanced' window named 'OnDemand'? Personally, I think this would go a long way to making TBirds IMAP capabilities near perfect...

Gerry, is this what you were working on in bug 470624, making that policy option specific for folders? There is now bug 516884 working on a migration frontend where you can choose not to synchronize certain accounts, but unless you go into the advanced settings (with bug 510707 and follow-up bugs providing a front-end to the time limit), that probably still means "all or nothing" as a choice. Also, in line with comment #10, that only affects migrating profiles, a counterpart of that dialog would be desirable also when creating new accounts.

Just to be clear... This Feature Request has nothing to do with Time/Space issues, limited bandwidth issues, or anything like that. Ok, I just remembered that I had a slightly different idea about how this could work a few months before ( back in May), but never came here to document it. I'd like to now modify this Feature Request , and I'll attach a new screenshot of how the GUI could look. This will also alleviate the request to make this a per-folder setting, so no mods to the per-folder UI would be necessary either. Since my primary concern about having everything set to full offline mode has been, not just email messages, but the fact that a lot of my messages have large binary attachments, it is, in fact, just the large binary attachments that are of concern. With that in mind, I posed a question on the dovecot mailing list back in May, and Timo provided an excellent response as to why this should not be very difficult to do, so I'm hoping you guys will take a few minutes to go read the exchange here: http://www.mail-archive.com/dovecot@dovecot.org/msg18866.html With this in mind, I would like to modify this feature request such that: 1. 'Sync-On-Demand' should only apply to messages and/or folders, and 2. it should only apply to messages that have *large* binary attachments, with an option to apply only to binary attachments over a certain size This way, full headers *and* body text would be available for Gloda search capabilities, but I wouldn't have to download the GBs of binary attachments that I would rarely, if ever, need access to on my secondary PC's (my netbook, home PC, and two other locations where I have TB installed). The attachments should only be downloaded/sync'd if I intentionally right-click > download or dbl-click > open them.

Graphical depiction of Sync-On-Demand only large binary attachments

Hi rsx11m. As it happens I just got back to the world of 24 electricity and running water two weeks ago :) In reply to comment 13. In bug 470624 I was trying to implement a 'sync now' feature that supported strategies and would thus give the same results as auto-sync. I hadn't yet tackled folder level policies/strategies and I'm going to leave 'sync now' for the time being as I think other related bugs/features are more important. My greater goal is for Thunderbird to "Only download what I ask for and only download it once" <https://wiki.mozilla.org/MailNews:Minimizing_Bandwidth_Usage> and IMO comment 14, bug 439731, bug 345832, bug 506024 and others would be solved with this user experience. IIRC technically there are two key features that need to be implemented. 1. User defined strategies for auto-sync (i.e. download large messages/attachments on-demand) 2. Support for partial messages (mime parts) in the offline store (1) is now possible globally and AFAIK could be implemented at the server/folder level without too much pain. (2) has been worked around by auto-sync (it downloads and stores the entire message) and bug 439731 caches mime parts. I believe the workarounds although a step forward are incomplete because AFAIK a message that is excluded by a strategy (e.g. not saved to offline store) will not be available offline and thus not indexable by gloda and when the cache expires mime parts will need to again be downloaded. Support for partial messages could be quite a big job.

(In reply to comment #14) Now you make good point since most people don't really need attachments until they try download one, only important is text, therefore this decrease sync time because we will sync only text. Personally I would like to see this implement too.

Hi Gerry, I'm confused... (1) would, iiuc, still require support for partial messages, since binary attachments are simply different mime parts... or am I missing something? Also, what about the ability to set a size threshhold? This way, small attachments (text/emails, tiny signature graphics, etc) would still be downloaded/indexed by Gloda... Thanks for listening and for all you guys' hard work!

>I'm confused... (1) would, iiuc, still require support for partial messages, >since binary attachments are simply different mime parts... or am I missing >something? You are correct. To gain fine control over auto-sync it would need to know about mime parts and to be able to sync them individually. However this may increase the complexity of auto-sync significantly and may not be necessary. If the offline store supported mime parts you could get quite close to the behaviour you are describing. First you override the default auto-sync strategy in an extension and exclude messages > X Kb (see bug 470624 for an example). Then when you view the body of a message it will be saved in the offline store and be indexable by gloda. Large non-inline attachments (e.g. pdf files) would not be downloaded or stored unless you click on them. What this doesn't solve is old messages. If you don't view a message the body will never be saved in the offline store.

Wow, this bug started off with a well-defined suggestion and is getting more complicated by the minute. :-) You are right, bug 439731 utilizes the disk cache for short-term storage of messages and attachments based on a most-recently-used scheme. While testing that patch, I've noticed that some messages are cached as a whole, whereas others were split over multiple entries for message body and attachments. Apparently, this depends on the message/attachment size and the disposition of an attachment. Meaning, "true" attachments were more likely to end up as a separately cached item than small attachments or those with inline disposition, e.g., images as part of the message or separate attachments. Thus, some logic is already present to split up messages (likely relating to bug 345832 as well) but currently not tied with the offline storage in any way. I think the overwhelming task is to sort out the different usage scenarios, including connectivity and disk-resource considerations, then come up with a set of preferences and functions to make as many users happy as possible.

There are many interrelated bugs which affect IMAP mail. What about pulling it all together on the wiki? I've made a page covering my concerns <https://wiki.mozilla.org/MailNews:Minimizing_Bandwidth_Usage>, but this could be expanded to cover other issues. It would need to be renamed, what about "Downloading, syncing and storing IMAP mail"?

Re: Gerry / comment #21 I'll be happy to help with editing the page... And yes, I agree the page name should be changed, and your suggestion sounds fine... I'll wait until you say you're ready for others to start editing... Thanks!

Re: rsx11m / comment #20 Actually, I thought this modification of the original request would *simplify* the overall request... Also, maybe this wasn't obvious, but this should also pretty much also totally disable the 'cache', since everything would be downloaded to the offline store, correct?

Re: comment 22 Charles I sent you a PM, but I suspect it may have been filtered so posting it again here. On second thoughts a new wiki page may be better. The minimizing bandwidth page contains quite a few references to the (to date vapourware) Incommunicado extension and would not be relevant, but I'd like to keep somewhere. To save time you could take parts of the wiki formatted body from my page to get a head start. I've been following many interrelated bugs (see related bugs section) and I have gone through most of the code at some stage and can contribute in putting the technical info together and making it human readable. A good resource and my original starting point was Emre's wiki on Better, Faster, IMAP <https://wiki.mozilla.org/MailNews:Better_Faster_IMAP_Plan>. If you've got time why don't you kick it off. I should be able to start working on TB3 again this month. If you have any other ideas let me know.

Hi Gerry, Sorry, I got really busy at work and kind of dropped out for a while, but things have slowed down a bit, and now since 3.0.1 was released, I'm ready to start working on this again. I tried an upgrade to 3.0.1 and whoah... big troubles, because my TB2 profile has so many IMAP accounts about 16), and most of them have multiple GB of messages, some with 10+GB, and most with dozens of folders, and some with hundreds... It crashed numerous times until I was able to disable all of the offline stuff - then I discovered that I could have easily done that with one click on one of the Migration Tabs that I closed immediately - heh... I'm even more adamant now though that TB should honor the existing offline settings. Like I said, I have a bunch of accounts, with hundreds and hundreds of folders, but only some few select folders set to full offline mode, and that was a bit of a pain to reproduce - and was even more painful (lots of cussing while I was doing it) with the knowledge that the only reason I was forced into having to redo those settings because someone decided that it was ok to *dishonor* my existing settings... that really peeves me off. I could understand if it was unavoidable, but this surely wasn't, it was just someone unilateral decision. As I discussed above, the main thing I want to focus on is the need for a different offline mode choice, that downloads *all* of the headers (not just the basic ones) *and* the body text, but *not* large binary attachments. This will make searching GLODA capable, and will allow message filters to work on full headers, without having to download the whole message (in cases of messages with large binary attachments). Anyway, I'm ready to start updating your wiki, if you're willing to let me at it (I already have a wiki account so can start right away)...

Ugh, that crashiness thing is particularly nasty. Did you get a chance to file a bug on that? If not, would you please, CC me, and nominate to block tb-3.1? Any relevant crash-stats.mozilla.com would be extra helpful...

clarkbw: if implemented, does the screenshot attached here seem right, or would you recommend any adjustments?

Dan: yeah, it was wonky, but the thing is, it never actually 'crashed' - it just went into la-la land, totally unresponsive, and I waited a long time (15+ minutes), then I killed it. So, I wasn't able to report the crashes. Does it write anything anywhere locally that might help when something like that happens? Also, I have a few questions about the current behavior to get things straight in my mind... 1. When 'Synchronization & Storage' > 'Keep messages for this account on this computer' is *not* checked, what, precisely, does TB3 download? Is it only the 'Normal' headers and not the full headers? Does it download any of the body text? 2. What exactly is all that activity/indexing going on? Mine was going crazy for 3+ hours, even with offline mode for all accounts disabled. 3. Following up on question #1, where does it store what it downloads when 'Synchronization & Storage' > 'Keep messages for this account on this computer' is *not* checked? In the offline store or the cache? 4. Am I correct that setting 'user_pref("mail.server.default.autosync_offline_stores", false);' in user.js will only change the default of 'Synchronization & Storage' > 'Keep messages for this account on this computer' to unchecked for *new* Accounts? 5. If the answer to #2 is yes, then if I add this pref to the user.js file in a TB2 profile prior to upgrading to TB3, will that pref remain in effect or get over-ridden by the profile update process the first time I run TB3 on that profile? 6. Is there any way to relocate the global-messages-db.sqlite file? This file is in my roaming profile, and is 450MB in size. This is gonna cause my logout times to be really long. (Yeah, I know, we need to implement redirected folders - it will be done with the new Domain Controller upgrade, but thats at least a few months away.) Thanks guys... as frustrating as this has been, I am overall very happy with the IMAP improvements, and I do realize that not everyone has this many accounts/folders/messages...

Re: comment 25 Hi Charles, No problem I've had other priorities too. As I mentioned in my last post, because there are many intersecting issues I think it would be best to start a new wiki page along the lines of "Downloading, syncing and storing IMAP mail" However we could get a head start by drawing on these two wiki pages and relevant bug info: <https://wiki.mozilla.org/MailNews:Minimizing_Bandwidth_Usage> <https://wiki.mozilla.org/MailNews:Better_Faster_IMAP_Plan> As for a solution I believe the main hurdle is support for partial messages (mime parts) in the offline store. Once you have this you can sync message bodies while excluding attachments. I assume gloda would then be able to index that message.

Can anyone answer my 6 specific questions above about the current behavior? I'm wanting to start working on the wiki pages where we can discuss potential IMAP improvements, and also would like to open 2 (or maybe 3) simplified bug requests to replace this one, but I need to know what the current behavior is before doing so. Thanks...

(In reply to comment #28) > > 1. When 'Synchronization & Storage' > 'Keep messages for this account on this > computer' is *not* checked, what, precisely, does TB3 download? Is it only the > 'Normal' headers and not the full headers? Does it download any of the body > text? That setting only affects whether newly created folders will be set for offline use. It does not control autosync directly. Autosync will only download the bodies of messages in folders that are set for offline use. > > 2. What exactly is all that activity/indexing going on? Mine was going crazy > for 3+ hours, even with offline mode for all accounts disabled. > We still index the headers, and we look for any messages that might be stored offline. > 3. Following up on question #1, where does it store what it downloads when > 'Synchronization & Storage' > 'Keep messages for this account on this computer' > is *not* checked? In the offline store or the cache? Again, that checkbox doesn't control whether we download messages. It's the per folder setting (offline or not) that determines whether we download messages for offline use. If a particular folder is set for offline use, we will store messages you read in the offline store. In general, we should only be putting messages in the disk cache if the user clicks on a message and the folder is *not* configured for offline use, but sometimes messages will end up in both because of a bug. But eventually messages will get aged away from the disk cache. > > 4. Am I correct that setting > 'user_pref("mail.server.default.autosync_offline_stores", false);' in user.js > will only change the default of 'Synchronization & Storage' > 'Keep messages > for this account on this computer' to unchecked for *new* Accounts? No, it does not. There is no UI that sets the autosync_offline_stores pref. That pref controls whether autosync runs at all. If autosync does run, then it looks for folders that are set for offline use. > > 6. Is there any way to relocate the global-messages-db.sqlite file? This file > is in my roaming profile, and is 450MB in size. This is gonna cause my logout > times to be really long. (Yeah, I know, we need to implement redirected folders > - it will be done with the new Domain Controller upgrade, but thats at least a > few months away.) Not currently, though perhaps we should consider putting it in the local profile, and not the roaming profile, but that would also require putting the .msf files in the local profile, since gloda uses the .msf files to know whether messages have been added to gloda.

First - thanks very much for taking the time to respond to these David, especially after my going off like I did yesterday (apologies again). Also - please don't hesitate to point me to any documentation that answers my questions (In reply to comment #31) > (In reply to comment #28) >> 1. When 'Synchronization & Storage' > 'Keep messages for this >> account on this computer' is *not* checked, what, precisely, >> does TB3 download? Is it only the 'Normal' headers and not the >> full headers? Does it download any of the body text? > That setting only affects whether newly created folders will be set > for offline use. It does not control autosync directly. Autosync > will only download the bodies of messages in folders that are set > for offline use. Ah, ok, got it. >> 2. What exactly is all that activity/indexing going on? Mine was >> going crazy for 3+ hours, even with offline mode for all accounts >> disabled. > We still index the headers, and we look for any messages that might > be stored offline. Ok, so, to confirm I understand this correctly with offline mode disabled for a folder, the first time a user clicks on a folder: a) it downloads and indexes the headers? b) is it *just* the headers? c) is it the *full* headers (View > Headers > All), or just the 'normal' headers (View > Headers > Normal)? d) are these headers stored in offline storage (even though offline mode is disabled for that folder) or the cache? e) is any of the above behavior modified by the state of mail.server.default.autosync_offline_stores? (The reason I ask c) is I distinctly recall reading somewhere someone saying that only the 'Normal' headers are initially downloaded/cached when a folder is not set to offline mode.) >> 3. Following up on question #1, where does it store what it >> downloads when 'Synchronization & Storage' > 'Keep messages for >> this account on this computer' is *not* checked? In the offline >> store or the cache? > Again, that checkbox doesn't control whether we download messages. > It's the per folder setting (offline or not) that determines whether > we download messages for offline use. If a particular folder is set > for offline use, we will store messages you read in the offline > store. YEah, my question was wrongly worded due to my misunderstanding of what 'Synch & Storage > 'Keep messages...' did, so here it is correctly worded: When offline_mode for any particular folder is *disabled*, is *anything* stored in the offline store? If not, would an option to always store the full headers in the offline store regardless of this setting, thus making this setting only apply to the full body/attachments, sound reasonable to you? Or is there any technical reason this would be impossible or impractical? > In general, we should only be putting messages in the disk cache if > the user clicks on a message and the folder is *not* configured for > offline use, but sometimes messages will end up in both because of a > bug. But eventually messages will get aged away from the disk cache. Gotcha... >> 4. Am I correct that setting >> 'user_pref("mail.server.default.autosync_offline_stores", false);' >> in user.js will only change the default of 'Synchronization & >> Storage' > 'Keep messages for this account on this computer' to >> unchecked for *new* Accounts? > No, it does not. There is no UI that sets the autosync_offline_stores > pref. > > That pref controls whether autosync runs at all. If autosync does > run, then it looks for folders that are set for offline use. Gotcha again... >> 6. Is there any way to relocate the global-messages-db.sqlite file? >> This file is in my roaming profile, and is 450MB in size. This is >> gonna cause my logout times to be really long. (Yeah, I know, we >> need to implement redirected folders - it will be done with the new >> Domain Controller upgrade, but thats at least a few months away.) > Not currently, though perhaps we should consider putting it in the > local profile, and not the roaming profile, but that would also > require putting the .msf files in the local profile, since gloda > uses the .msf files to know whether messages have been added to gloda. Is there some technical reason they have to be in the same location? And that begs another question... I know the .msf files contain the indexes for the associated mail file, and that they now contain the per folder 'view' preferences for the message pane (ie, which columns are enabled/disabled, where column is where, how wide they are, etc) - but do they contain anything else? Again, David, I really appreciate you taking the time to answer these questions...

What is actual problem or actual request of this bug? (1) Default of mail.server.default.mime_parts_on_demand is true (may be mail.imap.mime_parts_on_demand=true is also used) So, there is no need of UI to request "enabling mime parts on demand". (If mail.server.serverN.mime_parts_on_demand=false is set, it means user changed it via Config Editor or requested by user.js or requested by prefs.js modification. Why UI for enabling is needed?) As already explained several times, in any case of (a) you changed "Keep messages on this computer..." in Account Settings/Synchronization&Storage" from Unchecked to Checked, (b) you changed "Keep messages on this computer..." in Account Settings/Synchronization&Storage" from Checked to Unchecked but chaged folder to Offline-Use=On, mime_parts_ondemand will automatically be effective by one of following. (c) Via Account Settings/Synchronization&Storage/Advanced, or (d) via Folder Properties/Synchronization, uncheck Offline-Use=On setting of folder(s) Above is done by (1) and followins; (2) default of mail.server.default.autosync_offline_stores=true (3) Auto-sync of Tb does do auto-sync on Offline-use=On folder only. (4) mime_parts_on_demand=true/false is absolutely irrelevant to auto-sync. mime_parts_on_demand=true/false is relevant to Offline-use=Off folder only. (5) Slightly confusing but; Change of "keep ..." from Unchecked to Checked at Synchronization&Storage : set default of new folder to Offline-Use=On change all folders of this account to Offline-use=On Change of "keep ..." from Checked to Unchecked at Synchronization&Storage : set default of new folder to Offline-Use=Off change all folders of this account to Offline-use=Off I you don't want "all mail data is automatically stored in offline-store file always by auto-sync", followig is already possible in recent Tb 17.(I don't know when chaged) (6) mail.server.serverN.autosync_offline_stores=false Mbox of the account : Offline-use=On Because entire mail data is fetched and stored in offline-store file when mail in the Mbox is viewed(because Offline-use is not Off) only "mail you viewed" is stored in offline-store file. (This was done by a change) And, because mail data is held in offline-store, the mail data is used upon later mail viewing in both Work Online mode and Work Offline mode. (7) Retention setting for offline-store file. Account Settings/Synchronization&Storage already can control offline-store file size by mail's age, mail size etc. So, even if you use auto-sync, offline-store file size can be limited. To bug opener of this bug and duped bugs : To problem reporters in this bug and dup'ed bug : Anyway, again, what is actual problem of this bug and what is your request? You don't want to use auto-sync and your request is improvements when auto-sync is not used(generic or many improvements when mail data is not saved in offline-store file)?

Hi WADA... The main point of this bug is, for people with lots of IMAP accounts and/or very large IMAP stores, a way to have the just the headers and body TEXT stored locally, but have large binary attachments NOT stored locally, *and* to also have UI options for managing these settings. I though that my simple graphical depiction of how the GUI could represent these settings spoke for itself.

(In reply to Charles from comment #36) > Hi WADA... Again, what is actual problem of this bug and what is your request? Please note that here is bugzilla.mozilla.org where to report actual Tb's bug(flaw in Tb's code) to developers, with poviding sufficient data for problem aalysys by developers.

Ok, I'll summarize things Monday morning, no time until then...

(In reply to Charles from comment #38) > Ok, I'll summarize things Monday morning, no time until then... can you revisit? :)

Ok, am in the process of reviewing this, but it is taking a while... Will respond once I'm done...

Ok, I'm closing this as invalid, because it is really confusing... I will open a new bug that will hopefully explain what I'm looking for much more clearly.