WADA:World Anti-bad-Duping Agency Reporter
	Description • 15 years ago

If IMAP offline-store file size excceds 4GB, mails written at over 4GB can not be read, and downloaded again & again, even if mail folder size is within 4GB.

Following is checked with Tb 3.0.

[Steps to reproduce]

(1) Create a 4GB file of text data (F0) in local IMAP mail directory
(2) Restart Tb, create an IMAP folder(F1), set offline use=on
(3) Emulation of bug 487992
(3-1) Rename F0 to F1
(3-2) Copy a mail(mail-1) to F1 folder
    => mail-1 is downloaded at offset=4GB of file of F1
    => No problem occurs.
       Mail data seems to be managed by two 32bits offset value;
       offset-1 : offset of fisrt mail in offline-store
       offset-2 : each mail's offset from offset-1
(4) Force overflow of offset-2 value.
(4-1) Terminate Tb
(4-2) Append 4GB text data to file of F1.
      i)   Get write-timestamp of F1
      ii)  Open F1 with "write append", append 4GB data to F1, close
      iii) Change write-timestamp of F1 back to value of i)
      Emulation of repeated mail add, mail delete, without folder compaction.
(4-3) Restart Tb
(4-4) Copy mail-2 to IMAP folder of F1
      => mail-2 is downloaded at 8GB+ of file of F1
         offset-2 of mail-2 overflows
(4-5) Click mail-2
      => fails to find in offline-store
      => fetched from IMAP server again, and saved in Disk Cache
      => mail-2 is displayed normally
(5) Emulation of removal of mail-2 from Disk Cache
(5-1) Terminate Tb
(5-2) Delete all files in Tb's Disk Cache
(5-3) Restart Tb, Work offline
(5-4) Clink mail-2
      => "Mail is not downloaded yet" like text is displayed
(6) Work online, clink mail-2
    => mail-2 is downloaded and appended to end of file of F1
       offset-2 of mail-2 still overflows
    => Sama as step (4-5)

Even if mail folder is smaller than 4GB, repeated download can happen.

If mail folder is smaller than 4GB, "Compact Folder" clears above situation.

If mail folder is greater than 4GB then offline-store size exeeds 4GB, fix of bug 462665 is required.
If mail folder is greater than 4GB, split of IMAP mail folder is required.
([Gmail]/All Mail case)
  1. Define Gmail Label of All-01, All-02, ... (IMAP folder of All-01, ...)
  2. Add one of All-01, All-02, ... to all conversations in "All" folder
  3. Disable offline use of [Gmail]/All
  Above can be done with user defined archive folder with archive_granularity.

	WADA:World Anti-bad-Duping Agency Reporter
	Comment 1 • 15 years ago

As written in bug 532323 comment #3, problem can be said;
  If total synced data size exceeds 4GB, wrap-around of offset value occurs.
  Bug 532323 :
    IMAP mail folder size is larger than than 4GB.
    Synched data size always exeeds 4GB, if size of aut-sync data is not limited.
  This bug(Bug 537498) :
    IMAP mail folder size is smaller than 4GB.
    Total synced data size exceeds 4GB. 
    (no need of 4GB mail in IMAP folder to test in this case)

	Wayne Mery (:wsmwk)
	Comment 2 • 15 years ago

we should try to get this for 3.0.2. 
you would think it's also affecting some trunk users

	WADA:World Anti-bad-Duping Agency Reporter
	Comment 3 • 15 years ago

Possible workaround for "larger than 4GB" case, if auto-sync of such huge folder is required.
 - Synchronization&Storage, Synchronize most recent NNN days.
   Periodical compaction may be needed to avoid this bug.
 - Options/Advanced/Network&Disk Space, up to 16000 MB of space for the cache.
   Unless cache is cleared, repeated download probably won't happen,
   except some special cases such as bug 246415, bug 516211. 
   (downloaded only twice. at over 4GB of offline-store, and in disk cache)
Can search be done on data in disk cache?

	WADA:World Anti-bad-Duping Agency Reporter
	Comment 4 • 15 years ago

FYI.
Disk Cache may not be a effective workaround due to bug 175600.

	Dan Mosedale (:dmosedale, :dmose)
	Comment 5 • 15 years ago

If I'm understanding correctly, I think this would make Thunderbird effectively unusable for anyone with a sufficiently large mailstore (which seems likely to be at least somewhat common given GMail and Yahoo data storage these days).  That said, there don't seem to currently be any markings on this bug that would indicate that folks are complaining about it regularly in support venues.  Roland, have you seen any complaints that might be this?  Marking as needed+ based on what I currently know and Wayne's comment.  Assigning to bienvenu for further triage, since he has more context in his head than I do.

	WADA:World Anti-bad-Duping Agency Reporter
	Comment 7 • 15 years ago

(In reply to comment #6)
> *** This bug has been marked as a duplicate of bug 532323 ***

To David:Bienvenu: Why DUP of bug 532323?
This bug is mainly for problem of Tb 3.0.x(1.9.1 builds) even after fix of bug 532323 by Tb 3.N(where N>=1, 1.9.2 builds or later).

	WADA:World Anti-bad-Duping Agency Reporter
	Comment 8 • 15 years ago

Same root cause as bug 532323, and fixed by patch for bug 532323. But the patch for bug 532323(64bits offset use) is for Tb 3.1 only.
Re-opening in order not to loose "folder smaller than 4GB" case and problem on Tb 3.0.x even after fix of bug 532323.

	WADA:World Anti-bad-Duping Agency Reporter
	Comment 10 • 13 years ago

Tb 3.0.0 is no longer supported. No need to keep this bug for Tb 3.0.0.
=> Wontfix