Open Bug 845475 Opened 12 years ago Updated 2 years ago

Data Loss with "Archive" Command moving large amounts of mail

Categories

(Thunderbird :: Untriaged, defect)

17 Branch
x86
macOS
defect

Tracking

(Not tracked)

UNCONFIRMED

People

(Reporter: codetraum+bugzilla, Unassigned)

References

(Blocks 1 open bug)

Details

(Whiteboard: [dupeme])

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.57 Safari/537.17 Steps to reproduce: Used archive command on several hundred emails in a personal folder. Actual results: Many messages were archived and sorted into subfolder by year correctly; however, many more disappeared and are also not present when viewing the archive folder via webmail interface for the mail provider. Messages present for first two years of messages, but not for last two. Cannot verify that all messages from first two years are present. Expected results: Messages should have been sorted by year and placed into archive folder.
Bug 757328 is for similar phenomenon of "archived message was not shown at Archives folder when archive from local mail folder to local Archives folder". Same question as bug 757328 comment #1. Which case of bug 757328 comment #1? Because archive is an "mail move between two mail folders", performance problem may be exposed if many mails are arcived(moved) at once. Did performance problem such as CPU hog, very large memory consumption, long time freeze, occur in your "archive of several hundred emails"?
Flags: needinfo?(codetraum+bugzilla)
Same question as bug 757328 comment #3. What is your actual archive setting?
I have the IMAP to IMAP (same acoount) case from Bug 757328. All folders in question are marked for synchronisation for offline use. With the exception of perhaps a dozen messages, all mails were > 6 hours old and hence should have been synchronized for offline use. (And all of the mails had been viewed, so they had been downloaded.) I was able to restore the lost emails by using another copy of Thunderbird open on another machine. I assumed that the messages hadn't been deleted there because of an idle machine not sychronizing properly. I took that machine offline, back upped the mailbox in question and then brought it back online to sync -- the messages that were successfully moved on the original machine synced to their new state properly and the missing messages were present in their original location. During the original archiving process, I did not notice any especially bad performance issues -- a small delay (perhaps a few seconds), but nothing surprising for IMAP operations on roughly 1700 messages, many with attachments. (Having restored what I believe to be all the messages, I did a quick check to get a more exact figure.)CPU usage on one core did spike a bit, but the machine in question has a quad core and 20GB of RAM, so anything less than truly extraordinarily memory consumption won't be noticeable. Archive settings via about:config: > mail.identity.idX.archive_enabled key not present > mail.identity.idX.archive_folder (anonymized) imap://username@my.imap-provider.com/Archives > mail.identity.idX.archive_granularity key not present > mail.identity.idX.archive_keep_folder_structure key not present > mail.identity.idX.archives_folder_picker_mode key not present Archive folder structure: subfolders by year (default, I believe) My guess based on skimming over bugs related to archiving or moving messages in rapid succession is that there is some sort of race condition tied into the aforementioned performance problem. Enough operations in rapid succession can trigger it on local operations, but network ops are even more susceptible. I find it curious that the second computer had synchronized all the other changes I had made, including the proper archiving of the first half of the messages and creation of the archive subfolders , but had neither copied the second half to the archive nor deleted the originals. (I'm assuming that move is "copy and delete the original", but I know little about the specifics of the IMAP protocol.)
Flags: needinfo?(codetraum+bugzilla)
Oh, it occurs to me that I have it set to "prompt before compacting local folders" on the second machine and that I was prompted to compact several times before doing the restore, but intentionally didn't so as to preserve the local cache.
(In reply to codetraum+bugzilla from comment #3) > I took that machine offline, back upped the mailbox in question > and then brought it back online to sync > -- the messages that were successfully moved on the original machine synced to their new state properly > and the missing messages were present in their original location. Archive at single Mbox == (number of target Mbox) * (move mails from the single source Mbox to a target Mbox(Archives/yyyy), and a "move from source Mbox to a targrt Mbox" is batch type "move". "Move" in archive is 1 * ("copy N mails" + "delete N mails"), or N * ("copy a mail" + "delete a mail"). If I understand correctly, copy/move in IMAP utilizes offline-store file at least when Offlin-Use=On; copy/move locally, sync with server, issue "uid x,y,z copy Archives/yyyy" and "uid x,y,z store +Flags \Deleted". "uid x,y,z move Archives/yyyy" may be used , if server supports Move Extension. And, IIRC, Tb has problem in Offline/Online detection in Mac. If only one Arcive is requested to Tb(not "repeated 'a' key pressing" case like bug 757328), it sounds for me phenomenon like following. During archiving, before "issue IMAP command" phase, network error occurs, and archive stops at mid of archiving. State change of Online->Offline is not notified to Tb or Tb fails to detect it. After the error, network interfaces comes back. State change of Offline->Online is not notified to Tb or Tb fails to detect it. By manual change of Work Online=>Work Offli=>Work Online, sync'ed normally. Above is similar to; Archive while Work Offline mode, then go Work Online mode If IMAP delete model=Just mark it as deleted, following is seen. 0. Show "Order Received" column. Value is MsgKey and is UID of mail if IMAP. 1. Two mails is held in MboxA, mail-a/UID=a, mail-b/UID=b. Both has Date=YYYY/mm/dd of same YYYY. Click Archives/YYYY to force open Archives/YYYY. Tb knows "Next UID=x". 2. go Work Offline mode 3. By other IMAP client, copy mail-x to Archives/YYYY. UID=x is used by mail-x. 4. At MboxA, select two mails, do Archive while Work Offline mode. => UID=a and UID=b is shown with strike-thru line at thread pane => two mails appear in Archives/YYYY. UID=x, UID=y 5. Undelete Message for UID=a => UID=a is shown without strike-thru line 6. go Work Online mode => three mails are shown in Archives/YYYY. UID is perhaps; mail-x : UID=x, mail-a : UID=(x+1), mail-b : UID=(x+2) => At MboxA, UID=a and UID=b is shown with strike-thru line at thread pane ("Undelete Message of UID=a" at step 5 is not traced) If you can reproduce problem of your comment #0, check with IMAP delete model="Just mark it as deleted"(Server Settings), please.
> (A) Bug summary: Data Loss with "Archive" Command (In reply to codetraum+bugzilla from comment #0) > Actual results: > (B) Many messages were archived and sorted into subfolder by year correctly; > (C) however, many more disappeared and are also not present when viewing the > archive folder via webmail interface for the mail provider. Question about phenomenon. What phenomenon do you call by "Data Loss" of (A)? (1) In Tb, some or all archived mails disappear from source folder, and no mail appear in target folder or some don't appear in target folder (2) In Tb, archive is nrmally ended, and all archived mails disappear from source folder, and all archived mail appear in target folder. When was (C) observed in Web mail? (3) Even after you did "go Work Offline mode, then go Online mode" when you experienced your problem. (4) Before you did "go Work Offline mode, then go Online mode" when you experienced your problem, and no problem after "go Work Offline mode, then go Online mode".
Server Settings: When I delete a message, move it to this folder: Trash ... Message Storage Neither Expunge nor Empty Trash on Exit selected. >What phenomenon do you call by "Data Loss" of (A)? (1) In Tb, some or all archived mails disappear from source folder, and no mail appear in target folder or some don't appear in target folder. which is why I checked webmail immediately after seeing that the messages were neither in the target nor in the source folder-- I was hoping it was a display error. No manual toggling of the online/offline mode on the machine in question. (I physically disconnected the second machine to prevent the second machine from connecting to the server.) Webmail showed the same status as Thunderbird -- messages messing in both the source and the target folder. Regarding network connectivity: the mail server was on the same local network as the first client, i.e. on the university's internal network. The second client, i.e. the one which still had the messages in the source directory, was at home, i.e., not on the same local network.
this is most definitely a duplicate
Summary: Data Loss with "Archive" Command → Data Loss with "Archive" Command moving large amounts of mail
Whiteboard: [dupeme]
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.