This is driving me INSANE. What human ritual do I need to preform to get you guys to fix this?

In the mean time, anyone know how to off line restore imap directories from backup?

Todd, what do you mean with "offline" and what kind of backups do you have? .... anyway, i think, thats the wrong place to discuss that offtopic.

but hey, Thunderbird-Team, you got new programmers, right? So this is really a major-bug and you really should focus on it! :-)

(In reply to Todd from comment #52)

In the mean time, anyone know how to off line restore imap directories from backup?

Please let's not cover support questions in bugzilla - please post in https://support.mozilla.org/en-US/questions/thunderbird

(In reply to Wayne Mery (:wsmwk) from comment #54)

Todd, what do you mean with "offline" and what kind of backups do you have? .

Answered to Wayne's eMail address

And More fun

Well, I "copied" a folder with about 1600 emails to a gmail folder and only copied about 900 messages one time. Then I deleted the messages from the gmail folder and tried again. This time it only copied about 700.

Since I don't run this laptop with offline storage, it appears that the messages are successively fetched from the source imap folder. Then at some point there is a problem and in the status bar I see something about trying to reconnect. At that point I think the partial folder content, which is probably concatenated into an mbox-like temporary file, is imap appended to the gmail folder, which happens quickly. I need to look closer at what is happening using imap:5 log and/or wireshark, but in any case, there is definitely a problem here and I understand all the complaints.

Edit: Doing this while recording IMAP:5 log, I see that the each email is fetched from the source folder in ascending UID order and immediately appended to the destination folder. So the order is fetch, append, fetch, append ...
Looking at the log, I see that somehow the connection to the source server is lost. Can't tell from the log why or who is initiating the disconnect, tb or the server. This time over 1000 emails were fetch and appended before the problem occurred.
I then transferred a small folder having offline store to gmail. With source offline store, nothing is fetched from the source server. Each message is just read from the mbox file and individually appended. So there is less to go wrong when you have offline store.
Anyhow, given this info, I need to peruse WADA's detailed posts above and see what they say.

Someone reported above that increasing timeouts helped. However, he didn't say which one. Also, reading some of WADA's comments mentioned bad effects of various background activities like periodic checks for new mail and autosync. So on all accounts I disabled checks for new mail and immediate notification for new mail and in preferences (config ed.) I disabled autosync. With these changes and tb restart to ensure they take effect, the 1600 emails copied OK. Not sure if this is just luck so I need to try the copy a few more times...

(In reply to gene smith from comment #59)

Someone reported above that increasing timeouts helped. However, he didn't say which one. Also, reading some of WADA's comments mentioned bad effects of various background activities like periodic checks for new mail and autosync. So on all accounts I disabled checks for new mail and immediate notification for new mail and in preferences (config ed.) I disabled autosync. With these changes and tb restart to ensure they take effect, the 1600 emails copied OK. Not sure if this is just luck so I need to try the copy a few more times...

IMHO it has been just luck. Anyway, the main issue here is not the process being interrupted; that could be acceptable under certain circumstances. The root of the problem is that TB does NOT warn correctly about the connection drop and, furthermore, it hasn't a way to resume the operation nor a proper rollback to get things just like they were before the failure.

Furthermore, 1600 is almost nothing if compared to real life. A typical mailbox (in the business environment) nowdays can have 20,000 emails per folder. and a hierarchy 300+ folders. ThunderBird is just a no-go for most enterprise users because its inability to properly deal with big amount of data.

Well, just did the 4th time with no errors so doesn't seem like luck, but who knows. Before it failed (didn't copy all 1600) 3 times in a row. Anyhow, this is complicated stuff so I can only do one step at a time...
Curious how you copy 300+ folders? All I see is you select one folder and copy or move the message in the folder to another folder on another server. Is there a way to copy or move 300 folders in one command that I don't know about in tb? Maybe an extension?
I really don't know how tb stacks up as in business environments since I just volunteered to help out with imap issues. All of it was designed by paid mozilla people who are now long gone and apparently even 10 years ago when some were still around there wasn't much concern about this issue.

Copying 300+ is a pain in the neck because it must be done one by one, so if you have to do 300 operations and it fails 200 times, it's an endless task!

gene, if you copy a folder normally all subfolders and included mails in this folders will be copied aswell, so the "command" to copy more then one folder is "built-in".

(In reply to Comfine GmbH from comment #63)

gene, if you copy a folder normally all subfolders and included mails in this folders will be copied as well, so the "command" to copy more then one folder is "built-in".

I see. There doesn't seem to be a menu command or right-click command to copy/move a folder but you can select a folder or multiple folders with ctrl click and drag it/them to do the move. You can also drag with ctrl pressed to just copy the folder(s).

I tried my test with 7000+ messages moved twice with no problem (all messages transferred from server A to server B and back to server A). I will repeat the test an go out of wifi range and see what happens.

(In reply to gene smith from comment #64)

(In reply to Comfine GmbH from comment #63)

gene, if you copy a folder normally all subfolders and included mails in this folders will be copied as well, so the "command" to copy more then one folder is "built-in".

I see. There doesn't seem to be a menu command or right-click command to copy/move a folder but you can select a folder or multiple folders with ctrl click and drag it/them to do the move. You can also drag with ctrl pressed to just copy the folder(s).

I tried my test with 7000+ messages moved twice with no problem (all messages transferred from server A to server B and back to server A). I will repeat the test an go out of wifi range and see what happens.

Just reproduce the failure by disabling the network some seconds. The problem is not the interruption itself, but the absolute lack of information when that occurs and a usable mechanism to resume the operation.

With weaker wifi, my copies don't work quite as well. This summarizes what I see in the log when the copy from source server to destination server finishes prematurely:

Many fetch/appends occur and succeed, ie, messages copied from source to destination.
Append fails at at destination. Connection was dropped per log.
Tb makes new connection to retry the queued "append from file" url.
Append retry succeeds
Log message occurs: [30784:Main Thread]: I/IMAP CopyNextStreamMessage: Copying 458 of 7651
Fetch next message from source fails. Connection dropped.
Tb makes new connection to source to retry the queued "fetch" url.
Fetch retry succeeds.
Connection to destination reports that it has dropped. This is before the append has started.
New connection to destination made. "Append from file" url was never queued, per the log.
Tb fetches headers for all the emails in destination mailbox and seems to conclude, erroneously, the "copy" operation is done.
Only 457 messages copied and no error notification occurs.

I see this with laptop setting outside under trees. Don't have debugger or network sniffer. Also, this is still with autosync and checks for new mail disabled.

This and bug 720158 are commonly reported, so -> critical

(In reply to Wayne Mery (:wsmwk) from comment #67)

This and bug 720158 are commonly reported, so -> critical

bug 720158 also reports a hang. I never see a hang or tb lockup, just tb gets an unrecoverable network error and stops the copy and moves on like everything is OK.

I tested this for several days on two systems and usually the copy (7000+ messages) worked OK but sometimes not. What I think happens is that the destination or source IMAP server does not expect so much data for so long a time from tb and drops the connection(s) and refuses new connections. This prevent any retries from being effective. This may depend on how busy the server is since I noticed that the time of day made a difference (afternoon bad, evening better).

A major complaint above is that it is difficult to recover from the error without completely starting again. One way I found is to sort the source and destination folders by "Order Received". Tb copies in ascending UID order so if a partial copy occurs, you can look in the destination folder and find the highest numbered "Order Received" value (really the message UID) and note the subject and date. Then go to the source folder and copy one past that same message to the end (highest Order Received/UID). This will avoid duplication and having to completely start again.

Note also that when tb does a "move" to another server, nothing is marked \deleted unless the move 100% succeeds and completes. Then all the message in the source are marked \deleted. Of course, this assumes you are using move mode and not copy.

Disabling autosync and checks for new mail while doing this also seems to help, see comment 59.

Another complaint is that there is no indication when copy/move is done or if it has stopped due to failure. A pretty good indication of completion is that the status bar goes blank and shows no activity. Of course this still requires the user to check the destinations for complete copies or incomplete copies.

Also mention is problems for business or enterprise users. I tend to think that the sysadmin might use a dedicated 3rd party tool (e.g., https://imapsync.lamiral.info/) that transfers and sets up all the employee's folders at the new server for them rather than having them each copy or move 20,000 messages in 300 folder, for example, manually in tb.

In conclusion, several people have commented on this bug and state that is a reason to not use TB. I don't think for most users this is a showstopper (critical) since it is a pretty rare event to need to move messages to a new server and most user can do this if it's necessary and work-around the problems. Implementing a "restart" mechanism is probably very difficult. However, it seems like popping up an error message that the copy or move did not succeed should be fairly easy and don't know why that doesn't currently occur.

I frequently move data from one email account to another rather than copy.

It would have helped a lot if TB would remove copied messages along the way and not just at the end once the copying completed successfully - that way you would know how much was moved even when it fails.

I use it with Gmail where moving something twice doesn't result in duplicates. So I worked around this bug by moving emails in increments of 1000 emails or so at a time. When it completes these 1000 emails are removed from the source and I do the next batch. If it fails I just repeat it.

Still a pain but it worked to move my daughters' school accounts after they changed schools.

(In reply to gene smith from comment #68)

(In reply to Wayne Mery (:wsmwk) from comment #67)

I tested this for several days on two systems and usually the copy (7000+ messages) worked OK but sometimes not. What I think happens is that the destination or source IMAP server does not expect so much data for so long a time from tb and drops the connection(s) and refuses new connections. This prevent any retries from being effective. This may depend on how busy the server is since I noticed that the time of day made a difference (afternoon bad, evening better).

A major complaint above is that it is difficult to recover from the error without completely starting again. One way I found is to sort the source

Difficult? Nah. It is impossible.
In the real world some users have thousands of folders in complex hierarchies of dozens of depth levels.

Disabling autosync and checks for new mail while doing this also seems to help, see comment 59.

Why should users disable autosync for daily and trivial operations?

Another complaint is that there is no indication when copy/move is done or if it has stopped due to failure. A pretty good indication of completion is that the status bar goes blank and shows no activity. Of course this still requires the user to check the destinations for complete copies or incomplete copies.

Again: when you have 1,000 folders in a chaotic hierarchy that's not an option.

If only TB deleted the emails that have been sucessfully moved... it would be easy to retry and finished the remaining emails. But since the deletion is done only once the whole operation is finished, at the end of the day you get lots of duplicate stuff across the source and target folders.

Also mention is problems for business or enterprise users. I tend to think that the sysadmin might use a dedicated 3rd party tool (e.g., https://imapsync.lamiral.info/) that transfers and sets up all the employee's folders at the new server for them rather than having them each copy or move 20,000 messages in 300 folder, for example, manually in tb.

Yeah, you got it. Many of us have started using a third party tool. It's called Microsoft Outlook.
Moving a big folder to another place is something a regular user should be able to do; definitely not a sysadmin task.

In conclusion, several people have commented on this bug and state that is a reason to not use TB. I don't think for most users this is a

Ok, if you don't think it's a showstopper we will continue using Outlook then.

Believe me... thousands of companies might have stopped using Thunderbird because of these kind of issues. Thunderbird is not a reliable tool to be even considered in a professional environment. Just from my side, I had to remove Thunderbird at two different companies I've worked for; almost 2,000 users in total. Think about it.

(In reply to Markus Mobius from comment #69)

I frequently move data from one email account to another rather than copy.

It would have helped a lot if TB would remove copied messages along the way and not just at the end once the copying completed successfully - that way you would know how much was moved even when it fails.

At one time, from what I have seen WADA and others post, drag/drop between servers always did a copy. Now it seems to default to move unless you do a ctrl-drag/drop. I suspect they are all set at the end to \deleted since there would have to be an imap transaction sent for each message to mark it deleted instead just one message at the end that sets all the messages to \deleted, so you have fewer "round trips".

Probably this was not designed with moving many and huge folders in mind.

I use it with Gmail where moving something twice doesn't result in duplicates. So I worked around this bug by moving emails in increments of 1000 emails or so at a time. When it completes these 1000 emails are removed from the source and I do the next batch. If it fails I just repeat it.

Still a pain but it worked to move my daughters' school accounts after they changed schools.

Sounds like a reasonable workaround.

(In reply to gene smith from comment #71)

(In reply to Markus Mobius from comment #69)

Still a pain but it worked to move my daughters' school accounts after they changed schools.

Sounds like a reasonable workaround.

Yup. It's reasonable for a girl's school account. But not for a call center or a sales manager.

(In reply to José Ángel Morente from comment #70)

Disabling autosync and checks for new mail while doing this also seems to help, see comment 59.

Why should users disable autosync for daily and trivial operations?

A single user moving thousands of messages and folder to another server is a daily operation?

If only TB deleted the emails that have been sucessfully moved... it would be easy to retry and finished the remaining emails. But since the deletion is done only once the whole operation is finished, at the end of the day you get lots of duplicate stuff across the source and target folders.

Maybe that could be done (mark as deleted when the append of messages completes). TBD.

Also mention is problems for business or enterprise users. I tend to think that the sysadmin might use a dedicated 3rd party tool (e.g., https://imapsync.lamiral.info/) that transfers and sets up all the employee's folders at the new server for them rather than having them each copy or move 20,000 messages in 300 folder, for example, manually in tb.

Yeah, you got it. Many of us have started using a third party tool. It's called Microsoft Outlook.
Moving a big folder to another place is something a regular user should be able to do; definitely not a sysadmin task.

In conclusion, several people have commented on this bug and state that is a reason to not use TB. I don't think for most users this is a

Ok, if you don't think it's a showstopper we will continue using Outlook then.

Believe me... thousands of companies might have stopped using Thunderbird because of these kind of issues. Thunderbird is not a reliable tool to be even considered in a professional environment. Just from my side, I had to remove Thunderbird at two different companies I've worked for; almost 2,000 users in total. Think about it.

Where I worked (retired now) Outlook was standardized on not because Thunderbird (actually Netscape back then) was so bad but because everybody else was moving to Microsoft and nobody got fired for using Microsoft. Only the company's Microsoft Exchange server was allowed for use so there was no reason to need to copy stuff to random imap servers. So not sure what you say is the main reason few companies use TB (at least here, maybe more in Europe)

So without major redesigned, things that can be fixed are maybe:
-Record an error if copy/move fails so something pops up and is recorded in activity manager.
-Mark successfully appended messages as deleted in the source folder when operation is move.

comeon guys, what we dont need is discussion about if TB is used by a schoolgirl or a manager or a sysadmin, that want to move messages from A to B, it's pain in the ass if a operation just quit without errors before completion, regardless if you copy 20 mails or 20.000, for sure, but we should focus on the BUG here!
now as this bug was classified as critical, i hope, there is a big chance that this long story might end within 2019!
and i am sure that there is no need of a redesign. everything that is needed is a intelligent error-handling when working copy/move/delete mails via IMAP and the possibility to press a "try again" button to complete the operation, that has been started but not 100% finished. i am sure, that everyone from the TB-dev-team knows exactly what to do, if they will just focus on this bug!

Just a correction to what I wrote in comment 71:

At one time, from what I have seen WADA and others post, drag/drop between servers always did a copy. Now it seems to default to move unless you do a ctrl-drag/drop. I suspect they are all set at the end to \deleted since there would have to be an imap transaction sent for each message to mark it deleted instead just one message at the end that sets all the messages to \deleted, so you have fewer "round trips".

I checked this again and drag/drop of an imap folder to a different imap account does a copy of the source folder and not a move. So you should never see messages inside the source folder or the source folder itself be removed. But if you select messages inside a folder and drag them (or use the right click menu to move them) to a different account, they are removed from the source folder (marked as imap \deleted) and the source folder is not removed.

Therefore, having messages "disappear" as an indication of the progress of a move is only possible (as currently designed) when multiple messages are selected in a single folder are moved.

Maybe off topic but other problems observed:

1. Drag of a folder to another folder in the same account works: The source folder becomes a sub-folder of the destination folder. This is done simply using the imap "rename" command. However, ctrl-drag/drop seems to be a noop -- nothing happens. So it seems it is not possible to copy a folder using drag/drop within an account. (Of course, right-click copy on individual messages works within an account.)

2. Sometimes a dragging a folder between accounts is a noop -- nothing happens. Seems to be random and can only be fixed with a tb restart. Then the drag/drop works again. I've seen this with my trunk build on desktop system and with release 60.7.1 running on laptop.

bug 601900 mentioned in comment 26 above seems to be the same problem as this bug. However, it has been closed as fixed and the patch addresses problems with a virus scanner grabbing the temp file containing the message to be appended at the destination server. Also, it only seemed to affect Windows systems. That may be why I rarely see this when testing on linux. Of course it doesn't address the secondary issues that several here have loudly mentioned.

Referring to the problem described in the original post, it's not just Windows. I had the same problem on OS X (with Thunderbird 60). It's keeping me from switching to Thunderbird so it's definitely a critical bug. I'm using an iCloud account with IMAP. If I log into it using Thunderbird, I get the same problem when I try to migrate my large local mail folders to Thunderbird from Mail (I've been using e-mail for a long time). Small folders not a problem. I don't have a virus scanner. I can't really join the Thunderbird team until this is resolved. Meanwhile I'm looking at alternatives, which is too bad as I really do prefer Thunderbird but I can't afford to give up all my e-mail archives. I'm surprised that a bug of this magnitude hasn't been addressed in ten years. It's not confidence building.

(In reply to jdien07 from comment #78)
I looked at this about 2 months ago and concluded that this is not a trivial thing to fix but it does need serious work. See my comments starting at comment 58.

Can you tell me your approximate folder structure and how many messages are in the various folders? Also, what procedure you used to set up thunderbird and copy messages from account A to account B? Are you wanting to move/copy messages from iCloud imap account to another account in tb?

You also mention local folder from Mail to Thunderbird. Not sure what that means or if it involves IMAP.

Thanks so much for looking into this! So my starting point is I'm using Mail under OS X 10.14.6 and the iCloud mail service (IMAP). I'd like to immigrate to Linux so I'm considering Thunderbird as my new e-mail client. For this initial trial effort, I am trying to migrate to Thunderbird while still on OS X to minimize complications. I last looked into this in June with Thunderbird 60. It was quite a nightmare with it quitting partway through. I'd try doing it in batches of 1000 messages but somewhere along the way it'd mess up, often by duplicating all the messages I had just tried to copy so I had to start all over again.

In response to your query, I tried again, but this time with the new 68.0 release to provide you with up-to-date information. I'm pleased to say so far things are going better so the new update may have fixed things (mostly)! Just to be timely, here are my initial observations. I'm using a desktop connected via ethernet to a gigabit switch which in turn is connected to a 50mps Fios internet connection so wi-fi is not an issue, unlike some of the prior posts on this thread. I have some very large folders so I keep only the most important ones on iCloud. The rest are local. A typical local folder has 14,367 messages. It is the contents of a list serve that I follow and it is helpful for me (as a scientist) to have my own copy rather than to have to rely on the limited tools available on the web archive. So I have a copy of Thunderbird set up and it is working fine for the messages on iCloud. Since Thunderbird's import tools for Mail no longer work (they were not updated to keep up with changes in OS X), the only way for me to accomplish this migration is to first make a copy of this directory on iCloud and then from there to make a copy on Thunderbird's local installation. The first step of making a copy on iCloud using Mail went smoothly. For today's test, I am using Thunderbird to select all 14,367 messages in the GUI and then right-clicking and choosing "Copy To...Local Folder" to copy them into an empty local Thunderbird folder. Thunderbird says Downloading message ... of 14418 and went through the entire set without a hitch. It does seem to be having some trouble keeping synched with changes on the server when I then deleted the files and did some other things via Mail. More once I have a clearer sense of what is going on.

jdien07, I don't know of any changes between 60 and 68 that would directly affect copying many files in one imap operation. The only network change I know of added a tcp keepalive feature: Bug 1535969. However, this wouldn't seem to affect the problem reports for this bug since large copies will cause continuous network activity with no intentional idle connections.

Anyhow, glad it seems to be working for you, at least so far. So let me know how it goes.

The message count at the bottom of the Thunderbird window often undergoes prolonged pauses, presumably while it's doing some kind of cleanup or indexing. My guess is that prior to the keepalive addition it was timing out during these pauses, that in turn activated the bug you're examining, having to do with handling of errors. So it appears I'm good now. I appreciate your looking into this error handling bug. I expect I won't always be checking my e-mail under ideal conditions. Let me know if I can be of further assistance.

Interesting theory on why keepalive might actually help.I'll do some experiments and see what is happening. But even if it helps, others have pointed out that there is still no notification to the user that the copy (or move) failed or a way to restart a failed copy without completely starting over and maybe duplicating already copied messages at the destination.

jdien07, I do have one question. Can you tell me how many messages get copied before you see the "prolonged pauses"? Right now I have a test folder with about 7K messages and have done copies using it. Is that enough to see the pauses? (The messages are fairly short and from an old mailing list archive I found.)

My impression was that it was about a thousand. Perhaps it was doing the compression thing? But I tried it again with another folder and this time instead of counting down smoothly it is giving the number of messages in increments of about one hundred so harder to judge. I make a screen recording I could send you. Also I could send you the listserve messages themselves as a test case. They're fully public.

As a further possible test case, I'm currently trying to move the contents of an online folder with 40,0000 messages to a local folder. Thunderbird has already made a local copy of the online folder as far as I can tell, so this should be straightforward for TB; however, if I try to select and move more than about 1300 messages at a time (using the right-click method), it does nothing. So once again no error message. But at least there's no longer data corruption with this version of TB.

It would seem easy for tb to copy 40,000 messages from a folder with offline storage to local folders since no network should be involved. What do you mean by "does nothing"?

Also I could send you the listserve messages themselves as a test case. They're fully public.

If you can zip them all into 1 file and attach to an email and send them to me, that would be good. However, I don't want to receive 40,000 emails :).

I right click and then nothing further happens. The messages are still selected. There are no further messages in the status line. I leave and come back in an hour, still nothing different. With less than about 1350 messages, the status line immediately announces it is moving the files, provides a running count, and at the end of it they have been moved. I don't know how to confirm that TB really has made a copy offline of the iCloud messages, but looking inside the ImapMail directory of my Thunderbird Profiles directory, I do see a 1.34GB file with the name of the folder so I assume that is the offline copy. I'd be happy to zip the messages to you. I assume you mean you want me to zip all 40,000 and e-mail that one .zip file to you but you don't want me to forward all of them individually to you? :) I could zip the file that is in the ImapMail directory. Would 1.34GB bomb your e-mail account though? How about I send you a Box link?

Edit: oops, I saw 1.34 and was thinking 1.34 MB. So that probably wouldn't work via email. Also, missed the "Box" link suggestion. Not sure what that is but probably better than trying to email a gig file. Thanks.
Sure go ahead and send it zipped. I always need more emails to bring tb to its knees. :) Send it to w4.k9.vws@gmail.com which is almost unused, storage wise. If that bounces we can try my default email I used for bugzilla. I've sent myself bigger emails. (Or if you have a file sharing site like dropbox or google drive, you could put it there and send me the link. I think firefox has something like that too, not sure.)

Testing with jdien07's mbox file he sent me with about 17k messages, when doing a mass copy from Local Folders to a gmail imap folder, something seemingly random occurs. Each message is "appended" to the gmail mailbox and with the initial append request sent by tb there is a length, e.g., {1000} indicating 1000 byte are to be send by tb. Tb starts sending and sends 1000 bytes correctly but then 2 more bytes are sent. This causes gmail to respond with an error and the copy stops.
This seems to occur with random messages and can occur at any point during the mass copy.
Looking at the source mbox file in Local Folders, I can see that the message that fails actually has a length of 1002 bytes. So somehow tb is determining the wrong length for the email before sending it but sends the correct number of bytes.
It may also be possible for tb to tell the server that the length of the message to be appended is too large. This will cause the imap connection to timeout since now the server is still waiting for bytes that are never sent. I haven't actually seen this occur yet.

I'm also seeing some corruption of messages. For example, the date of one message went from 2000 and 1935 during a copy by TB. I'm really glad you're tracking down these issues! I think I'll hold off switching to TB until you've had a chance to fix them. Let me know if there is anything else I can do to help, like beta-testing.

(In reply to gene smith from comment #90)

Testing with jdien07's mbox file he sent me with about 17k messages, when doing a mass copy from Local Folders to a gmail imap folder, something seemingly random occurs. Each message is "appended" to the gmail mailbox and with the initial append request sent by tb there is a length, e.g., {1000} indicating 1000 byte are to be send by tb. Tb starts sending and sends 1000 bytes correctly but then 2 more bytes are sent. This causes gmail to respond with an error and the copy stops.

Ok, I was wrong about this. Actually, when the message length is X bytes, X bytes are sent but the append requires that two more byte be sent, CRLF to "terminate" the append. This is illustrated in the append example in rfc 3501 but you have to count the bytes to see the the two extra CRLF bytes are transmitted and are not part of the message byte count in {}'s.

However, gmail does respond with an error saying "BAD Invalid Arguments: Unable to parse message" but there is no apparent error in the data sent. (Of course, I can't really see the data the gmail receives.) I can retry the append and the exact same data is sent and gmail is happy. I have added a retry to the tb code to handle this error and do the retry. With this in the code I have been able to upload the complete mailing list to gmail (17k message some with fairly big attachment) several times with no error. (It is currently just test code and a formal patch has not been submitted to tb project.)

I have tested this with two other servers (my isp on openwave imap server and a dovecot server on localhost) but never see the same error that gmail reports. I have copied the same 17k messages twice to openwave and once to dovecot with 0 errors.

Tests so far have been from Local Folders to imap folders on gmail, openwave and (localhost) dovecot. I need to test to other servers now and then copy between imap servers. I am monitoring the transfer with wireshark and recording IMAP:5 log with tb to see where the errors occur.

This seems to occur with random messages and can occur at any point during the mass copy.

This is true, at least for gmail. Haven't seen it with openwave or localhost dovecot but need to check some other servers.

Looking at the source mbox file in Local Folders, I can see that the message that fails actually has a length of 1002 bytes. So somehow tb is determining the wrong length for the email before sending it but sends the correct number of bytes.

Checking again, tb is sending the correct message length and then sending the CRLF to terminate the append. So tb is doing the right thing.

It may also be possible for tb to tell the server that the length of the message to be appended is too large. This will cause the imap connection to timeout since now the server is still waiting for bytes that are never sent. I haven't actually seen this occur yet.

Still TBD on this.

(In reply to jdien07 from comment #91)

I'm also seeing some corruption of messages. For example, the date of one message went from 2000 and 1935 during a copy by TB. I'm really glad you're tracking down these issues! I think I'll hold off switching to TB until you've had a chance to fix them. Let me know if there is anything else I can do to help, like beta-testing.

On each append, tb sends the date that the message should have. Not sure why that is necessary since the date is already in the header that gets sent. Not sure why 2000 would be changed to 1935 but maybe 1935 is the start of the "unix epoch" or something like that. I'll check on this.

once you're ready with the new code, I'd be happy to beta test with iCloud and do some careful testing and report back on what I see.

gene, thank you very much for taking care of this issue!

we had this problem with our mailserver in the past, this is courier/exim. if it helps i can create a mailbox for you and give you the credentials for testing.

Comfine GmbH,
Thanks a test account on a real server is usually a good way to see a problem. So yes, that would be fine. You can send me creds via email to my bugzilla profile address, if that's OK.
Can you also describe the copy/move scenario for you that doesn't work or often fails.

gene smith,
i sent you Login-Data to your mail. The problems happend for me personal when i was moving a large ammount of mails from a folder of one account to a folder in another account. the move-process just stopped somewhere without response but not all mails have been moved - the source-folder was not empty. so i selected "the remaining mails" and started again to movie until i realized after the second time the process just aborted, that the mails at the destination were there two times while some was missing and some was remaining at the old folder, so i cleaned up that chaos and moved just small ammount of mails which worked fine than.
this is years ago but i noticed this behavier from time to time while transfer mails for customers (different providers/servers) also. hope this helps.

Thanks Comfine G. for the test account. I will use it in my tests real soon.

After testing copying from Local Folders to an imap folder I noticed this behavior: see bug 1579341 comment 31.
No one answered the question, but I determined the reason on my own. The full message fetches occur due to enabling "Enable adaptive junk mail controls for this account". This causes the most recent message bodies, on initial sync, to be scanned to detect spam. With this "Enable adaptive..." disabled, only a selected subset of message headers (From, To, Date etc.) are fetched for all message on initial sync with no offline storage and full message fetch does not occur until a message is opened.

Comfine G, I was able to copy jdien07's 17k message mailing list to your server from Local Folder. Ran overnight with no errors. Now copying it to from your server to my ISPs server. Going ok so far. Monitoring with wireshark and TBs IMAP:5 log to detect any errors or stoppages.

Interesting factoid: It seems that when I try to copy to yahoo imap or aol (both use the same server and some ISPs use yahoo too) I always get an error on the append indicating [SERVERBUG]. However, this seems to be somewhat variable in that sometimes it works but only rarely for me when I copy only a single message. So mass copy to yahoo based servers (uses imap append) seems almost impossible.

Copy to my ISP server from comfine server worked with no errors.

Another interesting factoid: My ISP's server seems unable to fetch the requested header items and returns empty. However, all message are present and completely accessible when opened, just subject, correspondent etc are blank on the message list. I think this is because the headers from jdien07's mailing list are too complex or maybe too long for my ISP's server (Openwave) to parse.

Now copying back the 17k messages to comfine server from ISP's Openwave is in progress...

Well, didn't go so well:

1. 3260 messages copied OK. Message ID 3261 fetched from Openwave OK.
2. Tb appended message ID 3261 at destination comfine server. Good.
3. Tb fetched next message ID 3262 from openwave, got only part of 226550 bytes message and
4. Openwave issued a tcp FIN causing a disconnect. Don't know why.
5. Tb immediately opened a new connection and fetches flags as expected.
6. Tb then fully fetched message 3262, all 226550 bytes using the new connection.
7. No append to comfine occurs as it should, so copy stops.
8. Last progress logged "CopyNextStreamMessage: Copying 3261 of 17257
9. Connection to comfine times out after a while due to inactivity as expected.

So this is an example of how a copy starts, runs for a while and then just stops. No errors are logged or notified to the user.

However, I'm not sure this is what the typical commenter to this bug is doing. For one thing, I am running the openwave and comfine accounts without offline store. So copy between them has to fetch the messages from the source server before appending to the destination server. When the bulk copy is from Local Folder or from an imap folder with offline store, the message to be appended is obtained directly from file and no fetch on the network of the message typically occurs. So having offline store should eliminate the error seen above, and I've not seen this when the source folder is in Local Folders.

However, the copy process should be able to fully recover from the above problem. It currently partially recovers by re-fetching the incomplete (due to openwave disconnect) message; but does not continue on and append the correctly fetched message to the destination server. The question is why not?

Edit: Tried to resume the copy by selecting message 3261 to the end on openwave (source) server and copy to comfile (destination). Copy would not start and nothing logged. Tried twice. Had to restart tb to resume the copy. I think this was mention by others above. Apparently a variable is set and locking out further bulk copies. More to look for...

I have some more information to add to the scenario described in comment 100. When tb fetches the message to be copied from the source server (Openwave) it streams the received message to a temporary file. At step 4 when openwave issues the FIN, the temp file contains the partial message. However, even though tb retries the fetch and successfully receives the whole message at step 6, the data never gets re-streamed into the temp file (the partial temp file is not deleted and a new temp file is not created). I think what is happening is that the temp file that was partially written remains open and the fetch retry essentially streams to /dev/null (bit bucket). Also, there is no completion signal that would cause the partial temp file to be streamed out (read) and imap appended to the destination server (comfine) and then deleted. Instead the whole thing hangs and requires a tb restart to allow additional copies from source to destination servers. Testing this multiple times, I see multiple nscpmsg* files accumulating in the /tmp directory.

I haven't been able to find a solution for this so any suggestions would be appreciated. Also, this may be not be the typical copy/move scenario described by commenters and reporters to this bug.

One other thing noticed after dragging a folder from Local Folders to root level of an imap account: If the copied folder in the imap account is deleted and then the drag/drop is done again, nothing happens. And even if a different folder in Local Folders is dragged, the copy doesn't occur. If the drag is to an existing folder in the same destination imap folder (destination not root level), the copy occurs OK. To drag again from Local Folder to that imap account root level requires a tb restart.
Note: Dragging between accounts only does a copy. Move between accounts (Local Folders is an "account") never occurs.

Is this ancient bug any closer to getting fixed?

I'm still seeing this problem regularly.

A common job I have to do is migrate people off of Windows Live Mail.
So I set them up with Thunderbird, import their stored mail into "Local Folders" (if not all on IMAP) using ImportExportTools (note: it's dissappointing that Thunderbird has so few built in import tools), then drag/drop or Copy the folders to the IMAP structure.
This is very very flaky and often has to be repeated many times, failing at different points with no error message. Whereas it should simply be a case of doing the drag/drop or copy/move and walk away and leave it to sync.

A "copy/move without allowing duplicates" would also be good.

Is this ancient bug any closer to getting fixed?

I've been working on this bug off and on since last summer. I didn't see any problem when moving/copying from local folder to imap folders, but sounds like you do. I do see problems copying between folders on different imap servers, but only when the source server setup in tb has no offline store and the source server decides to disconnect.

I have a local folder with 17k messages, some very large, that I have copied to several imap servers with no problems. So I didn't think this is a problem. Can you describe in more detail what you see, like how many messages, copying a single folder or tree of folders, drag/drop or using the menu, etc.

A "copy/move without allowing duplicates" would also be good.

Unfortunately preventing duplicates and/or restarting the copy where the failure left off would be very hard to put into the existing tb code. So I have mainly been trying to make the existing copy activity respond to errors better rather than just stopping like it sometimes does now.

I imagine an important part of the equation is which IMAP server. I have problems quite reliably and the server is Apple's iCloud service. Gene, have you had a chance to try out that one yet? It's a free service so you could easily set one up for testing purposes if you haven't already. David, what IMAP server are you using?

I'm doing this migration for a customer.
Their email is sky.com which is Yahoo based.

Thunderbird syncs up OK with the Sky/Yahoo IMAP server.
I am able to import WLM mail to Local Folders (Using ImportExportTools)
But when I drag/drop or Copy mail from Local Folders to the IMAP structure, it starts to work but then eventually stops randomly, giving me an incomplete copy. No error message. Fails at different point most times.

I have tried copying single folder and a tree of folders.
One folder I'm having problems with only has 1700 messages, which isn't that high.

Related question:-
I've always thought that for Thunderbird to succeed, importing mail from other mail clients should be much easier.
I'm a tech so using ImportExportTools isn't beyond me, but would it be possible to have good import filters built in to Thunderbird (that work) for common email clients like WLM. The need to import from WLM must be more common now that MS have retired it.

(In reply to jdien07 from comment #107)

I imagine an important part of the equation is which IMAP server. I have problems quite reliably and the server is Apple's iCloud service. Gene, have you had a chance to try out that one yet? It's a free service so you could easily set one up for testing purposes if you haven't already. David, what IMAP server are you using?

Thanks for the tip. I have an iCloud account but I've only used it on a mostly dormant 2008 MacBook Air for simple tb tests, not bulk copies. Do you see problems copying to or from the iCloud account? Does it involve Local Folders too?

(In reply to David Hale from comment #108)

I'm doing this migration for a customer.
Their email is sky.com which is Yahoo based.

I think I tried some bulk copies from Local Folders to yahoo and worked OK. Will try again. Also I think AOL uses yahoo server, FWIW.

I have tried copying single folder and a tree of folders.
One folder I'm having problems with only has 1700 messages, which isn't that high.

Guess if each message has megabytes of attachments it could be pretty big. With normal messages, yeah, not that big. This is also from local folders to the yahoo based server?

Related question:-
I've always thought that for Thunderbird to succeed, importing mail from other mail clients should be much easier.
I'm a tech so using ImportExportTools isn't beyond me, but would it be possible to have good import filters built in to Thunderbird (that work) for common email clients like WLM. The need to import from WLM must be more common now that MS have retired it.

Good point but maybe out of the scope of this bug. If doesn't already exist for this, maybe an "enhancement" bug should be entered. I don't know much about "wlm" users. Are they on a non-imap server like exchange?

Yes my copes were from Local Folders to the Yahoo based IMAP server.

I have now finally got all the mail uploaded but it was quite painful.
I got lots of failures but when I re-tried and completed it last night it went without a hitch. Makes me wonder if it was a gitch on the IMAP server that got solved? I still think it's a general problem though as I have had these problems many times when doing migrations from other mail clients where their mail wasn't all on IMAP. And if there are glitches on the IMAP server, Thunderbird need to be able to handle it or at least give an indication.

WLM is just a mail client, so the user could be hooked up to any mail service over either POP or IMAP.
The only way I've found so far to import the mail that's not on IMAP is to use ImportExportTools.

AT&T just forced me to revise Thunderbird from POP mail servers to IMAP mail servers in order to continue using my 20 year old bellsouth.net email addresses. Creating the new accounts in previously trouble free Thunderbird was not a major issue but now I am experiencing seemingly unresolvable issues as discussed above transferring the mail folders from the old POP to the IMAP accounts. My inbox has almost 9000 emails and there are about 20 sub-folders under the Inbox for specific email categories and the number of files in each vary from a few hundred to few thousand. I also installed the 64 bit version of Thunderbird (68.3.0) but that solved nothing.

Using Windows Explorer, I tried finding the actual file folders containing my Thunderbird emails history but could not locate them or at least find anything I could identify. Is it possible to work around this problem by simply copying the appropriate folders using Windows Explorer to the appropriate new file location? I would think with 32 GB RAM and all the horsepower my CPU provides this machine should not choke if I simply do file copy/pastes using Windows explorer. Any suggestions on how to locate and identify these folders would be appreciated. I am sure they exist somewhere as I regularly use Mozbackup and it obviously knows how to find them and where to put them if it is used to restore.

I don't think you can or should try to fix it by just copying the pop storage files to the imap storage location on tb. FWIW, the pop files should be somewhere like under c:\users\<your name>\application data\mobile data\thunderbird\<your profile>\mail. I'm not on windows now so don't remember the exact path; you might also see it in tb by looking under your account's server settings near the bottom which works for imap, not sure about pop.

I assume you want to duplicate your pop folder structure under the new imap account. Also, I assume the imap account is working ok and can receive emails into Inbox. So have you tried to copy a single top level folder from the pop account to the new imap account? When you do this, what happens? This should cause the copied pop folder to be copied, mail by mail, to the new folder on the imap server. Note: To copy a folder you must drag it to the imap account. Before dropping, hold down ctrl key and see a small '+' to ensure it is copied and not moved (using ctrl key probably not needed since between accounts drag/drop always just copies, but suggesting this just to be careful).

Wow! Thanks for the quick response. It is certainly appreciated.

After writing the message above out of frustration I looked again and in a path similar to what you suggested I did find both the IMAP and POP accounts and did find files named "Inbox" and "Inbox.msf" in both accounts (except "inbox was all in caps in the IMAP folder). There were also "Sent" and "Sent.msf" files. The very large size of the files without the .msf extension were obviously the files with all the emails. However, although tempted, I am hesitant to just copy them from the POP to the IMAP folder as I am not sure of the function of the .msf files and if they need to be copied too. Be curious about your thoughts of doing this.

All your assumptions are exactly correct. With respect to the Inbox it seemed I had to copy the contents from the POP inbox to the IMAP inbox and this does not work as described at the beginning of this thread. I also dragged some of the sub folders with varying degrees of success but not so successful that it is proving viable. In all cases the email or folder transfer just stops. Sometimes most of the emails get transferred, sometimes only a small percentage, but never all of them. Very frustrating for something I was not expecting any problem with.

Its obvious from reading all the posts above you have extended very significant effort to solve this problem and I need to tell you how much that is appreciated. I have been using Thunderbird ever since MS discontinued Outlook Express and this is the first Thunderbird has let me down so I am hoping to resolve this problem and continue managing my email as usual.

Yes, copying the files and .msf would not work. Tb needs to see the messages on the imap server and fetch them from the imap server and then create its own files in the imap directory based on what it obtains from the imap server. The act of copying from the pop folder to the imap account/folder actually just sends the messages to the imap server using a command called imap "append". Then when the messages are all copied (appended) to the server's folder, tb fetches them and creates the storage files in the imap directory.

Let just try to get Inbox fixed before moving to other folders. How many messages are in Inbox? It sounds like maybe the server is dropping the connection and refusing even the automatic retry that tb does. You might consider recording a tb IMAP:5 log while doing a copy to know what is really happening: https://wiki.mozilla.org/MailNews:Logging . Then attach the log using the "Attach file" link above.

Another thing to try is add a column to pop and imap inbox, using the widget at the far right, called "order received" and sort on it. When doing the copy, tb starts with the lowest number and works up. This way you can more easily tell how far the copy went before the server refused to cooperate and only copy from that point when you try again to avoid duplicates. I realize this isn't optimal but right now that's all there is. Otherwise you will probably need to delete all the messages copied so far in Inbox and copy Pop/inbox again and hope they all get copied before the server craps out to avoid dupes.

For such cases, just move the message files to under the Local Folders, then inside thunderbird UI, move them to the IMAP server.

Any non-Inbox folders he has created in the pop account are basically the same as "Local Folders" so moving them explicitly to Local Folders before online copy (append) to imap wouldn't be any different (at least that's what I assume). Comment 111 and others report problems copying from specifically Local Folders to imap folder.

Sounds like the recent poster has had the same problems I reported recently when trying to copy large numbers of email to an IMAP account.

You may be right in that's it may not be a Thunderbird bug as such, it could be the IMAP server dropping connection during the copy of a large number of messages. But could there be a way to make Thunderbird handle such server connection drops better?

Rather than simply failing, perhaps come up with a dialogue box stating that the server dropped connection, and allowing you to retry from where it got up to, or quit? Possibly also a tick box allowing it to "keep retrying".

I just encountered this problem as well. Could someone maybe update the documentation to tell people not to do this?

https://support.mozilla.org/en-US/kb/switch-pop-imap-account

Thanks.

(In reply to David Hale from comment #118)

Sounds like the recent poster has had the same problems I reported recently when trying to copy large numbers of email to an IMAP account.

You may be right in that's it may not be a Thunderbird bug as such, it could be the IMAP server dropping connection during the copy of a large number of messages. But could there be a way to make Thunderbird handle such server connection drops better?

Rather than simply failing, perhaps come up with a dialogue box stating that the server dropped connection, and allowing you to retry from where it got up to, or quit? Possibly also a tick box allowing it to "keep retrying".

Exactly. You put the finger on the sore spot.
The problem is not that connection sometimes breaks; that may happen, most of the time due to network or server failures. But Thunderbird MUST be able to deal with it by: a) informing the user in a proper way and b) having the ability of resuming the broken operation. None of these issues are currently solved in my opinion. I also think it should be a basic feature and can't understand why it isn't properly implemented after 16 years of development (!!).

Blaming the IMAP server for this is a cheap excuse. The problem is a blatant poor design of Thunderbird.

It's hard to tell if I was successfully able to swap emails between Local Folders and IMAP. How do I get a count of read and unread emails in each location?

(In reply to Mike H from comment #119)

I just encountered this problem as well. Could someone maybe update the documentation to tell people not to do this?

https://support.mozilla.org/en-US/kb/switch-pop-imap-account

You mean don't use imap or don't try to copy from pop to imap?

(In reply to Mike H from comment #121)

It's hard to tell if I was successfully able to swap emails between Local Folders and IMAP. How do I get a count of read and unread emails in each location?

Not sure what you are asking. When a folder is selected, be it imap, pop or local, in the status along the bottom there is an "unread" and "total" count displayed.

Anyhow, how many messages and/or folders were you moving? Was it between pop or "local folders" to a new imap account? Any other details you can provide?

(In reply to José Ángel Morente from comment #120)

Blaming the IMAP server for this is a cheap excuse. The problem is a blatant poor design of Thunderbird.

I've never said it is entirely the server's fault. But otherwise I agree. There are no hooks I can find that notify the user when a problem occurs and no way to restart from where the last copy failed. Tb just stops. When bulk copies/moves are occurring internally tb really doesn't know this is going on and is busy doing lots of other things. What it needs is a dedicated bulk copy/move mode and UI where that is all that is happening, similar to the dedicated program imapsync.

I had the same problem. Tried to move 17k messages from IMAP inbox to an 'archive2019' called IMAP folder. Got into an endless loop (tb tried to move messages which have been already moved) as soon as I hit the inbox after restart of tb. What did stopped tb from trying to move the messages was to repair the inbox - then it suddenly stopped trying to move already moved messages.

I've read through the thread and I am also experiencing the issue described.

My usage scenario: I've been using TB since ~2011, with a < 1GB IMAP limit until recently so I've been creating annual archive folders to keep within IMAP quota. My company recently moved to Outlook 365 with 100GB storage and I also received a new work PC. As such I want to move all my offline emails archived in TB back to my inbox so I can use Outlook or a new install of TB on another PC as well as access all my email online. My annual archives range from 3,000-7,000 emails each.

My steps: Go to an archive folder, e.g. "2016", select all emails, right click -> Move To -> me@mycompany.com -> Inbox.

Observed results: Status bar shows "Copying message x of y to Inbox..." with green status bar. x starts at 1 and increments towards y (y being 3k-7k in my case). At some point before they're all copied (i.e. before x == y), the copying stops with no error message. Some but not all of the messages are copied. I don't know how many exactly copied, but I've seen the status bar showing close to 1,000 done at times. None of the source messages that were copied are deleted if x didn't reach y.

If I try this with a small number of messages, say 1-100, the process completes successfully, i.e. the messages are all copied, and the source copies are deleted.

This all seems to be inline with the previous comments so I'm not looking for further diagnosis of my failure.

One recovery proposed above is to sort source/destination folders by received date and compare to see which emails copied successfully, delete them in source, and then restart. I don't think that's feasible for me. I'll likely need 5-10 attempts for each archive folder, times 8 years, times however long it take to parse after each failure which adds up.

What I am wondering:

There is some discussion/ description of this move process (copy all selected messages, then delete source) being called bulk move/copy. I assume from this that a different process where you copy one message at a time, then delete its source, then copy the next message, etc, could be referred to as non-bulk move.

Based on this thread, this bug when bulk moving many messages has existed for at least 10 years with no resolution. As such, is it possible to disable bulk move and put TB into a mode where when you select multiple messages and tell it to move them, it would do each one separately as a non-bulk move? I.e. copy msg 1, delete its source, copy msg 2, etc,? If this is not currently an option, could that be added as a feature while the designers continue working on fixing real bulk move?

That would at least make it easier to continue the move process after a failure since you wouldn't need to parse through your source and destination folders comparing email dates, then deleting the source emails that have already been copied. This could be an advanced, non-default option so that the majority of users moving small numbers of emails would continue to get whatever advantages bulk move offers.

Thoughts?

Thanks.

Hi spsilk,

My steps: Go to an archive folder, e.g. "2016", select all emails, right click -> Move To -> me@mycompany.com -> Inbox.

Are you moving between different imap accounts here or withing the same account. The reason I ask is because when the move/copy is intra-server, the sever does most of the work but when inter-server, tb must send all the messages to the other server via imap append.

There's not really a "bulk" mode vs. single message move/copy mode. When tb move/copies messages to another server it always just appends each message regardless of how many.

Also, move doesn't always delete the messages at the source. I think when you drag a folder to another server, tb always leaves the source folder in place. However, intra-server d/d does remove the source folder. (But I'm not 100% sure on this.) Anyhow, nothing is ever deleted if the move fails or hangs, I'm sure of that.

One recovery proposed above is to sort source/destination folders by received date and compare to see which emails copied successfully, delete them in source, and then restart.

My suggestion was to sort by "Order received" not necessarily by date. This orders the messages by imap UID and tb always copies from lowest to highest UID. (But maybe someone suggested order by date in the many comments above. :))

Anyhow, thanks for the inputs.

(In reply to gene smith from comment #125)
Hi Gene,
Thanks for the reply.

Are you moving between different imap accounts here or withing the same account.

I am moving from TB's "Local Folders", i.e. locally stored files. So, not moving between two IMAP accounts. To be clear, these are emails that were initially in my IMAP account, and were moved offline via TB's Archive action.

There's not really a "bulk" mode vs. single message move/copy mode. When tb move/copies messages to another server it always just appends each message regardless of how many.

OK, I'm not familiar with the inner workings of email protocols, but what I've been thinking of as copying, you're referring to as appending (to the IMAP Inbox?). In that case, my question still stands: couldn't there be a setting that makes TB append on at a time, deleting the source after each append? Rather than appending all the selected emails before beginning deletion, wherein you run the risk of the silently erroring out during the bulk append, leaving the source emails intact.

Thanks.

I've discovered an interesting thing when copying message from Local Folders to an imap server. When the message is large and/or the server is slow, tb sends all the data via imap append (in multiple segments) and waits for the final OK response from the imap server. If it takes more than 20 seconds for the OK to be received, the transaction times out, tb drops the connection and does a retry, which also fails and the copy stops. Here's where the timeout is set:
https://searchfox.org/comm-central/rev/6f8371b5697c7e34a8a594a5f449e1e51f68fe35/mailnews/imap/src/nsImapProtocol.cpp#2124
When copying large messages to outlook.com, which is slow, I notice that sometimes it takes 60 seconds or more for the final OK to be sent by outlook.com. So if I change the code to allow appends to timeout after the default of 120 seconds, I can successfully copy messages to outlook.com that quickly timed out before. (I'm not sure why the code implies that append response should almost be instantaneous.)

Another thing is there is an add-on called "copy folder" that is referred to in comment 26 above. I tried it and it and since it also uses the same imap code, the append timeout remains 20 sec so it also fails for large messages to outlook, but it keeps going and copies subsequent messages instead of just stopping. It does 5 retries instead of the default 2 and appears to have the ability to detect existing messages in the destination folder and not duplicate them when the copy occurs again. However, even the latest version called "copy folder mod" only works up to version 60 and not with 68. See https://addons.thunderbird.net/en-US/thunderbird/addon/copy-folder-mod/?src=ss . I haven't studied how it works internally but I'll take a look and see if maybe it's functionality can be incorporated into mainline tb.

I have a YUGE migration to g-suite in the works. Since g-suite will not migrate over folders, but only the inbox, I have to move the customers folders from backup into their local folder and then restore them by copying them over. Because of this bug, I will be running my customer bill up through the roof, not to mention testing my own sanity. PLEASE FIX THIS!!!!

Todd, are you actually seeing the problem where the copy from Local Folders stops mid-way when copy to g-suite? If so, how many messages in source folder and their approx. average size?
I haven't test this with Local Folders --> gmail recently to see how it behaves. (I'm assuming gmail and g-suite would behave in a similar way.)

I am going to have to do this shortly with imap.gmail.com. I have recently been put through this bug with exchange.charter-business.net. About drove me insane.

Can you describe more details of what problem you see. "This bug" describes several issues and not sure exactly which problem you had or are expecting to have in future projects.
FWIW, here I'm on charter but not charter-business. It does OK when I copy to it but messages with big headers when copied in don't show the headers in tb. All other servers display the messages OK in tb so doesn't seem to be a tb problem. Charter (aka, Spectrum) uses openwave imap server, don't know about charter-business.

Except for the short timeout of 20 seconds described in comment 127 which is still present, copy from tb local mbox files should be fairly reliable. A big problem occurs when tb has to first fetch the messages from the source server 'A' before they are sent to server 'B' using IMAP append (see around comment 100). Therefore, it is best to first make sure the source imap folder has a local copy of the messages downloaded/sync'd before doing the copy. Of course, Local Folders and POP3 folders always have a local copy but optional with imap.

I am copying folders over from local to an account on g-suite now. time out like crazy! Attachment cause it to barf

I am going to have to do this shortly with imap.gmail.com. I have recently been put through this bug with exchange.charter-business.net. About drove me insane.

Okay, anything with an attachment oer 3.0 MB will reproduce this issue

Todd, I'm still not sure of the problem you are seeing, but assuming it is the timeout of the IMAP append when copying from Local Folders to an imap folder. I am right now building a modified ESR 68 build that increases the timeout described in comment 127. I don't know your computer type so I am building win32, win64, linux64 and osx64 at the "try" server site here:

https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=38d97b3ca0e7c63771baf67893c1201a90a679c1

To obtain the executable, click on the green B next to the appropriate architecture. Then when the options appear after clicking the green B, select the "Job Details" column. You will see a long list of item most of which a not relevant. For windows you would use the "target.installer.exe" link or for linux you would use the "target.tar.bz2" link; osx uses "target.dmg".

It takes maybe an hour for the try build to complete so if you are working on this right now you will have to wait until you see a bold green B next to the computer type.

I changed mailnews.tcptimeout from 200 to 500 (500 of what, I do not know), and I am able to transfer 8 GB attachments now

I just got away with a 22 MB attachment

Probably the time in ms that tb will wait for a tcp ack response; just a guess. I didn't see that in my test but at least it's adjustable unlike the imap append timeout that I set in the try build.
Edit: It is time in seconds and it is how log tb will wait for data to be sent by the imap server measured from when the last data was sent to the server by tb.

I am doing anohter large migration today. mailnews.tcptimeout helps a lot, but some messagea have to be moved by themself.

I notice just before the time out, I get a "sending logon information" note on the lower left

(In reply to Todd from comment #139)

I notice just before the time out, I get a "sending logon information" note on the lower left

That would be expected if a connection to imap server is dropped. Tb then has to make a new connection to continue on and that requires logging on to the server.

I don't think you have mentioned the tb version you are using. If you are at >=68, Bug 1535969 added an imap keepalive setting in the config editor. I'm not sure it would affect the tcp timeouts you were maybe having.

Also, I have mailnews.tcptimeout set to 1000 now or 1000 seconds. It times out at about 15 seconds.

Thunderbird 68.4.2 64 bit for Windows

mail.imap.tcp_keepalive.enabled = true
mail.imap.tcp_keepalive.idle_time = 100
mail.imap.tcp_keepalive.retry_interval = 5

Should any of this be altered? And is idle_time milliseconds or seconds?

I think it's OK as is. The idle time is seconds meaning if no tcp activity on the imap connection for 100s a keepalive packet will be send. If no response to the keepalive, tb will be resend on a 5 second interval. Not sure how many times it will resend it if no response. (I'll have to re-read Bug 1535969.)

Here is an interesting tid bit. The customer I am working on right now had 10 Mbps down and 2 Mbps up speed. I could not move anything larger that 5 MB. I asked the ISP to max me out: 30 Mbps down and 5 Mbps up, and now I am able to transfer 15 MB files, but not 20 MB files

Todd, If you are copying emails from Local Folders or from another imap folder to a destination imap folder, you may be having a timeout waiting for the imap server to process the final response to the imap append. For big messages this can take a long time, maybe up to a minute for some servers. This can result in a timeout since the mailnew.tcptimeout gets set down to 20s for the append even if you have it set larger. So have you tried to tb version referenced in comment 134 above and see if it makes a difference? It keeps the timeout at your set value during the append.

comment 134 does not pick up my old profile and seems to be a stand alone program

had to mess with profile.ini and find the nightly exe on program files

Another way to run it is to just download the target.zip (https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/THZZ2LLrR06H8qufQr8Wvw/runs/0/artifacts/public/build/target.zip), unzip it and with a cmd window, cd into the new directory and run "thunderbird" from the cmd line. Requires no change to already install tb.

Anyhow, if you ran with the "try" version, did it help at all?

Edit: I tried the above link on win10 and I had to run "thunderbird --ProfileManager" and select the profile. Otherwise it wanted to create a new profile. Then when I went back to run the default install of tb, I had to add the parameter --allow-downgrade to the shortcut. So maybe not as easy as I claimed.

with the 134 build, I an transfer lanrge attachment one at a time. If I try too many, it times out with no warning

after running 134 on your profile, you can not go back to 68.4.0 general release. I had to install a nightly.

(In reply to Todd from comment #150)

after running 134 on your profile, you can not go back to 68.4.0 general release. I had to install a nightly.

Yes, that's why I had to add the -allow-downgrade parameter. This is an annoying "feature" that comes from mozilla code, from what I understand.

Hi Gene,

Did you recompile 134 with that switch? And if so, is it up for download yet?

Also, that was 68 ESR Proposed that told me 68.4.2 was a downgrade. Am I missing something?

-T

The "try" build referenced in comment 134 has nothing to do with -allow-downgrade switch. It only prevents the mailnew.tcptimeout from being set by the code down to 20s during imap appends. With the "try" version the timeout remains at whatever you have it set in config editor.

What I mean is that if tb complains on startup about the profile being old and telling you to make a new one (which is not really true when you switch between my "try" build of comment 134 and a recent release build) you now have to start release tb with the -allow-downgrade switch, either at the command line or in the windows shortcut for the desktop icon, e.g.,

thunderbird -allow-downgrade

If you need more info or this or what I'm saying is not clear, feel free to send me email since this is a bit off-topic for this bug. Thanks.

I just got done troubleshooting a customer with all kinds of connection issues over his cell phone hot spot trying to send, trying to copy to his sent bin, and imap socksts errors. He is running 68.5. I used comment 135 and it also fixed the issue.

See the attachment for a list of his errors.

Chuckle, I had to use -allow-downgrade to get comment 134 to work.

Hi Gene,

I have used your #134 spin several times now on various customer sites. Thank you!

Do you know which general release your wonderful modifications will be included in? And any idea when that will be?

Many thanks,
-T

(In reply to Todd from comment #156)

Hi Gene,

I have used your #134 spin several times now on various customer sites. Thank you!

Do you know which general release your wonderful modifications will be included in? And any idea when that will be?

Many thanks,
-T

Sorry, been meaning to get back to this. Glad to know it has helped you. I have submitted a separate bug for comment 134 (bug 1618455). I don't consider this a complete fix for the this bug (When attempting to copy or move large numbers of local messages to an IMAP server, it fails without any error or status message). I hope to look some more at it real soon.

TB user Xavier contacted me via email and reported that the "try" build from comment 134 is no longer is available. I did the build again and he reported that his bulk copied worked better. Here's the info I provided to Xavier for the builds:

You can find a modified 68.7 build here:
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&collapsedPushes=689680&revision=74bd64740e539f12f94b0bbfc77067f4583b5eff

You can also find a modified trunk build here:
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=fbcb811f3cd8cc9e228919d2355cab89f7ce1fd6

I built it for all architectures since I didn't know which one(s) you need. Also, it appears that 68.7 is building OK. Note: I haven't tried to run any of these builds.

To obtain the executable, click on the bold green B next to the appropriate architecture. Then when the options appear after clicking the green B, select the "Job Details" column. You will see a long list of item most of which a not relevant. For windows you would use the "target.installer.exe" link or for linux you would use the "target.tar.bz2" link; osx uses "target.dmg".

I just came across this thread and by god was I surprised to see this is an ELEVEN YEAR OLD issue.

Truth being told I had ceased to use TB definitely, now due to a project where I need to keep an eye on 20 different inbox folders ... decided to go back at TB and give it a whirl.

I started by making a backup of two inboxes previously configured on TB (with approximately 200k email messages) onto a backup inbox we use just to archive old emails.

Moving/coping or whatever in TB is excruciating. It has been one of the most brutal experiences I've gone through lately. Doesn't really matter if it's gmail, as it also happened with an outlook/hotmail account.

Emails are not copied or moved, the process breaks half way, nothing more happens. We have to move the same emails again. Duplicate email finder seems to stop working as I've erased more than 3000 duplicate emails manually.

One of the worse things I've seen tho, are the "notifications" that only show two lines and you can't see the whole notification.
Not only you can't see the only notification, it seems there isn't a single place where you can access these notifications later, with the full text.

All these made me start looking around and quickly found this thread.
It's an eleven year old issue. ELEVEN. I mean, wtf. Goodbye again.

This is a little unfair to Gene, who has been working diligently on this for the past year or so (not his fault the ten years before that). I am a little surprised this isn't getting more priority though. The fixes I'm seeing making it onto the release notes don't generally seem as urgent as this one. I would have thought that any bug that leads to data corruption and loss of e-mails would get more attention. Gene, how are decisions made about what to prioritize? Is there someone there that we could petition to make this more of a priority so you could focus on taking this over the finish line? At any rate, I'm also holding off on adopting Thunderbird for now. I'm not sure how much longer I can wait before I need to make a decision.

Joe

Thanks for your comment. I really don't want to blame Gene nor think he's the one to blame, on the contrary, kudos for him for picking this up and working diligently on it.
My rant is towards Mozilla Foundation or whomever has responsibility on this project.
There aren't many great email clients, there's definitely space for TB to claim a greater marketshare if it wasn't for the perceived abandonment of fixing serious bugs and more innovation on developing important features and the look and feel of TB.

According to the Email Client Marketshare on (emailclientshare.com) TB doesn't even show on the 10 list. I believe if this was cared for properly, could be on the top 5.

One bug it's driving me crazy and I've seen other people complaining about are the damn notifications (macOS) that don't show the whole error text and doesn't matter if you click on the notification or go through the activity manager, debuggers, anything, you can't see the errors anywhere. I can't even express how riled up it leaves me because it makes me feel like we're still in 2008.

But anyway sorry for using this topic to rant about all this, these two issues are killing me and I'm really disappointed with thunderbird.
All the best for Gene thanks for having this on your desk.

Joe, Thanks for the mailing list archive you provided. It has lots of emails with big headers and many large attachments (brain scans!) that are a good test bed that I have copied a lot. Yes, and I do need to get back onto this.
A while back I updated this addon https://addons.thunderbird.net/en-US/thunderbird/addon/copy-folder/ for TB 68:
https://github.com/smitgd/addon-copyfolder-tb-esr68 . Fixing tb to do what this does is not trivial. Unfortunately TB's built-in move/copy stuff doesn't have robust retry or duplicate prevention and there is very little feedback if something goes wrong. My addon at github only works with 68. I've never worked with addons before but managed to port the <60 one to 68. I'm not sure what it will take to change it for the next release or current betas and daily. Anyhow, the last time I checked it, the ported addon worked very well with tb 68.
I can't speak for TB priorities but I think most effort goes into keeping tb current with the latest firefox/mozilla codebase and fixing or adding calendar functions. So this is looked at as somewhat as a niche bug since it doesn't affect most typical users. At least that's why it was basically ignored when reported about 11 years ago.

I just tried my ported copy-folder addon with 68.8 and it still works. By default, it copies the folder recursively. But on the pref screen for the addon you can tell it to not copy subfolders but just the selected folder. There are a few other options that currently don't affect anything.
I've attached the addon.xpi file if anyone dare try it.

Note: This doesn't fix the tcptimeout problem that can occur with some servers (e.g., outlook.com) when the imap append response takes too long. But it will try a few more times which will also probably fail. It will then move on to the next message. The failures will be noted in the final summary. Then if you run the copy again, only the missing messages will be tried again.

Hi Gene thanks a lot for your message.

The add-on compatible with TB68 is the one at GitHub right? because this - https://addons.thunderbird.net/en-US/thunderbird/addon/copy-folder/ - says isn't compatible with my TB 68.8.1 so just to make sure. thanks

The add-on compatible with TB68 is the one at GitHub right?

Right. The official copy-folder addon was never ported to 68. The one I put on github is ported to 68 and works only with 68.*. You don't have to "clone" or download anything from github unless you want to. You can just use the attachment "addon.xpi" from comment 163 and install it from file on the TB addon manager screen. (No warranty or guarantee!)
Once installed and tb restarted, right click the folder you want to copy, select the new (Addon) copy that will now be present and select the destination folder. Oh yes, before starting the copy, click on a folder at the destination sever to make sure you have a connection to that server. This is mentioned in the readme on github.

Alright I actually have moved all the big folders already but I'll keep this in mind.
I truly hope tho, that this makes it to TB code base instead of being an "add-on".

Thanks Gene!

I'm being plagued by this problem at the moment. I've got over 18000 to copy to O365, and I just can't get consistent results. I'm going to try splitting it into folders of 1000, and copy whole folders at a time, so at least I can tell which ones failed.

I assume there are good reasons why this bug is still not fixed after 11 years. Perhaps a way forward would be to add some checks to the process so that users are at least warned there's been a failure. It would also be great if it was possible to identify which messages didn't get copied, so one could just try again.

I am several thousand emails away from manually moving tiny batches of emails to gmail over and over again because of this bug and the only thing keeping me going is the thought of never using Thunderbird again. I can't believe this bug has existed for 11 years. Cool that people have decided to fix it in the last year but it is inexcusable for a piece of software to be this fundamentally broken for so long. Honestly pathetic.

Here's the install file for my copyfolders add-on, updated for tb 78, described here: https://github.com/smitgd/addon-copyfolder-tb-esr68/tree/WL. Note: the version for TB 68 is on the "master" branch and the install file for 68 is attached above at comment 163.

Note: With 78, the add-on options are found under the Tools menu item. There is really only 1 option you can currently select: Recursive vs. non-recursive folder copy.

I would like to implement similar functionality in the mainline TB. Unlike current TB copy/move methods, this handles each copy separately so you can get somewhat finer-grained error indications.

If you are willing to try this, let me know if it seems helpful in doing the mass copies of whole folders.
Thanks and sorry for the extreme delay in getting this bug fixed.

P/S: copyfolder add-on attempts to preclude dupes in the destination folder after the copy. Another add-on tool that might be helpful is this: https://addons.thunderbird.net/en-US/thunderbird/addon/removedupes/?src=ss

(In reply to gene smith from comment #169)

Gene Smith, have you done any comparisons to compare the speed of your one by one copy with the standard copy? Obviously it's going to work out faster than doing it over and over till it's right, but it would be good to know if it's going to run more slowly.

(In reply to gene smith from comment #134)

Todd, I'm still not sure of the problem you are seeing, but assuming it is the timeout of the IMAP append when copying from Local Folders to an imap folder. I am right now building a modified ESR 68 build that increases the timeout described in comment 127. I don't know your computer type so I am building win32, win64, linux64 and osx64 at the "try" server site here:

https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=38d97b3ca0e7c63771baf67893c1201a90a679c1

In case others are finding this bug, I wanted to cross-post a comment and point to the other bug tracking it with an update.

This patch worked great! However, the change got backed out by a later commit and the treeherder artifacts expired. There isn't a way to test/use these patches at this time.
Info is on the other bug report:
https://bugzilla.mozilla.org/show_bug.cgi?id=1618455#c4

(In reply to showard314 from comment #171)

See bug 1618455 comment 5 for my proposed patch. Sorry for the delay on this.

Had this bug as well (TB 78.7.0, Win 10) - tried moving thousands of mails from local folder to an IMAP (gmail) account, and the process always stopped (waay) before completion. Now trying extension 78-copyfolder.xpi from comment #169. Will report back if it works or not.

In the meanwhile, let me note that while it is running, status/progress is strange: it restarts counting sometimes after a few hundred, sometimes after 1-2-3 mails because of indexing happening.

(In reply to csanad from comment #173)

Had this bug as well (TB 78.7.0, Win 10) - tried moving thousands of mails from local folder to an IMAP (gmail) account, and the process always stopped (waay) before completion. Now trying extension 78-copyfolder.xpi from comment #169. Will report back if it works or not.

In the meanwhile, let me note that while it is running, status/progress is strange: it restarts counting sometimes after a few hundred, sometimes after 1-2-3 mails because of indexing happening.

Seems to work close to perfectly! Copied ~60k mails from a local folder to an IMAP (gmail) account on the first try. Apparently ~1% of the mails were duplicates, and there seems to be a loss of ~0.1% - or these 0.1% of mails could not be copied somehow in the first place, or the mail count within a folder (as given in the respective column next to the folder's name in TB) is only accurate up to such precision.

I am now trying another, ~80k folder, it also goes well so far.

Note that simple copying failed 100% of times (out of 10-15 attempts), so this is a HUGE improvement, thanks a lot!

csanad,
Glad to here that the addon is helping. Yes, I noticed the counters resetting too but haven't had a chance to look at it yet. I've notice a few other problems too.
I would like to eventually port the addon to mainline tb. To me, it makes sense to right click a folder and be able to copy it one message at a time. With mainline tb, you have to drag the folder which seems to be basically undocumented and any error applies to the whole folder rather than the individual messages in the folder.
FYI, bug 1618455 is fixed and is now in tb daily and it may help with certain problems related to mass copies even when not using the addon.

Gene,
Thanks for your reply. Indeed this folder copy should be in the main TB version as well, as it is a missing feature otherwise, as you say; or even worse, the feature appears to be there but is quite buggy.
Also, now I tried it on another machine and somehow many times it just responds that X (sometimes X=0, sometimes many thousands) messages were copied, and the process was aborted. It would be good to somehow tell why it was aborted, to understand and be able to solve the issue.

Potentially related https://mzl.la/3skNjlz

This seems to be working wonderfully for me too! My one issue is that when I uploaded my inbox (it was a local folder) to my gmail imap server's inbox, it's now in a sub-folder of my gmail inbox.

Is it really easy to just copy the messages from this sub-folder since it's copying within the server and not from my computer? Or am I missing something with the recursive/non-recursive copy and should I be copying the individual messages differently somehow so I don't end up with a sub-folder on the server?

I assume you mean by "this" the addon.

Anyhow, glad to hear it is working OK for you. Again, it still needs work and may not work correctly with future releases greater than 78 due to changes in how addons works. So it will need some adaptations.

The addon does currently just copy the folder name and all internal messages to the destination and not just the internal messages. So, yes, you can just copy the messages from the new subfolder of gmail's Inbox into inbox. Then when the copy finishes you can delete the subfolder. (You could also use the move function but, just to be safe, I'd do the copy and then delete the subfolder yourself when you are sure everything copied OK.)

I'm using the addon 78-copyfolder.xpi with TB 78.11.00 (32bit) and it works without any stop. But it is very slow, like 1 email per minute.
Do I have to do the copy with TB in offline mode or there is any way to speed up the process? Is it so slow also for you?

The solution Gene gave worked perfectly for me. The way you have yours set up is exactly how mine was too but it went way faster than 1 email per minute. You might try emptying your gmail trash? I remember that caused some issues if the trash was too full—or at least I thought it did.

Also, you're highlighting just the folder and right-clicking on it to access the addon right? You're not highlighting all of your individual emails?

Last thing I can think of is that your internet speed is just really slow.

Just to update the problem seems to still be there with Thunderbird 91.1, ie to give my case exactly pretty impossible to move large amounts of messages from local folders to IMAP (Google) since timeouts very often.

With the addon and using older Thunderbird 78 from a snap (since the addon seems to install but isn't functional on 91) I seem to be able to workaround the problem of moving back a folder with 4811 messages from local to IMAP (moved away to temporarily make up space on account), even though it's indeed very slow (maybe 10 messages per minute).

(In reply to timo.jyrinki from comment #182)

Just to update the problem seems to still be there with Thunderbird 91.1, ie to give my case exactly pretty impossible to move large amounts of messages from local folders to IMAP (Google) since timeouts very often.

Must be somewhat network or location dependent. I've been working on another bug where I've copied more that 5G of messages to gmail Inbox with no timeouts using the built-in tb menu (not my addon).

With the addon and using older Thunderbird 78 from a snap (since the addon seems to install but isn't functional on 91) I seem to be able to workaround the problem of moving back a folder with 4811 messages from local to IMAP (moved away to temporarily make up space on account), even though it's indeed very slow (maybe 10 messages per minute).

I haven't attempted to update or even test my addon recently to see if it works with 91 or later. I'll try to get to it ASAP.

TB 91.2.0 64-bit on MacBook Pro just recently purchased - the problem is still there. I'm trying to migrate a user from one G-Suite based office to ours (also G-Suite). This user has 64,000+ messages in her Inbox and 92,000+ messages in her Sent folder that date back to Jan. 2015. It's imperative I get these emails copied to her new G-Suite account a.s.a.p. Both G-Suite accounts are IMAP so we can't "move" the emails - they have to be copied so we can leave the originals in the source account. The Copy-To option in TB only runs for about 5 minutes before it stops without any error message. Now when I try to pickup where it left off it won't copy any messages at all. PLEASE . . . is there an addon for this version to make this work. I'm running out of time.

(In reply to John K. from comment #184)

TB 91.2.0 64-bit on MacBook Pro just recently purchased - the problem is still there. I'm trying to migrate a user from one G-Suite based office to ours (also G-Suite). This user has 64,000+ messages in her Inbox and 92,000+ messages in her Sent folder that date back to Jan. 2015. It's imperative I get these emails copied to her new G-Suite account a.s.a.p. Both G-Suite accounts are IMAP so we can't "move" the emails - they have to be copied so we can leave the originals in the source account. The Copy-To option in TB only runs for about 5 minutes before it stops without any error message. Now when I try to pickup where it left off it won't copy any messages at all. PLEASE . . . is there an addon for this version to make this work. I'm running out of time.

If it's Google workspace to workspace migration why not just use the Migration service from Google? I have used it for exactly that purpose and it worked well.

https://support.google.com/a/answer/9476255

I never got the TB copying to work without errors (with moving it at least deletes the original so you know where to restart but i understand you want to just copy not delete).

Thanks Markus - I would have used the migration tool as well (have had success in the past using to migrate nearly 50 users in the past), but it was imperative that the admin of the source account/domain not be aware of the "move" (it's somewhat sensitive). Since I couldn't guarantee her current admin wouldn't know of the migration I went with the TB option instead.

John K.
You might try my addon in comment 169 above. I haven't gotten back to adapt this to 91 so you will need to temporarily downgrade to 78. (You will need to run 78 wih --allow-downgrade command line option unless you make a new profile since you have already run 91 with the profile.)

Gene, shouldn't your "addon" be integrated to the core? copy large ammounts of mails is just a basic-feature of a software. are you part of the development-team? if not, is someone of them reading here?

I have to admit that I'm somewhat of a novice admin here. That being said, I'm unaware of how one might implement a command line instruction on a Mac Powerbook Pro.
Minor update: the copy-to function seems to be working with small sets of of messages. Keeping to message count down to 250 at a time seems to be working just fine. Downside . . . I'll have to do this 625 times to get all her emails :-(

(In reply to Comfine GmbH from comment #188)

Gene, shouldn't your "addon" be integrated to the core? copy large ammounts of mails is just a basic-feature of a software. are you part of the development-team?

I'm an unpaid volunteer and I mostly just look after the low level IMAP issues when they come up. Although bulk copy/move involves IMAP, it also reaches into to all the other protocols and involves a lot of UI stuff that I'm not a specialist on.

if not, is someone of them reading here?

If you look at the CC list above you will see mkmelin and wsmwk. They makes most of the decisions as to what has priority to be added or fixed in tb by paid staff. Apparently they don't deem this an issue that requires action (even though it has been a problem for at least 12 years with comments still coming in).

Also, thanks for letting me test with your server a while back!

(In reply to John K. from comment #189)

I have to admit that I'm somewhat of a novice admin here. That being said, I'm unaware of how one might implement a command line instruction on a Mac Powerbook Pro.

I'm not sure either. I have an old "Air" that I haven't fired up in over a year with "Mavericks" OS. I suspect it's similar. I'll take a look and let you know how it's done.

Minor update: the copy-to function seems to be working with small sets of of messages. Keeping to message count down to 250 at a time seems to be working just fine. Downside . . . I'll have to do this 625 times to get all her emails :-(

Ugh...

John K.
I'm assuming you know how to find and install the 78 version of tb on mac. Since you have already installed and run 91, if you try to run 78 you will be requested to create a new profile to continue. You can do that if you want but you will have to download all the messages in the accounts again. So to avoid this, you need to use the --allow-downgrade option on tb 78.

You can run the installed tb 78 by going to "Utilities" and running the "Terminal" app. In the terminal app you can run thunderbird like this:
open /Applications/Thunderbird.app --args --allow-downgrade.

There seems to be no "easy" way to add a command line option to the TB icon you normally click to run TB on macOS/OSX. So you have to run tb from a terminal.
There may be other ways but it involves writing "scripts" and running them as apps, but this seems to be the easiest for just a temporary use.
Re: https://superuser.com/questions/16750/how-can-i-run-an-application-with-command-line-arguments-in-mac-os

Edit: You can get the last 78 versions here (Assuming en-US locale): https://archive.mozilla.org/pub/thunderbird/releases/78.14.0/mac/en-US/
I installed from file my addon in comment 169 and ran it in terminal with "open /Applications/Thunderbird.app --args --allow-downgrade" and it worked OK with v78.14 when I copied a folder to an icloud account. Actually, I had never tried running the addon with macos until now.
FWIW, on this old macbook air with "mavericks" the os is too old and tb 91 won't install but 78 still does.

If someone can pinpoint what's wrong, it would be nice to fix. What does the add-on do differently that fixes it?

Magnus, It's been a while but I think the main problem with tb built-in copy is that it sends all the imap UIDs in one big append chunk. So if it fails for some reason, you end up with just a partial copy to the destination and usually it just stops with no error or warning to the user. If you start it again, you get duplicates. A couple years ago I tried to understand and fix it but it was beyond me, so I found an old addon that had not been updated to new TB and adapted it.

With the addon, each individual message is handled atomically and you get an error message if there's a problem but it retries a few times before giving up on the single message copy. (I don't remember if it continues on to the next messages if one fails.) Then if you restart, it tries not to re-copy what has already been copied so it just fills in any missing messages.

Thank you everyone - the response here has been awesome - far better than I had hoped it would be. Gene - a special thanks to you and your continued contributions here! Being a novice and having so much time already invested in this, I'm going to stick with what I know is working. On a positive note, I'm up to copying about 500 emails at a time without a timeout/stall so I've just cut my number of copy groups in half :-)
Less than 300 groups of 500 to go. I run a group of 500 every 10 min.s or less (or about 50 in an 8-hour day shift). I'm sneaking these in between regular work so I should be able to put this in the done pile by the end of next week (sooner if I keep it going over the weekend).

I've updated the addon to now work with 78, 91 and later (tested with 78, 91 and 97a daily). https://github.com/smitgd/addon-copyfolder-tb-esr68/tree/WL
This is branch WL (Window Listener). Only the "master" branch works with tb 68.

(In reply to gene smith from comment #196)

I've updated the addon to now work with 78, 91 and later (tested with 78, 91 and 97a daily). https://github.com/smitgd/addon-copyfolder-tb-esr68/tree/WL
This is branch WL (Window Listener). Only the "master" branch works with tb 68.

HI Gene,

Is there an XPI out there of your add-on somewhere?

Edited comment:

Is there an XPI out there of your add-on somewhere?

I submitted it to the tb addon site and it still says "awaiting reveiw". Not sure what that means but this does seem to work:
https://addons.thunderbird.net/EN-US/thunderbird/addon/copyfolder-for-esr-78/

Submission was rejected since the addon uses legacy API and a "Window Listener Experiment" to adapt it. I didn't know that they require new addons to use "Mail Extension" API. So I just put the working XPI here for now: https://github.com/smitgd/addon-copyfolder-tb-esr68/releases/tag/v2.2.2

Just wanted to note for all concerned that this behavior still exists in the latest version of Thunderbird, 102.7.1.

Same thing that everyone else has described over the past 10+ years. If you select a bunch of messages and try to move them to another IMAP folder, the operation will begin and some number of messages will be copied, but then the operation just... stops. There's no error message, no indication of what has been copied and what hasn't, and generally if you retry it a couple of times you will end up with a bunch of duplicates in the destination mailbox/folder.

The type of IMAP server on the far end does not seem to matter. I have seen it happen with Gmail's IMAP implementation and Dovecot running on my local server (across several versions of Dovecot at this point). I have also watched the Dovecot logs as I tried a big copy, and there is nothing in there to indicate an error on the destination side.

I can only imagine that more and more users will run into this issue, as (at least in my experience, and my friends/colleagues') more people have very large IMAP accounts today than 10+ years ago, and there are now people—myself included—who are running into the limits of free storage on services like GMail and are looking to migrate their messages, often many GB, out and onto other services/servers or self-hosted setups. Without the ability to do large copy operations, users are stuck using stuff like Google Takeout and then dealing with the resulting mbox files directly.

Well, just to show how difficult it is to run tests using valgrind, here is an example of an incomplete test.
I reported Bug 1824691, it is about a runtime error when C-C TB is compiled using gcc-12 compiler suite.
I thought it was a simple miscompilation. (It could be.)
But then I suddenly recalled that there WAS a mysterious valgrind memory error reported in the hashtable implementation and it had something to do with string allocation some years ago.
And hashtable is in the stack in bug 1824691 and the error occurred int the string allocation.
So I thought there may exist uninitialized memory issue which may happen due to the particular manner the code is compiled.
It may happen sometimes and it may not show up sometimes depending on the manner the code is compiled/optimized.

So I tried to run the particular test under valgrind, but my attempt to test it failed even before the code hits the problematic assertion.
See below.
The point is

• the test involves a future activation of an event. It specifies 30 seconds for a future event, it seems.
  Usually, everything else finishes by then, and TB and the test framwork will wait for this event that happens at 30 seconds into the future.
  However, valgrind slows down the execution of involved program(s) to the point that the test framework still waits for "brower"(?) when 30 seconds passed. And indeed the test screen has an outline of the rectangle of TB window but only the frame lines only, nothing else, it is black empty rectangle with only the visible frame. That must be the so-called "browser" screen (in which, actually TB's content is displayed.)
  I wonder what would happen to the set alarm.
  Maybe when everything becomes ready under valgrind, the timer has already fired and TB cannot seriously wait for that event any more.
  Since nothing happens soon, I went to sleep.
  And this morning I found the whole test timed out.
• The runtests.py leaves room for improvement for timeout handling.
  I have hacked runtests.py to extend time out values here and there. But maybe the logical issue that the 30 second timer into the future in the original test may not be properly caught because the TB executes very slowly under valgrind needs to be addressed first.

So in order to run test(s) under valgrind in reasonably satisfied manner, various timeout values in TEST java code files themselves,
C/C++ source code, and runtests.py need to be adjusted. (I did it for |make mozmill| test framework of C-C TB by trials and errors several years ago, and it worked rather well for a few years. But new mochitest framework for C-C TB is very difficult to handle in this regard. I have run many of the xpcshell tests under valgrind, but for mochitests, I found too many tests timed out and so I could not get meaningful test coverage. This has been the state of affairs since C-C TB transitioned to mochitest framework a couple of years back. For mochitest, runtests.py seems to require much hack as far as C-C TB timeout issue is concerned. AND tests themeselves.
I tried address-sanitizer instead to check memory issues.
It can deal with out of bound type access issues, but cannot check for uninitialized value issues.

So unless someone who is fully employed to work on C-C TB steps in, I don't think any serious testing of C-C TB under valgrind would happen.
I don't even know if monthy run on tryserver for C-C under valgrind happens at all. I think it happens for FF, but I need someone familiar with FF development to learn more about it.

E.g., right now, I think the passing of valgrind arguments to valgrind on |mach| command line to run C-C TB under valgrind broke again [I found I cannot pass the arguments correctly any more last year], and so I had to use an outside proxy program that runs instead of C-C TB binary and adds desired valgrind arguments to an instance of valgrind that runs the original C-C TB binary with the originally passed arguments to C-C TB.
This passing worked two or three years ago for at least a few months if I recall correctly.

Oh well. Here it is the log snippet. I show the log about the scheduling of one-off event 37232 millisecond into the future happened.
This is for comm/calendar/test/browser/browser_calDAV_discovery.js which is one of the mochitest unit tests that experienced the bug 1824691.
Oh wait, I now notice another future event: " 10:25.73 GECKO(167286) [CalMetronome] Scheduling one-off event in 43341ms"

09:21.54 GECKO(167286) --167290-- failed to stat64/stat this file
09:30.86 GECKO(167286) --167290-- memcheck GC: 83263 nodes, 18636 survivors (22.4%)
09:31.82 GECKO(167286) console.debug: Calendar:
09:31.86 GECKO(167286)   [CalMetronome] Scheduling one-off event in 37232ms
09:49.45 GECKO(167286) --167290-- memcheck GC: 83263 nodes, 11726 survivors (14.1%)
09:58.40 GECKO(167286) [Parent 167290, Main Thread] WARNING: 'NS_FAILED(rv)', file /NEW-SSD/NREF-COMM-CENTRAL/mozilla/netwerk/base/nsSimpleURI.cpp:628
09:58.40 GECKO(167286) ==167290== WARNING: new redirection conflicts with existing -- ignoring it
09:58.40 GECKO(167286) --167290--     old: 0x04c04600 (__memcpy_chk_avx_una) R-> (2030.0) 0x048487d0 __memcpy_chk
09:58.40 GECKO(167286) --167290--     new: 0x04c04600 (__memcpy_chk_avx_una) R-> (2024.0) 0x04848170 __memmove_chk
09:58.42 GECKO(167286) {debug} SetSpec failed. : aSpec=messenger/otr/chat.ftl
10:04.89 GECKO(167286) --167290-- Reading syms from /usr/lib/x86_64-linux-gnu/gdk-pixbuf-2.0/2.10.0/loaders/libpixbufloader-svg.so
10:04.90 GECKO(167286) --167290--    object doesn't have a symbol table
10:04.92 GECKO(167286) --167290-- Reading syms from /usr/lib/x86_64-linux-gnu/librsvg-2.so.2.48.0
10:04.97 GECKO(167286) --167290--    object doesn't have a symbol table
10:12.99 GECKO(167286) --167290-- WARNING: Serious error when reading debug info
10:12.99 GECKO(167286) --167290-- When reading debug info from /memfd:mozilla-ipc (deleted):
10:12.99 GECKO(167286) --167290-- failed to stat64/stat this file
10:12.99 GECKO(167286) --167290-- WARNING: Serious error when reading debug info
10:12.99 GECKO(167286) --167290-- When reading debug info from /memfd:mozilla-ipc (deleted):
10:12.99 GECKO(167286) --167290-- failed to stat64/stat this file
10:25.71 GECKO(167286) console.debug: Calendar:
10:25.73 GECKO(167286)   [CalMetronome] Scheduling one-off event in 43341ms
10:39.84 GECKO(167286) --167290-- memcheck GC: 83263 nodes, 22867 survivors (27.5%)
10:41.23 GECKO(167286) [GLX] window 40006c has VisualID 0x50
10:42.14 GECKO(167286) GL_VENDOR: Mesa/X.org
10:42.14 GECKO(167286) mVendor: Unknown
10:42.14 GECKO(167286) GL_RENDERER: llvmpipe (LLVM 15.0.6, 256 bits)
10:42.14 GECKO(167286) mRenderer: Unknown
10:42.14 GECKO(167286) mIsMesa: 1
10:42.17 GECKO(167286) [Parent 167290, Renderer] WARNING: robust_buffer_access_behavior marked as unsupported: file /NEW-SSD/NREF-COMM-CENTRAL/mozilla/gfx/gl/GLContextFeatures.cpp:630
10:42.17 GECKO(167286) [Parent 167290, Renderer] WARNING: Robustness supported, strategy is not LOSE_CONTEXT_ON_RESET!: file /NEW-SSD/NREF-COMM-CENTRAL/mozilla/gfx/gl/GLContext.cpp:994
10:42.17 GECKO(167286) [Parent 167290, Renderer] WARNING: robustness marked as unsupported: file /NEW-SSD/NREF-COMM-CENTRAL/mozilla/gfx/gl/GLContextFeatures.cpp:630
10:42.57 GECKO(167286) [WARN  webrender::device::gl] Missing optimized shader source for gpu_cache_update
10:47.26 GECKO(167286) --167290-- WARNING: Serious error when reading debug info
10:47.26 GECKO(167286) --167290-- When reading debug info from /memfd:mozilla-ipc (deleted):
10:47.26 GECKO(167286) --167290-- failed to stat64/stat this file
10:47.61 GECKO(167286) --167290-- WARNING: Serious error when reading debug info
10:47.61 GECKO(167286) --167290-- When reading debug info from /memfd:mozilla-ipc (deleted):
10:47.61 GECKO(167286) --167290-- failed to stat64/stat this file
10:47.66 GECKO(167286) --167290-- WARNING: Serious error when reading debug info
10:47.66 GECKO(167286) --167290-- When reading debug info from /memfd:mozilla-ipc (deleted):
10:47.66 GECKO(167286) --167290-- failed to stat64/stat this file
10:47.67 GECKO(167286) --167290-- WARNING: Serious error when reading debug info
10:47.67 GECKO(167286) --167290-- When reading debug info from /memfd:mozilla-ipc (deleted):
10:47.67 GECKO(167286) --167290-- failed to stat64/stat this file
10:50.47 GECKO(167286) --167290-- REDIR: 0x491f5b0 (libstdc++.so.6:operator new(unsigned long, std::nothrow_t const&)) redirected to 0x483efd9 (operator new(unsigned long, std::nothrow_t const&))
13:13.37 INFO runtests.py | Waiting for browser...     <---- See, this is already close to 2:30 after the initial 30 second alarm was set.
                                                                                    or for that matter, the later  Scheduling one-off event in 43341ms.
14:33.35 GECKO(167286) --167290-- memcheck GC: 83263 nodes, 23239 survivors (27.9%)
_read(): after the while now 1680137194.8400612
_read(): after the while timed_out True
_read(): after the while timeout= 1680137194.831557
_read(): after the while output_timeout= 1680140464.303057
_read(): Timeout occurred, but should ignored for valgrind run?
_read(): stdout_reader= <Thread(ProcessReaderStdout-/NEW-SSD/moz-obj-dir/objdir-tb3/dist/bin/thunderbird, started daemon 139700052424384)>, stderr_reader=None
calling timeoutHandler(): timeout=  3700.0
calling timeoutHandler(): proc=  <mozprocess.processhandler.ProcessHandlerMixin object at 0x7f2eb849b1d0>
calling timeoutHandler(): utilityPath=  /NEW-SSD/moz-obj-dir/objdir-tb3/dist/bin
calling timeoutHandler(): debuggerInfo=  None
calling timeoutHandler(): browserProcessId=  None
calling timeoutHandler(): processLog=  /COMM-CENTRAL/TMP-DIR/tmpbr_pejfwpidlog
  File "/usr/lib/python3.11/threading.py", line 995, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/NEW-SSD/NREF-COMM-CENTRAL/mozilla/third_party/python/sentry_sdk/sentry_sdk/integrations/threading.py", line 67, in run
    return old_run_func(self, *a, **kw)
  File "/usr/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "/NEW-SSD/NREF-COMM-CENTRAL/mozilla/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 1231, in _read
    self.timeout_callback()
  File "/NEW-SSD/NREF-COMM-CENTRAL/mozilla/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 1072, in __call__
    e(*args, **kwargs)
  File "/NEW-SSD/moz-obj-dir/objdir-tb3/_tests/testing/mochitest/runtests.py", line 2917, in timeoutHandler
    traceback.print_stack(file=sys.stderr)
MochitestDesktop.handleTimeout():timeout=  3700.0
21:44.10 INFO Buffered messages finished
21:44.10 INFO TEST-UNEXPECTED-TIMEOUT | automation.py | application timed out after 3700 seconds with no output

Maybe I should re-check if my original set of patches to extend timeout values in runtests.py is still correct.
The above test did not seem to time out due to 3700 seconds timeout to be exact. I think it hit 10minutes timeout somewhere.
But I could be wrong.

(In reply to ISHIKAWA, Chiaki from comment #200)

Hi Chiaki,
You should win the prize for 200th comment on the bug! :)
Anyhow, not sure what the connection of what you describe is to this bug (imap copy between servers of many messages at a time fails with no feedback and, on restart, copies duplicate.). Something fundamentally wrong in tb or the compiler used to build tb?
-gene

(In reply to jwtuttle from comment #199)

Just wanted to note for all concerned that this behavior still exists in the latest version of Thunderbird, 102.7.1.

Yes, I wouldn't expect 102 to make any improvements on this. I haven't looked at this for over a year.
Are you saying you see this problem when copying between tb and a dovecot server on a local network?
Is the source folder using offline store? If so, tb won't have to fetch from the source server each message that is transferred to the (local?) destinations.
About how many messages and approximate total size are you trying to copy at once?

(In reply to gene smith from comment #202)

Are you saying you see this problem when copying between tb and a dovecot server on a local network?

Most of my tests have been from one IMAP server to another IMAP server (Gmail to Dovecot, or Dovecot to Gmail, or Dovecot to Dovecot), but in several of those tests the messages were synced to Thunderbird's local store, so they did not require retrieval of each message before sending them to the destination server.

Is the source folder using offline store? If so, tb won't have to fetch from the source server each message that is transferred to the (local?) destinations.

I've tried it both using messages that were synced offline, and ones that were not. It didn't seem to make a difference. The issue seems to happen with the uploading process to the destination server, not the retrieval, I think.

About how many messages and approximate total size are you trying to copy at once?

The folder that I have been using for test purposes contains around 10,000 messages. However, I've also tried copying subsets of the messages in the folder and the silent failure issue seems to happen at a few hundred, although I haven't found exactly at what point. A 500-message selection set usually seems to fail.

(In reply to gene smith from comment #201)

(In reply to ISHIKAWA, Chiaki from comment #200)

Hi Chiaki,
You should win the prize for 200th comment on the bug! :)
Anyhow, not sure what the connection of what you describe is to this bug (imap copy between servers of many messages at a time fails with no feedback and, on restart, copies duplicate.). Something fundamentally wrong in tb or the compiler used to build tb?
-gene

I was complaining about reporting how difficult to run TB under valgrind. But I might have filed the comment to a wrong bugzilla entry, I need to look at my e-mail folder to see where the message should have gone. Hmm...

In any case, I have not been able to run TB under valgrind during mochitest for a couple of years. In the past, I could in the days of |make mozmill| test framework.
I have a feeling that a few memory-related bugs have crept in and wanted to see if there are a few, or TB is free from such bugs during mochitest. This test may have suffered from such problems.

But somebody has to run TB under valgrind during mochitest, but so far I have failed due to misplaced timeout that is NOT extended properly by |mach| even if I tell it I am using valgrind (--valgrind valgrind-binary-file-name), and there can be test programs in mochitest that do not cope well with slowdown with mochitest.
Now I realize, I might have tried to post the comment to a generic entry for running TB or FF under valgrind. Sorry for the noise. I will investigate where the comment should go.

I've made a few changes in TB that may address a main issue here. I'll fork (clone) a new bug to present my proposals there rather than extending the discussion here way beyond the 200 comment mark.
See bug 1828372.

Was experiencing this bug but disheartened to find this 14 year old post. I can successfully move up to 500 emails from local folders to another IMAP account but anymore will usually fail. I do get an error message which displays briefly and is attached for info.