Closed Bug 1787687 Opened 2 years ago Closed 1 year ago

Thunderbird fails to sync folders for Outlook IMAP and triggers TRYCREATE errors

Categories

(MailNews Core :: Networking: IMAP, defect)

Thunderbird 102
defect

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: cedricmartinezcampos, Unassigned)

References

Details

Attachments

(6 files)

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36

Steps to reproduce:

In Thunderbird (version 91 and 102), add a new Outlook account:

  • IMAP, outlook.office365.com, 993, SSL/TSL, OAuth2;
  • SMTP, smtp.office365.com, 993, SSL/TSL, OAuth2.

Actual results:

After login, the main tab shows several folders, all in sync but three, which appear in slanted gray (archive, sent, trash). Sending or deleting trigger TRYCREATE errors. (More details in the next comment.)

Expected results:

All the folders should sync and appear in upright black. Sending and deleting should perform satisfactorily such actions without triggering any error. (More details in the next comment.)

Attached image bug-tb-folders.png (deleted) —
Attached image bug-tb-trycreate-send-server.png (deleted) —
Attached image bug-tb-trycreate-send-client.png (deleted) —
Attached image bug-tb-trycreate-trash-server.png (deleted) —

Upon account creation (see above), the following folders appear in the main tab. In order of appearance, with original Spanish name:

  • "Bandeja de entrada" (Inbox)
  • "Borradores" (Drafts)
  • /"Enviados" (Sent)/
  • /"Archivos" (Archives)
  • /"Papelera" (Trash)/
  • Other folders

Some folders sync other don't:

  • Drafts, sent, archive and other folders appearing in upright font do sync.
  • Those surrounded by slashes appear in grayed italic font and do not sync.
  • Folders previously created in Outlook do not appear in Thunderbird and, therefore, do not sync.
  • Folders created in Thunderbird do appear and sync.

The folder names shown in Thunderbird that no not sync differ from the ones shown in Outlook. The correspondence is the following Thunderbird/Outlook):

  • "Enviados" / "Elementos enviados"
  • "Archivos" / "Archivo"
  • "Papelera" / "Elementos eliminados"

Upon sending an email, an error notification is shown, "[TRYCREATE] Folder is not found", and Thunderbird complains. The email is nonetheless sent, what fails is to store in the sent folder.

Upon deleting an email, an error notification is shown, "[TRYCREATE] The destination mailbox could not be found", and the email is not deleted.

Upon archiving and email, a folder is created under "Archivos" (Archives), where the email is stored. Meanwhile, a new folder is created in Outlook, "Archives" (note it the difference). Folders under "Archivos" in Thunderbird sync with folder under "Archives" in Outlook.

There is a problem with the default folder names in Outlook being different in Thunderbird translation related. I prefer the ones given by Thunderbird but, as the imap URI is directly taken from the folder's name, this prevents sync. As I can't seem to be able to edit the URI, I can't solve the sync problem by myself.

Aside of this, "Archivos" is a bad translation for "Archives" since it is understood as "Files", a better translation would be "Archivo" (no 's') or "Archivados".

Finally, I do not understand why folders created in Outlook do not appear in Thunderbird.

I've checked that the problem is not due to internationalization but that Thunderbird does not recognize non-commonly-used folder names (mailboxes). According to section 5.1 of IMAP4rev2 (RFC 9051) and previous revisions, the only standard mailbox name is INBOX (fully capitalized), the interpretation of all other mailbox names is implementation dependent. For instance, the name for the sent mail mailbox is usually "Sent", whereas in Outlook is "Sent Items", reason why Thunderbird does not recognize it. Although Thunderbird allows to edit IMAP's URIs, it resets them upon restart. I don't know if there is more to it, but I cannot narrow more the problem.

Help is greatly appreciated to find a workaround while this is fixed.

Attached image bug-tb-folders-fake-to-sys.png (deleted) —

I've performed more tests. My account is an institutional account. I have created a personal one and Thunderbird successfully recognized and synced all the mailboxes (folders). I therefore deduce that there is a difference in how personal and institutional accounts are managed in Outlook servers. I don't know if Outlook renames the folders for internationalization or uses standard names for the folders and gives aliases (e.g. "Sent" is mapped to "Sent Items" in English and to "Elementos enviados" in Spansih"), in any case, Thunderbird does not succeed to recognize the setup.

Moreover, within Thunderbird, I have created a mailbox (folder) with a random name and then changed it to a system name, for instance, I create a folder "asdf" and then change its name to "Elementos enviados" ("Sent Items"). Thunderbird complains about the server issuing an error, "Target folder already exists". The folder with the original fake name is kept but a new folder (in Thunderbird) is created that syncs with the server and is recognized as a system folder with the proper icon (see screenshot). The folder solely remains for the session and disappears after Thunderbird restarts.

Changing the IMAP URIs for the default folders would perhaps solve the problem, but changes in the configuration editor are somewhat ignored.

I downloaded the ES-es TB version from release archives and installed it and don't see a problem with outlook folders being grey like you see. The only difference is that I just used a plain password instead of oauth2, but don't see how that would matter.
All the the "special" folders have Spanish names as you describe with appropriate icons and all in in full color and none are grey. I can also open them with no error. Also able to send a message and it got saved in Enviados (Sent) folder
Is this a brand new account or have you used it before OK with previous TB versions? Don't know anything about "personal" vs. institutional accounts in outlook.
Make sure all the folders you want to see are "subscribed" in TB.
Also, you can explicitly designate folders in "server setting" for trash: "When I delete a message move to this folder...." then select a folder you want for trash. Then under "Copies and Folders" you can select which folder you want for Sent, Archives, Drafts and Templates. Usually on account setup the imap server indicates which folders are these "special" folders and tb using that info, but sometimes the server is wrong so you have to fix up the mistakes using "server setting" (for trash) and "copies and folders" for the others. You should be able to select any folder for any of the purposes and tb will put the appropriate icon by it.

If this doesn't help maybe an IMAP:5 log that you record while accessing the outlook server would help: https://wiki.mozilla.org/MailNews:Logging

As an experiment, I changed the sent folder from Enviados to an existing normal folder called Sent2206 and sent a message. Then "Sent" icon went to Sent2206 and Enviados now has a normal folder icon. The message went to Sent2206 and to Enviados after I sent it. I noticed before that when I sent the message there were 2 copies of it in Enviados, so apparently outlook automatically puts your sent mail in Enviados without help from TB ! So you could just disable saving Sent mail in TB if going to Enviados is OK.

My apologies for not following up, semester has just begun and I am a bit busy.

I've been doing some testing with office mates. So far, results are the same. Before I'll post results, I'll do some more testing and info gathering at my institution.

I'll keep you posted.

Regular password vs OAuth2 is not per se a problem, however few clients support OAuth2.

The Spanish names are not a problem either. As I stated, TB does not sync with "system/special" folders if they don't have the defacto name which, depending on the client, might coincide in English. Therefore the problem should be present in other languages too.

Your tests does not apply to my case since you are not using an institutional account, accounts from institutions that use Microsoft services (e.g. universities).

All the folders are "subscribed". Allow me to stress out that I use a clean setup from scratch and I do check that the default setting is «subscribed» for each folder.

The "system/special" folders are not listed in TB and therefore I cannot select them.

I followed your idea and created "Mis enviados" in TB, which then syncs with Outlook. Then I set it as the default sent folder in TB. Sending an email is successful. The email appears en the folder in TB and synced with Outlook. Strangely, a copy appears in "Elementos enviados" in Outlook (the default sent folder in Outlook that does not sync with TB). Although this could be a workaround, it does not fix the issue since Outlook's sent, trash, draft and archive folders are not synced.

I undid the setting and did another experiment. I created a folder with a random name, let's say "TempFolder", in TB (syncs with Outlook). I changed its name to "Elementos enviados", the server complains and the folder keep its original name, "TempFolder". But the folder "Elementos enviados" then appears in TB, which is recognized by Outlook as a sent folder with a green icon, and syncs with Outlook. I set this folder as the default sent folder. I sent and email, which is successfully sent, and strangely two copies of it appears in "Elementos enviados". The folder sadly disappears when TB restarts and a Sent folder appears in its place, which does not sync with anything.

I am going to ask my institution to create a "fake" account so you can test it yourself.

I have been logging as suggested, but logs keep personal information. Any specific information that you might be looking for?

As I said, this affects all the workmates with whom I have tried, only 2, a total of 3 with me.

Vitamin C, Sorry for the delay in getting back to you.

Your tests does not apply to my case since you are not using an institutional account, accounts from institutions that use Microsoft services (e.g. universities).

I think the imap server used by your school should follow internet imap standards so it should work just like regular "free" outlook that I can test here.
I did have a temporary non-free office365 account that project leader Magnus set up for me (gene@mozthunderbird2.onmicrosoft.com) but it seems to have expired again. Maybe it is more like your institutional account. But when it was still working, I never saw the issues you describe, that I recall. Maybe Magnus can renew the account?

I am going to ask my institution to create a "fake" account so you can test it yourself.

Yes, that would be the best way to test this. You can let me know the credentials to access the test/fake account via my bugzilla profile email address.

Let me ask one more question just to make sure:
This is just a pure imap account, right? You're not trying to access the 365 using a proprietary protocol like activeSync and you are not using any addons that support this such as tbSync with "Provider" addon.
I'm pretty sure your answer is "no" based on what you have posted so far, but again, just want to make sure.
(Someone on TB support site had a problem with 365, using imap I presume, and reported that the addons tbSync+Provider solved the issue but the addons are not yet working with tb 102.)

Yes, pure IMAP, specifically IMAP 4rev1 with OAuth2, nothing else.

I think that the bug to which @Wayne referred is linked to this, that is, IMAP subscription in Office 365 is tricky. (I do not know how to link people and/or issues in this platform.)

I'll keep you posted about the possibility of an email account.

Ok, good to be sure on that...
Anyhow, Magnus enabled a new outlook365 account for testing and I don't see any problems like you describe using it. So must be something special about your imap provider.
The only issues I see are:

  1. Had to use manual setup to enable oauth2 for imap and for sending (smtp).
  2. Had use TB's subscribe dialog to subscribe all but Inbox and Trash to see the other folders.
  3. When sending email, two messages get stored in Sent folder so disabled TB saving sent mail to Sent folder.
  1. Same here.
  2. From which dialog? At account setup I can instruct TB to use "Enviados" (Sent) but this is not true sent mailbox in Outlook, which is, "Elementos enviados". I can trick TB however to use it for a while.
  3. A similar thing happens here when I use "Enviados" (Sent).

Isn't the IMAP provider Microsoft?

BTW Did you receive my message?

(In reply to Vitamina Cé! from comment #17)

BTW Did you receive my message?

Yes, I received it. Thanks!
Looking forward to seeing what is going wrong (I think :))

Attached image urjc.es-account-results.png (deleted) —

Received the credentials and setup went OK. Don't see any problem like are reported such as grayed folders or failure to sync. See attachment running todays daily (self-built). I also tried it with newest release 102.3.1 with no problems.
Setup was essentially the same as with the outlook365 account Magnus provided with the same minor issues as listed in comment 15.
Don't see any TRYCREATE errors when copying or moving or sending messages.

Probably just for Magnus:
The only unexpected problem I had was with my just-built daily, when the oauth2 password entry screen popped up, daily crashed due to an assert (see below). I then shutdown daily and started my installed tb91 version and it didn't crash and I authenticated oauth2 OK. Then I restarted daily and since it didn't need to oauth2 authenticate again, it ran OK for test account. Not sure if this is a known bug like with apache server running, but my apache is stopped.
The above is probably just a question for Magnus. Is this a known bug?

Here's the daily console output for the oauth2 crash:

JavaScript error: resource:///modules/OAuth2.jsm, line 167: NS_ERROR_NOT_IMPLEMENTED: Component returned failure code: 0x80004001 (NS_ERROR_NOT_IMPLEMENTED) [nsIRequest.name]
JavaScript error: resource:///modules/OAuth2.jsm, line 167: NS_ERROR_NOT_IMPLEMENTED: Component returned failure code: 0x80004001 (NS_ERROR_NOT_IMPLEMENTED) [nsIRequest.name]
JavaScript error: resource:///modules/OAuth2.jsm, line 167: NS_ERROR_NOT_IMPLEMENTED: Component returned failure code: 0x80004001 (NS_ERROR_NOT_IMPLEMENTED) [nsIRequest.name]
Assertion failure: false (Receiving unexpected Principal), at /home/gene/mozilla/dom/ipc/ContentParent.cpp:1312

The nsIRequest.name error is known (we think it's harmless even if certainly annoying). I don't know of an assert failure for OAuth2

Nice, this is a great advance! But strange...

I checked, double checked and triple checked "Subscription" on my and other accounts several times, and the full list of folders was not showing. The last time I did this check was on TB 91 on Friday 30, now I am on TB 102 and, supposedly, Microsoft has updated his servers to end regular password support. This might be related to the OAuth2 errors.

If you look at the list of folders, you will see "Sent" and "Trash". These folders were created by Thunderbird at first run. They are NOT the system folders for sent and trashed emails, which in fact are "Elementos enviados" and "Elementos eliminados", respectively. The TRYCREATE error is issued if you do not subscribe to the system folders, because they are not what they seem to be. Perhaps archiving an email may issue a TRYCREATE error since you have no folder associated to that action.

So, now, the situation, as I see it, is perhaps as you mentioned for the onmicrosoft.com account:

(1) Subscribe to all folders; and
(2) Setup account to use these folders for regular actions.

Not performing (1) will trigger TRYCREATE errors and scare non-aware users away. Not performing (2) will lead to two sent folders: one, "Sent", where sent emails are stored when using TB and, the other, "Elementos enviados", when using Outlook's web client (similarly with "Trash" and "Elementos eliminados"), which might also shoo non-aware users. Moreover, (1) and (2) are not obvious because the setup needs different dialogs (three, at least), which again is non-lazy-user friendly.

Anyhow, I am very happy to see that, finally things are working, although it might not be immediate for a new/regular user. Still wondering about the "subscription" though.

(In reply to Vitamina Cé! from comment #21)

Nice, this is a great advance! But strange...

I checked, double checked and triple checked "Subscription" on my and other accounts several times, and the full list of folders was not showing. The last time I did this check was on TB 91 on Friday 30, now I am on TB 102 and, supposedly, Microsoft has updated his servers to end regular password support. This might be related to the OAuth2 errors.

For me I could see all the possible folders to subscribe to. For the attachment above I subscribed to all of them and they show up (even though I don't know what many of them mean :)). If they don't show up after you subscribe, collapse and expand the folders of the top level account name using the "carrot" icon. If that doesn't work, restart tb and they will appear.

If you look at the list of folders, you will see "Sent" and "Trash". These folders were created by Thunderbird at first run. They are NOT the system folders for sent and trashed emails, which in fact are "Elementos enviados" and "Elementos eliminados", respectively. The TRYCREATE error is issued if you do not subscribe to the system folders, because they are not what they seem to be. Perhaps archiving an email may issue a TRYCREATE error since you have no folder associated to that action.

I had to do the following to get everything looking OK:

Eliminate "Sent" folder and have only sent icon on "Elementos enviados":
In Folder and Copies, select "When sending a message ... Place a copy in" checkbox
Select "Other" and then select folder "Elementos enviados"
Go back to main screen and delete folder "Sent"
Go back to Folders and Copies and un-select checkbox "Place a copy in" so only one copy appears in "Elementos enviados"
Back on main screen, empty Trash

Make trash folder "Elementos eliminados":
In Server settings, select "Elementos eliminados" for "When I delete a message move it to this folder"
Back at main screen, trash can icon should appear on "Elementos Eliminados"
Delete folder "Trash" and empty new trash "Elementos Eliminados"

Make sure Drafts and Archives (Borradoras and Archivo) are set the same way on Folders and Copies page.
Once you select them to the Spanish names, the English folder names Drafts and Archives, if present, can be deleted and emptied from "trash".

Not sure, but since you are using a Spanish version of TB if all this is necessary but I'm using the English version right now so I'm getting a mixture of language names showing up.

So, now, the situation, as I see it, is perhaps as you mentioned for the onmicrosoft.com account:

(1) Subscribe to all folders; and
(2) Setup account to use these folders for regular actions.

Not performing (1) will trigger TRYCREATE errors and scare non-aware users away. Not performing (2) will lead to two sent folders: one, "Sent", where sent emails are stored when using TB and, the other, "Elementos enviados", when using Outlook's web client (similarly with "Trash" and "Elementos eliminados"), which might also shoo non-aware users. Moreover, (1) and (2) are not obvious because the setup needs different dialogs (three, at least), which again is non-lazy-user friendly.

Before I did the steps above to set the archive to "archivo", I did see an error on my first archive attempt that "Archive" folder didn't exist. However, it seemed that outlook server went ahead and created it and emails were moved to Archives/2022/ folders like they should. When I pointed TB to the Spanish "Archivo" folder, there was no error pop-up. I wasn't recording an IMAP log so I'm not sure exactly what happened.

Anyhow, I am very happy to see that, finally things are working, although it might not be immediate for a new/regular user. Still wondering about the "subscription" though.

Again, the subscription seemed to work OK from here.

I think the problem is that outlook365, unlike most other providers, has all folder unsubscribed by default. So TB can't tell which are the "special" folders like trash, archives, drafts, templates etc. So I recreated the account using this procedure to auto-discover the special folders:

On account setup, after entering name, email addr and password, click "Continue"
Then on next page, click "Configure manually"
Change "Normal password" to "Oauth2" for both Incoming and Outgoing servers and then click "Advanced Config"
Answer the warning as "OK"
Under the new account settings, select "Server setting" and then click the "Advanced..." button about half way down the page
On advanced screen, un-check "Show only subscribed folders" then OK
Return to the main TB screen
In the new account, click on Inbox (or Spanish inbox name). You may get an Oauth2 pop-up at this point if brand-new setup.
If all folders don't appear, collapse and expand the folder list using "carrot" icon, then they should.
All "special" folders will have the appropriate icon and you will see all the pre-existing folders that outlook has.
Go into TB's subscribe dialog and unsubscribe folders you are not interested in. Don't unsubscribe any "special" folders like Spanish inbox, trash, sent, archive, drafts etc unless you really don't want them, except definitely need inbox so never unsubscribe it (not sure you can actually).
Go again to Server Settings, Advanced... and re-select/check "Show only subscribed folders" and click OK
Go back to main TB screen and again collapse and then expand the folders from the top level using the "carrot" icon.
The now unsubscribed folders will disappear while all others remain.
Done.

Well the idea is simple but still a lot of steps. But probably easier than manually pointing TB to the "special" folders like described in my previous comment 22.

As it was once said, "Microsoft products tend to not be full IMAP compliant and add limitations on top of IMAP".

Although "showing all the folders" might help if folder discovery is not working properly, it is perhaps excessive, but still a good idea.

Note that there is an extra benefit to folder subscription, it is persistent. On one of my PCs, I deleted ~/.thunderbird/ and ~/.cache/thunderbird/ to start anew. After initial account setup, TB automatically subscribed to all the folders I had previously subscribed to (I infer that the server stores information about folder subscription), and, aside of "Archivo" (Archive) and "Plantillas" (Templates), all were correctly mapped.

In general terms, the way to go is, as you say, to map manually the system folders, either after subscription or after showing all folders. This is in fact described at Strandford's helpdesk. The initial problem was that folder discovery was not working for some reason (TB, Outlook/Office365, IMAP, admin), but now that it does, folder mapping allows correct operation.

There is one last detail. After sending an email, two copies will be stored in the sent mail folder "Elementos enviados" ("Sent Items"). One is the copy put by TB, the other is a copy put by the server. Every email sent through Office365's SMTP is copied to the sent mail folder. Therefore, the solution is to simply uncheck "Put a copy in" at the "Folder and copies" dialog.

I don't know if all of this can be automated by TB or if it is worth it, nonetheless it should be documented. A regular user must surely won't know what to do if not. I also recommend to revamp the configuration dialog. Folder discovery, subscription and mapping are scattered all around. They should be all together and, from my point of view, "Copies" should be apart.

BTW, do you still need the test email account?

BTW, do you still need the test email account?

It's always good to have a "temporary" account provided by users remain active for when more problems come up on that server type. But if you need to disabled it, I understand, no problem.

Yes, I see that a full setup of o365 in TB is a pain. I guess that's why having a central "sys-admin" to provide a site specific setup is good. (Maybe that's what you are for your University.) There's also an option to not use imap but use Exchange protocol and maybe activeSync in TB but I think they require some special and maybe non-free addons. So this potentially adds more documentation variations.

I don't know if all of this can be automated by TB or if it is worth it, nonetheless it should be documented.

Perhaps some special steps could be added to the TB's account manager that are specific to o365 that would help. One would be to always select oauth2 for sending and receiving. Maybe also force an initial subscribe to all folders. But I don't work in that area so that would be up to TB's project managers. Some are CC'd on this bug.

BTW, do you still need the test email account?

Actually, I may need it a while longer for another bug. It may require an account with the "Junk" folder in another language causing a problem. And your o365 account is all setup and might make debugging bug 1793599 easier.

(In reply to gene smith from comment #26)

BTW, do you still need the test email account?

Actually, I may need it a while longer for another bug. It may require an account with the "Junk" folder in another language causing a problem. And your o365 account is all setup and might make debugging bug 1793599 easier.

I can duplicate the bug on my local servers so don't need the account for that.

Hi, Gene! My apologies for not getting back for such a long time. I got trapped by the semester.

I've written a guide to configure Office 365 Enterprise accounts (in Spanish). After recent updates on both sides (Microsoft/Thunderbird), things are almost similar as they were some time ago (see Standford's guide, in English). I would simply remark that to save a copy of sent emails in Thunderbird is not needed when sent through Microsoft's SMTP server since, in such case, the server itself takes care of the copy.

Thank you very much, Gene, for the kind help.

Status: UNCONFIRMED → RESOLVED
Closed: 1 year ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: