User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0 Build ID: 20170628123938 Steps to reproduce: I have a mailbox somewhere and multiple clients access it. A new mail gets send to that mailbox. I can see the new mail. Someone else deletes that mail. I still see that mail (sometimes the mail gets marked as read). When I press "Repair Folder" the mail disappears. Expected results: When someone else deletes an mail that mail should be removed from my message view without having to repair the folder.

Can you confirm that the e-mail disappears when: a) the /other client/ compacts the folder? (The TB folder may need to be updated first) b) you compact that folder in TB?

(I'm not sure what that has to do with compaction.) So we have two IMAP clients accessing the same mailbox. One deletes an e-mail and the second one doesn't see the e-mail disappear. Surely "repair" will solve the issue since that checks every e-mail in the folder against the server. The behaviour depends on which IMAP server you're using and what the deletion model is in the two clients. Gene, more thoughts?

(In reply to Jorg K (GMT+2) from comment #2) > (I'm not sure what that has to do with compaction.) I can confirm a problem with (delete- and other) flag updating. If the compacting-workaround helps with Kim, this is the real bug.

(In reply to Jorg K (GMT+2) from comment #2) > (I'm not sure what that has to do with compaction.) > > So we have two IMAP clients accessing the same mailbox. One deletes an > e-mail and the second one doesn't see the e-mail disappear. Surely "repair" > will solve the issue since that checks every e-mail in the folder against > the server. > > The behaviour depends on which IMAP server you're using and what the > deletion model is in the two clients. > > Gene, more thoughts? This sounds similar to WADA's comment from the "force select for charter" bug at Comment 72. However, I think that problem only occurred when imap IDLE is not supported, so forcing a select will only help then. In this case, it appears that the reporter's imap server *does* support IDLE. I just tested a server that supports IDLE and deleted a message from INBOX on TB on my desktop and with the same account with TB on my laptop, the message almost instantly went away. (Jorg, This is with that t.testov@... account using the MDaemon imap server.) I have done similar experiments with gmail on my phone and TB on desktop and that works OK too. Is it possible the reporter's imap server does not correctly support IDLE? Also, is IDLE enabled in TB (allow immediate notification)? -- I think it is enabled since IDLE command appears in the log. Also, repairing the folder seems like an drastic fix, especially if the INBOX is large. Did the reporter also try "get new mail" before repairing? Also, expunge (compact) may also cause the message to go away if IDLE is not working correctly. Restarting tb should also cause the message to go away. Another fix is to reduce the number of connections from default 5 down to 1 and restart tb. Then, after other client deletes the email, select another folder with the mouse and then go back to INBOX and see if the message goes away. This also forces a imap SELECT on the inbox folder.

I just compared the reporter's log with the log I see here when other client deletes an email in inbox. It is similar to what the reporter sees. I think an important thing is the untagged EXPUNGE response that occurs. This should tell tb that the email is deleted. However, in my log, the EXPUNGE response is followed by a "5 EXISTS" response. The previous EXISTS response before the delete was "6 EXIST". Maybe this message count reduction is also required for the message to disappear? There is only one "14 EXISTS" response in the reporter's log so the total count never goes down and maybe this is why the message remains. According to the example transaction in the IDLE rfc https://tools.ietf.org/html/rfc2177 the EXPUNGE and EXISTS are always paired.

Reporter, as you can see from Gene's detailed analysis, we really need to know which IMAP server you're using. It appears that it is not working properly or at least not "collaborating" well with TB.

(In reply to gene smith from comment #5) > I think > an important thing is the untagged EXPUNGE response that occurs. This should > tell tb that the email is deleted. However, in my log, the EXPUNGE response > is followed by a "5 EXISTS" response. The previous EXISTS response before > the delete was "6 EXIST". Maybe this message count reduction is also > required for the message to disappear? <https://tools.ietf.org/html/rfc3501#section-7.4.1> "EXPUNGE Response" | The EXPUNGE response also decrements the number of messages in the | mailbox; it is not necessary to send an EXISTS response with the | new value.

(In reply to Alfred Peters from comment #7) > > <https://tools.ietf.org/html/rfc3501#section-7.4.1> "EXPUNGE Response" > | The EXPUNGE response also decrements the number of messages in the > | mailbox; it is not necessary to send an EXISTS response with the > | new value. Right, I should have looked there too. Tb code seems to respect this and doesn't really use the EXISTS value after an EXPUNGE is detected received.

Just to be sure, I tested again but with an official release version 52.2.1 (64-bit) on fedora 25 and it works as expected. Before I was testing with a development version that I built myself.

(In reply to gene smith from comment #9) > Just to be sure, I tested again but with an official release version 52.2.1 > (64-bit) on fedora 25 and it works as expected. Before I was testing with a > development version that I built myself. The 'real' deletion works here (trunk) as well. Only (as indicated in comment 3) if the other client does only mark as deleted, it has no effect on TB.

(In reply to Alfred Peters from comment #10) > > The 'real' deletion works here (trunk) as well. Only (as indicated in > comment 3) if the other client does only mark as deleted, it has no effect > on TB. I set the other client to only mark /DELETE (no expunge). On tb under test the message goes away when "deleted" from other client. Then on other client I "undelete" the message, and it comes back on both clients. I don't see a problem here. Just to be clear, the "other client" is also tb but with delete mode set to "Just mark it as deleted". The other tb under test where I watch the message go away and come back is set to delete mode "Move it to this folder: Trash". As another test, I set both tb's to delete mode "Just mark it as deleted". When I delete on one tb the other tb show the deleted message with a red X and line through it. When I undelete on one tb the red X and line disappear on both. So tb seems to be working exactly as expected. Am I missing something? I don't understand the bug you might be seeing on Comment 3 and Comment 10. Please explain again. Thanks.

(In reply to gene smith from comment #11) > I set the other client to only mark /DELETE (no expunge). On tb under test > the message goes away when "deleted" from other client. Then on other client > I "undelete" the message, and it comes back on both clients. I don't see a > problem here. What a ####! ;-) The TB asks even with [F5] always only for flags for new messages: | > 71 UID fetch 14:* (FLAGS) | < * 7 FETCH (FLAGS () UID 13) | < 71 OK UID FETCH is now completed This led me to: <https://dxr.mozilla.org/comm-central/source/mailnews/mailnews.js#504> | [...] | // The extra select insures that new emails are automatically detected by servers | // requiring it. Also, if a server does not support IDLE, setting this to "yes" | // can insure messages are marked as "read" after being read in other email clients. | pref("mail.server.default.force_select", "auto"); Setting this to "yes" cures *my* problem (In fact, I have none, because I normally use no second client). To see if it could help "Kim", he would have to answer Comment 2 first.

(In reply to Alfred Peters from comment #12) > To see if it could help "Kim", he would have to answer ~Comment~ ~2~ first. Comment 1 of course.

(In reply to Alfred Peters from comment #12) > | pref("mail.server.default.force_select", "auto"); > Setting this to "yes" cures *my* problem ... Not available in TB 52 yet. I'm still considering uplift.

Hello, at first I'm sorry for the late response and thank you very much for your help! > Can you confirm that the e-mail disappears when: > > a) the /other client/ compacts the folder? > (The TB folder may need to be updated first) > > b) you compact that folder in TB? I've tried both and the mail was still there. > The behaviour depends on which IMAP server you're using and what the deletion model is in the two clients. The Mailserver uses dovecot. I'm not sure what you mean with "deletion model". Could you please give me a hint here. # * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS LOGINDISABLED] Dovecot ready. # * CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SPECIAL-USE > Did the reporter also try "get new mail" before repairing? Yes, didn't help. > Restarting tb should also cause the message to go away. No it didn't. > Another fix is to reduce the number of connections from default 5 down to 1 and restart tb. Unfortunately this also didn't help. > Setting this to "yes" cures *my* problem (In fact, I have none, because I normally use no second client). I haven't had this setting in my config editor and added it with "yes" as string value. Unfortunately this also didn't help. Additional: I've tested it with 52.2.0, 52.2.1 and also 45.8.0. This drives my to the conclusion the mailserver is a part or the cause of the problem. But with another mailclient (Trojitá) I don't have this problem. Even if I disable "Allow immediate server notifications when new messages arrive" for that account and press the "Get M essages" Button the mail does not dissapear. It just get marked as read.

(In reply to Kim from comment #15) > > > The behaviour depends on which IMAP server you're using and what the deletion model is in the two clients. > > The Mailserver uses dovecot. I'm not sure what you mean with "deletion > model". Could you please give me a hint here. > Deletion model or deletion mode just means what happens when an email is "Deleted". Tb provides 3 choices: 1. It is copied to Trash and expunged (permanently deleted) from inbox. This is tb default. 2. It is just marked as deleted and not expunged or copied. Tb puts red X in front and draws a line through the email summary. 3. It is just marked as deleted and disappears from the summary. In tb, 2 and 3 you can expunge the deleted emails by using the compress right-click menu. I assume the "other client" you are using to delete the mails is doing something like 1, 2 or 3 above. Do you know? > # * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE > STARTTLS LOGINDISABLED] Dovecot ready. > > # * CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE > SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT > MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS > LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN > CONTEXT=SEARCH LIST-STATUS BINARY MOVE SPECIAL-USE Yes, your imap server supports IDLE so email deleted by other client should by deleted in tb. However you must have "allow immediate email notification" enabled for IDLE command to be issued to your imap server, which I think you are doing. Just curious, is the dovecot server local or at your isp? Any chance you could provide me with a temporary guest account so I can test this? > > > Restarting tb should also cause the message to go away. > > No it didn't. That's seem weird. If the other client deleted the email, tb should re-establish the mbox states on restart and any deleted or moved/copied email should be gone or in the new place, e.g., Trash. (Just to be sure, when restarting tb you have to make sure *all* windows are closed.) > > > Another fix is to reduce the number of connections from default 5 down to 1 and restart tb. > > Unfortunately this also didn't help. Well, if tb restarted didn't help, this won't either since it seem the mbox states are not being made known to tb. Also, since you are doing IDLE, this shouldn't be needed. > > > Setting this to "yes" cures *my* problem (In fact, I have none, because I normally use no second client). > > I haven't had this setting in my config editor and added it with "yes" as > string value. Unfortunately this also didn't help. The code that supports this is not yet in the release tb so making a variable won't help. You have to do a custom build from tb repo source and you will get these config items only then. But, having IDLE working should also make this unneeded. > > Additional: > > I've tested it with 52.2.0, 52.2.1 and also 45.8.0. This drives my to the > conclusion the mailserver is a part or the cause of the problem. But with > another mailclient (Trojitá) I don't have this problem. Is "Trojita" the other client that is doing the deleting? > > Even if I disable "Allow immediate server notifications when new messages > arrive" for that account and press the "Get M > essages" Button the mail does not dissapear. It just get marked as read. I think you mentioned before about the deleted emails only getting marked as read. Is that always true or only when "allow immediate notification" is disabled (that really means don't use IDLE)?

Kim, One more "long shot" thing to try. Toggle your setting on "mail.server.default.use_condstore". I think it defaults to false. Whatever it is, set it to the opposite of what you see and completely restart tb. I have seen old bugzilla discussions on this and sometimes they mention dovecot. I see CONDSTORE mentioned in the 2nd capability response you provided.

(In reply to gene smith from comment #16) Regarding: mail.server.default.force_select: > > I haven't had this setting in my config editor and added it with "yes" as > > string value. Unfortunately this also didn't help. > The code that supports this is not yet in the release tb so making a > variable won't help. Available in TB 55 beta and TB 56 Daily.

> 1. It is copied to Trash and expunged (permanently deleted) from inbox. This is tb default. Mail gets marked as read but doesn't dissapear on the other client. > 2. It is just marked as deleted and not expunged or copied. Tb puts red X in front and draws a line through the email summary. Mails get marked as read but doesn't dissapear or gets a red X. After "repair folder" the mail does not show up again. > 3. It is just marked as deleted and disappears from the summary. For me the 3rd option I can choose is "Remove it immediately". With this the mail gets marked as read but doesn't dissapear on the other client. > However you must have "allow immediate email notification" enabled for IDLE command to be issued to your imap server [...] I have. Also the mail is nearly immediately marked as read on the other client so I'm pretty sure IDLE is working. > Just curious, is the dovecot server local or at your isp? Any chance you could provide me with a temporary guest account so I can test this? At the isp, I'll check if I can get you an account. > (Just to be sure, when restarting tb you have to make sure *all* windows are closed.) Yes, I'm 100% sure all windows were closed. Also there is always this 3-5 seconds waiting time when Thunderbird starts. I would have noticed if that didn't happen :) > Is "Trojita" the other client that is doing the deleting? No, it's just Thunderbird. I've just tried another client because I wanted to test if the client is part of the problem. > I think you mentioned before about the deleted emails only getting marked as read. Is that always true or only when "allow immediate notification" is disabled (that really means don't use IDLE)? In the last ~30 tests (with using IDLE) it was only getting marked as read. I thought there was some case where it didn't but I can't remember. > I have seen old bugzilla discussions on this and sometimes they mention dovecot. I see CONDSTORE mentioned in the 2nd capability response you provided. Yes, I've seen that too and played a lot with that setting before I've opened that bugreport. I even did let "CONDSTORE" get removed from dovecot capability response. (I've reset that setting to default before starting this bugreport). My next steps now are creating a new mailbox and test if it happens there too. If that's true I'll try to give you access to it.

(In reply to Kim from comment #19) > > Mails get marked as read but doesn't dissapear or gets a red X. After > "repair folder" the mail does not show up again. Repair folder seems to re-download all the imap info again from server and re-create the headers and email body files, I think. One thing I didn't ask: are you only storing headers locally or both headers and the full messages? This is set with "Synchronization & Storage / Keep messages for this account on this computer", just in case you don't know. If unchecked, you are only storing headers; otherwise you are storing headers and messages locally which is default for new accounts. You can also fine-tune this setting for individual folders with the "Advanced" button. Just need to know how you have this configured. > > > 3. It is just marked as deleted and disappears from the summary. > > For me the 3rd option I can choose is "Remove it immediately". With this the > mail gets marked as read but doesn't dissapear on the other client. Yes, 3 is what really happens when you configure "remove it immediately". It just marks it deleted with no expunge and it only appears to be removed. (You can actually get it back if you select 2 and restart tb. Then it will display with the red X and line then you can un-delete it.) > > At the isp, I'll check if I can get you an account. That would be great if you can. Otherwise I might try to set up a localhost dovecot server, which is probably not real easy, and may not exactly match your isp's setup. > > My next steps now are creating a new mailbox and test if it happens there > too. If that's true I'll try to give you access to it. Thanks! Let me know what you find out.

While evaluating bug 1385229 (subscription issues) I noticed that the imap server mail.runbox.com reports its server type with the ID response, like this bug, as "dovecot" (but no version returned). I had set up a test tb account on device A to work on bug 1385229. I also set up a runbox tb account on another device B. I can delete an email from inbox on device B and it disappears from device A inbox and appears in device A and B trash. Then on device B I move the email from trash back to INBOX and it reappears in device A inbox and disappears from device A trash. So with that dovecot server mail.runbox.com the problem reported by this bug does not exist.