Closed Bug 594106 Opened 14 years ago Closed 11 years ago

Email read status not synced with Gmail after some time that Thunderbird has been open

Categories

(MailNews Core :: Networking: IMAP, defect)

Product:
MailNews Core
Component:
Networking: IMAP
Version:
1.9.2 Branch
Platform:
x86
macOS
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED DUPLICATE of bug 693204

People

(Reporter: ehsan.akhgari, Unassigned)

References

(Blocks 1 open bug)

Details

This has happened multiple times for me.  I have an IMAP Gmail account set up.  After a while that Thunderbird is left open, it stops syncing the read status of emails for my folders, which means that if I read an email through Gmail's web interface, or Thunderbird on my other machine, the original Thunderbird instance fails to mark that message as read.  Things work correctly when I restart Thunderbird and go to the folder.

Not sure what the correct component for this bug is, just filing it in Mail Window Front End for now.
Component: Mail Window Front End → Networking: IMAP
Product: Thunderbird → MailNews Core
QA Contact: front-end → networking.imap
Version: 3.1 → 1.9.2 Branch
Ehsan could you log AutoSync , and imap and give us the log when statuses aren't updated ?
(In reply to comment #1)
> Ehsan could you log AutoSync , and imap and give us the log when statuses
> aren't updated ?

Of course, but I don't know how to do that!
(In reply to comment #3)
> https://wiki.mozilla.org/MailNews:Logging should have the info you need.

Sorry I thought you knew ...
OK, running with the logging option turned on now...  I'll attach a log when I see that the problem has started to happen again.
I ran Thunderbird with:

NSPR_LOG_MODULES=ImapAutoSync:5,timestamp NSPR_LOG_FILE=/tmp/tb-imapautosync.log /Applications/Lanikai.app/Contents/MacOS/thunderbird-bin

But nothing seems to be going into the log file...  Am I missing something?
Do you have those imap folders set to be checked for new messages? (folder properties dialog). Otherwise, we're not going to go out of our way to keep the counts in sync. Autosync will occasionally check (at most, once per hour), and if you have a cached connection to an imap folder, and the server supports IDLE, we'll get informed of changes, until the connection times out (gmail I think times out after 10 minutes, but it could be less).
(In reply to comment #7)
> Do you have those imap folders set to be checked for new messages? (folder
> properties dialog). Otherwise, we're not going to go out of our way to keep the
> counts in sync. Autosync will occasionally check (at most, once per hour), and
> if you have a cached connection to an imap folder, and the server supports
> IDLE, we'll get informed of changes, until the connection times out (gmail I
> think times out after 10 minutes, but it could be less).

I do have that setting enabled.  After a couple of hours, this is what I have in my log:

deliver mode: 7
deliver mode: 7
deliver mode: 7
deliver mode: 7
deliver mode: 7
deliver mode: 7
deliver mode: 0
2010-09-10 14:14:16	autosyncActivities	ERROR	onDownloadCompleted: TypeError: this._syncInfoPerFolder[folder.URI] is undefined
deliver mode: 0
deliver mode: 0
deliver mode: 7
deliver mode: 7
deliver mode: 0
deliver mode: 7
deliver mode: 7
deliver mode: 0
deliver mode: 0

The error looks intriguing!  :)
2010-09-10 14:14:16    autosyncActivities    ERROR    onDownloadCompleted:
TypeError: this._syncInfoPerFolder[folder.URI] is undefined

well, it may or may not be intriguing - that's from the activity manager, which is relatively harmless.

an IMAP log, not an autosync log, is more likely to be interesting, so

NSPR_LOG_MODULES=IMAP:5

In theory, every time biff (check for new mail) fires on your gmail inbox (depending on what your biff interval is), we'll do a STATUS on all the folders you've set to get checked for new messages. Also, clicking get new mail will cause the same thing.
(In reply to comment #10)
> 2010-09-10 14:14:16    autosyncActivities    ERROR    onDownloadCompleted:
> TypeError: this._syncInfoPerFolder[folder.URI] is undefined
> 
> well, it may or may not be intriguing - that's from the activity manager, which
> is relatively harmless.
> 
> an IMAP log, not an autosync log, is more likely to be interesting, so
> 
> NSPR_LOG_MODULES=IMAP:5

I'll try that.

> In theory, every time biff (check for new mail) fires on your gmail inbox
> (depending on what your biff interval is), we'll do a STATUS on all the folders
> you've set to get checked for new messages. Also, clicking get new mail will
> cause the same thing.

You mean I should get the same result if I click on Get Mail?
> You mean I should get the same result if I click on Get Mail?

I mean that clicking get new mail should make us update the counts for folders that you've set to be checked for new messages. I.e., the automatic check is the same as the manual check (but not the same as clicking on the imap inbox)
So, I have a log which should demonstrate the problem.  I'm not sure if the log contains private data or not.  If it does, I can mail a link to it to the developer(s) who want to investigate this issue.  Otherwise I can include the link in a comment here.

(Warning: the log is huge, 11MB compressed, 255MB uncompressed)
Ehsan, it most likely does contain private info - why don't you e-mail me a link.
Following is log for next operation with Gmail IMAP.
(0) IDLE and "automatic mail check" is disabled for ease of check.
(1) Restart Tb, Open Inbox, Mark folder read. Largest UID=11885
(2) At Gmail Web, add Gmail Label of AAA to 7 mails in Inbox.
(3) At Gmail Web, select all mails of Inbox, Mark As Unread.
(4) At Tb, click Gmail IMAP account, Get Mail
    ==> Following log is obtained.
        Gmail looks to return Flags of mail of highest UID.  
        If Inbox is current folder at folder pane,
        only "UID fetch 11886:* (FLAGS)" is issued by Tb.
    Mail of UID=11885 only is marked Unread automatically,
    and unread mail count of Inbox is set to 1 at folder pane
    This is consistent with with IMAP log.
> [5556] 396[4dfcf40]: 7895800:imap.gmail.com:S-INBOX:ProcessCurrentURL: entering
> [5556] 396[4dfcf40]: 7895800:imap.gmail.com:S-INBOX:ProcessCurrentURL:imap://yatter%2Eone%40gmail%2Ecom@imap.gmail.com:993/select%3E/INBOX:  = currentUrl
> [5556] 396[4dfcf40]: 7895800:imap.gmail.com:S-INBOX:SendData: 49 noop 
> [5556] 396[4dfcf40]: ReadNextLine [stream=80bdbe0 nb=15 needmore=0]
> [5556] 396[4dfcf40]: 7895800:imap.gmail.com:S-INBOX:CreateNewLineFromSocket: 49 OK Success 
> [5556] 396[4dfcf40]: 7895800:imap.gmail.com:S-INBOX:SendData: 50 getquotaroot "INBOX" 
> [5556] 396[4dfcf40]: ReadNextLine [stream=80bdbe0 nb=24 needmore=0]
> [5556] 396[4dfcf40]: 7895800:imap.gmail.com:S-INBOX:CreateNewLineFromSocket: * QUOTAROOT "INBOX" "" 
> [5556] 396[4dfcf40]: ReadNextLine [stream=80bdbe0 nb=37 needmore=0]
> [5556] 396[4dfcf40]: 7895800:imap.gmail.com:S-INBOX:CreateNewLineFromSocket: * QUOTA "" (STORAGE 101774 7677003) 
> [5556] 396[4dfcf40]: ReadNextLine [stream=80bdbe0 nb=15 needmore=0]
> [5556] 396[4dfcf40]: 7895800:imap.gmail.com:S-INBOX:CreateNewLineFromSocket: 50 OK Success 
> [5556] 396[4dfcf40]: 7895800:imap.gmail.com:S-INBOX:SendData: 51 UID fetch 11886:* (FLAGS) 
> [5556] 396[4dfcf40]: ReadNextLine [stream=80bdbe0 nb=40 needmore=0]
> [5556] 396[4dfcf40]: 7895800:imap.gmail.com:S-INBOX:CreateNewLineFromSocket: * 67 FETCH (UID 11885 FLAGS (NonJunk)) 
> [5556] 396[4dfcf40]: ReadNextLine [stream=80bdbe0 nb=15 needmore=0]
> [5556] 396[4dfcf40]: 7895800:imap.gmail.com:S-INBOX:CreateNewLineFromSocket: 51 OK Success 
(5) At Tb, click folder of AAA, in order to see read/unread status is propagated to all Gmail Labels.
    => Unread count=7 as expected.

To know Flag change of all existent mails upon each new mail check, "UID fetch 1:* (FLAGS)" is required upon each mail check. New mail check is new mail check, not re-syncronization of flags of all mails.

"UID fetch 1:* (FLAGS)" is issued upon IMAP folder open(select "folder", then fetch for flags of all mails. re-synchronization).
AFAIK, there is no way to force closing of selected folder at an cached connection in order to force re-select and re-synchronization(no UI for it, and as CLOSE forces expunge, CLOSE is probably not used by Tb).
If other folder than Inbox is selected at a cached connection by auto-sync, message/junk filter, MsgPurge etc., "select Inbox followed by UID fetch 1:* (FLAGS)" is issued upon next mail check or next folder open of Inbox.

"Max cached connection count=1" is a way to force "select Inbox + UID fetch 1:* (FLAGS)" upon each new mail check.
"UID fetch <uid_next>:* (FLAGS)" is used when folder is already opened(select is already issued), because STATUS can't be used for selected folder.
Note:
Tb sometimes issued "CHECK" before getquotaroot "folder" and "UID fetch <uid_next>:* (FLAGS)".
It may be a reason why progress bar is displayed for relatively long time upon folder click, because "check" usually takes long. 

If folder is not opened(==not selected), STATUS is issued at a cached connection for other folder or connection of not "Selected Status". As UNSEEN count is returned, unread mail count at folder pane is refreshed upon new mail check. 
> imap.gmail.com:S-AA:SendData: 21 STATUS "AB" (UIDNEXT MESSAGES UNSEEN RECENT) 
> imap.gmail.com:S-AA:CreateNewLineFromSocket: * STATUS "AB" (MESSAGES 7 RECENT 0 UIDNEXT 8 UNSEEN 0) 
> imap.gmail.com:S-AA:CreateNewLineFromSocket: 21 OK Success
And, as folder is not opened, "UID fetch 1:* (FLAGS)" is issued upon folder open, so mail status of all mails is sync'ed by folder open. (ordinal re-syncronization).

Is "select XXX at connection where XXX is already seleced" permitted?
Is there any way to change a folder to "unselected status" at a conection in order to use STATUS, by other than CLOSE, without selecting other folder?
Third "interval" is needed for "flag re-sync", in adition to "new mail check interval" and "interval or sleep time of auto-sync"?
A way to manual "flag re-sync" is sufficient?
To bug opener:

Is your comment #0 for opened IMAP folder? Or for not-opened IMAP folder?
Bug 581707 can happen if something wrong happens on a folder during new mail check, and bug is probably fixed by Tb 3.1.3.
What is your Tb version?
(In reply to comment #17)
> To bug opener:
> 
> Is your comment #0 for opened IMAP folder? Or for not-opened IMAP folder?

It happens on opened folders as far as I can tell.  But closing them and reopening again doesn't help.

> Bug 581707 can happen if something wrong happens on a folder during new mail
> check, and bug is probably fixed by Tb 3.1.3.
> What is your Tb version?

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.10pre) Gecko/20100911 Lanikai/3.1.5pre
(In reply to comment #18)
> But closing them and reopening again doesn't help.

Inbox? Or other than Inbox?

Connection for Inbox looks always kept if max cached connections is greater than or equal 2. I couldn't observe "select <folder>" at cached connection for Inbox and "select Inbox" at a cached conection for folder, by folder switch form Inbox to other and vice versa.
I could observe "unselect of Inbox"(select of other than Inbox at connection for Inbox) and "re-sync of flags of Inbox by select + uid 1:* fetch flags upon Get Mail" by next test only with Gmail.
  Max cached connections = 1
  At window-1, open Inbox, mark all mails in Inbox as Read
  => Inbox is selected
  Copy all mails in Inbox to AA.
  Open new Tb window-2 via "Open" of context menu of account, open AA.
  => AA is selected at only connection, thus Inbox is unselected.
  At AA, mark all mails Unread (mail in Inbox is changed to Unread by Gmail)
  Get Mail
  => "select Inbox, uid 1:* fetch flags" at only connection,
     then unread count is refreshed at folder pane,
     and at window-1, all mails in Inbox is shown as Unread at thread pane
(In reply to comment #19)
> (In reply to comment #18)
> > But closing them and reopening again doesn't help.
> 
> Inbox? Or other than Inbox?

I've mostly observed this with folders other than Inbox.
Do you enable IDLE command use? (default=enabled)
Tb has problem like "connection loss during IDLE is not recognized". So, if connection is lost during IDLE is active, Tb possibly wait for OK response to DONE. If a folder is selected at the connection, request for the folder may be queued and may no be executed.

How frequently does your problem occur?
Can "go Work Offline" then "go back to Work Online" be an effective recovery procedure?
If IDLE command use is disabled, will frequency of problem decrease?
(If you relied on IDLE for new mail detection of Inbox, enable automatic mail check, please)
(In reply to comment #21)
> Do you enable IDLE command use? (default=enabled)

I have no idea.  I haven't enabled or disabled it explicitly.

> Tb has problem like "connection loss during IDLE is not recognized". So, if
> connection is lost during IDLE is active, Tb possibly wait for OK response to
> DONE. If a folder is selected at the connection, request for the folder may be
> queued and may no be executed.

How can I verify this in my profile?

> How frequently does your problem occur?

It occurs every time that Thunderbird is left open for a few hours.

> Can "go Work Offline" then "go back to Work Online" be an effective recovery
> procedure?

I haven't tried it, but I doubt it.  The problem happens if I move from one wifi hotspot to another, which should have the same effect, right?

> If IDLE command use is disabled, will frequency of problem decrease?
> (If you relied on IDLE for new mail detection of Inbox, enable automatic mail
> check, please)

Like I said, I don't know if I have the IDLE command enabled or not...
(In reply to comment #22)
> > Can "go Work Offline" then "go back to Work Online" be an effective recovery
> > procedure?
> 
> I haven't tried it, but I doubt it.  The problem happens if I move from one
> wifi hotspot to another, which should have the same effect, right?

I stand corrected.  Going offline and then coming back online does seem to fix the problem.
(In reply to comment #22)
> > How frequently does your problem occur?
> It occurs every time that Thunderbird is left open for a few hours.

> The problem happens if I move from one wifi hotspot to another, (snip)

Is "move from one wifi hotspot to another" always involved in your "Thunderbird is left open for a few hours" and your report of comment #0?
How about "sleep/wakeup" or "suspend/resume"? (Power saving feature of PC/OS)

About IDLE command use setting.
  Server Settings, "Advanced..." button,  [?] Use IDLE command ...
Does your problem occur with the option disabled(unchecked)?
DHCP is used with ordinal configuration/set up. Check assigned IP address to your PC by "ipconfig" at Terminal, please.
Is IP address changed when problem occurs?
(In reply to comment #24)
> (In reply to comment #22)
> > > How frequently does your problem occur?
> > It occurs every time that Thunderbird is left open for a few hours.
> 
> > The problem happens if I move from one wifi hotspot to another, (snip)
> 
> Is "move from one wifi hotspot to another" always involved in your "Thunderbird
> is left open for a few hours" and your report of comment #0?
> How about "sleep/wakeup" or "suspend/resume"? (Power saving feature of PC/OS)

Neither of these have any effect on the bug.  If I open Thunderbird when I get to the office, for example, and leave it running for a few hours, I start to see this problem.

> About IDLE command use setting.
>   Server Settings, "Advanced..." button,  [?] Use IDLE command ...
> Does your problem occur with the option disabled(unchecked)?

I've disabled it to give things a shot.  Do I need to restart Thunderbird or does that pref take effect immediately?
(In reply to comment #26)
> Do I need to restart Thunderbird or does that pref take effect immediately?

Setting change doesn't clear "IDLE" status, but fetch or status will be executed sooner or later by automatic mail check and IDLE will not be issued, so no need of restart. But I believe "do restart of Tb regardless of demand" is far quicker and safer than "do change, ask question, and wait for answer"...   

> Neither of these have any effect on the bug.  
> If I open Thunderbird when I get to the office, for example,
> and leave it running for a few hours, I start to see this problem.

Is AirMac or similar used at officce for internet connection?
Setting IDLE doesn't seem to have helped the issue.

(In reply to comment #27)
> > Neither of these have any effect on the bug.  
> > If I open Thunderbird when I get to the office, for example,
> > and leave it running for a few hours, I start to see this problem.
> 
> Is AirMac or similar used at officce for internet connection?

I have no idea, but this problem doesn't have anything to do with me being in the office or not.
(In reply to comment #18)
> It happens on opened folders as far as I can tell.  But closing them and
> reopening again doesn't help.

When you see problem next time, do next, please.
  Assume problem occurred on folderX.
  At Gmail Web Interface, add Gmail Label of folder-X to several conversations,
  mark them "Unread", and sign-out.
  New mails in folder-X are detected by new mail check?
  Open other than folder-X at any Tb Tab and Tb Window, after a while,
  open folder-X at a Tb window or Tb Tab.
  New mails in folder-X are detected by folder open?
Cc'ing Atul since he's become my GMail integration go-to guy. :)
So what is the current status for this bug? Do we have to "do" something for this?
If I recall correctly, we've been unable to figure out precisely why this happens, so yes, the first actionable item would be to find better STR than « wait overnight » :).
Firstly, I still see this bug every day.  One thing that might have an effect here is suspending and resuming your computer.  Thunderbird seems to lose its IMAP capabilities for a while after the computer resumes (at least 4 out of 5 times), and it takes a while before it can start fetching email etc.  I usually see this problem when TB figures that it's time to fetch mail again, and I'm reading mail (and marking what I have read as read) at the same time.
Note that I'm also willing to provide more information if you tell me what you need to know.
Blocks: 798667
Dlech, does checking flags more often will solve this problem? #JustCurious
Flags: needinfo?(david)
Yes, I think that fetching flags more often or providing a way for the user to manually trigger a fetch would solve this problem.
Flags: needinfo?(david)
I'm seeing the same problem with Thunderbird 17.0.7 on Linux Gentoo and on Ubuntu 13.04.
Yes, also seeing this problem with Thunderbird 17.0.7 on Windows 7.  Was just "switched" to gmail by Berkeley and took a bit to figure out what was happening here.  It is a pretty annoying bug, since I routinely read email on both Laptop and desktops.

Any thoughts on a fix?
As for Gmail IMAP, following can be a solution, because Gmail IMAP started to support CONDSTORE(see bug 885220 for CONDSTORE support by Gmail IMAP).
  If CONDSTORE is supported,
  "uid fetch 1:* Flags()" upon Mbox open,
  "uid fetch 1:* Flags() CHANGEDSINCE( ...)" upon new mail check
  at cached connection where Mbox is selected.

FYI.
Known Workaround == Force "uid fetch 1:* Flags" upon next Mbox open :
(a) Max cached connections=1, and re-open Inbox
(b) As already stated in Comment #23 by Ehsan,
    Going offline and then coming back online, and re-open Inbox

Anyway, read bug 512745 well to understand "what is cause in this kind of phenomenon", before add your "Me too" comment or complaint to this bug, please.
> Anyway, read bug 512745 well to understand "what is cause in this kind of
> phenomenon", before add your "Me too" comment or complaint to this bug,
> please.

No offense intended.  Just putting in a plug for a better fix than going offline and back online, since my normal usecase is to keep Thunderbird open in multiple places for quick email reading.
Hi, got the bug here too,

> FYI.
> Known Workaround == Force "uid fetch 1:* Flags" upon next Mbox open :
> (a) Max cached connections=1, and re-open Inbox
> (b) As already stated in Comment #23 by Ehsan,
>     Going offline and then coming back online, and re-open Inbox

Can you please explain how to use this workaround ? I don't understand it (not an IMAP nor Tb wiard). The workaround "going ofline and then online" is a bit "time consuming" and some time we don't even know if we're touched by the bug, until a "Wow, I already read this one".
Still happens in Thunderbird 24.0 and 24.0.1.
This happens with me as well.  I assume this is the case for the other posters reporting the same thing, but for clarification it's more than just the 'read' / 'unread' status of a message.  Basically, along with that issue, Thunderbird fails to sync with Gmail properly when asked to do so via the "Get Mail" button.  So, that button is essentially is useless to me.  As any messages that I 'archive' or 'delete' in Gmail's website (or the app on my Android) will not be made aware to Thunderbird for a while.  In this I mean that my Thunderbird seems to finally sync up with Gmail and any changes I made to my e-mail about every 20-30 minutes, but is only done so by Thunderbird on its own internal timer, not by me hitting the "Get Mail" button.  Like the others have stated, the only 'fix' is to close down Thunderbird and reopen it, which is annoying to do so.

Another person on the interwebs also echo my complaint as noticed here:  https://getsatisfaction.com/mozilla_messaging/topics/thunderbird_not_syncing_gmail_imap

I am running Thunderbird v24.1.1 on Ubuntu Linux v13.10 (64-bit).

P.S. This has been an issue for me for the past year, and throughout that time-frame I've repartitioned and formatted my Ubuntu system.  So, I know it's not a new glitch.  Also, in my opinion this should be a high-priority bug, as it makes me nervous to suggest anybody else to use Thunderbird until this gets fixed.  I love Mozilla and their philosophy, but this bug needs to be squashed.
Hello - Going offline then online did not resolve the issue. I'm on Ubuntu 12.04.3 x64 with Thunderbird 24.1.0. I upgraded to 24.1.1 but the issue persists.

Strange that it only affects one single Google account in Thunderbird when there are 4 others syncing just fine.

Case and point: I was away all last week using another Ubuntu 12.04 laptop with Thunderbird. .thunderbird folder originally copied to laptop from main desktop. Read mail from all accounts all week, Thunderbird on desktop at home left open. When I returned tonight, all messages received in the affected mailbox are unread and have no forward/reply symbol/flag until I click into the message to mark as read. The other main google account in Thunderbird does not have this issue and correctly reported the read/unread messages. 

Note the affected one is a Google Apps and the working one is Gmail. The other 2 are Google Apps and working fine. 

I use Thunderbird across three Ubuntu 12.04 x64 machines - all with the same original .thunderbird folder from the desktop. The two laptops seem to not be affected by this the last I checked. 

If I can provide any kind of logging or something from my setup let me know.
I've found a solution to this issue on another forum.  http://forums.mozillazine.org/viewtopic.php?f=39&t=2072543&sid=63b608344a7c19761a74d317cf5ad7cf&start=30

I had to change.

Tools->Account Settings->Server Settings->Advaned->Use Idle Command=false (untick it)
Max Server Connections To Cache = 1 (default is 5)


Tools->Options->Advanced->Config Editor

mail.server.default.check_all_folders_for_new=true
mail.imap.use_status_for_biff=false (otherwise it only updates when I changed folders)
(In reply to Squillow from comment #44)
> Going offline then online did not resolve the issue.
Gmail IMAP started to support CONDSTORE from this year, so problem in Tb's CONDSTORE support is perhaps exposed to all Gmail users(see bug 912216, bug 885220).
IIUC, workaround of "Going offline then online" is affected by the problem and the workaround perhaps doesn't work as expected(see bug 885220 cmment #76).
Disable Tb's CONDSTORE support, please.
  mail.server.server#.use_condstore=false
"IDLE does not send unsolicited responses for flag changes" is not Gmail IMAP only issue.
Issue of "Tb is not torelant with 'IDLE does not send unsolicited responses for flag changes'" is not Gmail IMAP only issue.
Because bug 693204 was processed after analysis of many bugs, that bug is crispy than other bugs.
So, duping this bug to bug 693204.
If duping is wrong, re-open this bug, please.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → DUPLICATE
No longer depends on: 512745, 518581
You need to log in before you can comment on or make changes to this bug.