Using _both_ Netscape Communicator 4.76 and Mozilla 0.9.2 I have last attachment in the list truncated (if it's only one attachment than it happens only sometimes). That happens to all types of attachments (pictures - you can see them truncated, Word documents). Outlook Express views/downloads thouse files normaly. Same happened in older Communicators (4.7 etc) and Mozillas (M18 I think). Using Microsoft Exchange 5.5 Build 2653.23 SP4 with default Internet Mail Service settings.

.9.3 is due out in about a week. Can you test against that? We fixed some attachment discovery bugs between .9.2 and .9.3. At: http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#imap you'll find instructions for generating an imap log which will show us information about why we may not be showing your attachment. Just click on a message giving you this problem. Then quit the app and send us the log file. Thanks!

Using 2001072403. While playing around to create a good test example I noticed that it actualy truncates the whole file sometimes. Say I attach 4 files, and it only shows me 3, also downloading only part of third file. Attaching log file.

Marking NEW.

Sounds like a similar problem that I have. A workaround I have found is to copy the message (usually by control-dragging) to the local Drafts folder, and then back to the INBOX. All of a sudden it can read the entire message again.

Yes Bob, I can verify that, and it makes my life a little bit easier. However, that only works in NN4 for me. After that I am able to view attachment from the local folder and IMAP folder using NN4. Mozilla is still not able to view that new e-mail's attachment correctly.

Hmmm, perhaps we have slightly different experiences. The thing I have noticed today is using Mozilla, that I can copy it to my IMAP folder named Drafts, instead of the local version, and then copy it back, and be able to read the new copy of it. I wonder if there is something special about Drafts. I also have an IMAP folder that I created called Unreadable that I put these kinds of things to see if I can detect a pattern. They are still unreadable there. I got used to using Drafs as a staging area to do things like delete attachments with NN4. The other thing I find curious about all of this is that if I do a View Source, I don't get the entire message from the IMAP folder. Is this showing me what was actually downloaded, or what was parsed? I have played around with imaplib in Python, and that seems to have no problems downloading the entire thing, as does mutt for me, and pine for others. Any suggestions as to see what is actually coming back from the Exchange server?

Ok, to clarify my experience 1) The same behavior is shared by Netscape Navigator (4.78) and Mozilla 2) The most obvious way to see it is by looking at attached picture, whether inline or by clicking on it. It's truncated (note that it doesn't happen always). 3) That doesn't happen always. 4) In NN, dragging message from IMAP folder to any local folder fixes the problem. Same thing doesn't work in Mozilla (tested on today's nightly) To answer the last question, you can find instructions on enabling logging here http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#imap And I have a couple of logs attached to this bug already.

I'm almost certain that what you're seeing is bug the same thing I can see in bug 105606. I've had this bug for ages in Netscape 4.x and in Mozilla. This bug is the main reason why many of my co-workers switched to Outlook since copying mails to local folder just isn't a solution that you can have in your daily work. So to clarify: this bug is also present in Mozilla build 20011018

I went into details of IMAP RFC and Henrik's note that Outlook uses different command also helped. I believe that Mozilla behaviour is OK, it's Exchange problem. Check out http://www.faqs.org/rfcs/rfc2683.html part 3.4.5 I think it will require tweaking Mozilla specificaly to work with Exchange.

I had read about the size issue long ago in some fetchmail documentation. Knowing that Microsoft will most likely not change Exchange any time soon, does this not fall into the are of "quirks" vs. evangelism? The one thing I don't understand is why it works if you copy it into another IMAP folder. Perhaps Exchange calculates the size correctly then. Time to learn how to make IMAP logs. Is there any chance of this being implemented in Mozilla as a quirk?

One more workaround... File/Offline/GetSelectedMessages will download the whole thing. Sometimes you have to select another message and come back to see it, but it is there.

I forgot all about this bug. I suspect it is fixed. What is the process for closing a bug? Did someone have to claim to have fixed it?

Well since your not seeing it anymore I will go ahead and mark it WORKSFORME. Thanks for getting back.

this is not working. I'm hit on a daily basis by this bug.

ANything reproducible? I could test it at my end.

this normally happens when trying to open large attachments like Word documents without first saving them to the disk. Because of this Exchange feature the document are chopped... You can also just use view source where you can see that some stuff (the last bytes) are missing from the mail

Basicaly Mozilla uses martial retreival method (as described in bug 105606): UID fetch 4519 (UID RFC822.SIZE BODY[]<0.10240>) and Outlook Express retreives all: UID FETCH 4519 (BODY.PEEK[] UID) RFC I refered to (rfc 2683 part 3.4.5) does not require IMAP server to report attachment size correctly, but Mozilla still uses that data and retreives only part (depending on the size Exchange reported)

I see this corruption almost daily. One noticable case is image attachments being cut off (lower part of .jpg is missing.) I'm using Linux Build ID: 2002051409 I'm hitting against a "Microsoft Exchange IMAP4rev1 server version 5.5.2655.37" mail server.

I am having the same problem with the packaged 1.0 release. We recently switched to Exchange as our internal mail server from iPlanet's Messenging server... Now, some attachments (but not ALL!) will start downloading, and never finish. hitting stop doesn't stop the mail client from spinning, but I never get the download. While some attachments work, I am unable to get any to work once I try to download one that fails.. Any kind of diagnostic that I can provide to help track this down? (My workaround is either to copy the message to a local folder, or switch to the outlook client for the problem messages, neither of which is really helpful) Regards.

Please see my comment at http://bugzilla.mozilla.org/show_bug.cgi?id=105606#c2 that explains how this bug squashes inbound S/MIME.

Changing summary from "MS" to "Microsoft" to make this bug easier to find.

In Exchange 2000 SP3 on Win2k SP3, there is a toggle for 'Enable fast message retrieval'. The help menu says this about this option: <snip> Use this check box to send only approximate message sizes to IMAP clients. This will speed the retrieval of MAPI messages in IMAP4 clients that do not require exact message sizes. Important Some IMAP clients require exact message sizes and may not function properly if this option is enabled. <snip> The option is found using the Exchange System Manager > Administrative Groups > [Name of your Admin Group] > Servers > [Name of your Server] > Protocols > IMAP4 > [Virtual Server isnstance Name] <Properties I don't know if this is enabled by default. (I think it is) or if this option exists in other versions of exchange.

Mail triage team: Gary, can we reproduce this easily in-house?

using commercial trunk Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.3b) Gecko/20021219 on XP. I am using Microsoft Exchange IMAP4rev1 server version 5.5.1960.6. I AM able to reproduce this problem. Imap mail acnt, in html format, I attach 3 jpeg files in this size order (4kb, 541kb, 81kb) and send it to myself. The last jpeg (81kb) is cut off. Tried exact same test case on Netscape Mesg server 6.1 and no problems displaying all 3 images. Tried mail mesg w/6 attachments (word, jpg, excel,gif,txt,html) the total size of mesg was only 42kb. and no problems opening or displaying inline any of the attachments Tried another mail mesg w/2 attachments but 3 meg word file and 1/2 meg jpeg and no problem displaying the jpeg or opening word file via double click. This bug was reopened on 5/14/2002. I can reproduce this problem with Jpegs. I have not seen the problem with the word file.

I have test images and the order of the attachments in my blues directory (folder is bug92111).

Another way to easily reproduce this bug. Send an email to yourself, and digitally sign the email using S/MIME. Then retrieve the email from Exchange via IMAP, and the S/MIME attachment will not be retrieved.

MS Knowledge Base says they solved a seemingly related problem with this in Exchange Server 5.5 SP 4. http://support.microsoft.com/default.aspx?scid=kb;en-us;271199 Can anyone say whether that applies to this problem or not? That is, are we still having this problem with Microsoft's latest version of Exchange?

By the definitions on <http://bugzilla.mozilla.org/bug_status.html#severity> and <http://bugzilla.mozilla.org/enter_bug.cgi?format=guided>, crashing and dataloss bugs are of critical or possibly higher severity. Only changing open bugs to minimize unnecessary spam. Keywords to trigger this would be crash, topcrash, topcrash+, zt4newcrash, dataloss.

Trunk build 2002-12-17 and the more recent 2003-02-05: WinMe, it appears to be working for me. I used gary's attached files and some other files and in all cases the full images appeared. Reporter (and anyone else), can you still reproduce this problem with a more recent build?

Where is the "blues" directory with the test images?

blues dirctory is a local directory. sorry you can't access it. but if you can still reproduce this problem please let us know

Mail triage team: nsbeta1-

jacekchmiel, if you don't know how to use flags then please don't set them. Thanks.

Still encountering attachment truncation on Mozilla 1.4 milestone release (Build ID: 2003052908) connecting to MSExchange IMAP4rev1 version 5.5.2653.23. However, it WORKED for me when message was moved to Local Folders (like in comment #9)

I confirm the problem using Mozilla Build 2003022516 (and some newer ones that I don't have right now) and "Microsoft Exchange 2000 IMAP4rev1 server version 6.0.6249.0". I've sniffed the IMAP packets using this Mozilla and Evolution to compare and see what makes one fail and the other succeed. The problem is exactly the one mentioned at least on comments #11, #19, and #24. (I disagree, however, with #19 when it says that RFC2683 doesn't require the IMAP servers to report the message size correctly. Section 3.4.5's last paragraph is positive in asserting that this is a protocol violation.) More specifically, Mozilla asks for the message size like this: 9 UID fetch 13241:13257 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To)]) <...> * 32 FETCH (UID 13257 RFC822.SIZE 464450 FLAGS (\Seen) BODY[HEADER.FIELDS (From To Cc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To)] {317} The number 464450 is the size of the message components added but disregarding any MIME headers and base64 conversions. Believing in this information Mozilla goes on to fetch exactly 464450 bytes from the message in a sequence of commands like this: 11 UID fetch 13257 (UID RFC822.SIZE BODY[]<0.10240>) 12 UID fetch 13257 (UID RFC822.SIZE BODY[]<10240.10240>) <...> 29 UID fetch 13257 (UID RFC822.SIZE BODY[]<460800.3650>) The problem is that, in this case, the single attachment has 634006 bytes when converted to base64. Hence, it ends up truncated. The problem doesn't arise with Evolution because it disregards the message size as offered by Exchange and fetches it with these commands: A00030 UID FETCH 13257 BODY.PEEK[HEADER] A00031 UID FETCH 13257 BODY.PEEK[1.MIME] A00032 UID FETCH 13257 BODY.PEEK[2.MIME] A00033 UID FETCH 13257 BODY.PEEK[1] A00034 UID FETCH 13257 BODY.PEEK[2] This way, it doesn't need to know the message size. Actually, fetchmail isn't affected either, but I didn't sniffed it to see how it fetches the message. I haven't yet had the chance to check out the solution mentioned in comment #24. However, even if this is a confirmed Exchange bug, it's one that's hard for the average admin to find out about and it is usually thought to be a Mozilla problem. How hard it would be to reimplement Mozilla's algorithm for fetching messages so that it doesn't care anymore about the message size as reported by the IMAP server? Reading RFC2683 gives the impression that there is more than one IMAP server with this problem, and avoiding it in the client could be the best solution, given the context.

I just checked the solution proposed on comment #24 and it works. Now I have a bunch of happy Mozilla users thanking me. I still think it worthwhile to evaluate the possibility of changing the fetching algorithm.

Although comment #24 works for non-ssl IMAP, it does not help secure IMAP. My non-MS mail client users have set up fetchmail on solaris to remove thier messages from exchange, and they point their clients at the fetchmail server. There are some funky fetchmail tweeks to get it to properly retrieve the messages from exchange. I don't know what they are, but it works.

Gustavo, you are right about my comment #19. I was reading it couple of months after I made it, and was like "what was i thinking?". I was waiting for someone to catch it :) Yes, RFC states that size has to be reported correctly, and it's an Exchange bug.

Regarding comment #39, I'm personally using fetchmail to fetch my email from Exchange for a long time and so far I only had one problem which has nothing to do with the one being discussed here. (I looked into this problem because of user complains, not my own.) This is my fetchmail configuration: set daemon 1200 set postmaster gustavo set logfile /l/disk0/Mail/fetchmail.log poll smaug: proto imap, timeout 2400, user gustavo, fetchall, ssl, mda "/usr/bin/procmail -Y -d gustavo"; Note that I use ssl. The only unusual thing here (I guess) is the mda directive. I use it to feed procmail directly and not to feed sendmail. I've never had any problem regarding truncated messages. BTW, I'm using fetchmail 6.2.0-3 (RH9's rpm) right now, but I use this configuration for two or three years I guess.

I have been seeing this again recently on Mozilla 1.5, but have also moved Exchange servers so I might be on one without the fix. Some additional comments. As for changing the fetching algorithm, I assume having the size allows the client to control memory allocation as it downloads the message. For clients that don't check size, is there an issue with a giant message being transmitted? Another thing I have done that works, but blows the size thoery out of the water is to blow away the summary file and let it get recreated. It is a major pain as everything has to be downloaded again, but it has fixed the symptoms for me a number of times. The other thing that sometimes seems to work is to move the message to another folder on the server.

taking.

(In reply to comment #43) David, is Thunderbird Bug 243548 same problem?

yes, it all sounds like the same problem, with exchange reporting the wrong attachment size. Thunderbird/Seamonkey can be configured not to fetch messages/attachments in chunks, and that might be the easiest work-around for users with Exchange servers: user_pref(mail.server.serverXX.fetch_by_chunks, false); where serverXX is the corresponding imap server. Can people try that hidden pref and see if it fixes the problem for them? Setting this pref will make it so we fetch the whole message/attachment at once, which will mean it's more painful to click on a large message and then change your mind and click on a different message, but that's less of a problem than this bug.

I am running TB 0.7.1 under Windows 2000 Pro SP4 and have experienced the problem. I have one of the messages on my IMAP Server with two attachments, the first is a short README txt file and the second is a gzipped file. When I view the message and select File -> Attachments -> Save All, only the first small text file gets saved. The second gzipped file is about 227K and either does not get saved or is created with a 0 byte size. I set the preference specified and noticed no difference. An IMAP trace shows an transmitted request "11 UID fetch 200 (BODY[3])", followed by a received "31 FETCH (UID 200 BODY[3] {229046}", followed by the entire base64 encoded stream, ending with a received "11 OK UID FETCH completed". Using a FILEMON (a system file monitoring utility, http://www.sysinternals.com/ntw2k/source/filemon.shtml), I can see TB creates the file, issues a flush, and then closes the file.

*** Bug 242062 has been marked as a duplicate of this bug. ***

I can confirm this issue on Thunderbird 1.5.0.4 with Exchange Server 2003. We are in the process of trying the workaround described in comment #24.

(In reply to comment #48) > I can confirm this issue on Thunderbird 1.5.0.4 with Exchange Server 2003. We > are in the process of trying the workaround described in comment #24. > We have tried the workaround described in comment #24 and we can confirm that this works; Attachments are now succesfully retrieved!

The bug remains unsolved in Thunderbird 2.0.0.0, where the user has no access to Exchange server configuration. The problem is obviously caused by "Fast message retrieval" option of Microsoft IMAP4 server implementation. (Exchange 2003) Administrators of large systems have little interest in disabling this option (increases server load) and the RFC unfortunately doesn't require the server to report exact size. I suggest duplicating Outlook's message retrieval routine.

As a workaround this DOES solve the problem. Tested on Thunderbird 2.0.0.0 @ Exchange 2003 > > user_pref(mail.server.serverXX.fetch_by_chunks, false); >

I'm just entering this to show continued problems. We have over 50 Thunderbird users, using versions from 1.5 - 2.0.0.4, using Exchange 2003. Several have reported one of these two problems: 1. Message sent from Outlook 2003 to Tbird, with attachment of Word doc, is blank, and no indication of attachment. Forwarding the "blank" email reveals the contents, and the indication of attachment, but the attachment is corrupt (too small). 2. Same as above, though message is not received as blank, but attachment is corrupt (again, too small). I tried fix in comment 45 above, and it seemed to work, though one user reported a .pdf file was corrupt. Unclear if that was accurate or not. Just now turned off "Fast message retrieval" option on Exchange server. I'll report what the results are. I should note that we recently moved to an Exchange server from a sendmail installation. We had never seen this issue on that platform.

(In reply to comment #19) > RFC I refered to (rfc 2683 part 3.4.5) does not require IMAP server to report > attachment size correctly FYI. It's wrong. RFC 2683 part 3.4.5 says; (http://www.faqs.org/rfcs/rfc2683.html) > 3.4.5. RFC822.SIZE >(Description that some servers returns estimated size,) >(and explanation about bad affect on client when estimated size) >(snip) > The protocol requires that the RFC822.SIZE value returned by the > server be EXACT. Estimating the size is a protocol violation, and > server designers must be aware that, despite the performance savings > they might realize in using an estimate, this practice will cause > some clients to fail in various ways. If possible, the server should > compute the RFC822.SIZE for a particular message once, and then save > it for later retrieval. If that's not possible, the server must > compute the value exactly every time. Incorrect estimates do cause > severe interoperability problems with some clients.

Should be bug marked OS=ALL? Has an evangelism bug been filed to Microsoft? :)

Adding RFC822.SIZE and "RFC violation" in summary for ease of search.

Are the following bugs dupes of this? 301541 390795 also related? 220281 365675

Even though it's acknowledged to be an MS bug in the spirit of enabling TB to survive in the corporate environment, could a compensating patch be rolled in TB to detect and compensate until MS gets around to addressing this? Otherwise the unintended consequence will be to frustrate users into using another client which works.

I strongly support the suggestion in comment #57. Exchange usually advertises its presence and version in the IMAP banner, so detection shouldn't be too hard, and TB's behavior should be to automagically disable chunked transfer. I feel that instances of Exchange IMAP server whose banner gets changed so that it doesn't advertise itself as such are quite rare so that we could have at least a roughly functional workaround.

Is there any chance of an easy way to force a reload of attachments which have already been truncated, without having to delete the files for the entire folder and force a reload of everything?

Gavan, if you're talking about a message that has been stored for offline use, one trick you can do is to move the message to an other folder that's not configured for offline use, and then move it back to the original folder - this will make TB forget that it had an offline copy of the message.

I have encountered the same bug - or at least I think so - please see the forum thread at http://forums.mozillazine.org/viewtopic.php?t=617915 I noticed this behaviour sporadically on Tbird/Win XP over the years, but on OS X it is simply infuriating, so right now I'm doing exactly what comment #57 suggested: desperately looking for another IMAP client that has decent HTML message rendering / authoring. No luck so far.

I have the same problem with Thunderbird 2.0.0.9 under Windows XP Pro at work, I'de like to continue to use TB and promote it in our exchange structure, if it worked. The only solution I found was using the MS Exchange webmail to see the attached files correctly, but it's quite annoying.

FYI, I have just discovered that Seamonkey does not have this problem. Just installed 1.1.8 on the exactly same system, imported everything I could from Tbird (accounts, identities, imap cache, gpg keys, extensions etc) and 1. all the attachments are displayed for messages I am absolutely sure are wrong in Tbird 2. the entire message body is fetched as expected

As much as I would like this bug tackled and solved, I must say that I routinely use TB against a MS Exchange server using IMAP, and worked around this bug by disabling chunked transfer within TB's preferences. I'd strongly suggest to the posters of comment #61 and comment #62 to try this workaround and post their experience.

P.S. AFAIK the pref to set still isn't available in the GUI, but can be found at "mail.server.serverXX.fetch_by_chunks" and should be set to FALSE to work correctly with Exchange 2003. Anyone out there that can test against Exchange 2007?

Don't have Exch 2007 available to test, but: please note that as mentioned in the initial problem description, things worked nicely on XP with Exch 2000 and 2003. It is on Mac OS X only that everything went wrong. Also please note that Seamonkey and Tbird installed side by side, the first one does not have this problem.

Thanks Mauro, it seems to work with chunks disabled (>50KB). I'll post here if the problem happens again :)

In response to Comment #66: various sources (including myself with Win XP for instance) confirm that this problem is not limited to Mac OS X. It is by all accounts (as comment #67 points out) related to a MS Exchange setting that should speed up response by the server, allowing it to report just an "approximate" size for the message, and rely on the client to send a fetch for the complete message. This is definitely a protocol violation, and an example of the contempt with which MS usually regarded communication standards (including their own, actually). TB instead takes the reported size at face value and thus when doing chunked transfer fails miserably. Everything works fine disabling chunked transfer (or disabling fast message retrieval on Exchange IMAP), and this regardless of OS.

summarizing status (...) of bugs mentioned in comment 56. > Are the following bugs dupes of this? > bug 301541 (not exchange) > bug 390795 (dependent) > > also related? > bug 220281 (WFM) > bug 365675 (incomplete)

(In reply to comment #68) I still have the problem, I think I didn't do the right thing; How do you disable chunk transfers and/or fast message retrieval? I tried to modify the boolean in the configuration of TB using: mail.imap.fetch_by_chunks (made it false)

Xavier: please read David Bienvenu's explanation of the workaround in comment #45 (if the setting isn't there, just create a new boolean and set it to false). On my part, I may add that my setting is as follows: user_pref(mail.server.default.fetch_by_chunks, false); which reverses the TB's behaviour disabling chunked transfer, unless there's an explicit setting on the server. Since I mostly use Exchange servers, this works very well for me and doesn't require me to find out the server number within prefs.js.

What I just wrote brings an idea to my mind: couldn't we just pull in some UI for these settings and get over with this bug? We could possibly advertise the thing in the server settings page as "Exchange 2003 workaround" or "Set this if your downloads are corrupted most of the time" and then add some detail in the help file. Also, it would be nice to have an explicit UI for the default setting. If we agreed on this, we could close this bug with WONTFIX and open an enhancement bug for the UI.

I think there is a bit of a misunderstanding here: #1 The workarounds suggested (to disable chunked transfers on the client, and to report accurate attachment size on the server) do help, but they do NOT solve the problem, I am still getting only an empty file, and still not seeing all the files attached to an e-mail (only 2 out of 8 f.ex.) #2 The problem is intermittent, and manifests a lot more often on OS X than on XP #3 SeaMonkey does NOT have this problem, I am reliably seeing all the attached files, and can download them I am using always the latest stable versions of TBird, SeaMonkey, Windows 2003, OS X and Exhange 2003, with all the updates installed for each.

A collegue told me to try to put the following strings to false: mail.imap.fetch_by_chunks mail.server.default.fetch_by_chunks It looks like it works now, I recently had truncated attachments, and with this trick the truncated things are now ok. I'll report here if I still have the issue. Thanks you :)

(In reply to comment #73) Helminthe: this bug _is_ definitely related to Exchange behaviour, that does violate RFC standards and can be reliably worked around with the suggestions given above. The behaviour you're reporting seems completely different, and therefore has most likely quite different causes and eventual solutions. I'd advise you to open a new bug.

Better yet. Wouldn't it be nice if Thunderbird simply disabled the mail.server.default.fetch_by_chunks option when it detects that is fetching messages from an Exchange server? It could do it by guessing from the server's greating: $ nc mailsrv1 imap * OK Microsoft Exchange 2000 IMAP4rev1 server version 6.0.6619.12 (MAILSRV4) ready. I suppose there may be some people that changes this greating, but they would simply not be benefitting from this kludge. In the end, I think this coulnd't have any bad side effects, and only save time and frustration for many people.

(In reply to comment #76) > Better yet. Wouldn't it be nice if Thunderbird simply disabled the > mail.server.default.fetch_by_chunks option when it detects that is fetching > messages from an Exchange server? This workaround has been proposed in comment #57 and #58 but nothing happened :-( The UI solution is less brilliant, but IMHO more future-proof and, arguably, more respectful of user choice (by not using hidden assumptions).

This problem is not specific to MS Exchange. It's widely reported by gmail IMAP users, and I'm experiencing it with a dovecot IMAP server. Is there some way that Thunderbird can test the server's behaviour and choose to use chunking or not accordingly? This is a 7 year old bug with severity=critical!

BTW, why does this bug still have a status of NEW?? Multiple reports confirm it - at least, if this boils down to a philosophical issue, do close it as WONTFIX!

I forward a radical motion: let's just do away with this fetch_with_chunks madness. Let's comment away all the code and the preferences, bandwidth and cpu power are increasing faster than attachment size so who cares about being bandwidth friendly, and if that's needed maybe the CPU stack will take care of it. It would be better if we lived in a nicer world but we don't.

We are having the same problems her with multiple users where images get cut off part way though when displayed inline, and images and even some other kinds of documents can't be saved when the user is interfacing with our Exchange server over Imap. I have found that either changes as suggested in <a href="show_bug.cgi?id=92111#c45">comment #45</a> or mail.server.default.fetch_by_chunks change to false from <a href="show_bug.cgi?id=92111#c71">comment #71</a> and <a href="show_bug.cgi?id=92111#c74">#74</a> worked for solving the problem on a user by user basis. I think either dumping the preferences as suggested in <a href="show_bug.cgi?id=92111#c80">comment #80</a> or doing a test like suggested in <a href="show_bug.cgi?id=92111#c76">comment #76</a> would be the easiest from an end users point of view, because they could be blissfully ignorant of the situation. However I don’t agree with changing mail.server.default.fetch_by_chunks, as suggested in <a href="show_bug.cgi?id=92111#c76">comment #76</a>, as I believe it is a global change in mail.server.default.fetch_by_chunks and might potentially mess up another of the users email accounts. (If chunking is even necessary anymore) I would instead suggest that changing settings of user_pref(mail.server.serverXX.fetch_by_chunks, false); as suggested in <a href="show_bug.cgi?id=92111#c45">comment #45</a> as it is specific to a single mail server would be better. I have also submitted a new feature request <A href =”https://bugzilla.mozilla.org/show_bug.cgi?id=449416” >bug 449416</a> which suggests that they change the email account set up & the preferences such as there is an option for exchange servers and it sets the setting as in in <a href="show_bug.cgi?id=92111#c45">comment #45</a>. Also as a Note I testing only changing the mail.imap.fetch_by_chunks set to false from <a href="show_bug.cgi?id=92111#c74">comment #74</a> and it did nothing. Since I thought it was more specific than the default settings I submitted a new bug <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=449407">Bug 449407</a>

There are several reasons and also few possible solutions for solving this issue. - MS Exchange violates RFC and provides only approximate size of the message for "performance reasons" - I saw also other mail servers providing incorrect RFC822.SIZE value So sticking with this value is not a good idea. It is also discouraged by IETF author in RFC 2683 http://tools.ietf.org/html/rfc2683 section 3.4.5. Solutions: - in Exchange disable "Fast message retrieval" function/RFC violation. http://support.microsoft.com/kb/191504 But it will have performance implications. - follow RFC 2683 recommendations and in Thunderbird fix the code and use message size provided by "FETCH RFC822.SIZE" command only for informational purposes. For message size validation and possibly truncation use e.g. size reported by "FETCH RFC822" as it seems to be correct value all the time.

I've found a useful addon that gives a workaround to this issue, attachment extractor seems to download the attachments fine: https://addons.mozilla.org/en-US/thunderbird/addon/556

(In addition to Comment #84) Pine/Alpine looks suffering from Exchange too. > http://lumbgaps.blogspot.com/2009/04/exchange-2007-imap-rfc822size-and.html Following looks the question to MS. > http://social.technet.microsoft.com/Forums/en/exchangesvrgeneral/thread/21adbe96-e21b-458a-8242-2c3894b9d7cf > http://www.office-outlook.com/outlook-forum/index.php/m/509237/ Related Exchange setting seems; > You can enable the exact size for everyone with: > Set-ImapSettings -EnableExactRFC822Size:$true > Or only for a specific user: > Set-CASMailbox "IMAP User" -ImapUseProtocolDefaults:$false > -ImapEnableExactRFC822Size:$true Exchange's Help for above setting. > Exchange 2007. > http://technet.microsoft.com/en-us/library/aa998252(EXCHG.80).aspx > Exchange 2010. > http://technet.microsoft.com/en-us/library/aa998252.aspx > http://technet.microsoft.com/en-us/library/bb125264.aspx

Linkify pointer to workaround in White Board. Workaround: See Comment #24 for Exchange setting. (Comment #86, if Exchange 2007/2010) See Comment #45 for Tb side setting. user_pref(mail.server.serverXX.fetch_by_chunks, false);

FYI. UI for mail.server.serverXX.fetch_by_chunks is requested by Bug 449416. xref.

I see no resolution of this bug or target for development. University of Maryland changed to Exchange back server from a Unix system, and my Thunderbird constantly has this problem, particularly on thin network links. I have up to date Thunderbird, and can volunteer testing to get this resolved. The 'fetch-by-chunks' has no effect on this. Any attachment of substantial size(>100K) hangs indefinitely on thin links (WiFi remote connections) -- appears to eventually recover on fat network links, such as in office on ethernet. Using a docking system, I can characterize this bug quite well. Any recent work on this?

Stan, don't get wrong but I think you should ask Microsoft same question, University of Maryland already pay for product so why no just get support from Microsoft and get fix in first place. I really doubt any core developer of Thunderbird will work to get some workaround of this issue, because its not Thunderbird itself problem but Exchange. But I believe if anybody contribute patch to "fix" that issue, developers will accept it for review gladly. It's just too common see users come to open source product and ask to "fix" other vendor product.

Nikolay, Three thoughts on what you submitted - somewhat conflicting. (1) I like Thunderbird as my preferred client - and choose to use it. I have no choice in my Universitiy's primary email system. I am not an IT manager or in the field any longer, just a somewhat knowledgeable EE. To survive as a viable alternative client to Outlook or others, Thunderbird must somehow work well with Exchange server. No doubt, its not a fun task. I have multiple email accounts, and a heavy user. (2) How do I bring such a bug to the attention of MS as I am not in the university IT department, but a simple user. Is there an equivalent to BUG page for MS users? I will pursue if knowledgeable. I was under the impression, maybe somewhat mistakenly, that the service request would need to come from the IT department of the Univsersity - and their response is 'nobody else has reported this issue, why not try Outlook.' (3) My preferred work around would be to find a good IMAP host and forward all mail to it. Any recommendations? I need to research this.

Stan, my understanding is that there is a setting on the Exchange server to make it behave correctly, so it's more a matter of contacting your IT dept. You can also tell Thunderbird to stop using mime parts on demand, which should ameliorate the issue, at the cost of downloading the full message whenever you click on it (tools, options, config editor, type mime_parts and toggle mail.server.default.mime_parts_on_demand to false by right clicking on it and picking toggle). There's a lot of information in this bug, e.g., the comment that links to http://support.microsoft.com/kb/191504 Gmail can host imap and I believe retrieve mail from other imap accounts.

though maybe gmail just does other pop3 accounts - not sure. It definitely supports IMAP for normal gmail accounts.

(In reply to comment #95) > though maybe gmail just does other pop3 accounts - not sure. It definitely > supports IMAP for normal gmail accounts. David, I will look into the other setting in Thunderbird and read up on the link you provided - maybe it will give me the info I need to work with my IT department. Note on GMAIL - it does support an IMAP interface, but beware if you use many folders in IMAP and then look into GMAIL through a web interface. Not very pretty.

Does it still happen at Fox Chase? I am not sure I have moved into another sub-department of IT & don't deal with either the exchange servers or the Mac users on a Day to day basis. I believe at this point in time most of our users are using OWA, apple mail, entourage or Outlook. I have put a call into Jim our senior macintosh support person to find out if anyone is still using Firebird & if so if they are still experiencing the problems. I will provide an update if he gets back to me.

(In reply to comment #94) > Gmail can host imap and I believe retrieve mail from other imap accounts. Gmail definitely has a functional IMAP interface: I use it as my primary email server (actually, the only one right now) and has no interoperability issues with Thunderbird whatsoever. Gmail can download mail from other accounts only through POP3, which Exchange does support, unless it's been explicitly disabled - but it's much less common than disabling IMAP, since Exchange disk space is at a premium and sysadmins would much more have their users offload the servers than the other way around, so POP3 is good and IMAP is not. I reckon, though, that if Gmail is the primary email server it's much better and less confusing to have it actually collect the messages so one only has to deal with one folder hierarchy (or, in Gmail-speak, only one tag set). If all else fails, Exchange can easily be configured to automatically forward all messages to another address, through a server-side rule that can be enabled via OWA, too. Exchange 2007 and 2010 do it almost seamlessly, 2003 gets a bit more in the way but it works although it's not as pretty.

Is there any special award for bugs over a decade old?

Another question is, can we install a workaround automatically if an Exchange server is detected?

While I think it should really be properly fixed because it is my understanding that other mail servers return estimated sizes instead of actual sizes and as such the size should should not be relied on, I like the idea from comment 101.

I like the comment from 101 as well, but I am not an active programmer, so just a request. Exchange has been forced upon me by the university, and Outlook is the only client that handles it well. Outlook's IMAP interface is functional - but lacking. Gmail's IMAP interface is fine, but it's native tag system drive's me nuts. For what its worth.

This shouldn't be hard to work around. For Evolution we just "notice" that the server actually returns more data than it claimed was available in RFC822.SIZE. We expect the final 'chunk' to be truncated to the stated size, but in fact it isn't. At that point we *know* the server lied to us, so Evolution falls back to just asking for more and more chunks until the server really *does* stop giving us data. In the common case of a non-broken server and an email which isn't a precise multiple of our chunk size, there's no performance hit. When the claimed RFC822.SIZE happens to be a multiple of the chunk size, we do need to do one extra speculative fetch, which will return zero bytes on a non-broken server. https://bugzilla.gnome.org/show_bug.cgi?id=621262 covers this. I have to say, there is a lot to be said for simply declaring that Microsoft Exchange IS NOT an email server, and anyone trying to use it as one deserves all the breakage they get ☺

I like David's idea better than idea 101 as idea 101 assumes that only Microsoft exchange violates RFC 3501 and I have read about other IMAP servers doing the same thing and this would follow the previously mentioned recommendations of RFC 2683 section 3.4.5 "RFC822.SIZE" that clients can't rely on the RFC822.SIZE returned as there are a number of non compliant servers out there. http://www.ietf.org/rfc/rfc2683.txt .

The same issue with the Seamonkey 2.7.2 mail and the google's IMAP servers. • Almost all MS file types (XLS, PPT, etc.) are corrupted (most of the time) when received as an attachment on a x86-64 ubuntu PC. • Another PC (x86-64, Linux Mint) with the same exact version, Seamonkey 2.7.2, receives the same attachments correctly. Also, using the google's own web-mail client, the attachments can be extracted correctly. • This issue is more apparent with the last few versions of the Seamonkey mail-client on my system.

First cut of patch to read IMAP bodies by chunks until the server stops sending more, to compensate for bad servers (Exchange, Google Mail) that lie about the message size in response to the RFC822.SIZE request. Still needs: - test cases: - validation that it works for messages that end exactly on a chunk boundary - test that it works on chunks where \r\n splits across chunk boundary - works on messages that don't end in \r\n? - Correctly store "size of message as downloaded", or some similar change, to make sure that we can re-use the downloaded message from the necko cache

Comment on attachment 618681 [details] [diff] [review] WIP, read message chunks until the server stops sending more thx, looks reasonable - as you say, we'll need tests! I'm happy to help with that. Testing imap from xpcshell can be really really tricky. The XXX comments will need to be addressed or at least unXXX'd before review...

First version of this patch wasn't correctly handling non-chunked messages. Still need tests, and still need to deal with the necko cache message size issues.

Added test cases, discovered that we weren't correctly handling CR/LF split across chunk boundary, fixed that too. Also some clang warning cleanup; please double-check the added parentheses at nsImapProtocol.cpp:6524 (copied below); I changed the logic slightly, but I think it's better than it was: @@ -6544,18 +6518,19 @@ bool nsImapProtocol::FolderIsSelected(co void nsImapProtocol::OnStatusForFolder(const char *mailboxName) { if (FolderIsSelected(mailboxName)) { PRInt32 prevNumMessages = GetServerStateParser().NumberOfMessages(); Noop(); // OnNewIdleMessages will cause the ui thread to update the folder - if (m_imapMailFolderSink && GetServerStateParser().NumberOfRecentMessages() - || prevNumMessages != GetServerStateParser().NumberOfMessages()) + if (m_imapMailFolderSink && + (GetServerStateParser().NumberOfRecentMessages() + || prevNumMessages != GetServerStateParser().NumberOfMessages())) m_imapMailFolderSink->OnNewIdleMessages(); return; } This patch also includes the change to canonical line endings proposed by David for bug 740453, in order to make test_chunkLastLF.js pass on Mac/Linux; alternative if we don't want that change would be to change the test to disregard line endings or convert the test to use the same mechanism as test_imapChunks.js.

Comment on attachment 625173 [details] [diff] [review] Download all of chunked messages, with test cases This breaks handling of non-chunking case when server returns an invalid size, right, and doesn't replace it with the correct counting of the download size, right? So it probably doesn't belong in this patch. The unrelated parentheses changes look OK, but it's probably better to have those in a separate patch. in case of regressions, it's better not to land patches with a bunch of unrelated changes... other than that, this looks OK. - // if we are in the process of fetching everything RFC822 then we should - // turn around and force the total download size to be set to this value. - // this helps if the server gaves us a bogus size for the message in response to the - // envelope command. - if (fFetchEverythingRFC822) - SetTotalDownloadSize(fSizeOfMostRecentMessage); -

Removed unrelated warning cleanup changes. As discussed on IRC, the removed call to SetTotalDownloadSize() in nsImapServerResponseParser::msg_fetch() had no effect on correct handling of mis-stated message sizes.

Comment on attachment 627991 [details] [diff] [review] Correctly handle chunked messages, with Bienvenu's feedback this looks good, in general. one nit - brace below else here: + else { + numberOfCharsInThisChunk = 0; + } unfortunately, test_chunkLastLF.js is failing on my debug windows build. The failure is here: WARNING: NS_ENSURE_TRUE(thread) failed: file c:/builds/tbirdhq/objdir-tb/mozilla/netwerk/base/src/../../../../../mozilla/netwerk/base/src/nsSocketTransportService2.cpp, line 115 WARNING: unable to post continuation event: file c:/builds/tbirdhq/objdir-tb/mozilla/xpcom/io/../../../../mozilla/xpcom/io/nsStreamUtils.cpp, line 438 WARNING: nsExceptionService ignoring thread destruction after shutdown: file c:/builds/tbirdhq/objdir-tb/mozilla/xpcom/base/../../../../mozilla/xpcom/base/nsExceptionService.cpp, line 166 ###!!! ASSERTION: Using observer service off the main thread!: 'Error', file c:/builds/tbirdhq/objdir-tb/mozilla/xpcom/ds/../../../../mozilla/xpcom/ds/nsObserverService.cpp, line 110 <<<<<<< PROCESS-CRASH | c:\builds\tbirdhq\objdir-tb\mozilla\_tests\xpcshell\mailnews\imap\test\unit\test_chunkLastLF.js | application crashed (minidump found) Crash dump filename: c:\builds\tbirdhq\objdir-tb\mozilla\_tests\xpcshell\mailnews\imap\test\unit\64888d62-4d6e-4c08-a79d-70acd1a50688.dmp I suspect this means that test is either not cleaning up after itself, or finishes before we've stopped talking to the server. Have you tried this on a debug windows build? If you can't recreate the error, I can poke around a bit.

Comment on attachment 627991 [details] [diff] [review] Correctly handle chunked messages, with Bienvenu's feedback clearing review request because of test failure, which, at least for me, can be fixed by disabling autosync in the test.

Fixed test as recommended by David, cleaned his nit, tested on OS X and Windows.

Comment on attachment 628593 [details] [diff] [review] Correctly download chunked email, Windows tests fixed looks good, thx!

Added an additional fix to handle the case of fetching BODY.PEEK[] as well as BODY[]; earlier version of this patch was sensitive to an existing difference between those two code paths. Also removed two spurious assertions that were checking for conditions that occur (and are handled correctly) in normal operation.

fixed on trunk - http://hg.mozilla.org/comm-central/rev/5a6bee217340

Well, that took ELEVEN YEARS to fix.

No, it took eleven years to work around. The server is *still* broken.

I would not say this bug is "fixed". I am using Tbird 16.0.1 on Win XP Pro SP3. I received two large attachments--one approx 2 MB and one over 600 KB--from a Postfix server which I access via IMAP. Tbird truncated both messages to approx 414,000 bytes. Searching the forums for a solution, I found this bug and the suggestion to turn off fetching by chunks. I made the appropriate change in my configuration, and had the messages re-transmitted to me. This time, Tbird was able to receive them in full. So I believe that this bug should either be reopened, or reclassified as "Won't Fix" because the suggested workaround still works.

(In reply to Neil Parks from comment #125) > I would not say this bug is "fixed". Writing to offline-store(offline-use=On, mail.imap.mime_parts_on_demand is irrelevant) and writing to Memory/Disk Cache(offline-use=Off, mail.imap.mime_parts_on_demand is relevant) has different mechanism. Offline-use=On folder? or Offline-use=Off folder? By patch for this bug, new IMAP log such as following was introduced. > + PR_LOG(IMAP, PR_LOG_DEBUG, ("FetchTryChunking: curFetchSize %u", downloadSize)); Can you get IMAP log and check IMAP level flow? If you can obtain IMAP log, open separate bug for ease of problem analysys by developers, please If log analysis by IMAP people is needed, before attach log file to bug, remove/replace private data and remove many log lines for mail data except mail header, please. Remove also log lines for access of irrelevant Mbox, access of irrelevant server's Mbox, please.

(In reply to Neil Parks from comment #125) My parents are having with I think is the same problem using TB 16.0.1 on Win7. I do not see it on Ubuntu (for the same message). My server is Postfix with Dovecot for IMAP.

This bug is fixed (it fixed at least one case). There may be other issues of course, but file new bugs for those if you still see problems.