Open Bug 477799 Opened 16 years ago Updated 2 years ago

Body of message is missing when copying a message that contains the NUL character from local folder to imap folder. Message imported from Outlook

Categories

(Thunderbird :: Folder and Message Lists, defect)

Product:
Thunderbird
Component:
Folder and Message Lists
Platform:
x86
Windows XP
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

People

(Reporter: Tom.Bohler, Unassigned)

References

(Depends on 1 open bug)

Details

(Keywords: dataloss, Whiteboard: [penelope_wants])

Attachments

(2 files)

Folder "troubles - modified" from local folder
(deleted), text/plain
Details
Wireshark imap snoop
(deleted), text/plain
Details 
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.6) Gecko/2009011913 Firefox/3.0.6 (.NET CLR 3.5.30729)
Build Identifier: version 2.0.0.19 (20081209)

The body of a message that contains a NUL character between the header and body section is missing in the destination email after copying the original email from a local folder to an imap folder. Messages copied after such a bad copy operation are empty. You have to restart Thunderbird to restore correct operation of copy operations.

We noticed this problem while migrating messages from an Outlook POP account to a Thunderbird IMAP account. A lot of messages could not be migrated due to this error. Uploading complete folders was unsuccessfull as most messages were empty when they arrived on the server.

The problem is not messaging server related. I verified by snooping the imap communication.

Reproducible: Always

Steps to Reproduce:
1.Import the attached mailbox (folder called "troubles - modified")
2.Try to copy the message from the local folder to folder located on an imap account

"troubles - modified"
------------------------
From - Mon Jan 1 00:00:00 1965[CR][LF]
X-Mozilla-Status: 0001[CR][LF]
X-Mozilla-Status2: 00000000[CR][LF]
Date: Wed, 15 Mar 2006 16:36:19 +0100[CR][LF]
From: "Test" <test@test.etat.lu>[CR][LF]
Subject: test qwertz[CR][LF]
To: "Test" <test@test.etat.lu>[CR][LF]
Message-ID: <001001c64846$34f4b200$6b206e94@MC17124>[CR][LF]
MIME-Version: 1.0[CR][LF]
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2869[CR][LF]
X-Mailer: Microsoft Outlook Express 6.00.2900.2527[CR][LF]
Content-Type: multipart/alternative;[CR][LF]
	boundary="----=_NextPart_000_4ED8_01C66525.2AB13650"[CR][LF]
X-Priority: 3[CR][LF]
X-MSMail-Priority: Normal[CR][LF]
X-SpamPal: PASS A-WLIST EMAIL[CR][LF]
X-Wlist-Pattern: test@test.etat.lu[CR][LF]
[NUL][CR][LF]
[CR][LF]
This is a multi-part message in MIME format.

------=_NextPart_000_4ED8_01C66525.2AB13650
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

test qwertz

------=_NextPart_000_4ED8_01C66525.2AB13650
Content-Type: text/html;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=3DContent-Type content=3D"text/html; =
charset=3Diso-8859-1">
<META content=3D"MSHTML 6.00.2900.2802" name=3DGENERATOR>
<STYLE></STYLE>
</HEAD>
<BODY bgColor=3D#ffffff>
test qwertz</BODY></HTML>

------=_NextPart_000_4ED8_01C66525.2AB13650--
Actual Results:  
The body of the copied message is missing.

Expected Results:  
The copied message should be identical to the original email.

I can upload the file to reproduce the problem
MBOX file that can be used to reproduce the problem
Is the imported mail in local mail folder displayed as expected?
If same mail is uploaded to same IMAP server by Outlook, problem won't occur? 

Tb probably uploads mail data as-is. "Empty message body" itself may be produced by server. Get IMAP log for mail upload & mail fetch, and check IMAP level flow.
See Bug 402793 Comment #1. See RFC 3501 for IMAP command/response.
> http://www.faqs.org/rfcs/rfc3501.html
If Tb's fault is seen in log, and if log analysis by developers is required by developers, attach log file to this bug(never paste, please.)
Note: "transfer of [NULL] via IMAP" can be said "Tb's RFC violation".  

By the way, initial mail creator looks to be MS O.E 6, but I can't think MS O.E. is generator of the [NUL].
> X-Mailer: Microsoft Outlook Express 6.00.2900.2527 
Because following headers exist just before [CR][LF](separator of mail header and mail payload), I guess culprit is spam checker of server who received the mail first.
> X-SpamPal: PASS A-WLIST EMAIL[CR][LF]
> X-Wlist-Pattern: test@test.etat.lu[CR][LF]
> [NUL][CR][LF]
But no Received header is seen. Did you remove them for testing or pasting?
>>Is the imported mail in local mail folder displayed as expected?
Yes it is.

>>If same mail is uploaded to same IMAP server by Outlook, problem won't occur?
I don't know. I do not have the original PST file. I have only received the resulting Thunderbird files and cleaned them to remove every "confidentiel" data.

I will upload the wireshark snoop.

>>Because following headers exist just before [CR][LF](separator of mail header
and mail payload), I guess culprit is spam checker of server who received the
mail first.
Yes the culprit is the spam checker, but it is not our own. The message was received by our server in this way. Thunderbird should habdle such messages in an intelligent way.

>>But no Received header is seen. Did you remove them for testing or pasting?
Yes I have deleted everything from the message that is not needed to reproduce the problem. The message has been "anonymized".
Attached file Wireshark imap snoop (deleted) — 
Password has been replaced with the string "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> 8 append "test" (\Seen) "15-Mar-2006 16:36:19 +0100" {1360+}
> Date: Wed, 15 Mar 2006 16:36:19 +0100
>(snip)
> X-SpamPal: PASS A-WLIST EMAIL                                              
> X-Wlist-Pattern: test@test.etat.lu                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
>
> * 1 EXISTS
> * 1 RECENT
> 8 OK [APPENDUID 1234337587 1] Completed

Data before "* 1 EXIST" is [NUL]? [CRLF]?
"*  EXISTS" and  "* 1 RECENT" are data sent by Tb during the append? Or response to other command for other connection?
Does the log mean "mail body data" is not sent to IMAP server during the append? 
(I know nothing about "Wireshark imap snoop")

Can you reproduce problem with Tb trunk-latest build?

Workaround: 
  Keep backup, compact folder, remove [NUL] from mail folder file by script.
>>* 1 EXISTS
>>* 1 RECENT
>>8 OK [APPENDUID 1234337587 1] Completed
This data is sent by server. If the network snoop via wireshark is working correcty, then thunderbird does not upload the "mail body data".
The number of characters (8 append "test" (\Seen) "15-Mar-2006 16:36:19 +0100" {1360+}) that are contained in the message only considers the characters till the NUL character. I do not believe it is a mail server related problem.

>>Workaround: 
>>  Keep backup, compact folder, remove [NUL] from mail folder file by script.
I have tested this workaround before opening this bug. It works.

>>Can you reproduce problem with Tb trunk-latest build?
The Thunderbird 3 version downloaded from http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-trunk faces the same issue.
Attachment #361513 - Attachment mime type: application/octet-stream → text/plain
Problem was reproduced with Tb trunk 2009/2/08 build, with attached test mail, using Gmail IMAP.

(case-1) Original data ( 0x00200d0a )
> imap.gmail.com:S-Test-001/X-01:SendData: 9 append "Test-001/X-01" (\Seen) "16-Mar-2006 00:36:19 +0900" {1360} 

(case-2) 0x00 is changed to 0x01 ( 0x01200d0a )
> imap.gmail.com:S-Test-001/X-01:SendData: 11 append "Test-001/X-01" (\Seen) "16-Mar-2006 00:36:19 +0900" {2060} 

My guess was wrong. Tb didn't violate RFC rule of "don't send 0x00", but was the culprit of "empty body".
Tb looks to stop data transfer at 0x00. Tb uses 0x00 as data terminater in send buffer?

Confirming.
Status: UNCONFIRMED → NEW
Ever confirmed: true
FYI. If stand alone 0x0a(LF) instead of 0x00(NUL), Bug 301010 can occur.
When the mail with 0x00 in local mail folder is copied to local mail folder, data is cut before 0x00 and padded with 0x20. (Checked with Tb trunk)
> X-Wlist-Pattern: test@test.etat.lu            [CRLF]
> <------ 811 bytes --------------------------->[CRLF]

This indicates: if 0x00 exists in mail data stream, and if moved/copied to local mail folder by filter, data after 0x00 is lost. Some of "empty body display" bugs may be this case.
I think 0x00 in mail data should be rejected(or removed) by Tb upon import/RETR/FETCH, in order to avoid further problems due to 0x00 like this bug.
FYI.
Bug 83396 seems first report of "stand alone LF" related issue, which occurred on mail imported from Outlook family, which was exposed by Cyrus IMAP server who doesn't permit RFC violation by mail client.
See Bug 83396 Comment #54. It's applicable to both stand alone LF and NUL.
FYI.
"Local folder to local folder" case produced different phenomenon/problem.
So I opened separated Bug 495404 for it.
No longer blocks: 495404
Whiteboard: [penelope_wants]
Keywords: dataloss
Depends on: 495404
Summary: Body of message is missing when copying a message that contains the NUL character from local folder to imap folder → Body of message is missing when copying a message that contains the NUL character from local folder to imap folder. Message imported from Outlook
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: