User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; ja-JP; rv:1.7.5) Gecko/20041108 Firefox/1.0 Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; ja-JP; rv:1.7.5) Gecko/20041108 Firefox/1.0 Symptom: Japanese attachment file name gets garbled in the following condition. Condition: - Japanese character set used in file system is ShiftJIS (i.e Windows environment). - The 2nd byte of ShiftJIS character is less than 0x80 (i.e range that overlaps with ASCII). ShiftJIS characters may contain 0x40-0x7E and 0x80-0xFC in 2nd byte. - The 2nd byte of ShiftJIS character is escaped. 0x5C is the case. Related code: /intl/uconv/src/nsUTF8ConverterService.cpp nsUTF8ConverterService::ConvertURISpecToUTF8(const nsACString &aSpec, const char *aCharset, nsACString &aUTF8Spec) { -snip- PRBool written = NS_UnescapeURL(PromiseFlatCString(aSpec).get(), aSpec.Length(), esc_OnlyNonASCII, unescapedSpec); -snip- return ToUTF8(unescapedSpec, aCharset, aUTF8Spec); } What is going on: 1. For instance, the URI of attachment file name is "file:///C:/Documents%20and%20Settings/me/%83f%83X%83N%83g%83b%83v/%83%5C.txt " 2. Calls AttachmentPrettyName(). 3. Calls NS_UnescapeURL() with esc_OnlyNonASCII flag. 4. Does not unescape 0x5C. 5. Calls ToUTF8(). The 2nd byte of ShiftJIS character is still escaped. 6. "5C" shows up in mail composition window. Refer to the attached screenshot. Reproducible: Always Steps to Reproduce: 1. Compose a new message. 2. Attach a file with Japanese name that contains some particlar character codes. 3. Displayed attachement file name is garbled. Actual Results: Raw character code shows up like "5C". Expected Results: Readable Japanese characters show up.

Screenshot of garbled attachement file name.

I can reproduce this bug. Nominating blocking 1.0

This bug should be fixed ASAP. Many people make file with Japanese name and attach it. Please reconsider blocking 1.0.

is this a new problem or something that's always been there?

I think this is a 0.9 regression bug.

jshin can you take a look at this? I don't see any changes in our codebase to our UTF8 converter since 2003 so I don't understand how this is a recent regression unless the change is with our escaping code. Possible sources could be (although after looking back at that patch, I doubt it) https://bugzilla.mozilla.org/show_bug.cgi?id=268219

This is new since 0.9. I checked http://bugzilla.mozilla.gr.jp/ that is a local bugzilla in Japan. They report this issue on 0.9 and 1.0RC1. It appears 0.7.3 works fine. There is no report regarding 0.8. NS_UnescapeURL() is a bit different between these versions as follows. 0.7.3: ignores esc_OnlyNonASCII flag and unescapes the 2nd byte of a ShiftJIS character. 0.8: I haven't reviewed code. 1.0RC1 and probably 0.9: handles esc_OnlyNonASCII flag and doesn't unescape the 2nd byte of a ShiftJIS character.

Ahn, this bug looks like the culprit: https://bugzilla.mozilla.org/show_bug.cgi?id=243504 That looks pretty suspicious to me. What do you think?

Sorry I didn't mean that bug. I meant this one: https://bugzilla.mozilla.org/show_bug.cgi?id=263462 which was a I18n change to our unescaping code. I'm pretty sure this is the culprit. Thoughts?

do any of you have a debug build where you can quickly try backing out: https://bugzilla.mozilla.org/show_bug.cgi?id=263462 and confirming that this starts working again?

alternatively, can someone send me an e-mail with an attachment that has a name in japanese that contains these particular characters that get corrupted? I have japanese fonts on my machine, so if you send it to me, I can drag it to my compose window and hopefully see the corruption.

Scott, I sent you an email with attachment named in Japanese via web-based Yahoo mail. Then I noticed that Firefox 1.0 too causes character corruption when I save attachement with Japanese name to my local disk. At ordinay web site, it is rare to download files with Japanese name. But at web-based email site, we often find Japanese attachment name. I will report this unless it is already reported.

I got the e-mail attachment just fine. My problem is that on Windows XP when I save the file, windows converts it into my native charset so the file on my OS looks like __.txt. I'm trying to figure out a way to get my Operating system to save it without losing the japanese characters.

ok, in about 15 minutes, a test build will be up on the ftp servers: ftp://ftp.mozilla.org/pub/mozilla.org/thunderbird/test/ThunderbirdSetup.exe This test build backs out the patch for Bug #263462. Can one of you guys try it out and see if that fixes this issue as soon as you can? If we can confirm that it does we'll look into re-spinning our 1.0 bits with the back out. Thanks!

Mscott, it works fine!

Works fine on my environemnt too. Thank you! The issue I mentioned in https://bugzilla.mozilla.org/show_bug.cgi?id=273109#c12 appears to be a duplicate of https://bugzilla.mozilla.org/show_bug.cgi?id=252309

we are respinning the final 1.0 bits to pick up the backout cited above. Marking fixed-aviary1.0. Leaving open and re-assigning to jshin so he can fix his original change in Bug #263462 to not break on certain japanese characters.

Patch for Bug 263462 is backed out on Thunderbird 1.0 release build. This probably revive Bug 243504 and Bug 239192 (problem in file attach when "#" or ";" in file path). Both bugs seems to be resolved by patch for Bug 243504 in conjunction with patch for Bug 263462 (Bug 243504 depends on Bug 263462). See post in MozillaZine forum. ( http://forums.mozillazine.org/viewtopic.php?p=1048564#1048564 )

Change summary(add description on Bug 243504 and Bug 263462 according to Comment 14 by Scott).

Attached is a patch to leave ascii symbols like # escaped. Applying both of this and jshin's patch resolves all of Bug 239192, Bug 243504, Bug 263462 and Bug 273109.

Perhaps esc_FilePath looks better than the combination of esc_Scheme | esc_Host | esc_Directory | esc_FileBaseName | esc_FileExtension. I just didn't find documentation about it in nsEscape.cpp.

Do Mozilla Suite(Trunk and 1.7branch) have same problem? Patch of Bug 263462 was check in for them too. but not backout.

Add to list in Bug 274366 (regressions meta-bug)

Comment on attachment 168486 [details] [diff] [review] Patch to leave ascii symbols escaped Yes, this should , along with my patch, fix all those things. IDN host name could be a problem, though. We also have to be careful as to which set of flags to use.

This is fixed on trunk (bug 274264) and the patch will be fixed in branches asas I get approved. I'm resolving it as a dupe of bug 274264 (because there's a landed patch). No need to keep two bugs around. *** This bug has been marked as a duplicate of 274264 ***

This bug is for Thunderbird, and was closed as DUP of bug 274264 which is for Mozilla App Suite. But bug 274264 was reopened on 2005/1/21 due to backout of fix for it. So reopen this bug in order to request blocking1.8rc1 (Thunderbird 1.5 rc1).

This bug's problem have revived on Tb 1.5 Beta2. (See bug 274264 comment #27) This is due to backout of patch for bug 274264 from trunk (See bug 274264 comment #26)

This is not a bug of Tb. It's Core bug. *** This bug has been marked as a duplicate of 274264 ***