User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1) Gecko/20030114 Debian/1.2.1-9 Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1) Gecko/20030114 Debian/1.2.1-9 Lots of my friends use Mozilla Mail, but they still send mail with incorrect encoding :( Problem is that incorrect encoding warning dialog is not user friendly - it isn't intuitive and users don't know how to choose correct encoding and also can't choose encoding directly from warning dialog. The simplest (and the best in my opinion) solution would be adding button "Use UNICODE" to the incorrect encoding warning dialog. I think users and programers should use unicode (UTF8) more intensively (for example in Opera 7 mail program are no possibility to select encoding - all mail with non iso-8859-1 is send as UTF8) More advanced solution is to add list of suitable encodings to the incorrect encoding warning dialog. (suitable encodings list is made from all encodings list thowing away encodings not containing symbols found in email) Reproducible: Always Steps to Reproduce: 1. Press CTRL+M in mozilla 2. Choose ISO-8859-1 encoding and write some characters, that aren't in this encoding (for example àèæëáðøûþ ÀÈÆËÁÐØÛÞ) 3. Press "Send" button Actual Results: Incorrect encoding warning dialog without instructions how to select correct encoding and without posibillity to choose correct encoding is displayed. Expected Results: Incorrect encoding warning dialog with posibillity to choose correct encoding in that dialog (for example UTF8 always is correct) or at least instructions how to select correct encoding should be displayed. If you don't have time to solve this problem quickly - at least add instructions in incorrect encoding warning dialog how to select correct encoding (for example: "Select View->Character Coding->Unicode(UTF8)"). I think this bug should be solved (at least partially) ASAP. Maybe in Mozilla 1.3 final version instructions can be added and in Mozilla 1.4a posibillity to choose correct encoding in the dialog in incorrect encoding warning dialog can be added ?

I wrote patch for this bug. Please test in and include to mozilla. I changed the OK button. Now it automatically changes message encoding to the UTF-8 and sends message - it will be always correct. You do not need to think which encoding to use. A lot of people do not know even what is it "encoding" and press OK and send message with incorrect encoding. Now it will be allways correct.

It's hard to include 2 lines patch ?

It's not hard, but you have to ask for review and superreview. (please, read http://www.mozilla.org/hacking) Besides, people working on Mozilla-mail was not on the CC list. I like the patch, but some people may want to have 1. Convert to UTF-8 and send 2. Send any way 3. return and choose a different encoding BTW, this may be a dupe.

I don't have a strong opinion - whatever you think is good, Jungshik. If you want a r/sr for this, just let me know.

We seem to be on an evolutionary path. Referring to jungshik's options, we had up to now 2 & 3. The proposal here is to add 1 and leave 3 as is. I think we shoul try next 1 & 3 without 2. I don't see much advantage in leaving option #2 in. Sometimes it will end up creating unreadable msgs. Other times, it will work OK. That's not a kind thing. The warning language needs a bit of editing. I'm reviewing it now and will make suggestions later.

I would like to suggest the following rather than the proposed wording in the patch in comment #1. "The message you composed contains characters not found in the selected Character Coding. While you can choose a different Character Coding, it is always safe to use Unicode for mail. To send or save it as Unicode (UTF-8), click OK. To return to the Composer window where you can choose a different Character Coding, click Cancel." Two changes from the patch: 1. Use the uppercase "U" for Unicode. 2. Place the adverb "always" in a better position and build a slightly more explanatory sentence.

Thanks, David and Kats for comment and help. There a couple of remaining issues. Stupid web mail services such as Hotmail, Yahoo mail and many country-specific ones (and open source web mail programs widely deployed at schools and elsewhere these days) don't support UTF-8 well (how about AOL mail?) I haven't checked Hotmail and Yahoo mail recently, but I'm pretty sure that in the message display pane of those stupid web mail services, subjects and senders in UTF-8 (RFC 2047 encoded or not) would be garbled. They made a classical mistake of tightly binding UI languages to character encodings. that is, if you choose English as your UI language, they assume that you'd be content with ISO-8859-1/Windows-1252. In case of the message body, it'd be initially garbled, but I guess either that there's a button/menu ('this message is 'foreign language' or 'different encoding'. to read it, press this button) or that end-users have to change the character encoding manually. Given this, I think we have to 'reword' 'always work' part slightly. The other problem is that if the message format is text/html, it's possible to include any Unicode characters in the message body (text/html part) in any character encoding and Mozilla is actually able to do that with NCRs. (however, characters in the message headers are turned into question marks. Here, RFC 2047 encoding with UTF-8 can be used.). In addition, stupid web mail services work well in this case. So, it's a bit more complicated. How about this (assuming that it's not much work. I have to take a look to see how it can be done) A. text/plain: offer 1 and 3 B. text/html : offer 1, 2 and 3 C. multipart/alternative (text/plain + text/html) : I'm not sure. In B-2, characters in headers would be garbled so that the warning (as it is) is valid.

*** Bug 104064 has been marked as a duplicate of this bug. ***

I implemented what I wrote (for text/plain and multipart/alternative, it's the same as now while for text/html, three options are offered), but I don't like it much because nsIPromptService::Select() turned out not to be what I wanted. Either I have to write an xul/js file (a la askSendFormat.xul/js) or just have to go with attachment 135553 [details] [diff] [review] plus rewording by Kats (with 'always' replaced by 'in most cases' :-))

jungshik, you're trying to spec Mozilla mail behvaior against a moving target. Sure, some of the web mail may not be able to handle UTF-8. But if you send the msg as is, no web mail will be able to handle the headers because they are encoded incorrectly. Maybe the body can be displayed. My point is that we are not going to get completely correct display unless both headers and body are labeled correctly. UTF-8 offers the only sane choice. Also I say it's a moving target because (I believe) most web mail will be improving. A quick check shows that you can display UTF-8 mail (headers & body) with English version of Yahoo web mail. Usually the English version is ahead of country specific versions. I would think it would be a matter of time before country specific Yahoo mail implement the same ability with UTF-8 mail. With hotmail, you can view UTF-8 mail headers and body with the encoding set to UTF-8. I used Japanese version of hotmail and lost the UI language because it's set to Shift_JIS but UTF-8 header and body became readable when I switched the encoding. I suspect that webmail will be changing. Hopefully, Mozilla's decision to go with UTF-8 as the deafult in this case should help expedite that process. There are already other mailers such as Outlook Express, AOL Communicator, AOL Mail that send UTF-8 when the characters input don't match the default encoding of the client. Mozilla's joining this rank will further accelerate the trend and that is the right thing to do at this point. For these reasons, I vote for doing only 1 and 3 and making it easier for average users.

OK. I'm sold :-) partly because I don't like the idea of having to write two files (xul and js) to implement what I wrote in comment #7 (the selection box generated by nsIPromptService::Select() looks akward on Windows). However, I think we have to replace 'always' with 'in most cases' :-) simply because it's not the case. Good to hear that things are getting better in Yahoo-mail, but it's unfortunate that Hotmail hasn't changed a bit since the last time I checked (needless to say, changing the encoding of the browser to UTF-8 always worked with the UI part garbled). It's frustrating that they're slow in making changes (see http://bugs.horde.org/show_bug.cgi?id=1052 A couple of years ago, I patched Horde IMP and put it up at my computer and offered it to my friends as an alternative to my school's web-mail interface which doesn't know anything other than Latin-1.) I'll upload a patch in a moment (virtually identical to attachment 135553 [details] [diff] [review] except for word-smithing thanks to you).

In comment #11, jungshik says: "OK. I'm sold :-) ... However, I think we have to replace 'always' with 'in most cases' :-) simply because it's not the case." Thanks. I would like to suggest the following wording: "The message you composed contains characters not found in the selected Character Coding. While you can choose a different Character Coding, it is usually safe to use Unicode for mail. To send or save it as Unicode (UTF-8), click OK. To return to the Composer window where you can choose a different Character Coding, click Cancel." For UI language, I think "usually" sits better than statistical sounding "in most cases".

patch as I promised (with 'usually')

It turned out that we should not ignore fallbackCharset because there's a pref. entry for 'promoting' charset (i.e. ISO-8859-1 -> Windows-1252). See http://lxr.mozilla.org/mozilla/source/modules/libpref/src/init/all.js#707 http://lxr.mozilla.org/mozilla/source/mailnews/compose/src/nsMsgCompose.cpp#4780 (and follow the code path from there) The dialog only comes up if even the fallback charset (Windows-1252 for ISO-8859-1) fails to cover characters in the message. Only in that case, we have to use 'UTF-8'. In this patch, I also included the patch for thunderbird.

Comment on attachment 138104 [details] [diff] [review] update asking for r/sr. I just tried including U+2018 and U+2019 (not covered by ISO-8859-1 but by Windows-1252) with ISO-8859-1 and Mozilla properly 'fell back' to Windows-1252. When I included a Korean character, the dialog box came up as it should and pressing OK worked as intended.

Thank you all. fix checked into the trunk.

sorry for bugspam. David, do you think I have to ask for a1.6 as well? It's a low risk item, but drivers may think it's too late in the 1.6 cycle.

"To send or save it as Unicode (UTF-8), click OK. To return to the Composer window where you can choose a different Character Coding, click Cancel." why not just label the buttons "Send as UTF-8" and "Return to compose window"?

it's fine to ask for 1.6 approval - I'll ask Seth.

Comment on attachment 138104 [details] [diff] [review] update a=sspitzer

fix checked into 1.6branch. re: comment #18. that's a good idea. let's deal with it for 1.7alpha.

*** Bug 209200 has been marked as a duplicate of this bug. ***

*** Bug 204735 has been marked as a duplicate of this bug. ***

I have a problem with this solution. I come across a situation where the encoding selection dialog come up and reports that my message contains characters not in the selected encoding. In my situation the message does not contain such characters but perhaps some of the characters are mis-identified (perhaps they are already unicode encoded). My problem is mostly that this prevents me from sending a mail in the encoding that I need to send. In this case I am trying to send a mail in shift_jis encoding but the program give me only the option of sending in unicode or change the encoding (which is already correctly set and should contain only characters in that encoding). I have been using Outlook Express but I wanted to change to Thunderbird, for many reason one of which is that it correctly displays sender names and subject in mail with encodings outside the local codapage. In a similar situation Outlook offers the choice of sending as is (perhaps scrambling some characters), send as utf8 or cancel. This lack of choice (along with the apparent mis-identification of letters in the mail) renders Thunderbird useless to me. The reason for this is that I have to send a lot of mail to mobile phones and such in Japan, most (or all) of which don't support utf8 encodings for CJK characters. Hope there is someway to resolve this issue.

(In reply to comment #24) I have the same problem, too. In my case, the receiver is Lotus Notes via its SMTP gateway, this also doesn't support CJK with UTF-8 encoding. I just wanted to forward mail from Swedish people to my colleague with my Japanese comment. Previously I've used Netscape 7.1, and this works fine for me, selecting ISO-2022-JP encoding and send it, warning, but send it anyway, diacritical marked characters in the original mail are converted properly as I expected. But now, I'm using Mozilla 1.7.2, I have to manually retype all diacritical marked characters before choosing ISO-2022-JP before sending ! Please bring back the option "2. Send anyway".

(In reply to comment #24) I also have the same trouble. The situation is as follows; I wanted to forward the mail in English to my friends with my comments in Japanese (ISO-2022-JP encoding). When I tried to send it, warning dialog about encoding appeared and my mail was sent in UTF-8 encoding with no choice. After that, a friend of mine (Yahoo web mail user) complained that he could not read my message at all. In Japan, a vast number of mail client does not support UTF-8 except for very few clients developed outside Japan. Thus, current solution is quite problematic and harmful. Enforcing usage of UTF-8 is a dogmatic solution IMHO, at least, as of writing this message. As AZUMA said, I hope that Mozilla mail (and Thunderbird) behaves again like Netscape 7.1; I need the choice (sending "as is" or "as UTF-8" or "cancel").

Please, add your comment to bug 249530 (and add yourself to Cc instead just adding comment and driving away - drive-in comment) if you believe Mozilla needs to offer 'send as is'. This bug was resolved as fixed a long time ago !! > In Japan, a vast number of mail client does not support > UTF-8 except for very few clients developed outside Japan. It seems like you're talking in terms of email clients supporting or not supporting UTF-8. How about the 'market' share of email clients supporting UTF-8?

*** Bug 229209 has been marked as a duplicate of this bug. ***