** Observed with 5/3/99 Win32 M5 candidate build ** I'm filing this bug as a reminder of what we need to do for a future Milestone. Currently, if we paste in Shift_JIS text into the mail body using 5.0, it goes out in Shift_JIS. Netscape servers will generally Base64 encode such mail. Here's what the headers look like: ... Content-Type: text/plain; charset=iso-2022-jp Content-transfer encoding: base 64 X-MIME-Autoconverted: from 8-bit to base64 by netscape.com id TAA29656 (Then the B64'ed text follows..) ... 4.6 can display this kind of mail text but 5.0 cannot. Under 5.0, this kind of message shows as a blank text body. So, we have to assume something like the following. 1. We don't have Base64 decoder working for mail text. 2. We don't have a way to override the wrong charset tag even though we have Base 64 decoder working. Since the Base64-encoded attachment without charset label can be displayed currently with 5.0, it's more likely #2 we are looking at. In any case, since nhotta seems to have a plan for the mail charset override, I'm filing this bug as a reminder. I also heard today from our Mozilla Slavic Mail lead, Pete Cassetta, that many Cyrillic mail messages are mislabled as 'iso-8859-1' even today, and that the charset override is a must in 5.0 Mail.

Regarding the first issue for base64, we do not apply base64 for the body. We plan to do base64 for html attachment as 4.5 but attachment send is not supported in M5. Also in case we allow sending Shift_JIS body, it will be sent as 8 bit thus no base64 to be used. The second issue, for sending it just send as the whatever the menu selection (no override needed). About viewing, the plan is to use a pref to use (honour) MIME label or not. If not using MIME label then the display will be controlled by the charset menu. I am accepting the bug for the viewing control by the pref. Also cc to rhp@netscape.com as he implements this in libmime.

rhp@netscape.com checked in the changes last week. The following should be available. There is a new pref "mail.force_user_charset". If this is true then libmime does not do charset conversion at all. The conversion is controlled by the charset menu selection. If the flag is false then libmime will apply the main body charset in case the message has multi-part and no charset labels. There are a couple of remaining issues. 1) A multi-part message with different charsets can only be seen as one charset (i.e. may not show all the attachments correctly). The problem should be resolved partially if we integrate auto charset detection (could be filed as a separate bug). 2) Charset override only affects the body. Headers cannot be controlled by the charset menu. This requires libmime to get a charset through libnet. The current plan is to do this with a new libnet integration in M7 or later.

** Checked with 5/22/99 Win32 build ** The current fix (using the pref option) to enable the Character Set menu control is now working. In this sense, this bug has been fixed. However, there is more general issue of how the mail viewing charset override should be done even when the charset-honoring is turned on. This part requires a spec and I'll provide one for discussion soon. In this latter sense, we need to keep this bug alive. Therefore I'm going to confirm that the current fix is working for M6 and then move the remainder of issues to M7 or later. The ones nhotta mentioned could be dealt with in M7. The general charset override implementation could come later -- this also needs to be coordinated with the proposal to do charset override in Browser (and late in Editor). If not all the issues are resolved at N7, then move this bug forward to a later Milestone. Re-opening it with these conditions.

Charset override feature has been disabled since the META breakage in M6. It is possible to restore the pre-breakage feature but I would like to co-ordinate with browser to have the unified feature. Moving to M8.

Moving to M10.

Dependency info: There is a pending issue of passing (override) charset to libmime from the menu (then webshell). The discussion was done once in mail-news mozilla newgroup (6/29) but not resolved. It is currently discussed under libnet mozilla newsgroup.

Added 11965 as a dependency (see my previous comment).

M15

Adding mscott for the issue of charset passing from webshell to libmime.

I am trying to summarize the remaining issues around this bug. The override feature is needed when the mail contains incorrect charset label (e.g. us-ascii for Japanese), this is not unusual. The current issue is to pass a charset from webshell to libmime. nsMessenger::SetDocumentCharset get the charset through JS. That needs to be passed to libmime. I will reassign the bug to mscott for this to be implemented. Additional info about override: In libmime, there are already two fields defined and used for override (override_charset) and default (default_charset). See, mimetext.cpp MimeInlineText_rotate_convert_and_parse_line() Those fields are currently not set thus not used. The default charset is needed to read mails without charset specified. I will file a separate bug for this and assign to rhp. I think this can be done by defining a pref and set it to default_charset in libimime. There is one more feature needed. When libmime decide a main body charset to use, the charset name to be feed backed to the user by putting a mark on the charset menu item. I will file a separate bug for that.

Removing 7886 from depend.

bulk move to M16 per selmer.

Putting on [nsbeta2+][5/16] radar. This is a feature MUST complete work by 05/16 or we may pull this feature for PR2.

I have this feature implemented in my tree. I'll be checking it in when the tree goes green today. If you select a a character set from the menu then we'll reload the currently displayed message, passing in this new charset as the over- ride charset into libmime. By the way, I noticed that it takes a *noticeable* amount of time to bring up and tear down the I18N charset menu. is that a known bug? It made me think that something suspicious may be going on when we were building and dismissing this menu.

I checked in my changes for this feature. Naoki came by my cube and we verified that it appears to be working (at least with some simple cases). You can now use the charset menu to force a charactet over ride for a particular message! *yeah*

Scott, you are right, we do something every time we build the charset menu: we execute a piece of JS that places the checkmark on the menu item representing the character set of the current document. However, the performance impact should not be visible. I do not know yet why it is happening, I'll have to see if it's just the the fact that we are an RDF/XUL menu and then what's the hit of the extra stuff we are doing. There is a performance bug filled on this: #29552. Please feel free to add your comments/observations/ideas there.

I did a simple test using today's win32 build 2000051609. The second message in the attachment has a wrong charset label for Japanese (ISO-8859-1). It shows a garbage initially but after I changed the menu to ISO-2022-JP, it shows a correct Japanese text. Then after I select the other message and come back to the second message, it shows the garbage again. This follows the spec (override status should not stick).

I tested more combinations. Override works for attachments, it also overrides auto-detection. One case the override does not work is when html attachment has a META charset tag. Do we also want to override this? We can change mimetext.cpp not to use META when override charset is set.

This bug for the feature implementation is resolved. Let's log the issue you mention as a separate bug where we can discuss if it is valid or not.

I see that there are some minor bugs associated with eabling of this feature but these will be filed in separate bugs. verified to be working on Windows with 5/16/2000 build.

I found a problem when the overridden message is quoted. Filed a separate bug 39736 - charset override has no effect on quoting.

** Checked with 5/30/2000 Win32, Linux and Mac builds ** This feature is working with the above build in the following types of cases: 1. When the MIME-charset info is absent for main body or displayble attachments. 2. When the MIME-charset info is erroneous for main body or multi-part body. This does not currently override an erroneous meta-charset tag in an attached document. We should probably file this in a separate bug. Marking it verified as fixed.