** Observed with 9/13/99 Win32 M10 build ** This bug was discovered while looking at Bug 13454. There are some conditions under which you can succesffully input Japanese strings into the Subject field without the initial ASCII space. Under such conditions, when you send that mail. The received mail lacks the Subject header completely -- it's not found in the mail source file. To avoid this problem, you need to insert an ASCII space before you begin Japanese input into the subject line. (Any ther ASCII character would be OK to begin the line.)

Raised priority to P1 and set TM to M11.

I have been more or less successful in inputting Japanese into the Subject field without any prior ASCII space when I first input into the main body window. (This is not always true but works most of the time.)

The same problem occurs with Plain Text mail. So I corrected the summary line.

I think this is the same problem as 10940. I will change the code to skip unmappable unicode chars on conversion. Later we should either encode as UTF-8 or encode with multiple charset labels.

I think my change (in MIME encoder) is necessary and I will check in. But currently, there is a problem before getting to the encoder. In nsMsgCompFields::SetSubject, the vaule getting from JS is an empty string in case the subject starts with Japanese text. Works fine if I put 'a' before Japense text. Need more investigation.

I put dump statements in GenericSendMessage of MsgComposeCommands.js. Recipients2CompFields(msgCompFields); msgCompFields.SetSubject(document.getElementById("msgSubject").value); dump("subject = " + document.getElementById("msgSubject").value + "\n"); dump("subject = " + msgCompFields.GetSubject() + "\n"); They dump nothing when the subject starts with Japanese. If the text starts with Ascii followed by Japanese then it dumps Ascii plus dots (for Japanese). For me, it doesn't seem like a messenger but gfx widget problem. I'll try more generic test writing a simple html with js.

Reassign to Ender since this looks like a generic gfx widget problem. It is reproducable with an attached html. Try following 4 Japanese characters in case no IME installed on the machine. \u98a8\u6797\u706b\u5c71

sounds like an output

This is a DUP of 8427, which is currently waiting on a decision from I18n regarding how the unicode converter is supposed to work. *** This bug has been marked as a duplicate of 8427 ***

#8427 sched: [DOGFOOD] make copy work with Japanese characters Is this bug really related to copy?

Copy/Paste, Send, and Save all go through the same output converters, which rely on the unicode converters.

akkana, Question: Do you have any idea why this works if the text starts with Ascii followed by Japanese characters?

It may be a case of the sink stream getting tricked into using a unicode converter in that case where normally it might not.

The copy/paste problem in the sink stream has been fixed -- it no longer uses ToNewCStrings except in the all-else-failed case -- and the copy/paste problem is actually in the xpwidget clipboard code. That being the case, that means that this bug isn't a dup after all. Reopening. The problem is that I haven't been able to reproduce this problem myself. I have no way of typing Japanese input on a Linux box, and my NT box isn't set up for Japanese. nhotta is working on a way that I'll be able to reproduce it on Linux so I can try to find out what the problem is. It might be that that one pathological case in the output sink is the culprit ... but it might just as well be that the problem is somewhere else entirely.

I attached a new html file with two test cases which have hard coded Japanese texts so no IME is needed to reproduce. But they're actually working fine, the problem does not occure like it does with IME. I realized that whenever I type Japanese text, it is appended a white space like extra character. When I prepend or append any ascii chars, that white space is gone then my script works (as well as mail send). If I append ascii after Japanese then remove appended ascii later then I was able to have Japanese text with no white space appended. With that text, my test html worked no problem (I assume it's the same for mail header). So I assume that extra white space is something to do with this problem. Who is appending that white space and why the widget behave differently with it? It looks like only reproducable by IME (not by setting Japanese by JS code).

I spent a fair amount of time yesterday trying to reproduce this in the editor ... and I came to the conclusion that the problem probably isn't in the nsHTMLToTXTSinkStream, but somewhere else. The gfx text widget code also looks pretty solid (I saw no CString stuff where it shouldn't be). At this point, I'm guessing maybe the mime code that gets the contents of the subject line, but I can't trace it without being able to type Japanese characters. I'm afraid I have to send this back to nhotta to track down where the Japanese characters are being lost.

I debugged with Japanese NT and it seems like something got wrong around nsTextEditRules when doing Japanese text. Reassign to Akkana, I can help to debug but I cannot fix the problem with my very limited knowledge about Ender. Paste from mail, Naoki Hotta wrote: > Inside OutputToString, mRules->WillDoAction sets cancel to true then > do aOutputString = *(ruleInfo.outString); and returns (if the text is > Japanese only). > > In nsTextEditRules::WillOutputText, it got a case like below to set > aOutString as empty. > > else if (mBogusNode) > { // this means there's no content, so output null string > *aOutString = ""; > *aCancel = PR_TRUE; > }

Rules, and this one in particular, are Joe's.

the real deal here is that none of the ime stuff goes through the rules code at all, so the bogusNode never gets deleted when text is inserted. If you type "normal" text first, that goes through the rules code and causes the bogusNode to be removed, which is why things work in that case. I'll have to override SetCompositionString() in nsHTMLEditor, just so I can get access to the rules system (nsEditor doesn't have access). Then I'll have to add in a mRules->WillDoAction()/ mRules->DidDoAction() sandwhich to give the rules a chance to remove the bogs node. I should be able to get this checked in tonight.

fixed

Over to marina for verification.

There is a blocking problem with today's build. windows/32bit/x86/1999-09-20-12-M11 tested with the second attachment (09/13/99 16:21). The typed Japanese characters do not appear in the text field. Also the field itself is not selectable (and editable) until I reload the page. The Dos prompt shows dump as below (three lines for three JA chars I typed). Candidate window position: x=35, y=35 Candidate window position: x=35, y=35 Candidate window position: x=35, y=35 I am not sure if this is a result of this bug fix or something else.

Added teruko because this is not a mail/news specific problem.

does anyone know if this is fixed or not? I didnt get to test it (dont have a machine set up for japanese text).

As I mentioned before, Japanese input to text widget has been broken (cannot input any Japanaese at all) since 9/20 build. I don't know if that is caused by the fix for this bug or not. But it is possible because the check in of this was on 9/17 (Friday). I can check that by reverting the change on my local machine. Joe, could you send me files that I can revert your change?

** Checked with 2/17/2000 Win32 build ** We have not had a problem with JPN subject headers for a long time. I still don't see a problem with the above build. Marking it verified/fixed.