Tested 6-17-08 Win32 and Mac build. Step of reproduce 1. Open the above URL page 2. Select some Japanese text 3. Select menu Edit | Copy 4. Open new Notepad then, paste the Japanese characters Notice that Japanese characters you copied from the Browser is displayed ".........."

Per ftang, I reassign this to tague@netscape.com

reassigning to cut and paste/clipboard owner on windows

tague is going to have to step through with the debugger to see what is going on. The place to start is in nsPresShell.cpp in the DoCopy method. I think the problem is with the XIF converter. The selected content gets converted into XIF. buffer.ToNewCString is passing the XIF encoded content into the transferable. So the question is: Does XIF deal with double-byte chars? The XIF converter was written by the editor group (Greg K.) Akkana do you know much about this?

My understanding is that XIF is supposed to work with double byte characters, but I don't really know much about how that works. You might check the file /tmp/xif.html on Unix or c:\\temp\\xif.html on windows (these are supposed to be converted into standard temp file formats, and right now there's no equivalent on mac) for the XIF file, for debugging purposes. Maybe this will at least help pin down where the problem is happening.

The first question to answer is: Does XIF really support double byte characters?

I Akkana and Tague to figure out on the Japaneese platform if XIF is supporting double-byte chars, I can't do anything more with this bug until then.

Reassigning this to tague: I need tague to sit down with akanna (in the debugger) and see if the XIFConverter supports double-byte chars.

moving to M9

In the past Greg K discussed this with ftang, and the design was to support this. ftang is out on paternity leave for another week.

Changing the component from Internationalization to Editor.

Why editor? The test case had nothing to do with the editor.

I moved it to Ender because I thought Ender owned copy&paste, XIF and all that. What component should be used?

XIF will actually support double-byte characters, the problem is that nsPresentationShell.cpp::DoCopy line 1635 does a ToNewCString each time the data is transfered to the cliboard object. reassigning to layout since that's where the presentation shell lives.

Adding rods and pinkerton to cc list since they own clipboard.

*** Bug 5939 has been marked as a duplicate of this bug. ***

Now that Greg is gone, who owns XIF?

I own XIF. But this is a copy issue, not a XIF issue; XIF already handles multibytes and charsets (at least, Greg wrote code to do that though I doubt it's been tested much).

Just to repeat what I previously said, this is *NOT* a XIF issue. The problem is with string handling in the presentation context. The presentation context takes the nsString from the XIF converter and does a ToNewCString on it. It's the ToNewCString operation that is causing the data loss. The XIF converter deals with double byte characters correctly.

Tague -- I accept your comments at the end of this bug on face value, but I'd be greatful if you could send me a pointer as to where the ToCString bug that referring to is. Thanks.

I changed the PresShell to get a unicode string instead of a cstring. I think this is what is needed. I don't think he was trying to imply that there was a bug in nsString.

Just to clarify some confusion, what rods said is correct. The problem is not a bug in ToNewCString/ToCString but a bug resulting from using ToNewCString in this context. sorry for the confusion.

reassigning to rods since the data loss problem has moved from nsPresShell.cpp to nsXIFFormatConverter.cpp. (a) Plain Text Conversion: if (aToDataFlavor->Equals(kTextMime)) { if (NS_OK == ConvertFromXIFToText(srcText, text)) { *aToData = (void *)text.ToNewCString(); *aDataToLen = text.Length(); } ConvertFromXIFToText does the appropriate conversions; however, the call text.ToNewCString causes data loss for any non-ASCII data. The Unicode data in text needs to get converted to the system code page (windows), system script (Macintosh) before getting written to the clipboard. (b) HTML Conversion } else if (aToDataFlavor->Equals(kHTMLMime)) { if (NS_OK == ConvertFromXIFToHTML(srcText, text)) { *aToData = (void *)text.ToNewCString(); *aDataToLen = text.Length(); } ConvertFromXIFToHTML is not converting "double-byte" characters appropriately. It needs to generate an HTML fragment which includes the encoding used for the HTML and it also needs to convert the HTML to a reasonable encoding which can express the data.

Adding Pav to cc list, since he's run into this problem working on the gtk clipboard.

Reassigning to Stuart, he has signed up to solve this problem

this has become a blocking problem. we need this functionality so mail & news can test and debug Japanese mail support. without cut and paste we have no way of providing them with test cases or helping them to debug problems that we find

There is *no* way I will get this fixed for M9. I haven't even looked at the transferable code yet and won't have a chance for a while.

Yes, that is ok. Move it to M10. I briefed tague on this earlier. He did express a desired to have it fixed for M9, but understands the currect state of affairs.

it would be nice if we could at least fix this on windows so mail and news can make better headway. Pav :- this is still a blocker - it's preventing mail & news from doing testing and debugging of work we've requested.

fixing it on windows is going to require me to fix it everywhere. I don't have time right now to work on it.

rods made some changes today that fixed this problem for intra-application copy and paste. (thanks rod!) this still needs to be fixed for inter-application cut and paste, but rod's changes do take care of our blocking problem. i'm changing the severity, but i still would like to see this worked on for M10. ------- teruko - can you please do a quick QA on japanese cut and paste within seamonkey.

need estimated remaining duration in status whiteboard

Pav and I looked at this a bit yesterday. The nsHTMLToTXTSinkStream class does a ToNewCString in the case where there's no unicode converter, which is the case for ucs2 -- we have specific logic NOT to get a unicode converter for UCS2. Pav has some other issues inside the clipboard code that he's working on, but meanwhile I have two possible fixes for the html to txt converter which both seem to fix that part of the problem: 1. Go ahead and create a unicode converter even for ucs2 2. Continue not using a converter for ucs2, but in the case where we don't have a converter, do everything in unicode rather than using ToNewCString(). I can check in either or both of these fixes today (or whenever the tree opens next), but I'd like some feedback from someone who understands this issue better as to which of these is appropriate.

this is already assigned to me, i don't need to be on the cc list

akkana do you want this bug since I think all of the current remaining stuff you were working on?

akkana has two different solutions but doesn't know which one is right. could someone in i18n talk to her and find out which she should check in?

if you have questions or need some advice about potential solutions, send me email, call me or drop by my cube. this is a core bug, the core team needs to drive resolution of this.

I don't understand Tague's comment. Who are the "core team", and who on the "core team" is qualified to make decisions about which unicode converter is the right one to use?

let me clarify. this bug is the responsibility of the people who own cut and paste -- this bug is not owned by the international group, so it should not be re-assigned to me. if you or pavlov or anyone else working on cut and paste need my help in resolving this bug, it is glady given but the owners of cut and paste need to continue to be responsible for driving this bug to completion. if part of getting this bug fixed requires consulting time from me, the owner of the bug needs to take the initative to find me and ask the questions that he or shee have not just reassign this bug to me and expect for it to be fixed. /t

As akkana stated on 08/20/99, she has two possible fixes. If you want this fixed, then you need to advise her as she requested. If you don't want to own this bug, should I file another one saying that we are waiting on i18n to give us an answer?

*** Bug 13677 has been marked as a duplicate of this bug. ***

Update: the problems previously mentioned in nsHTMLToTXTSinkStream.cpp no longer exist and are not the culprit here. Turns out that after calling the sink stream to convert the plaintext output to an nsString, the nsXIFFormatConverter, in the plaintext case, does a ToNewCString(). This may be indicative of a fundamental problem with the clipboard APIs -- the fact that Convert() takes char* arguments rather than nsString arguments. But it seems to work okay for html copy/paste of Japanese characters (such as the ones at the bottom of the html editor init page).

Here are some results of the copy/paste with today's build (M10) using Japanese strings for copying. What works: 1. copy/paste from Editor to Editor 2. copy/paste from Editor to Mail compose body window 3. copy/paste from Plain text editor to Editor 4. copy/paste from plain text to Mail compose body window What does not work: 5. any text into Plain Text editor window (We see dots instead) 6. any text into Mail compose Subject field (We see dots instead) On Browser window: Copy function does not seem to have been enabled.

Agreed, that's also what I see. Any time the clipboard converts to plaintext, the Japanese characters get lost.

We need to keep in mind that in addition to these intra-application copy/paste, we also need to do a decent job of inter-application copy/paste for which the original bug report was filed. As of this buuild, we are not able to copy/paste properly from Mozilla into applications such as NotePad, Communicator 4 Editor, etc. On the other hand, IE5 can copy Japanese or Latin 1 strings into these applications though an application such as NotePad seems to be monolingual, i.e. cannot copy Latin 1 characters with accents into a Japanese Notepad without losing the accents. But IE5 can copy Japanese or Latin 1 into Comm 4's Editor as long as the Editor's charset menu is changed to Japanese, Latin 1 or UTF-8. What IE5 sends to the Win clipboard, however, does not paste well into Mozilla Editor. So we probably need to take care of this issue also.

Note that when I said "clipboard" I meant nsClipboard, which is used both for inter- and intra- application copy/paste. Since the code in this class doesn't handle Japanese, it will be broken for both cases.

I realize that inter-application copy/paste may not work 100% even under the best of circumstances but I would like to suggest that we should agree on what should work. I think a good start would be to say that what works for Communicator 4 shoud at least work. (So, for example, IE5 to Mozilla Editor should for non-ASCII strings.)

By the way when would nsClipboard code be able to handle Japanese (and other languages)? Is this on the schedule somewhere?

I can't speak for the xptoolkit people's schedule, but replacing that ToNewCString line with something more like what the kHTMLMime clause is doing shouldn't be that huge a deal, I wouldn't think ... it was probably just an oversight.

The clipboard code can handle one-byte (plain text) and two-byte (unicode) data depending on the flavor used. If the "text/plain" flavor is used, you get one- byte ascii with no encoding information (yet, tague said he'd do that). If you don't use that flavor, all should be ok.

What flavor should be used in order to get unicode plaintext? Perhaps the editor should be looking for that as well as for kTextMime for text insertions.

there is a kUnicodeMime type

I tried calling trans->AddDataFlavor(kUnicodeMime) just before calling trans->AddDataFlavor(kTextMime) in nsHTMLEditor::Paste. When I try to paste into a plaintext editor, I crash in: #0 strcmp (p1=0x0, p2=0x40561dd6 "text/plain") at ../sysdeps/generic/strcmp.c:38 #1 0x40553f2a in nsBaseClipboard::CreateDataFromPrimitive (aFlavor=0x0, aPrimitive=0x0, aDataBuff=0xbfffe5d4, aDataLen=1080677867) at nsBaseClipboard.cpp:180 #2 0x40537871 in nsClipboard::SelectionGetCB (widget=0x89add80, aSelectionData=0xbfffe97c, aInfo=0, aTime=0) at nsClipboard.cpp:787

i assume that means the 1st param to strcmp is null so it crashes (gotta love those c standard libraries). What is that param, and why is it null?

moving to m11

Not for dogfood as of PDT meeting.

We should do when we past text to the native clipboard ? on Window 1. put Unicode to the native clipboard with CF_UNICODETEXT see http://msdn.microsoft.com/library/sdkdoc/winbase/clipbrd_5nfo.htm http://msdn.microsoft.com/library/sdkdoc/winbase/dde_9f1u.htm http://msdn.microsoft.com/library/devprods/vs6/visualj/vjref/com.ms.wfc.app.Data Formats_f.html http://msdn.microsoft.com/library/devprods/vs6/vfoxpro/foxhelp/lnggetformat_meth od.htm 2. Put Text in ACP (ANSI CodePage) with CF_TEXT to native clipboard In order to do that, you first find the charset name by get the nsIPlatformCharset service. Call the GetCharset method with selector kPlatformCharsetSel_PlainTextInClipboard to get a charset name back. Then you get a nsIUnicodeEncoder for that charset. Then you convert the Unicode text by using that conveter. on Mac, 1. put the Unicode to the native clipboard with 'utxt'. see http://developer.apple.com/techpubs/mac/TextEncodingCMgr/TECRefBook-154.html for details. 2. put the text in system script to the native clipboard with 'TEXT', same as window, use the nsIPlatformCharset service.... on Linux- 1. convet to text in locale encoding by using the same way which got nsIPlatformCharset first... This IS a dogfood for IQA since they need to create test cases Japanese text in daily bases. And if this is not implement, they they cannot switch to use SeaMonkey as their daily browser to do their work.

Adding dogfood again; Copy and paste are in the dogfood list for English, right? Same for Japanese then. Please, explain what the reasons were to remove the dogfood criteria.

Putting on PDT+ radar.

mass-moving all m13 bugs to m14

How can a PDT+ bug be an M14 bug? This has to be fixed by Dogfood.

sorry, I was trying to not move any PDT+ bugs, must have overlooked this one. How did it get to M13 anyway? moving to M12

Pav, since you are currently 'most doomed', could someone else (Pink, RodS) take this bug?

yes

(pink reads over the torrid history of this bug) god, i guess i'll take it off of pav's hands. it will take me some time to come up to speed on the i18n issues and the correct solutions on the individual platforms. don't expect this one fixed by m12. moving to m13, accepting bug.

Pav recently (last week) fixed the gtk clipboard so that it handles Unicode pasting, and Naoki and I made corresponding changes to the editor to use unicode in preference to text. Has anyone checked this week to see if this bug is still a problem?

we don't copy and paste from other apps properly as far as i know

It cannot be moved to M13; this is a Dogfood PDT+ bug that affects international and it should be fixed by M12. Peter, please review target milestone.

We are aware of the bug's importance, and have triaged it accordingly, relative to our other priority work. Pink, could you please add the target fix Friday date to the summary whiteboard?

annotating tfd 12/3.

This bug has been metamorphosizing through the days and months of discussion. It is not very clear now just what we are supposed to be accomplishing. bobj, msanz, teruko, and I met and discussed this bug, and then teruko and I separately to agree onthe current facts. What we decided to do is to reduce the scope of this bug to just intra-application copy/paste with Japanese and other non-ASCII characters. teruko will file a new bug which will address inter-application copy/paste issues -- which might not be a Dogfood item but certainly part of Beta 1Intl requirements. Here are the current state of affairs with Mac, Win32 and LinuxM12 builds of 11/18/99 with regard to copy/pasting ** within Mozilla **. There may be a few minor differences among the platforms but there seem to be pretty good cross-platform consistency in what is and what is not working. 1. Copy/paste from Edit field/area (Ender) into another Edit field/area (Ender) is now working for non-ASCII characters. (There may be a few minor bugs left but these should be addressed in other new bugs.) 2. Copy/paster from non-Edit filed, e.g. Browser page and Mail message viewing window, into an Edit filed/area is NOT working. This function is disabled and not available through any means. We consider #2 to be a Dogfood item. International testing involves using data from many different languages which we may not be able to input easily through keyboards. Without copy/pasting working from a Browser page or Mail Viewing Window, our daily work is greatly hampered and in some cases such as Thai and Vietnamese, just totally blocked. The requirement of this bug then is to make item #2 working. Currently copy/cut menu or keyboard actions are totally disabled and not available. This must be made to work. There already may be a bug on this elsewhere calling for the copy/cut function to be enabled on the Browser and Mail Viewing windows. If so, make this bug dependent on it. However, this bug should NOT be made a duplicate of it since we don't know if non-ASCII copy/pasting will work once the function is enabled from these windows. The reqruirement for us is that A) the function be enabled from these non-Ender areas both from a menu and via keyboard shortcuts, AND b) such operation must be successfully applicable to non-ASCII data including Japanese and Latin 1.

There is not "cut" from Brower or Mail Veiwing windows -- so please disregard the inclusion of "cut" in my remarks above.

the fact that copy/paste are grey'd out is a different bug. that is an XP apps bug. this bug is meant for making sure that we can copy and paste i18n text to and from other applications.

Thanks. There seem to be a number of bugs open right now for Copy/paste in various components of Mozilla including one for Browser (Bug 14026), MessengerCompose (Bug 5063), etc. I don't see one specifically for Messenger Viewing window, though. I'm afraid that we need to hunt down all of them and make sure that non-ASCII copy/paste works when they are fixed. I changed the summary to reflect the fact that this bug is targeted to 'inter-application i18n concerns'. Sorry for the confusion. Do we need a summary of the current state of affairs for inter-application copy/paste?

Ok, well if we're waiting on #2 (which is out of my hands), then this is dependant on xpapps hooking up cut/copy/paste. Right now, that work (enabling the edit menu items for the content area) is not on the beta feature list, according to don, but maybe someone (davidm) can hook it up when he fixes 14026. I'll talk to davidm today and see if we can get that fixed and on his radar to do. I will leave this M13 since we don't have a timeframe for when this functionality will be testable. The fact that it works w/in editor gives me the warm fuzzies, however.

FYI: Bug 14025 is the bug on CCP being disabled in the browser window. See the dependency tree for that bug for other bugs.

Oops, that should have been 14026. I see that's already in this bug's dependency tree.

So, pinkerton, you probably can remove the PDT+ since the remainings of the bug are not dogfood. Thanks.

removing from pdt list, per talking with msanz. Yay! ;)

PDT- given comments at bottom of document.

I created a separate bug 21414 for XIF to kTextMime conversion.

Reassigning to momoi for verification if this is still a problem now that one can copy text out of the browser.

reassign back to pinkterton. Pinkerton, please do not assign this bug to QA engineer ( momoi, momoi and teruko are QA engineer. They won't FIX this bug. ) If you think you fix it, mark it fix and IQA will pick them up and verify them. If you are not sure about your fix. Send them mail in private or put your comment in bug report. Thanks.

I thought the "for verifcation" part of Pinkerton's comment was pretty clear...

Okay, fine. marking fixed. we'll play that game if you want.

Can you comment on to what extent we fixed this bug? I'm less concerned about copying from external application to Mozilla --> we should ask for Unicode or native clipboard, right? It is the exporting of data that concerns me. If an external application does not ask for Unicode data from the clipboard, is is supposed to work? For example, copying JPN data from Mozilla to Win Notepad works but it does not work to WordPad and a few non-MS applications I tried with 1/6/00 Win32 build. By the way, for clarification, I believe it is legitimate to assign a bug to a QA when a developer wants to know the current status of the problem so that he/she can guage how the matter stands at present without first marking it fixed.

It surprises me that it works in NotePad (which only understands text/plain) and doesn't work in WordPad, which understands unicode. Did you mix up the names by accident? This is opposite from what I expected to hear.

No, I described it right the first time. To elaborate, This is WinNT4 only. On Japanese NT4, Notepad.exe --> display JPN data correctly from Mozilla write.exe (Wordpad) --> displays JPN data from Mozilla incorrecty as dots rather than as characters. It is NotePad which is OK. Both are version 4.0 which get updated to with Service Pack 5. We really need to look at Win98/95 where there might, potentially, more problems than one would encounter on NT4.

Sounds like we're advertising plaintext ahead of unicode in the lists of flavors we support. Either that, or wordpad is asking for plaintext when it could be asking for unicode, in which case it would be a wordpad bug. Does wordpad definitely accept Japanese text from other applications (to rule out that possibility), applications which can also paste ascii?

Yes, it does. For example, we can copy JPN data from Notepad to Wordpad, and also from another JPN text editor (e.g. Hidemaru), which has the same problem accepting Mozilla JPN data, to Wordpad.

Here's what I've found: WordPad: asks for text/plain NotePad: asks for text/unicode I think the reason that WordPad doesn't work is that we don't have text/plain in the correct charset right now. I imagine (and Naoki can tell me if i'm right) that his changes that he submitted to me for charset conversion will fix this problem. Is this a correct assessment? Is there a bug filed on that yet? If so, then it should be a dependency of this bug. The fact that unicode copy/paste works gives me the warm fuzzies that we don't have any fundamental problems.

Yes, my change for 21414 is for pasting text/plain. I need a code review for that. BTW, my change does not change pasting to mozilla. Is that working now?

but that bug has to do with text/mime, not text/plain. Am i just crazy? I'll shut up if i am.

21414 is for kTextMime which is text/plain (see nsITransferable.idl).

my god, you're right. doh!

As expected, on Windows 98-Japanese the results were much worse. Here's the current status with Mozilla (1/6/99 Win32 build) and WritePad, NotePad and another application. Unlike on NT4-J, here WritePad performs slightly better. The following involves copying a Japanese string. 1. Copy from Notepad/Hidemaru Editor & paste to Mozilla -- Fail (looks like ASCII garbage) 2. Copy from WritePad & paste to Mozilla -- Pass 3. Copy from Mozilla to Notepad/Hidemaru Editor -- Fail (dots in place of characters) 4. Copy from Mozilla to WritePad -- Fail (dots in place of characters) I'll attach an .jpg image illustrating the above results using Mozilla composer. You can see what they look like. Note that there is only 1 Pass out of 4 cases under Win98-J.

I should add that there are applications like Communicator 4.x, Outlook Express 4/5 and Internet Explorer 4/5 which can easily do copy/paste to/from Mozilla without any corruptions on Windows 98-J. But by and large, the number of such applications is small. So a well-known application like PageMill-Japanese edition has problems similar to NotePad.

Hopefully nhotta's/my fix for text/plain and correct charsets will help those other apps that don't understand the CF_UNICODETEXT flavor. Try these tests again on Monday and let me know.

i was wrong about 21414 being fixed. i misapplied nhotta's patch. more work needs to be done there, but it shouldn't be that hard. anyway, QA should ignore my statements that all this should work on monday ;)

ok, 21414 is fixed (the right way this time). hopefully this should improve for the 1/11 build. can someone test for me then?

i had a flash last night watching tv. I think my fix for 21414 broke unicode copy/paste for non-roman languages. sigh. i know how to fix it, so don't be surprised if it regresses for a few days.

removing target fix date (heh) and pdt- for reclassification, now that we're off to beta and not m12.

As you say, non-ASCII copy/paste seems to be generally broken now, even the intra-application type.

Oh yes, it certainly is. I did a number to it. Fixes are all in my tree just waiting for the tree to go green and we should be back to "normal" but charset stuff should work with plain text.

Putting on PDT+ radar.

Can someone test this now and see if it is any better/worse wrt the i18n aspects. I checked in my fixes so all should be back to normal.

Adding 24010 to the dependency (copy from other app to mozila). I think copy to other app is now fixed (21414).

moving to m14 as the new dependency was just added.

Added beta1 in Keywords.

with 24010 fixed, no more work should be needed on this. marking fixed.

I tested this in 20000208 Win32 and 2000020410 Mac. Copy/Paste Japanese Editor (Mozilla) / Editor (Mozilla) OK / Mail (Mozilla) OK / Plain text editor(Mozilla) OK / Text area in Browser (Mozilla) OK / Notepad OK / Wordpad OK / Composer (Communicator 4.x) OK Notepad / Editor (Mozilla) OK / Mail (Mozilla) OK / Plain text editor(Mozilla) OK / Text area (Mozilla) OK WordPad / Editor (Mozilla) OK / Mail (Mozilla) OK / Plain text editor(Mozilla) OK / Text area (Mozilla) OK Copying Browser and test area does not work completely. And, I need to investigate the copying and pasting in Linux how it will work.

Added that work is being done to verify in Status Summary.

After I spent time on investigating Japanese Copy/Paste in linux, I do not think we have the right environment in Linux. We are using Redhad 6.0 US plus pseude Japanese locale. The other editer cannot handle Japanese characters. I will verify this bug since this works in Mac, Win32 build.

vi doesn't work if started in Japanese locale? What about using kterm?

Copy/paste between kterm and mozilla doesn't work either. 4.x on linux also has this problem.

I think copy/paste in Linux has problem. After the beta1, we will investigate more.