Mozilla should not export anything else on the clipboard layer than unicode text if the text is known to contain other things than the things that can be displayed on the text/ascii level. Mozilla copies text as 'MOZn', 'text', 'utxt' + a multitude of other 'MOZsomething' variants. The only external application --on MacOS-- which is able to use the 'utxt' layer when it receives something via the clipboard *from Mozilla*, is the editor Pepper. But also Pepper needs that one turn on the special preference "Use ATSUI" (ATSUI = Apple's API for editing and imaging Unicode). If one do not turn on ATSUI, non-latin text is pasted as question marks. This means that Applications, which are 100% capable of getting Unicode text via the clipboard, are unable to use the non-Western text i receives from Mozilla. And it becomes a task for specialists to be able to use text transportation from Mozilla via the clipboard. So what is the reason for this trouble? As mentioned Mozilla copies text to the clipboard in a multitude of MOZsomething-types, in addition to 'text' and 'utxt'. Perhaps this confuses other applications? Clearly, if the external application *prefers* 'utxt'-layer and sorts out all the MOZsomething-types/layers, there is no problem. The only reason for keeping the 'text' layer is that it makes it --according to nhotta-- (I doubt it correct though...) make i possible with "vanilla" copying and pasting from Mozilla of text which belongs to the system native script. I.e. Japanese 'text' on Japanese system. And cyrillic text on Russian Mac OS. Except nhotta himself, I have only heard that my japanese an russian friends complains that it is not possible. (We are of course speaking about MacOS Classic). If the 'text' support was simply dropped, it would probably be possible to transport text from Mozila to all applications which support unicode on the mac. These applications are e.g. such applications as Microsoft Office, MS Outlook, Apple's WorldScrip, ClarisWorks newer version(s), Style, all applications based on the WASTE engine etc. I am quite sure most users would be more satisfied with such a solution. Mozilla is modern application. So why disable its modern unicode features in intereaction with other applications via the clipboard by the incosisten 'text' layers, which simply destroys mulitlingual text? (I have also tested transportation text via the clipboard on Linux, from Mozilla to AbiWord and to the KDE Advance Text editor. I both cases text were pastes as question marks.But as we know both AbiWord and KDE are unicode capable.So it should have been possible, no?) Reported with Mozilla/5.0 (Macintosh; U; PPC; en-US; rv:0.9.5) Gecko/20011011)

two points: - the 'Moz*' flavors don't get in the way at all. to suggest they do betrays your lack of understanding of the clipboard on MacOS. - removing 'TEXT' makes mozilla incompatible with every app that doesn't understand unicode, which on os8/9, is just about all of them. I have no idea what you're talking about.

I changed the title and reopened the bug. What is it that you do not know? There is some info about the problem on e.g. Bug 79864.If you copy a bilingual text of russian english (e.g.), all the russian text is destroyed. For MacOS Classic I have only come accross *one* application, Pepper, from <www.hekkelman.com>, which is able to paste non-roman text from Mozilla correctly. The reason for this trouble is not that the other applictions are unable to use 'utxt' (well, yes, there are some few applications which are unable to use make use of 'utxt, yes), but that the other applications prefers 'text' over 'utxt'. Why do you say OS 8/9 ? I believed OS 8.5 was minimum requirement? utxt is at least available since 8.5 or 8.6. But we need not care about OS 8, since Mozilla is not designed for OS 8. The fact are: text destroys multlingual text whether system is Japanes or US. stxt (styled text) is not available in Mozilla utxt is available, but is hidden for other application because of the text layer. In all occations when one tries to copy multilingual text, text-layer is uneccessary an can be dropped.Wake up and realise the facts. So at least in those circumstanses text-layer can and should be dropped. I changed the title of this bug report accordingly.

i18n for triage. I suggest futuring, but lhs wants moz1.0.

You might say this is my opinion about what I think Mozilla 1 should be. Btw, another possible solution to this problem could be to drop 'text' completely on the Carbon build? At least if the Carbon build can also run on OS 9, which I guess it cannot... I also want to say that Mozilla *had* this feature in *very* limited way earlier, until the question marks were added (by nhotta I think). Because previously, if you copied *just* cyrillic (e.g.) letters, you could paste them into WorldText (of MacOS 9) since the clipboard were not used. Currently this is not possible anymore since clipboard is filled with questionmarks.

There are two way to file a bug- tell use what does NOT work for you and ask us to do a particular thing. In this case, you chose the later one, you ask us to do a particular thing- drop 'text' instead telling us to fix the cut & paste with other apps. From my point of view, what you really care is fixing the copy & paste regardless droping 'text' or not, right ? I agree with pinkerton that we should NOT drop 'text' In the mean time, I agree with lhs@russisk.no that we should support 'text' better. We currently convert the Unicode text to the 'text' and put the text in the system script. IF the unicode cannot be converted into system script, then we convert it to ?. The question is why your system script is not Cyrillic but MacRoman. Are you running on Russian MacOS? or are you running on Russian Language Kit? Did you register the app to the language kit registration ?

Frank Tang wrote: «I agree with pinkerton that we should NOT drop 'text' In the mean time, I agree with lhs@russisk.no that we should support 'text' better. We currently convert the Unicode text to the 'text' and put the text in the system script. IF the unicode cannot be converted into system script, then we convert it to ?.» Here is a suggestion: What to convert it into could be decided by the encoding of the html/mail header. E.g. if charset tag says: "iso-8859-5" or another cyrillic encoding, the text should be converted to MacCyrillic. If the header says some hebrew encoding it should be converted to machebrew etc. If the encoding says utf-8, then it would be better to copy as plain utf-8 text code (that is what netscape 4.x does), [and the user him/herself will be given the task of converting the entities or the utf-8 code into something readable]. Also if the text cotains entities outside the charset range, e.g. if encoding is iso-8859-1 but text also includes e.g. cyrillic html entities or other entities outside the MacRoman-range, those entities should be copied as is [and user can convert them him/herself if need be]. But let me also explain a tad more about dropping 'text'... The best thing would be to make use of MacOS Text Encoding COnverter. Then Mozilla only needed only copy text as unicode text and have Text Encoding Converter convert the clipboard into styled text *only* when/if user leave the application. I.e. some kind of export mode. However, if you simply dropped 'text' without adding this export mode, the user could create the "export-mode" himself simply by going to an application which supports unicode text, like WorldText (which comes with OS 9.1) and pasting the text into that application, and copy the text again from there. Just like it today also works the other way around: if you copy multilingual text from e.g. Nisus Writer (which only supports 'stxt') then to make it possible to paste this into Mozillia one only needs to go to WorldText and perform a paste. Then automatically WorldText adds 'utxt', so that one can paste even into Mozilla. (WorldText only require you to perform a paste or to open the clipboard, through that the conversion of the clipboard taks place. It is not needed to copy, paste, select and copy again.) I hope you see through this example why dropping 'text' is not all that stupid. It would only require that you required the users to use WorldText or similar applications. FT wrote: «The question is why your system script is not Cyrillic but MacRoman. Are you running on Russian MacOS? or are you running on Russian Language Kit? Did you register the app to the language kit registration ?» Norwegian MacOS with Cyrillic Language Kit. Registering MOzilla might help (I'm tried but do not rememeber result, I thinmk a bug was filed even on that...) on getting some cyrillic window titles in cyrillic. Otherwise, it has no effect on nothing. Really, what registering does -- in other apps -- is simply that it let you see the menu names in Cyrillic. THe acctually features (writing, copying; pasting etc) are usually not affected.Except if the application is extremely simple and only supports 'text' on all levels. If you wonder how to make sure that e.g. maccyrillic 'text' is pasted into another application with a maccyrilic font face, then the answer usually is: switch to a cyrillic keyboard first and then perform the paste. Or, switch to a cyrillic font first and then paste. Anyhow: please keep the current feature where Mozilla itself prefers 'utxt' over 'text'. This makes it possible and easy to paste multilingual text *from* other applications into mozilla, like I described above.

About converting all text with e.g. cyrillic charsets to MacCyrillic 'text': I want to spesify that this is what Netscape 4.x does. I have never suggested Netscape 4.x feature simply because I wanted something better... And also with the 4.x solution users either needs to make sure he switches to cyrillic font/keboard before pasting text into third part application to be able to use this text. Allthough I don't think this always works, because 'text' seems often to be *tied* to system script encoding. In which case he needs a program which is able to *force* the text into a MacCyrillic font face (like Nisus Writer). As you see this require a level of knowledge or use of speical programs from the user... It would be much simpler if the only thing we required were use of WorldText or of similar programs, like SUE <http://homepage.mac.com/tkukiel/sue.html>. And getting Mozilla to comply with WorldText (and perhaps the Style editor) would probably be far more simple for you the programmers than to enable Text Encoding Converter support, 'stxt' support or any other alternative. Simply reveal 'utxt' to WorldText, and we are there! The only disadvantage I see with this attitude (*if it require 'text' to be removed*) is that WorldText and SUE requires OS 9. But fortunately we have Style <http://www.merzwaren.com> which works on OS X and which uses the immensly popular text engine WASTE. Hopefully, if you comply with WOrldText, you will also comply with Style. Please note that I say 'comply with WorldText' instead of 'using UTXT' because you allready use 'utxt'... It is just that it is hidden from WorldText...

My original idea, where 'text' is dropped *only* if the the text to be copied contains other text than the that of the system script, is still the best. Because otherwise one will need 'utxt'-supporting applications even to copy plain ascii/english text. Dropping 'text' in those circumstances *is* an improvement (if you make sure WorldText accept it -- which means *no* trace of 'text' can be present, otherwise WorldText will prefer it.) since currently such non-system script text is simply destroyed. But even going back to a more Netscape 4.x 'text' treatement, with the improvements I suggested above, is of course better than todays situation. Preferrably on could put on the 'text' layer a 4.x solution and *still* have WorldText use the 'utxt' layer. But whether that is a possible instruction to give to WorldText, I don't know.

rename the summary from "'text' layer should be dropped in favour of 'utxt' when copying multilingual texts" to "choose a better clibpard encoding for 'text' format depend on the page charset." reassing to nhotta for better desing. We have dicussion the following possibility 1. as is- which is depend on the system script to decide the encoding. Usually, the system script is depend on the region code in the 'vers' resource. 2. convert to the KeyScript when paste 3. depend on the charset of the document - however, there are cases that we find a charset fot the document. for example, in the URL bar, what should we do then, use 1, or 2 as fallback? we also have one problem with 3. The current interface may not allow us to pass down the doucment charset down or pull the charset from the document.

please consider this as a m98 bug. mark it as P2 since it lost data.

the other thing we have consider is use the sniffer in the Unicode converter to decide the encoding (from the installed script code) and generate a 'STYL' resource with it.

The api name is TECSniffTextEncoding

> 2. convert to the KeyScript when paste I think this is confusing as we do not do script keyboard synchronization. We could do this only if we encounter the conversion error. > 3. depend on the charset of the document - however, there are cases that we >find a charset fot the document. for example, in the URL bar, what should we do > then, use 1, or 2 as fallback? I think changing conversion behavior differently depends on the context is confusing (similar to the keyscript solution) >TECSniffTextEncoding I think this is also good for a fallback solution.

Does MacOS support specifying the encoding in the clipboard? Both Windows and Unix/X have support for this (though both are underutilized). If you are pugging "plain text" on the clipboard in an encoding other than the system encoding, how on earth is the receiving application going to know what the sending application intended as an encoding? Since the clipboard will contain 8-bit text, the recieving application will have to interpret it - which means it must know its encoding or it must choose one - choosing one generally means using the system encoding. If MacOS supports a clipboard format which specifies the encoding, Mozilla should support this for both copy/cut and paste. When pasting to Moz from the clipboard it should prefer the clipboard format in this order: Unicode, plain text with encoding info, plain text with no encoding info.

The main problem is not how to paste *to* Mozilla, because like you said, it prefers Unicode and if you know how to do it, it is easy to get the system to recognize your text as of type Unicode before trying to paste into Mozilla. Mozilla will then, currently, prefer Unicode. The main problem is that if you try to transfer text *from* Mozilla, via the clipboard, then there are almost no other MacOS applications which prefer Unicode over plain text. And that is very critical. Netscape 4.x copies all text as plain text without coding information. And then I mean *all* in the sence "everything". But Mozilla, if you copy e.g. a cyrillic text, will only copy to the plain text layer that text which ISO-8859-1 and ISO-8859-5 have in common, namely punctuation and spaces... Plus, of course the Aa-Zz-letters. When I think about it, MOzilla should probably, on the plain text level, try to regain the features of Netscape4.x. "plain text with encoding info" exists in the form of a styled text layer, where e.g. cyrillic fonts belong to MacCyrillic script of MacOS. THis is the most compatible multiencoding (for that is what it is) clipboard transfer method on the mac. But with the coming of MacOSX it is becoming less relevant, and probably we won't get it therefore...

*** Bug 124678 has been marked as a duplicate of this bug. ***

Note for implementing keyscript fallback in case of the conversion failure using the platform charset. * nsPrimitiveHelpers need to return a result code so the caller can know about the conversion error. http://lxr.mozilla.org/seamonkey/source/widget/src/xpwidgets/nsPrimitiveHelpers.h#53 * The callers need to check the error and try another conversion using keyscript. http://lxr.mozilla.org/seamonkey/source/widget/src/mac/nsClipboard.cpp#154 * Example code which already using keyscript for Unicode conversion. http://lxr.mozilla.org/seamonkey/source/widget/src/mac/nsMacEventHandler.cpp#1085 Other implementaion possibility: * Implement the fallback inside nsPrimitiveHelpers. * Need a cross platform code to return a charset name for the current keyscript (i.e. impelent "kPlatformCharsetSel_KeyboardInput" for nsIPlatformCharset). http://lxr.mozilla.org/seamonkey/source/intl/uconv/public/nsIPlatformCharset.h#60

Bug 141248 was filed for implementing "kPlatformCharsetSel_KeyboardInput".

A couple of more fallbacks possible in addition to the keyscript fallback. * ConvertFromUnicodeToScriptCodeRun() for pasting from Mozilla. * Get a script of the first run if 'styl' is available. This is for pasting to Mozilla.

This is a working patch. Need testing. Also need to check availablity of the API in classic environment.

http://developer.apple.com/techpubs/macos8/TextIntlSvcs/TextEncodingConversionManager/TEC1.5/TEC.1.html > The Text Encoding Conversion Manager software can run on Mac OS System > 7.1 or later. The converter libraries and associated files are installed by default > as part of Mac OS 8 and as part of Mac OS Runtime for Java. So it should be okay to use the patch for the classic build.

summary: * nsPrimitiveHelpers changed to return a result code so the caller can try fallback changed convert from Unicode to try keyscript in addition to the system default * nsMacControl added functions to convert between Unicode and Mac native script * clipboard, dragdrop copy from Mozilla, added fallback to try non system default scripts copy to Mozilla, check 'styl' and if available use that script instead of using the system's default

i would prefer that these new functions be in their own class, not lumped in with nsMacControl. That way, we can re-use the file in chimera which doesn't use nsMacControl at all. + // if 'styl' is available, we can get a script of the first run + // and use it for converting 'TEXT' + loadResult = GetDataOffClipboard ( 'styl', &clipboardData, &dataSize ); if we're going through all the hassle to get the styl data, why not use it for text as well rather than dumping it and then going after the TEXT flavor? Is that not possible? It would also fix some bugs we have with office where it only puts styl on the clipboard, not TEXT. + for (ScriptCode i = 0, j= 0; i <= smUninterp, j < numberOfScriptCodes; i++) this probably doesn't do what you want it to do. the comma operator only returns the value of the last expr, in this case, the comparison of |j|. do you want to check both of these values? If so, use &&. + ScriptCodeRun *scriptCodeRuns = (ScriptCodeRun *)nsMemory::Alloc(sizeof(ScriptCodeRun) * numberOfScriptCodes); rather than mucking with all these return values and forcing ourselves to clean up early in exception cases, why not use an auto_ptr here? that way, when you return, the data will be automatically freed if it was constructed. much cleaner. auto_ptr<ScriptCodeRun> scriptCodeRuns = new ScriptCodeRun[numberOfScriptCodes];

'style' seems to contain style info only. The field 'scrpStartChar' contains an offset to a text data, I think that is 'TEXT'. I tested MSWord 98 with my patch and both 'styl' and 'TEXT' were put in to the clipboard from Word.

Comment on attachment 86673 [details] [diff] [review] Changed per reviewer's comment except for the first one since no text data in 'styl'. The indenting should be fixed here (else line): + factor = 6; + else + factor = 2; I prefer these lines to be about 6 lines lower: + ByteCount sourceRead; + ByteCount unicodeLen; For these methods, is there some reason that the strlen params are signed rather than unsigned ints? ConvertScripttoUnicode ConvertUnicodetoScript need a space after 'j': for (ScriptCode i = 0, j= The licenses in the new files don't look the same, should they be? Check mozilla.org for latest license to use.

Using signed integer the length is to match with nsPrimitiveHelpers and eventually intl/uconv interfaces. Not sure why uconv uses signed but changing them would require all converters to change. So not changing for this patch.

Comment on attachment 89131 [details] [diff] [review] Updated with reviewer's comment. > if (!scriptCodeRuns.get()) { > ::DisposeUnicodeToTextRunInfo(&unicodeToTextInfo); > return NS_ERROR_OUT_OF_MEMORY; > } tabs on these lines. other than that, looks good. I assume you've put both dnd and the clipboard through their paces to make sure things still work in cases both where the converters are used and where they are not. r=pink.

Comments: + StScrpRec *scrpRecP = (StScrpRec *) clipboardData; + ScrpSTElement *styl = scrpRecP->scrpStyleTab; + ScriptCode script = styl ? ::FontToScript(styl->scrpFont) : smCurrentScript; There is a chunk of code including this fragment that is repeated twice in the patch (once for clipboard, once for D&D). Can it be factored? In nsMacNativeUnicodeConverter.cpp: ConvertFromTextToUnicode() and ConvertFromUnicodeToScriptCodeRun() are usually called in a loop which repeats while not all the input has been consumed, and while the output buffer is too small. Should they be called that way here?

Simon, about looping to call the converter, we got all the input data at once and the converted result has to be provided as a whole data, so I don't see a reason not to call the converter once.

As long as you can be sure that your output buffer is large enough, fine.

Simon, the code to extract a script code from 'styl' data, I cannot find a good place to put a new function which can be shared by clipboard and D&D. The code is only needed by clipboard and drag&drop. I don't think we are going to have more places which need it. Can I leave the code as the current way?

*** Bug 148395 has been marked as a duplicate of this bug. ***

Comment on attachment 89131 [details] [diff] [review] Updated with reviewer's comment. sr=sfraser

Comment on attachment 89131 [details] [diff] [review] Updated with reviewer's comment. a=asa (on behalf of drivers) for checkin to the 1.1 trunk.

checked in to the trunk

mach-o build has a problem with the patch, so I backed out the patch.

Need to add the new file to Makefile.in. Index: Makefile.in =================================================================== RCS file: /cvsroot/mozilla/widget/src/mac/Makefile.in,v retrieving revision 1.22 diff -u -r1.22 Makefile.in --- Makefile.in 9 Jun 2002 00:05:37 -0000 1.22 +++ Makefile.in 17 Jul 2002 00:51:03 -0000 @@ -86,6 +86,7 @@ nsWidgetFactory.cpp \ nsWidgetSupport.cpp \ nsWindow.cpp \ + nsMacNativeUnicodeConverter.cpp \ $(GFX_LCPPSRCS) \ $(NULL)

r/sr=sfraser on the makefile patch

checked in to the trunk again it passed the mach-o tinderbox build

Verified fixed with 07-25 trunk build on Mac OS 9.2.2-US and OS 10.1.5.

You might visit <URL http://www.lenk.no/ > for an image of me and a test page with text in both cyrillic and norwegian. Try to copy all the text on that front page. Then paste it into WorldText. Result: all the russian text is deleted. Even some of the english text. Otherwise, I am of course glad that all text contained in a single MacEncoding (e.g. x-mac-cyrillic) can be pasted into Mosilla. I tested with MOzilla 1.1b on MacOS 9.2.2

This bug is only checked into trunk, but not in branch build. Can you try it on trunk build? Naoki: Is there any way that we can land it into branch build? because this bug status is verified fixed.

The patch is included in 1.1 branch.

The milestone target is 1.1beta. So if it is not allready in the 1.1beta, the target needs be futured, I guess.

Ok, I saw nhotta's comment."Better clipboard encoding" does not mean "perfect", right? Perfect it will be only when bug 79864 (Mac style text 'styl' support for mozilla clipboard) is fixed.Hower that bug are only Assigned... The only thing that can be said about "better clipboard encoding for text" is that it should not silently delete text outside the charset. EIther it should inset questionmarks, like it did before, for the text it can't handle. Or even the text outside the charset should be translated to that "better encoding". I.e. if copying a page with some french and some cyrillic text, then the cyrillic text should also be converted to 'TEXT'. Of course, the user would need to edit the text afterwards to distinguish the french from the cyrillic.But then at least the text is there.

i've found a testcase that doesn't work correctly and i'm not sure if it fits under this bug or bug 108029. go to www.yahoo.co.jp drag text to the finder open the clipping on the branch, it's all question marks ("??????"). On the trunk, it's better, but still not displaying in japanese (even with the OS language set to japanese when creating the clipping). what is missing to get that case to work?

oops, that should read "this bug or bug 79864"

Mike, You use MacOS X, right? Even thought you use the MacOS X 1.1 branch build on MacOS with Japanese UI, the Japanese characters in clipboard are displayed as garbage. If you paste Japanese characters to TextEdit, Japanese characters are diaplayed correctly. This is still bug for MacOS X. I checked with IE. After I copied Japanese text, the Japanese characters in clipboard are displayed correctly.

so do we open a new bug, or is bug 79864 supposed to address these issues for OSX?

I open the new bug 163908.

we need this on the branch for an embedding client ASAP

Xianglan, please test if this is landed in 1.0.1 (will be 1.0.2) branch build while I am on vacation.

a=chofmann for 1.0.2

adt1.0.1+ (on ADT's behalf) approval for checkin to the 1.0 branch, pending driver's approval. pls check this in asap, the replace "Mozilla1.0.1+" with "fixed1.0.1". [Using adt1.0.1 keywords as a proxy, since no adt1.0.2/edt1.0.2 keywords have been created for the 1.0.1 branch]. thanks!

Marking as a=chofmann for 1.0.2 via email.

checked in to 1.0 branch add fixed1.0.2

Verified as fixed with 08/29 1.0 branch build on Mac OS 9.1, Mac OS 10.1 and Mac OS 10.2. Both copy/paste and drag-n-drop work fine.