** Observed with 6/8/99 Win32 build ** It doesn't look like Japanese folder names are supported currently in that they seem to be interpreted as Latin 1 or UTF-8. Try something like: テスト There is also a special problem with 0x5c JPN names in that folders containing such names do not show up at all. Example is something like: 日本語の表示

This bug is for local folder names, I think. Probably charset conversion is needed. I18n can provide infomation but I don't know about the current design and implementation of local folder name. Reassigning to putterman@netscape.com.

I don't think this is happening for m9, though I did change the GetName nsMsgFolder routine to support wide chars, I don't think we ever put wide chars in the name. please correct me if I'm wrong.

I don't think I'm doing that.

reassigning to M10

Triaging to M11

This will not be fixed for M11 and I question whether it will be fixed for M12. I will mark it M12 to allow for those who might want it fixed to comment on this.

We need this fixed for beta; Japanese users won't be able to display these folder names in their mail.

(target milestone is M11 or M12 - add to mail beta tracking bug)

Added [BETA] per 9/20 comment.

Hi, we need to get moving on this for Beta. I think what we need to do on this. 1. Make sure that when we display folder names on a system, we assume that the file names are in the file-system charset of the operating system Mozilla is operating on. 2. Make sure that when we create a new local mailbox names, we convert whatever we receive in input to the OS's file-system charset. The I18n group be able to provide info on how to query the system for the file-system charset. Bob/Naoki, we need to get this on our Beta radar.

*** Bug 23623 has been marked as a duplicate of this bug. ***

Here is the code to get the platform file charset. #define NS_IMPL_IDS #include "nsIPlatformCharset.h" #undef NS_IMPL_IDS nsCOMPtr <nsIPlatformCharset> platformCharset; nsAutoString aPlatformCharset; rv = nsComponentManager::CreateInstance(NS_PLATFORMCHARSET_PROGID, nsnull, NS_GET_IID(nsIPlatformCharset), getter_AddRefs(platformCharset)); if (NS_SUCCEEDED(rv)) { rv = platformCharset->GetCharset(kPlatformCharsetSel_FileName, aPlatformCharset); } For the charset conversion from unicode to the platform charset, ConvertFromUnicode (see nsMsgI18N.h) can be used.

Is this the same as bug 7130?

No. This is for Local folder names. Bug 7130 is for IMAP folder names. I believe the latter requires special conversion as IMAP requires the folder names to use modified UTF-7.

I think 7130 is about conversion between unicode (M-UTF-7 <-> UCS2). This is about unicode and file system charset. BTW, the sample code I put is already available as a function (nsMsgI18N.h).

Putting on PDT+ radar for beta1.

OK I will try this out. This isn't as bad as I had once imagined because as David said, we are now passing PRUnichar*'s around whereas I thought we were still passing char*. Now I just need to make sure they get converted correctly. At least the last time I looked, we were getting the folder name from the uri and we were getting the uri from the filename. Am I going to be able to test this out on my machine?

I have Japanese local mailboxes but these would not do for US Windows. I think for US Windows machines, Latin 1 accented folder names will have to do for debugging. For Japanese and other languages, if you let us know that you checked in a code that should work, we'll check it out for you. (You can change the system's default language via Control Panel but this tends to be very messy and not recommended.) I just tried migration of JPN mail fodlers from 4.x and it is botching a file with 0x5c 3nd byte. It chops off the name in half. I'll file another bug on this.

sorry, I guess what I was asking before is that I can view Japanese Imap folder names. Is there any reason why I shouldn't be able to view Japanese local folder names?

Yes, because the local folder names use the system charset, e.g. Shift_JIS on Japanese Windows while IMAP folder names are stored in modified UTF-7. We would need different conversion, don't we?

ok. I will try out what you suggested now using the code Naoki suggested.

I think one of the problems is that we build up the file names using nsFileSpec's and I think there are only char * interfaces. Won't this make this not work?

nsFileSpec reads and writes files using the OS's file system charset encoding. These encodings should be represented using char *. Does that answer your question? nhotta knows the details...

Yeah it does. I talked to Naoki about this earlier. I tried something out and according to Kat it was better but not there yet.

changing estimated date. I will need to debug on a Japanese machine. The hope is that this will be easy to find. Putting 2/25 down just in case it isn't.

After I applied a patch from Scott, I was able to see a Japanese folder on my localized Japanese windows. But cannot see the messeages in that folder. We can schedule and use my machine to debug.

Scott, I can help you on this one. As the MacOS charset is not ISO Latin 1, I should be able to reprocude this problem by just using some 8 bits characters on my "English" Mac.

Agreed, that should be easier than debugging raw Shift_JIS data on my Japaense NT.

I've just created a local forder named "L'été" and everything is fine except that the filename used for the folder was using the wrong charset.

I wonder if the patch I gave Naoki would fix JFD's charset problem? Anyway, I want to work on this today. Could someone give me a Japanese folder I can ftp over to my mac? Also, what should I be expecting to see in the debugger?

I failed to upload Japanese named file to the server, neither mail attachment succeeded. Could you pick up a floppy in my cube? Anyway, the folder contains two Japanese characters (0x82A0 and 0x82A2 in Shift_JIS), in unicode they are \u3042 and \u3044. If you debug on non Japanese machine, you also need to hard code the charset name (Shift_JIS) in msgCompFileSystemCharset().

The debugging tip I mentioned does not work on Windows unless using localized Japanese system. The file name turned to question marks so not able to convert even with hard coded charset name. But it works on Macintosh. The file name was passed in native encode (Shift_JIS) but not turned to question marks. I used MacOS9 with Japanese option installed.

per PDT, this will become PDT- after 2/25

The following patch makes it so I can see a folder with accents on my Mac without the recurring children that momoi was seeing (I saw this too so I know this fixes it). I can also select a folder and see all of the headers. Unfortunately, I can't read any messages. Naoki, could you test this out and verify that this works on a Japanese machine? ndex: nsLocalMailFolder.cpp =================================================================== RCS file: /cvsroot/mozilla/mailnews/local/src/nsLocalMailFolder.cpp,v retrieving revision 1.185 diff -r1.185 nsLocalMailFolder.cpp 72a73 > #include "nsMsgI18N.h" 218a220,221 > nsAutoString convertedFolderNameStr; > const nsString fileCharset = msgCompFileSystemCharset(); 223a227 > 225c229 < currentFolderNameStr = folderName; --- > currentFolderNameStr = folderName; 232a237,238 > ConvertToUnicode(fileCharset, folderName, convertedFolderNameStr); > child->SetName(convertedFolderNameStr.GetUnicode());

I applied the patch and ran on WinNT-J (Japanese localized NT). A folder with Japanese name was displayed correctly. But I was not be able to see the headers in thread pane. Probably that's because my tree is not up to date. I will try again tomorrow with updated tree.

Yeah, that's important. It needs the changes that bienvenu checked in to fix problems with uris and non-ascii characters. The reason I can't display a message is that strtok is finding a '/' in one of my characters and messing up the path. I think it needs to be escaped.

If I add the following code, I can display messages as well. This is the same code bienvenu wrote to replace strtok in nsMsgFolder. I wonder if this is affecting us in even more places. Naoki, when you get a new build, can you try this out as well. I think this will fix this bug. Index: nsLocalUtils.cpp =================================================================== RCS file: /cvsroot/mozilla/mailnews/local/src/nsLocalUtils.cpp,v retrieving revision 1.33 diff -r1.33 nsLocalUtils.cpp 189d188 < while ((*curPos)=='/') curPos++; 194,214c193,219 < < // for each token in between the /'s, put a .sbd on the end and < // append to the path < char *newStr=nsnull; < char *temp = PL_strdup(curPos); < char *token = nsCRT::strtok(temp, "/", &newStr); < while (token) { < nsCAutoString dir(token); < < // look for next token < token = nsCRT::strtok(newStr, "/", &newStr); < < // check if we're the last entry < if (token) { < NS_MsgHashIfNecessary(dir); < dir += sbdSep; // no, we're not, so append .sbd < pathResult += (const char *) dir; < } < else { < NS_MsgHashIfNecessary(dir); < pathResult += (const char *) dir; --- > > nsCAutoString newPath(""); > > nsCAutoString oldPath(curPos); > nsCAutoString pathPiece; > > PRInt32 startSlashPos = oldPath.FindChar('/'); > PRInt32 endSlashPos = (startSlashPos >= 0) > ? oldPath.FindChar('/', PR_FALSE, startSlashPos + 1) : oldPath.Length() - 1; > if (endSlashPos == -1) > endSlashPos = oldPath.Length(); > // trick to make sure we only add the path to the first n-1 folders > PRBool haveFirst=PR_FALSE; > while (startSlashPos != -1) { > oldPath.Mid(pathPiece, startSlashPos + 1, endSlashPos - startSlashPos); > // skip leading '/' (and other // style things) > if (pathPiece.Length() > 0) { > > // add .sbd onto the previous path > if (haveFirst) { > newPath+=".sbd"; > newPath += "/"; > } > > NS_MsgHashIfNecessary(pathPiece); > newPath += pathPiece; > haveFirst=PR_TRUE; 215a221,230 > // look for the next slash > startSlashPos = endSlashPos; > > endSlashPos = (startSlashPos >= 0) > ? oldPath.FindChar('/', PR_FALSE, startSlashPos + 1) : oldPath.Length() - 1; > if (endSlashPos == -1) > endSlashPos = oldPath.Length(); > > if (startSlashPos == endSlashPos) > break; 217c232,234 < PL_strfree(temp); --- > > pathResult+=newPath.GetBuffer(); >

I applied both patches and I can see Japanese folder name and headers in thread pane also messages in the view pane. So it is working for Japanese (tested on localized Japanese Windows NT). The only problem I saw was that a Japanese folder name which contains 0x5c (backslash) is not shown correctly (a character contains 0x5c is truncated). No thread pane header displayed for that folder either. I think this special case can be filed as a separate bug.

changing status to need approval to checkin

I assume that bienvenu and nhotta have already reviewed this, correct? Have ftang review this too. He's been tracking down lots of nsString misuages which might be causing the 0x5c problem. After ftang's review, you can check-in on my approval. (Run the pre-checkin tests, yada yada. You know the drill...)

I cannot review the of mozilla/mailnews/local/src/nsLocalMailFolder.cpp becuase I cannot see the context. Can you attach -c version here. Also, you should take a look at the patch in 6770 and try the new PRUnichar*/nsString flavor of GetLeafName()

also, when you have time (not necessary before you check in.) apply the patch in bug 28424 (remember to change the DEBUG_ftang to your name) and verify there are no illegal nsString usage in your code.

>Also, you should take a look at the patch in 6770 and try the new > PRUnichar*/nsString flavor of GetLeafName() I'll do this since it needs Japanese system in order to verify.

I include the patch of 6770 and rebuild. I also change the patch of this bug to use the unicode based GetLeaf method. I was able to see the folder name which contains 0x5c but cannot see the headers in thread pane. Looks like more place need change in order to resolve 0x5c bug.

As I mentioned, there are more than one problem to fix 0x5c problem. I got bobj's approval for the check in. Scott, please check in your patch.

Note: 6770 has been fixed. Per ftang, the additional method is: GetLeafName(nsString); not GetLeafUniName(nsString);

I'll check this in tomorrow. What do I have to change in my patch? I don't have any way of trying out any changed code over the weekend, but if you tell me what the changes are, I can put them in.

I talked to bobj and agreed that your patch can be checked in without the new GetLeafName (nsString version). As far as I tried using the new GetLeafName, it solved the display problem of 0x5c contained folder (not able to see headers in thread pane). So not useful until we fix other problems around 0x5c (which we don't know exactly where). Basically by using the nsString version of GetLeafName, you do not have to do the conversion in your code. But if we change to use it we need to test (I cannot test over the weekend). And as I mentioned, that change itself just fixes the folder name display, messages still cannot be read.

fix checked in. Should we open up another bug for the 0x5c problem?

There is going to be a local mail specific change needed for 0x5c problem (to use nsString GetLeafName). But I also see that I cannot create the name contains 0x5c with IMAP and local. I will ask IQA to test and file bugs.

Using today's build (2000-02-28), I see two problems: 1. Can't create a Japanese folder with IMAP. 2. With local folders, there are two problems with the new created japanese folder name: 1). If the Ja folder name doesn't contain 0x5c chars, the Ja folder can be created, but the Ja folder name is displayed as dots. 2). If the Ja folder name contains 0x5c chars, the specified new folder is not created. Instead, a folder with the same name as the local folder is created. For example, 1. Highlight Local Folders and select File | New | Folder.. 2. Enter a japanese folder name containing 0x5c char, like hyouji. 3. Click on OK button You'll see under Local Folders, another Local Folders is created, not the Japanese one you entered. Do I need to file seperate bugs for these?

IMAP bug is a separate one. If this cannot be done, then it is a regression. We probably should open a new bug.

Sorry, the IMAP bug is already there. It is Bug 26745. It is still open.

Yes, please file separate bugs. Also, once the IMAP bug is fixed, please try a folder with 0x5c on IMAP (and file a bug if that exists).

You're referring by "this" and "it" to #2, right? We don't need a new bug for IMAP, at least I don't.

David. I just noted in Bug 26745 that we should check for 0x5c filename issue whenit is resolved. That should be sufficient as you say.

Bug 29543 is the general bug for creating JPN name folders. Bug 29546 is a specific problem with creating a JPN folder name with 0x5c in it. With these 2 bugs filed, this bug has been reduced to just the display one for migrated folder names. We still can't see any contents of migrated folders with 0x5C in it, but I take it that this is taken care of in another bug. For Windows 02-28-2000 M15 build, this bug then has been fixed. All Japanese names show correctly. Will Look at Mac and Linux next.

With Linux 3/6/2000 M15 build, I see Local mailbox names under EUC locale. With Mac 3/5/2000 M15 build, under Japanese Mac OS 8.6, I can see Japanese folder names correctly. On Mac and Windows NT/2000, you really don't need to have a Japanese OS to see Japanese file names. This latter issue is not addressed by the current fix. As I understand it, we have to deal with this latter problem at a more general level. With these provisos, the current fix has been cnfirmed. Marking it verified.