Tested on trunk, Linux/OSX (cannot get Windows/NTFS to give me a bad file name) The OS is set to use UTF8 for file names, as default for any recent distro. STR: Run attached xpcshell test case Expected results: No lines between "Enumeating..." and "done" Actual results: At least one entry (the file created near line 8) is listed The problem is that, although the filesystem supposedly is using UTF8, it can actually still hold files that are not. In this case, the file name is "\202.txt" (octal escaped; it's not a valid UTF8 encoding). The spec ends up as file:///tmp/%C2%82.txt ("%C2%82" is the escaped UTF8 encoding of \202); when attempting to read that file, it does not exist. However, file:///tmp/%82.txt does exist (if that is manually entered, it can be opened). Workaround: Do the Wrong Thing, and construct the spec as "file://" + nsLocalFile.persistentDescriptor (and be sad. and be scared of getting broken whenever somebody feels like it)

Problem is that native charset is set to UTF8 and filename isn't valid UTF8 string. Conversion UTF8 -> UTF16 fails when URL is build from nsLocalFile and latin1 -> UTF16 conversion is performed at http://mxr.mozilla.org/mozilla/source/xpcom/io/nsNativeCharsetUtils.cpp#747. 0x82 (latin1) == 0x82 0x00 (unicode) == 0xc2 0x82 (UTF8) My proposed solution uses native path in net_GetURLSpecFromFile() instead of UTF16 path and tries if it can be converted from UTF16 to the same native path back. If conversion is correct then it is safe to use UTF8 path name. Otherwise native path is used. I'm not sure if this problem can occur also when native charset is other than UTF8. If not then it can be simplified and conversion check could be omitted. nsURLHelperOS2.cpp can be fixed in the same way, but I can't test it so I didn't change it. But I have no idea how to fix this problem on OSX. The lossy conversion is done inside nsLocalFile implementation. nsLocalFile::GetNativePath() doesn't return native path but UTF8 path and it is already crippled. I think that this is a bug in OSX implementation of nsLocalFile.

Comment on attachment 328175 [details] [diff] [review] proposed fix for unix + // Convert to unicode and back to check correct coversion to native charset coversion -> conversion + ePath = NS_ConvertUTF16toUTF8(path); CopyUTF16toUTF8(path, ePath);

Comment on attachment 331472 [details] [diff] [review] patch changed according to comment #2 [Checkin: Comment 8] sr=bzbarsky, but please make sure to file the OSX bug, and hunt down someone to fix it!

Is it possible on OS X to have non-UTF-8 paths? At least http://en.wikipedia.org/wiki/HFS+ stores filenames in Unicode.

There is written in the description that the testcase was tested and failed on linux and OSX.

Can somebody test this bug under OS/2? OS/2 code is almost identical to Unix code, so I can create the patch for OS/2 too, but I'm unable to test it.

Comment on attachment 331472 [details] [diff] [review] patch changed according to comment #2 [Checkin: Comment 8] http://hg.mozilla.org/mozilla-central/rev/a81c47c501a8

R.Fixed. Maybe file a "followup" OS/2 bug too ?

Did the OSX bug ever get filed?

(In reply to comment #10) > Did the OSX bug ever get filed? Yes, #457702

(In reply to comment #11) > (In reply to comment #10) > > Did the OSX bug ever get filed? > > Yes, #457702 Sorry, that was #452516