In all places where a URL is displayed to the user, Mozilla should display the URL using the local character set, if possible. Only when a character cannot be displayed using the local character set, should it be displayed using %-encoding. Two of the most common places where a URL is displayed is in the Location field and at the bottom when the mouse is moved over a link. But also URLs are used in dialog windows. In current Mozilla I have seen URLs displayed using local character set, in %-encoding and in UTF-8. I can enter a URL in the Location field using local character set and I can click on a link containg a URL in the character set of the HTML document, but when I go to the page the URL is displayed using %-encoding in the Location field. This is not acceptable for a user. A URL must always be displayed using local character set. To get things to work well Mozilla should do like this: All URLs should, if possible, be converted into UTF-8 internally. The UTF-8 should be normalised using Unicode normalisation form C. All comparing and handling should be done using this standard format so that for example cookies work. If a URL cannot be converted to UTF-8, for example if the character encoding cannot be recognized and the user have given no information of default character set to assume, it will have to be retained in original form. When a URL is displayed to the user (in all places), a URL must have all character displayed using the local character set if the character can be displayed in it. If not, %-encoding is used. If a URL is not in UTF-8 format, the characters should be displayed using the local character set, if the byte value can be displayed as a character. For example, if the local character set is ISO 8859-1 and a non-UTF-8 URL contains character values in the range 0xa0-0xff, those values should be displayed as ISO 8859-1. Character in the range 0x7f-0x9f should be displayed using %-encoding. When transmitting URLs to web servers, they should send them in a character set supported by the server, if known. If unknown the default character set defined by the user should be used. The URLs need not be transmitted using %-encoding, if preferred by user the pure binary format may be used. A MacOS user entering a URL should not have the URL sent using a MacOS character set unless the user have defined this as the default. Summary: Internally Mozilla should use one format on all URLs so they can be compared - UCS normalised using Unicode form C and the encoded using UTF-8 is a good one. In all places where a user enters or is shown a URL, the URL must be displayed using the local (client) character set only encoding those characters not supported by the local character set as %-encoding.

->ftang

>Mozilla should display the URL using the local character set, if possible. Why? The following reason for Why Not? 1. the URL are from the net, not from the "local" 2. What is "local character set", we don't have such thing. 3. IETF is moving to use UTF8 for URL >All URLs should, if possible, be converted into UTF-8 internally. The problem is how to do so if you don't know the source charset of the URL. >If a URL cannot be converted to UTF-8, for example if the >character encoding cannot be recognized and the user have >given no information of default character set to assume, it will >have to be retained in original form. Tell us any place which we COULD convert to UTF-8 but we don't now. >When a URL is displayed to the user (in all places), a URL >must have all character displayed using the local character set >if the character can be displayed in it. Why ? If all the URL is UTF-8 then why we only display those characters in local character set ? >When transmitting URLs to web servers, they should send them >in a character set supported by the server, if known. How do you know what charset is supported by the server? in HTTP protocol? in IMAP protocol? in POP3 protocol? in FTP protocol? what should we do if they support more than one charset, which one we should choose? >If unknown the default character set defined by the >user should be used. Why? how could user know what charset should be default ? >A MacOS user entering a URL should not have the URL sent >using a MacOS character set unless the user have defined this >as the default. I don't think we ever do so, not even in Netscape 1.0 >Internally Mozilla should use one format on all URLs so they >can be compared - UCS normalised using Unicode form C and the >encoded using UTF-8 is a good one. In theory, we already use UTF-8 as possible as we can. We cannot do so in File url for backward compatability reason.

>>Mozilla should display the URL using the local character set, if possible. >Why? >The following reason for Why Not? >1. the URL are from the net, not from the "local" Some come in through the net, others are entered locally in a client. But the important thing here is that all characters in a URL that can be displyed for the, is displayed for the user insted of showing them as %-encoded ASCII. For example, when I om my web go to URL: /Affärsenheter/ Mozilla displays: /Aff%E4rsenheter/ in the localtion field. This is not right. >2. What is "local character set", we don't have such thing. The local character set is the character set the user is using on the system the client (mozilla) is run on. For example, on my Solaris system the local character set is ISO 8859-1 (all documents, web pages are in that character set) and on a MacOS 9 system the local character set is an Apple invented character set. (by character set I mean the mapping between character s and code points and it represents the characters that can be displayed as a character). On a Unix system the local character set is defined by the current locale. >3. IETF is moving to use UTF8 for URL Yes, UTF-8 is a good choice for interoperability, but as UTF-8 is not always the local character set, you have to convert netween UTF-8 and the local character set to be able to display the character for the user. >>All URLs should, if possible, be converted into UTF-8 internally. >The problem is how to do so if you don't know the source charset of the URL. Agreed. You have to guess to let the user tell you. >>When a URL is displayed to the user (in all places), a URL >>must have all character displayed using the local character set >>if the character can be displayed in it. >Why ? If all the URL is UTF-8 then why we only display those characters in local >character set ? Maybe I was unclear. See the example above. If that URL had also included a Korean character which cannot be displayed by my local character set ISO 8859-1, that character should be displayed as %-encoded. But all characters in the URL that can be displayed by ISO 8859-1, must be displayed without %-encoding. >>When transmitting URLs to web servers, they should send them >>in a character set supported by the server, if known. >How do you know what charset is supported by the server? in HTTP protocol? in >IMAP protocol? in POP3 protocol? in FTP protocol? what should we do if they >support more than one charset, which one we should choose? Yes that is a problem. I have no solution. HTTP should probably use UTF-8 or ISO 8859-1 unless user have said otherwise. IMAP has a protocol for sending non ASCII. FTP has UTF8 and local character set defined to be used. POP3 I do not think has any support for it. >>If unknown the default character set defined by the >>user should be used. >Why? how could user know what charset should be default ? Well, for example I have a MacOS user. As he knows that most servers he visits use ISO 8859-1, it is much better if the browser sent ISO 885-91 instead of MacOS-encoding. >>A MacOS user entering a URL should not have the URL sent >>using a MacOS character set unless the user have defined this >>as the default. >I don't think we ever do so, not even in Netscape 1.0 Oh yes it does. My web server has a special fix to recognise URLs sent in MacOS character set just because of this.

so... basically what you said is we should try to display the URL without % there, right ?

the problem is the URL bar is a text field, if we convert it into a for that can be display without %, we may LOST information when user click RETURN

>so... basically what you said is we should try to display the URL without % >there, right ? Yes. Would you like an A to be displayed as %41 in an URL? To have my non-ASCII letters displayed as %-encoded characters is as unacceptible for me as %-encoding all ASCII letters would be to you. >the problem is the URL bar is a text field, if we convert it into a for that can >be display without %, we may LOST information when user click RETURN Then you have to store that information somewhere else. The user want to see URLs in native character set and to enter them using native characters. If Mozilla handles all URLs in UTF internally, it is fine, but not when displaying them to the user. Today my cookies do not work in some places. This is probably because the path for the cookie is in an another format than the %-encoded format Mozilla puts into the Location bar. Apparently Mozilla do not convert all URLs to one standard format when comparing paths.

Reassign this to nhotta. nhotta, should we future this one ?

Whenever an URL is decoded without knowing a charset it may not work. I think the current behavior of percent encoded should be the default. If decode using locale charset or UTF-8, the user has to be notified about possible data loss. We may change the backend code first with a backend pref (e.g. encode, decode as UTF-8, decode as system charset) to experiment.

>I think the current behavior of percent encoded should be the default. No, absolutely not. In old Netscape, when I type a URL: .../Affärs in remains that way. In Mozilla it is immediately converted into: .../Aff%C3%A4rs. This is very ugly. Every time I want to copy and paste the URL into some other place I have to manually decode the %-encoding to get it right. I can't give other people a such ugly and unreadable URL. And what are all those who do not understand %-encoding and hexvalues going to do? They will just be irritated and complain. Change Mozilla so that the characters ABC also are translated into %-encodings and see how ugly it looks and works. And cookies do not work! It looks like Mozilla compares the %-encoded (or UTF-8 encoded) URL with the cookies path in ISO 8859-1. There clearly is much more work to be done to get URLs handled correctly (as a user sees it) in Mozilla. I think the best possible way for handling URLs is to everywhere internally convert them into UTF-8 (and leave them as invalid UTF-8 if it cannot be converted into UTF-8). And display them as good as possible when displayed, and send them to servers either as UTF-8 or, by user defined, character encoding.

>In old Netscape, when I type a URL: .../Affärs in remains that way. I think this was just working for ISO-8859-1 only. There is a bug about internal UTF-8 URL, bug 84032. Even using UTF-8 internal URL, it needs info about a charset, because the server's charset may not be UTF-8. >(and leave them as invalid UTF-8 if it cannot be converted into UTF-8 This does not work for user typed URL.

*** Bug 29698 has been marked as a duplicate of this bug. ***

Since the problem in bug 29698 is on all platforms which has marked as duplicated as this one, I'm changing platform to All.

OK, what if I have URL with cyrillic CP-1251 (windows) chars, but my local charset is KOI8-R (unix)? What about CP866 (dos encoding)? Huh, they all 8-bit single byte charsets, any string in one encoding can be displayed in another, but you'll get garbage. > I can't give other people a such ugly and unreadable URL. Why not? This is just URL, not Shakespeare's poetry ;-)) What the use of just reading URLs? Paste 'em into browser url bar or bookmark them, that's it! > And what are all those who do not understand %-encoding and hexvalues > going to do? They will just be irritated and complain. There's URL containing thousands of characters. Do you read them all? ;-)

>OK, what if I have URL with cyrillic CP-1251 (windows) chars, but my >local charset is KOI8-R (unix)? What about CP866 (dos encoding)? >Huh, they all 8-bit single byte charsets, any >string in one encoding can be displayed in another, but you'll get garbage. Yes, you cannot just take a character encoding and without translation display it using some other encoding. When a html page is displayed, it is translated from its character set into the one used in the web browser. You will have to do the same with URLs. If you have a URL in cyrillic CP-1251 and want to display it on a system using KOI8-R, you have to convert from cyrillic CP-1251 to KOI8-R before displaying it. You have the same problems when comparing URLs. Today Mozilla fails to handle cookies for me, because when I type in a URL like: /Tjänster/xxxx, Mozilla displays it as /Tj%E4nster/xxxx, but the cookie set by the program sets the path of the cookie to: /Tjänster and the matching routines in Mozilla do not do the comparing correctely. MS IE does the same fault when set to send all URLs as UTF-8. The best way to do thisng would be to convert all URLs from its character set into UTF-8 internally. All comparing should be done on the UTF-8 form. When displaying, the UTF-8 encoded URL is decoded and all displyable characters are displayed as itself, and others using %-encoding. When sending to server you either send them as UTF-8 or with the format the URL had when entered into Mozilla. There are drafts from W3C recommending URLs containg non-ASCII to use UTF-8, but just like html pages will be in different character sets, most URLs will be written in the same character set as its surroundings. In a html page it will be the same character set as the rest of the text, in a dialog it will be the local character set used to display and enter text. Mozilla will have to do the convertion between characters sets when URLs are displayed or used in different contexts. Just changing everything that is not english letters into %-encodings is not right, it will just make non-English people using Mozilla displike it and select a browser that works for international users. >> I can't give other people a such ugly and unreadable URL. >Why not? This is just URL, not Shakespeare's poetry ;-)) >What the use of just reading URLs? Paste 'em into browser url bar or bookmark >them, that's it! Many URLs will only be used in a context where they are pasted or just called from a web page. But there are many URLs that are used to make it possible for people to remember or easy to use. Should we say to mozilla users to go to: www.mozilla.org/r%65l%65%69s%65s to look for releases? Not easy to remeber, not easy to type. No, companies do not want to have to write %-encodings when advertising. URLs are not just a binary string, many of them are used to give information on what they relate to and to make them easy to remember. Try to turn on %-encoding on all letters in URLs and see how easy it is to use. Then you will get some feeling of what non-English speaking people will see if non-ASCII is displayed as %-encoding.

This is one of the series problem caused by darin's patch for bug 124042. Before this patch, we used to just escape the local string, and unescape it before display. After his patch, nsStandardURL::GetFileName, nsStandardURL::GetFileBaseName, (and probably more) are assumed to return string in UTF8 encoding. Now 2 problems arise: 1, when nsStandardURL (or its subclass/superclass) is initiated with string, those strings are not always converted to UTF8. 2, when the string is used for displaying, unescape is not done. We must fix both those 2 issues. There might be many places which need to be fixed. Bug 135854, 136221, and prossibly 127476 are just some examples.

This problem causes another problem which is that drag&drop their icon from URL to somewhere such as desktop. Created icon doesn't work beside, displays error message because can't understand file name/folder name to link.

Comment # 16 splited into bug 160236

See also bug 102984

*** Bug 187445 has been marked as a duplicate of this bug. ***

*** Bug 138951 has been marked as a duplicate of this bug. ***

*** Bug 137597 has been marked as a duplicate of this bug. ***

*** Bug 230421 has been marked as a duplicate of this bug. ***

*** Bug 234154 has been marked as a duplicate of this bug. ***

i'm not sure if this the same bug or not. if we put this wikipedia url to Mozilla http://th.wikipedia.org/wiki/╩цу╢у_╬╧аб╖╓Л it will point to wrong (empty) page http://th.wikipedia.org/wiki/%C2%BB%C3%83%C3%95%C2%B4%C3%95_%C2%BE%C2%B9%C3%81%C3%82%C2%A7%C2%A4%C3%AC but Internet Explorer can handle it correctly, leads us to http://th.wikipedia.org/wiki/%E0%B8%9B%E0%B8%A3%E0%B8%B5%E0%B8%94%E0%B8%B5_%E0%B8%9E%E0%B8%99%E0%B8%A1%E0%B8%A2%E0%B8%87%E0%B8%84%E0%B9%8C I think that other non-european languages also have this problem.

*** Bug 257198 has been marked as a duplicate of this bug. ***

>the problem is the URL bar is a text field, if we convert it into >a for that can be display without %, we may LOST information when >user click RETURN This problem is not new. Every wordprocessor and editor has to deal with it. If the character is in the users character set, display it properly, else keep it encoded.

Yes, and other languages have same problem, like Cyrilic alphabet. You can see on this image that only Opera show url in Cyrilic, i try this with Opera, Firefox, Netscape and IE. Image is here http://djevrek.on.panonnet.net/slike/LOOK.PNG.

See http://www.ietf.org/rfc/rfc3987.txt for the internalized resource identifier (IRI) recommendations. Konqueror and Safari currently behave as Sasa describes for URLS that are encoded in UTF-8 (and thus can be considered IRIs): they get displayed as unencoded text, while other encodings, eg ISO-8859-1, will still show hex codes. This generally provides a nice UI, although copy-and-paste from the URL bar can be annoying when you really wanted the encoded form. For non-ASCII characters the URL encoding is really, really ugly: 6 bytes per character for Cyrillic, 9 bytes per character for CJK expands to long illegible, non-memorable URIs. IRIs should if possible be presented to the user in a legible way, though this carries with it similar problems (and benefits!) as IDN hostnames.

I think we should at least unescape the URL when it's escaping URL-safe low-bit ASCII characters; i.e. if we wouldn't escape the character if you typed it in, we should unescape it if it occurs escaped. For example, if you paste http://www.mozilla.org/products/%66%69%72%65%66%6f%78/ into the address bar it's really the same as http://www.mozilla.org/products/firefox/ but you can't tell that. You can use this for example to obscure the actual address of websites that you send people via IRC/email/etc even after they've loaded the page. This of course doesn't apply to the hostname portion of the URL where an escaped character can mean something different from an unescaped one.

*** Bug 309025 has been marked as a duplicate of this bug. ***

Is there any news about fixing/trying to fix this bug ? I think that this option is very needed in FF 2.0, but because no one share my opinion. Please just try to add option for this in about:config for FF 3.0. Sasa Stefanovic

Workaround: The extension Locationbar² (see http://en.design-noir.de/mozilla/locationbar2/ ) decodes the URL.

In the patch for bug 366797, URLs are decoded this way: + try { + val = decodeURIComponent(urlParts[4]); + } catch(e) { + val = unescape(urlParts[4]); + } Please let me know if that is sane and if it would fix this bug.

Apologies for duplicating this in buglizza, which was kindly fixed by Magnus. It seems there has not been much discussion in this thread as of late. Doesn't firefox's "network.standard-url.encode-utf8" not fix this problem, and if so, why is it not set as the default? http://www.w3.org/International/O-URL-code.html says everything should be using UTF8.

Oops, I apologize for double posting, I also forgot to mention the setting "network.standard-url.escape-utf8", which should be false by default.

(In reply to comment #35) > Doesn't firefox's "network.standard-url.encode-utf8" not fix this problem No, see http://kb.mozillazine.org/Network.standard-url.encode-utf8 (In reply to comment #36) > "network.standard-url.escape-utf8", which should be false by default. That would be an improvement, but not the full solution, imho. I.e. set it to false and go to http://th.wikipedia.org/wiki/%E0%B8%9B%E0%B8%A3%E0%B8%B5%E0%B8%94%E0%B8%B5_%E0%B8%9E%E0%B8%99%E0%B8%A1%E0%B8%A2%E0%B8%87%E0%B8%84%E0%B9%8C What you see is that this pref won't cause already-escaped URLs to be unescaped.

I'm not sure if this is the same bug or if it should be filed separately -- but Firefox's History records escaped and unescaped URL's as two separate entries. For example http://en.wikipedia.org/wiki/Zombie_(disambiguation) and http://en.wikipedia.org/wiki/Zombie_%28disambiguation%29 are both listed. Opera doesn't do this and always unescapes characters in both the location bar and history.

Fixed as part of bug 366797.

The fix for bug 366797 was backed out. Please reopen this bug, too.

just to clarify why decoding URLs is important. most Wikis and many blogging platforms generate URLs based on article title, which means there is an increasing number of URLs in non ascii chars. and in the case of wikis one needs to edit the URL from time to time. as mentioned here before Konquerer decodes UTF-8 URLs.

<SphinX> http://www.degadget.com/laptops/iphone’s-multi-touch-technology-on-macbooks-in-october <SphinX> sorry... that apostrof should be like that: http://www.degadget.com/laptops/iphone%E2%80%99s-multi-touch-technology-on-macbooks-in-october <jruderman> interesting, safari's "feature" of decoding %-escaped URLs breaks pasting into irc [because irc clients don't consider the "smart apostrophe" to be a URL character] <jruderman> how do you get the escaped url back? <SphinX> c/p-ed that from the guy who gave me the link in the first place ;-) We should make sure power users have access to the escaped URL. Perhaps copying from the address bar should automatically use the escaped version.

(In reply to comment #43) > Perhaps copying from the address bar should automatically use the escaped > version. The patch in bug 366797 does that (or at least tries to), if you select the entire URL. Which build was SphinX testing? Presumably one of the builds that included the fix before it was backed out?

(In reply to comment #44) > (In reply to comment #43) > > Perhaps copying from the address bar should automatically use the escaped > > version. > > The patch in bug 366797 does that (or at least tries to), if you select the > entire URL. Indeed, and it always worked for me. > Which build was SphinX testing? Presumably one of the builds that > included the fix before it was backed out? Looks like he wasn't using Firefox at all?! (Jesse mentions Safari.)

(In reply to comment #45) > > Which build was SphinX testing? Presumably one of the builds that > > included the fix before it was backed out? > > Looks like he wasn't using Firefox at all?! (Jesse mentions Safari.) Oh, that makes sense. Glad you already have that case covered :)

Almost 6 years later, still discussing this bug... *sigh*

I guess with the exception of text-mode terminals, there are no longer such things as "display character set" and "input character set". Any character can be displayed (at least as a box with a number inside) and entered (at least with some Alt-numpad or XKB compose and dead keys). If I understand correctly, the application such as Firefox does not need to perform extra encoding of output and input characters. This means utf8/hex-encoded links can (a) always be decoded when showing them in the URL bar. On entering links, they (b) can always be encoded into utf8-hex. My Firefox at work is "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.4) Gecko/20061201 Firefox/2.0.0.4 (Ubuntu-feisty)". I did not see (a) utf8-hex decoding in Firefox's location bar yet. On input (b) in Linux, the location bar seems to unnecessary convert Unicode input into the LANG charset, then hex encoding the result. This should be changed to "convert Unicode input to utf8, then hex encode the result".

The patch in bug 366797 has been checked in again, so this is FIXED again.

I'm getting %-encoding in the address bar again.

Only UTF-8 addresses are decoded. (Assuming nothing has been backed out silently.)

(In reply to comment #53) > Only UTF-8 addresses are decoded. (Assuming nothing has been backed out > silently.) > Because that's the standard set down by w3c.

It appears that decoding of every percent-encoded character may result in non-equivalent URI. See example 2 of encoding the slash "/" character in http://www.w3.org/Addressing/URL/4_URI_Recommentations.html I guess the existing URI decoding algorithm in Firefox 3 URL bar is already cautious about this issue. From seeing some Wikipedia URIs in Firefox 3 It appears it leaves non-decoded even the comma character. Other than that extra precaution in the existing algorithm, I don't see any issue in Firefox 3 URI bar. I am not aware about Unicode invisible or similarly-looking characters, though.

How about readable non-ASCII urls in visited links drop-down list? New TT needed of this issue can be fixed within this TT?

I also think the visited links should be consistent with what is shown in the location bar, currently it is non readable - and you don't need compatibility with IRC clients there, it is a fully firefox thingy. I hope it will also be shown in a readable form.