We have two instalations of Chimera 0.4 . number1 is on a dual-1GHz OSX 10.1.5. A previous version of Chimera was on the machine and was overwriten by 0.4. number2 is on dual 500Mhz OSX10.1.5. A previous version of Chimera had been installed (probably older than the one that was on number1) but had been deleted some time ago. I installed Chimera 0.4 a few days ago. The problem is that on number2 Chimera is not sending any accept-language header. It is sending "accept-language = ("en-us, en;q=0.50");" on number1. This caused our localized WebObjects app to throw an exception (something we have to fix on our side). However this could be a problem with other sites. I'm not sure if accept-language is a compulsory header... Here is a full printout of the headers. number1: httpVersion=HTTP/1.1 headers={user-agent = (Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-US; rv:1.0.0) Gecko/20020724); keep-alive = (300); host = (www.foobar.com:8080); accept = (text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,video/x-mng,image/png,image/jpeg,image/gif;q=0.2,text/css,*/*;q=0.1); accept-encoding = (gzip, deflate, compress;q=0.9); accept-language = ("en-us, en;q=0.50"); connection = (keep-alive); } content-length=0 cookies=null userInfo=null>) method=GET uri=/cgi-bin/WebObjects/ElectronicCommerce.woa defaultFormValueEncoding=ISO8859_1 formValueEncodingDetectionEnabled=NO formValueEncoding=ISO8859_1 formValues={} number2: httpVersion=HTTP/1.1 headers={user-agent = (Mozilla/5.0 (Macintosh; U; PPC Mac OS X; rv:1.0.0) Gecko/20020724); keep-alive = (300); host = (www.foobar.com:8080); accept = (text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,video/x-mng,image/png,image/jpeg,image/gif;q=0.2,text/css,*/*;q=0.1); accept-encoding = (gzip, deflate, compress;q=0.9); connection = (keep-alive); } content-length=0 cookies=null userInfo=null>) method=GET uri=/cgi-bin/WebObjects/ElectronicCommerce.woa defaultFormValueEncoding=ISO8859_1 formValueEncodingDetectionEnabled=NO formValueEncoding=ISO8859_1 formValues={}

*** Bug 161343 has been marked as a duplicate of this bug. ***

have you tried blowing away your profile? Sounds like something we may have added inbetween versions. Pink, bryner, ideas?

I tried deleting the chimera directory in ~/Library/Application Support/ , no change. Is there anything else you might need? How does Chimera get its list of languages?

Is there anything I can use to view the accept-language header being sent?

I use the printout of my WebObjects application but you could use http://www.ElfQrin.com/binfo.shtml

According to the specs, Chimera is actually doing the right thing now... From http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html: "As intelligibility is highly dependent on the individual user, it is recommended that client applications make the choice of linguistic preference available to the user. If the choice is not made available, then the Accept-Language header field MUST NOT be given in the request." Check out bug 182610 for adding a preference to choose accept-language header.

The accept-language should be changed by using the international system preferences pane. This is what I think is the expected behavior by users. Mac OS X provides a built in selection of preferred languages that is what should be used, don't reinvent the wheel.

Problem is, I doubt the Mac OS Language setting uses HTTP language tags (defined in RFC 1766). If they do, I would be happy to reopen this bug, but otherwise there is no guarantee that a particular language chosen in the System prefs will correspond to a valid language tag.

Ya Ok I had another look at the preference pane and bug 182610. The international pane is only language so even if it was using http language tags it doesn't have the necessary country. let alone the variant.

Correction: RFC 1766 has been depricated by RFC 3066. Also, RFC 3066 is based on ISO-639 (two-letter and three-letter language codes) and ISO-3166 (two-letter country codes). BTW, I came across a page at unicode.org that shows a table of which Mac language codes correspond to which ISO language codes: http://www.unicode.org/unicode/onlinedat/languages.html I have no idea if this table is valid for Mac OS X, but if so, it may be possible to pull the language tags from the system prefs after all (as I believe the country and variant codes are optional). I'll go ahead and reopen the bug and leave it up to the developers to decide. Ideally, I would still prefer a separate preference (bug 182610) for the following reasons: * Apple's language pref is for "application menus and dialogs". * The W3C spec specifies that the choice should be available in the client application. It does not mention OS preferences being applicable. * People who develop web pages or web applications in several languages (such as myself) would love to have a way to test their pages without having to change the language for the entire OS. That said, if bug 182610 is deemed to be bloat, this bug may be a decent work-around, as having no language code at all is less than ideal. Of course, I'm still not sure if this bug is even doable. Perhaps one of the developers could give us a better idea.

Changing summary and severity to reflect repurposing of this bug per comments 7-10. I probably should have just opened a new bug, but I'm a lazy bastard :) Previous summary: "accept-language header missing" New suumary: "pull accept-language header from system prefs"

You "must" have a different dialog, it has nothing to do with the Sys Pref. In a lot of countries, users are obliged to use a program in one language, no translated version available, that does not mean they want to see the web pages in that language. In Belgium we have 3 languages, sites are using that very often, an exception apple.be :-)

RE: comment 12, the system preference lets you rank the order of languages which you want to use, so you should put your preferred language at the top, and the system will fall back to other languages if a localization is not available. You shouldn't set a language at the top just because it's the one that more programs have available. I don't see why these wouldn't be exactly the same settings to use for web browsing; it should be a list of all supported languages in the order the user is most comfortable with them. The real problem is that the system prefs don't contain all the possible HTTP languages. They probably do contain the ones 90% of our users will use. An alternate solution would be to use the international language settings to create the default settings, which then only the other 10% of users would have to change.

Steve, that's a one way solution. You think program then page, you can have also the contrary. Think about it you'll see you are obliged to have both, programs via SysPref and Page.

The user will always set up their program language preferences before their page language preferences because you set up the system preferences language during the Mac OS X installation process. But I still agree that we need a seperate setting since the system preference doesn't cover all the languages available for web pages, I just think we should use system prefs to set the default the first time a user starts Chimera. If a user's native language is available under system prefs, then it will already have been set as the first preferred language during system installation, and by setting it as their default accept-language, we're saving them a trip to the Chimera preferences.

*** Bug 187169 has been marked as a duplicate of this bug. ***

Some more commentary on this bug would be helpful. Any new perspectives?

Can somebody compare our current accept-language header behavior to that of Firefox on Mac and Windows?

Oh yeah - also a comparison with Safari would be nice.

Okay, a quick summary of the different browsers behaviour. Camino: Claim to understand US English fluently and all other English dialects only half as well. (Accept-language= "en-us,en;q=0.5"). No way of changing this without resorting to the source code. (If about:config worked the user could alter the intl.lanugages option, see bug 209070). Safari: Take the list of languages in the "International" pane of Systems Preference. Claim to understand the first language (English in my case) fluently. Claim to understand the rest of the checked languages less and less well (docking 1/n for each entry further down the list of n languages). For each language (apart from English, not sure it this is because its special, or because its my first language) claim to understand the primary dialect (eg. "fr-fr") slightly better (~0.05 units) better than the generic (e.g. "fr"). (Accept-language= en, ja;q=0.29, es;q=0.88, de-de;q=0.82, de;q=0.76, nl-nl;q=0.71, nl;q=0.65, it-it;q=0.59, it;q=0.53, da-dk;q=0.47, da;q=0.41, ja-jp;q=0.35, fr;q=0.94, ko-kr;q=0.24, ko;q=0.18, no-no;q=0.12, no;q=0.06) This is fine expect that the international pane has far too many languages ticked by default (sorry Norwegians, but you'll have to make do with my (very limited) Danish). Mozilla and Firefox: Take the languages in order from their own preferences dialog and divide up the unit space equally between them. (Accept language = en-gb,en;q=0.8,en-us;q=0.6,fr;q=0.4,de;q=0.2) Basically Camino is completely broken for those whose first language isn't English, and broken (but useable) for all those of us who consider English rather than American our preferred language. I'm not sure how many people will have edited their international preferences pane, certainly mine isn't as refined a list as I have set up in Mozilla at work). However it has the advantage of being what's used by the other big Mac browser, and it would avoid the need to write lots of UI. I think we should probably pick up the information from system preferences.

Okay, I think what we need to do is: 1. Pull the list of languages from system preferences (see http://developer.apple.com/documentation/Cocoa/Conceptual/Internationalization/Concepts/InternatSupport.html) for details of how to do this. Note that this page implies you get back a name for the language rather than an ISO code, but other pages imply that only ISO codes are supported. The code for WebCore might make this clearer). 2. For the top language in the list we need to work out what dialect we want to declare. We can do this by examining the user's region. (http://developer.apple.com/documentation/CoreFoundation/Conceptual/CFLocales/Articles/CFLocaleWorking.html#//apple_ref/doc/uid/20002241/BBCEFFGC) 3. Set up the intl.languages option in Gecko with something link L1-XX, L1;q=0.95, L2;q=(n-1)/n, L3;q=(n-2)/n, ... Ln;q=1/n Where L1 is the lanuage at the top of the list and XX is the region we determined, and L2...Ln are all teh other languages in the list. (I don't think we can sensibly determine the region for these, and I don't like the logic that Safari uses of "assume its the same as the language".)

cc'ing Ludo as its probably of interest to people interested in localising the browser! Are they currently recoding the language-accept header as part of localisation?

Taking this bug.

This patch implements the algorithm I outlined in my earlier post. Seems to generate a reasonable accept-language header. Something in necko (?) is truncating the numbers to only 1 dp, so the numbers generated are slightly different from Safari. I'm going to ask Ludo to do the first review of this one. It would benefit from testing by people who's primary language isn't English.

This patch unfortunately causes us to effectively ignore any setting for intl.accept_languages the user may have in their user.js file, in favour of our computed value. This is a bit nasty, but I can't see a way round this. If we only compute the value if the pref is empty then we will always end up using the values we compute the first time round, even if the user subsequently edits the languages list in System Preferences. Is there any way to determine if a preference was set in user.js or prefs.js? Also, I've realised the necko isn't doing rounding to 1dp I mentioned. Its actually computing its own q values. IMHO its wrong for necko to do this, and we should still compute q values in case necko ever changes its behaviour to accept q values from the pref.

We need some good test cases for this bug, becuase I really don't know how o test this to see if it works correctly.

This page seems to detect correctly the languages accepted: http://www.christian-gerner.de/computer/status406-lang.php

Comment on attachment 167885 [details] [diff] [review] Patch >Index: src/preferences/PreferenceManager.mm >=================================================================== >RCS file: /cvsroot/mozilla/camino/src/preferences/PreferenceManager.mm,v [snip] > > [self configureProxies]; >+ >+ // Now work out the user's preferred accept-language headers You are using tabs here, and you should not : see http://www.mozilla.org/hacking/mozilla-style-guide.html#Visual for your code formating within mozilla >+#if DEBUG Remove debug code please. Patch applies correctly. I have a few comments however on the way the patch works : I hadn't cared to modify my language list in the System preference hence my list of accepted language was long - this is unecessary we should limit this to say 5 languages (I'm giving here a number, we should discuss the value for this limit). and sorry for the delay bruce.

Something I ran into while testing this was that english was represented on my machine not as 'en' but as 'English', which isn't a valid Accept-Language token. I didn't see anything in the patch taking this into account (I'm guessing what I experienced isn't the case on all systems).

Also, I am curious as to why the line #import <Foundation/NSUserDefaults.h> is in the patch when importing <Cocoa/Cocoa.h> already imports the NSUserDefaults class.

(memo to self): The System Preference's lanuage list already includes some regional variants. However these are currently returned in the form "en_GB", and will need to have the underscore changed to a hyphen. We should probably also check that we don't send the same language twice with our "guess the country" approach for your primary language.

Updated patch addressing Ludo's review comments, Wevah's review comment. Also includes protection against human readable locale names that Wevah encountered on systems upgraded from 10.1 and deals with languages from the system preferences pane that include a dialect indicator. Requesting review.

Updated patch that returns nil rather than empty string for invalid locales for slightly cleaner code. (Thanks to Wevah).

Changing the null checks to if ( x ) rather than if ( x != nil ) to match rest of code.

As far as some languages coming through as their English names (as was my case until I fiddled), you might take a look at /System/Library/PrivateFrameworks/IntlPreferences.framework/Versions/A/Resources/EnglishToISO.strings for a nice conversion table.

Comment on attachment 171685 [details] [diff] [review] Updated patch with nil checks in "house style" >+ r = [NSString stringWithString:language]; >+ >+ [language release]; >+ return r; >+} >+ NSUserDefaults *defs = [NSUserDefaults standardUserDefaults]; >+ NSArray *languages = [defs objectForKey:@"AppleLanguages"]; >+ NSString* primaryLocale = [defs objectForKey:@"AppleLocale"]; >+ >+ // Set up the user's primary language using the locale (so we get dialect info) >+ NSMutableArray* acceptableLanguages = [[NSMutableArray alloc] init]; >+ if ( primaryLocale != nil ) { the if should be aligned. >+ NSString* primaryLanguage = [PreferenceManager convertLocaleToHTTPLanguage:primaryLocale]; >+ if ( primaryLanguage ) >+ [acceptableLanguages addObject:primaryLanguage]; >+ } >+ >+ // Now set up all the other languages the user understands >+ // (from System Preferences | International). Strip duplicates because the >+ // user may have their primary locale set up as a language too (e.g. British English) >+ for ( unsigned i = 0; i < [languages count]; ++i ) { >+ NSString* language = [PreferenceManager convertLocaleToHTTPLanguage:[languages objectAtIndex:i]]; >+ if ( language && ![acceptableLanguages containsObject:language] ) >+ [acceptableLanguages addObject:language]; >+ } >+ >+ // Now set up the accept-language header itself >+ // Note that necko will determine quality factors itself >+ NSMutableString* acceptLangHeader = [[NSMutableString alloc] init]; >+ for ( unsigned i = 0; i < [acceptableLanguages count]; ++i ) { >+ if ( i > 0 ) >+ [acceptLangHeader appendString:@","]; >+ >+ [acceptLangHeader appendString:[acceptableLanguages objectAtIndex:i]]; >+ } >+ [acceptableLanguages release]; Shoudldn't you call RemoveAllObjects before releasing the Array ? >+ >+ if ( [acceptLangHeader length] > 0 ) >+ [self setPref:"intl.accept_languages" toString:acceptLangHeader]; >+ >+ NSLog( [self getStringPref:"intl.accept_languages" withSuccess:NULL] ); should be removed

> Shoudldn't you call RemoveAllObjects before releasing the Array ? Releasing an NS[Mutable]Array releases any objects still in the array. There's therefore no need to do so explicitely by removing these objects before calling release. The other review comments (alignment and a stray NSLog that had somehow survived the cull) are fixed in this new patch.

Comment on attachment 172020 [details] [diff] [review] Updated patch r+

Comment on attachment 172020 [details] [diff] [review] Updated patch changing self to r not sr

Comment on attachment 172020 [details] [diff] [review] Updated patch Nit: In places like + for ( unsigned i = 0; i < [languages count]; ++i ) { I'd liek to see 'unsigned int' rather than 'unsigned', but I'll let pink decide whether it matters. Looks fine, r+

> + NSArray* localeParts = [anAppleLocale componentsSeparatedByString:@"_"]; > + > + [language appendString:[localeParts objectAtIndex:0]]; are we guaranteeed that |localeParts| will have an object at item 0? What if |anAppleLocale| is nil? > ++ (NSString*)convertLocaleToHTTPLanguage: (NSString*)anAppleLocale nit, please use |inAppleLocale| for the param name. > + NSMutableString* acceptLangHeader = [[NSMutableString alloc] init]; ... > + [acceptLangHeader release]; for things that are alloc'd/released in the same scope, I generally prefer that you autorelase on the alloc/init line. It prevents someone from accidentally adding a return between the alloc/release (introducing a leak) and is more consistent with what we do in the rest of the app.

Added nil check in the conversion locale subroutine to protect against nil (redundant in current code, but better safe than sorry). Renamed subroutine argument. Changed several local variables to be autoreleased.

Updated patch to address nit with NSMutableString initialisation that Mike raised on IRC. Verbal sr=pink received on IRC with this change.

Comment on attachment 172293 [details] [diff] [review] Updated patch one more nit ;) + NSMutableArray* acceptableLanguages = [[[NSMutableArray alloc] init] autorelease]; can just be ... = [NSMutableArray array];

Comment on attachment 172367 [details] [diff] [review] Absolutely the last patch - hopefully! sr=pink

landed on trunk

landed on branch