If you right click on a email address in a mail and select "Add to Addressbook" the New Card dialog appears. Here the "First" and "Last" fields are not auto filled with information from the added address. Only the fields "Display" and "Email" are filled. When adding fx: Henrik Gemal <gemal@dk.net> the New Card dialog should default with the following fields: First: Henrik Last: Gemal Display: Henrik Gemal Email: gemal@dk.net

reassigning to mscott.

polish bug == M18.

Adding myself to cc: list.

putterman owns the address book.

This looks great for English. I'm not 100% sure if this is localizable so I'm cc'ing nhotta and momoi. Also, I have some email where I get it "lastname, firstname <email>" and this also fails by putting the lastname in the firstname and vice-versa.

reassigning to chuang

Changing qa assign.

is it better to do something and hitting 80% of the cases instead of doing nothing? Perhaps something could be done here if we agree on what formats we should support. What about for a start: Henrik Gemal <gemal@gemal.dk> Gemal, Henrik <gemal@gemal.dk> gemal@gemal.dk (Henrik Gemal)

Ok once again: isn't it better to do something and hitting 80% of the cases instead of doing nothing? Perhaps something could be done here if we agree on what formats we should support. What about for a start: Henrik Gemal <gemal@gemal.dk> Gemal, Henrik <gemal@gemal.dk> gemal@gemal.dk (Henrik Gemal) Please someone reply, if I should do anything of this bug. I dont want to contribute something that just gets rejected right away. So basiclly: should I contribute anything??????????????????

Gemal, I would like to see the following: 1. We should have Type A and Type B langauges. Type A is of European type, Type B is an Asian type. 2. In Type A, Name1 Name2 --> First Last (i.e. Interpreet the left sequence as the righ sequence) 3. In Type B, Name1 Name2 --> Last First 4. In both Types A and B, Name 1, Name2 --> Last First 5. In Type A, Name1Name2 --> First 6. In Type B, Name1Name2 --> Last Can we have a pref item somewhere by which we can switch the behavior between type A and type B? Is this asking too much?

wow.. you answeer is weird. basicly I dont understand anything! fx: Name 1, Name2 --> Last First do you mean: Name1, Name2 --> Last, First ? What this about "Type A and Type B langauges"?

Let me clariy. Name1 and Name2 refer to strings you see in the Address field of a message. Last and First each refer to the entery data in Address Book fields. So if you have the following scheme: Name 1, Name2 --> Last First That means, "if you see the sequence of names "Name1, Name2", then add Name1 to the Last name field of AdBook, and add Name2 to First name field of AdBook. "Name1Name2" means that there is no separator of any kind between the 2 names. Chinese or Japanese names can be of this type though sometimes they insert a space between them. Is this clearer now?

QA-assign-to fenella.

reassigning to racham

Implements all the requested behavior. I was not sure how Asian languages handle Name1, Name2 so I made it work the same as European ones (First: Name1, Last: Name 2).

CC'ed ji & marina.

Naoki, can you make sure that the patch deals with CJK differently? Referring to comment #11, I am thinking of something like this: If you see the Addressee names encoded in any of the encodings in CJK, then A. For names that have spaces in between, e.g. Name1 Name2, Name1 Name2 Name3, assign the first Name to: Last Name field of AdBook Card assing all others to: First Name field of AdBook Card B. For names that have a comma separator, e.g. Name2, Name1 Name3, assign Name2 to: Last Name field of AdBook Card assign any names to the right of the comma to: First Name field of AdBook Card C. For single names, e.g. XXYYVVHH, assign this to: Last Name field of AdBook Card If the names are not encoded in CJK encodings, then we apply Western algorithm. This should give us something to begin with.

All of those behaviors are handled acceptably. There is a flag editCard.displayLastNameFirst which is based on the mail.addr_book.displayName.lastnamefirst pref that should get set to true on Asian style language versions of Mozilla. If you set that flag to true at the beginning of GenerateFirstAndLastName() you can test it. I did and all 3 test cases work as you describe. In theory, non-western encodings shouldn't matter.

better comments, full path to files

> In theory, non-western encodings shouldn't matter. I guess I am proposing something more sophisticated than the flag you talk about. Western names and CJK names have the same Last name First order if they are written in Latin Alphabet. But they would be in different order if CJK names are written in native script. So taking my name as an example, Katsuhio Momoi -- First Last (if written in Latin Alphabet) MOMOI KATSUHIKO -- Last First (assuming these represent names written in Japanese) This is why switching the editCard.displayLastNameFirst option probaly is not enough. That is, the order is based on what script is being used for that name NOT on what language version of Mozilla is used. Whether you use English version of Mozilla or Japanese version of Mozilla, if you see the addressee name encoded in Japanese as in "JName1 JName2", that name should be interpreted as "Last First" and added to the Address Book as such. Does your patch take care of this without taking into account what encoding the name uses?

To put it in another way, an Asian mail reader will receive messages from people whose names are written in Latin Alphabet and also from people whose names are in their own native script. We would like the order of the names in Address line to be distinguished based on what script is being used. This means Latin Alphabet names and names in Chinese/Japanese/Korean scripts will be given different order in adding to the Address Book. This has to be the case even when you use the English version of Mozilla. Otherwise some names will be incorrectly committed.

I follow you now, we'll need to add a CJK check. What is the best way to determine if the string contains CJK characters?

> I follow you now, we'll need to add a CJK check. What is the best way to > determine if the string contains CJK characters? This is where we would like nhotta to help with coding details. I know in outline it would be something like this: 1. Assuming that names are MIME-encoded, look for multi-byte encodings. (** Only CJK scripts and Unicode use multi-byte encoding) 2. If Unicode (UTF-8), look for CJK ranges. (CJK Ideograph, Hangul, Kana ranges). 3. In 1, what if the name is in 8 bit characters. This is where we may have to punt. See if there is a charset for the body, etc. and other creative solutions such as relying on the default folder encoding. I think nhotta can give us what it would take to accomplish this and other factors I haven't thought about.

Is this changing the auto fill behavior for the display name? 'editCard.displayLastNameFirst', does that come from the pref?

No, it is augmenting it. Currently the display name is filled in if the appropriate data is available. What is being added is that the first and last names get filled in as well. According to momoi's proposal, the address book can be more intelligent than using the editCard.displayLastNameFirst property (which is equal to the mail.addr_book.displayName.lastnamefirst pref). Since a person in a English language country may receive an email from a person in a Japanese speaking country for example, the address book should try and recognize (based on the character set used in the name) that the person is from a country that uses the Surname Firstname format and deal with it accordingly, rather than blindly using the pref and causing incorrect behavior. This should have no effect at all on the display name (currently, at least) since I generate the first and last name after the display name has been set. It might make sense to use the same behavior for the generating the display name by typing in first and last name, but that belongs in a different bug. In order to do this, what is needed is a method to determine if a string is encoded with a European or Asian encoding. From what I can tell the encoding can be defined using the escape sequence or with a "Content-type: text/plain charset name" header in the message.

ji, can you upload some sample e-mail messages which have sender names in 1. Chinese (Trad, Simpl), Japanese and Korean encodings. 2. Make also one language example under UTF-8 header encoding. 3. Name formats should include the following type: LastName Firstname LastNameFirstName 4. There should be all total of 10 sample messages. Note: If you don't have a CJK font on Windows, you can use the following large font containing CJK glyphs. Just by installing the font in your system, Mozilla should be able to display all these names as long as the headers are correctly MIME-encoded. ftp://ftp.netscape.com/pub/communicator/extras/fonts/windows/Cyberbit.ZIP

I think deciding the order by charset and the user cannot control is not good. We can have a pref that the user can choose the order (First/Last or Last/First).

> We can have a pref that the user can choose the order (First/Last or > Last/First). I thought we have this already. It is not meant for this purpose but according to Hampton, editCard.displayLastNameFirst revserses the order in which names are committed. But this also leaves names to be committed in one way only. Scriptual difference and use of punctuation is normally what determines the name order. We should build that logic into Mozilla.

Please unzip the attachment and place it under your Local Folders. After restart, you can see a mailbox under Local Folders that contains 10 messages for Japanese, Simplified Chinese (SCH), Traditional Chinese (TCH), Korean and UTF-8(Japanese). Each language has two messages with sender's name in "LastNameFirstName" and "LastName FirstName" format.

marking as enhancement. jumping in, as devils advocate, why are we doing this again? From filling in first and last name, I guess we allow for better first / last name searches. But I could always search display name, but that does result in more hits, I guess. this seems like icing on the cake, we've go so many bad bugs we should be focusing on instead. I don't want to discourage Hampton from fixing this, but I'd rather encourage him to work on other bugs (high priority or nsbeta1+ bugs)

> marking as enhancement. > jumping in, as devils advocate, why are we doing this again? Because the current behavior would not meet a product requirements in anywhere other than First-Last name countries. We are producing global software. Why would anyone be asking such a basic question is a mystery to me. Moving it back to major.

seth, since you raised the question, I should add that this feature would not be so critical if we allow for more flexbile search in Display name field. Currently auto-address filling can only match from the beginning of the Display name. It will not match Doe in "John Doe". If First and Last name fields are filled in, auto-mtach with "Doe" works. Unless this bug is fixed, filling in both names correctly for different lang scripts as requested here continue to be important.

This patch does eastern/western charset detection and handles the firstname lastname business correctly. If it can't find a distinct charset (e.g. Unicode) it falls back to the displayLastNameFirst pref value. Someone else can do glyph detection within unicode if they want. Notes: The charsets may not be 100% accurate. I've included Korean, Japanese, Chinese, Thai, and Vietnamese as Eastern charsets. Everything else that's not unicode is considered a Western charset. I was not sure if the default Western charset (ISO-8859-1) should be considered Western since for example it may be used by Japanese speaking people who write their name in Romaji. Currently I include it in Western. isWesternCharset() and isEasternCharset() need a home outside abCardOverlay.js, since they may be useful to someone else down the line. Where should they go?

As I commented before, I think deciding the order by charset and the user cannot control is not good. And the result is not accurate by using the mail charset. For example, I am always using "Naoki Hotta" and I don't change it to "Hotta Naoki" when I send Japanese mails.

> And the result is not accurate by using the mail charset. For example, I am > always using "Naoki Hotta" and I don't change it to "Hotta Naoki" when I send > Japanese mails. Is your sender header MIME-encoded in ISO-2022-JP when you use "Naoki Hotta"? If you wrie this name using Japanese characters, I don't believe you will write "Naoki Hotta". As I have been saying, the order in which the names are written is largely controlled by what language script you use. Each language script mirrors its name order convention. I think there are a couple of ways to understand my proposal. 1. Look for the MIME-encoding name for the sender header. If it is one of the CJK type of encoding, apply the mechanism suggested here. 2. The MIME encoded header must be decoded to be committed to the Address Book. So look for characters in CJK ranges and then apply the mechanism suggested above. Of the two, I think #2 is probably more accurate. If the sender headers is not MIME-encoded, then we should use the Western order for names. If it is in 8bit, then we might use the encoding for the mail body as a guide. My understanding is that our MIME-encoder does not encode ASCII names like "Naoki Hotta". So, the mechanism suggested here will not reverse the order.

>And the result is not accurate by using the mail charset. This is a comment to the patch. Whatever mechanism we use, the result is basically a guess because there is no standard way of specifying the order. I think we want to provide a way for the user to control the order of the auto-fill. In addition, we may provide a way for the user to swap First and Last name fileds (and in the display name) easily.

> Whatever mechanism we use, the result is basically a guess because there is no > standard way of specifying the order. What do you mean by "standard" way? If you mean that a language script does not have a well-accepted convention for name order, you are simply mistaken. Please tell us what you mean by "no standard way".

Guys, regardless of your bickering, I would appreciate it if some patch were applied. This bug has been sitting around for almost two years, and should have been patched 18 months ago. In this case specifically, I think the best is the enemy of the good. Patch the problem for Latin languages, then work on the fix for other languages. There's no reason to punish the majority of users just because the patch doesn't solve the problem for some.

I mean standard way to specify the order explicitly in a mail. Also language cannot always determine the order. For Japanese, I heard that the government recommendation recently changed from First/Last to Last/First for Roma-ji. The comment by Jason Bennett, I think it's fine as long as the original order is preserved in the display name filed. And there is already a pref to control the order, right?

> Also language cannot always determine the order. > For Japanese, I heard that the government recommendation > recently changed from First/Last to Last/First for Roma-ji. I have been using "script" to refer not to language but to the writing system itself. No one is going to write your name or my name with Kanji with FirstName LastName order. Roman letter is a different script. In the case of Roman script, it would have to follow its own script convention. I believe what we have currently for non-CJK scripts will catch 90% of the cases or more. That's a lot better than what we have now.

Detecting the convention based on charset range would be nice. If you want to point me towards docs that give ranges (for UTF-8), go for it. Even if we have this, I'm going to say 90% of the messages sent to a CJK person encoded in Baltic should probably ignore the pref and use the charset. Detecting by range for CJK people will give worse results for that case since western encodings are perfectly valid for composing CJK names, e.g. Katsuhiko Momoi. This is based on the possibly flawed assumption that unicode will maps all instances of a letter from different charsets to one numerical representation ('u' in western, roman, ibm, etc. would all get translated to 1 value) In the meantime, here is what I propose to get this patch in. 1. For CJK charsets, automatically use lastname firstname behavior. 2. For Western charsets, use firstname lastname behavior. 3. For unicode, use the pref. 4a. For the default Western charset (ISO-8859-1) and perhaps some other western charsets, use the pref 4b. Alternatively to 4a, for all western charsets just use the pref In defense of #1, I agree with Momo. People are not composing letters in Kanji with the firstname first. I am not sure about #2, but I am guessing that we will get more accurate results than using the pref if we detect charset. How many people are using Japanese names encoded in a German charset. Doing #3 requires ranges. It is tough. Solving it solves this bug near 100%. For #4a, it is quite conceivable to see CJK people using the default Western charset. I don't like the #4b alternative: see comments about #2. The code is there. It works. Let's get some functionality in there and deal with the rest later. I do not support adding prefs beyond what we already have. We should be accurate enough that a pref is not needed.

>In defense of #1, I agree with Momo. People are not composing letters in Kanji >with the firstname first. I agree with that. But I don't think the patch uses the charset of the header which includes the display name. We do not have an interface to return the charset of MIME encoded header since we switched to return the decoded string in UTF-8. The other issue is that we cannot assume that Last and First names are always separated by an ASCII space.

>We do not have an interface to return the charset of MIME encoded header >since we switched to return the decoded string in UTF-8. Indeed. I pass in the charset of the message itself (which is probably coming from the the Content-type: Charset: header) before the string is converted to UTF-8. It's the same as the selected value in the View -> Charset Coding menu. Are there two values, one for the message and one for the header/display name? >The other issue is that we cannot assume that Last and First names >are always separated by an ASCII space. Not a problem. I run the regex's on the UTF-8 strings. The Javascript implementation uses UTF-8 as the native string format, so it works fine unless there are delimiters that are not regarded as whitespace (regex: \s character) Please, can you guys please read the latest patch and test it out. You'll see that it works pretty well, 100% on the test messages that were uploaded.

Hampton, thanks for the patch. nhotta reviewed it and we discussed what can and cannot be accomplished with the patch. The mail body encoding, which is what you're using is not always the correct indicator of what is being used in the Sender header. A Japanese person sending a Japanes msg can be using a name in Roman script. The only accurate way to realize my proposal is to parse the MIME-encoded sender name with a provision to allow for the non-MIME encoded case. I had been assuming all along that we should have an API to query in what Unicode range, the decoded MIME-encoded header would be in. Since Unicode classifies character blocks based on script types, having such an API was what I had assumed. nhotta informs me that we may not have something like that. Without it, the behavior I wrote up in my specs above cannot be fully realized. I think non-CJK part will work well. So 2 things. 1. Let's look at your patch to confirm the expected behavior for CJK and see if we cam improve that. I assume that non-CJK part is good. 2. Let's ask shanjian if he knows such an API or service exists.

I can add charset detection trivially using the Javascript charCodeAt() function, so I'm going to finish this bug. This attachment contains all of the valid UTF-8 ranges taken from the link at the top of the attachment. I have included the proposed way to regard each encoding as the rightmost field - CJK for Eastern, WEST for Western, and NON for languages that are in neither category (basic latin alphabet, mathematical symbols, etc.). I was not completely sure that Tibetan and Myanmar are western languages. If someone would care to review my categorizations, feel free to. Planned behavior is to match any "Western" or "CJK" character. On match, immediately characterize as that charset. If charset is generic (e.g. Latin) fall back to pref since Japanese users may write their name in Latin and other such scenarios. Questions: where should I put isWesternCharset() and isEasternCharset() as they may be useful to others in the future.

patch that detects charset of email address string

Can someone take a look at this patch and see if it's ready or needs more grilling.

Is this based on Unicode Technical Report #24? http://www.unicode.org/unicode/reports/tr24/ http://www.unicode.org/Public/UNIDATA/Blocks.txt Bug 135853 was filed for that. Anyway, if the range is based on the technical report then it should be okay. But how did you get First and Last name order for each range? I think "isWesternCharset", "isEasternCharset" are misleading. It should be named to describe Last and First name ordering instead of Western/Eastern. For example, why "Tibetan", "Kannada" are Western?

mass re-assign.

*** Bug 214414 has been marked as a duplicate of this bug. ***

cc--> mscott, Is it still reasonable to think this patch can be worked to be used? A really quick analysis... Personally, I question it being fixable, at least without some serious logic. What about inline email addresses (within a paragraph)? Take the following email: ----- Snip ------- To harass Scott: call him, email <mscott@mozilla.org>, page, bang on his door at 2 in the morning asking for spare change. ----- /Snip ------- First Name: Call Last Name: Email Email: mscott@mozilla.org I think at the most, we should support replies: From: "Bob Dole" <bigstud@viagra.com> To: <iwannabebig@ed-supportformen.net> Perhaps look for a strict From: "First Last" <email> From: "Last, First" <email> I think it's better to avoid doing things we can't do well (until we can do it well). That doesn't cover encoding issues, language/customs associated due to localization and international use. In any regard, if there's some fruit after this bug, perhaps we should harvest before the bitrot sets in?

Thunderbird appearently attempts to splitting names into first and last names when automatically adding outgoing addresses - see bug 259289. The two ways of adding entries to the address book should probably use the same strategy for splitting the name (either do it the same way or not at all).

Still reproducible with Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1pre) Gecko/2008070102 SeaMonkey/2.0a1pre; the patch provided in attachment 78978 [details] [diff] [review] for abCardOverlay.js(1.38) would need to be updated. Splitting First/Last names in Collected Addresses works fine.

(In reply to comment #54) > Splitting First/Last names in Collected Addresses works fine. interesting point. so no objections then to doing this with "add to addressbook"? CAB name code at http://mxr.mozilla.org/seamonkey/source/mailnews/addrbook/src/nsAbAddressCollecter.cpp#222 patch line adjustments needed, roughly 124>176 438>556 - at SetCardDialogTitle(displayName);

Well, "fine" in the sense of it's filling in some values for the collected addresses. Currently, the last token is always filled into the "Last" field and everything else goes into "First". While this works fine for a "First Last" order, it is reversed for the "Last, First" case, and the ',' not removed. An entry "Last, First X" will split off the "X" as last name, and "Last, First" goes into the first name. Thus, there would be room for improvement (also keeping localization of names in mind) in the nsAbAddressCollecter::SplitFullName() implementation (see comment #53). The patch attached to this bug addresses those issues.

I do wonder if the approach isn't overly complex with charset detection and all, especially as I don't think you can tell by much certainty by that. E.g. comment 40 say the Japanese use both, and that's true in Finland too (unfortunately). In the end it's a dialog with a suggestion, and you'd just have to check if the info is correct.

parts of this are already picked up on trunk. perhaps someone can run it and enumerate any bits that are still missing?

Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1b5pre) Gecko/20090512 Shredder/3.0b3pre wfm on usual cases

In 3.0 beta first last and display are populated, and looks great for the trivial case of "firstname lastname". When sender contains "lastname, firstname" - display populates fine, but firstname and lastname populate backwards.

This is interesting (and inconsistent, and somewhat absurd): - For the *unattended* scenario, when we add addresses from outgoing emails *automatically*, we auto-populate the card's First and Last fields with a non-intelligent guess. This scenario is completely covert so chances are the user will never check if these fields have been filled correctly, and we very are likely to get it wrong (bug 259289). - For the *attended* scenario, when user takes action to add addresses *manually* from the message reader, we do *not* auto-populate the card's First and Last fields at all (this bug; comment 60 no longer applies). But in this scenario, imo it is much more likely that user will check for correctness right after creating the new contact, so it would be much safer to pre-populate. (In reply to Katsuhiko Momoi from comment #33) > seth, since you raised the question, I should add that this feature would > not be so critical if we allow for more flexbile search in Display name > field. Currently auto-address filling can only match from the beginning > of the Display name. It will not match Doe in "John Doe". If First and > Last name fields are filled in, auto-mtach with "Doe" works. > Unless this bug is fixed, filling in both names correctly > for different lang scripts as requested here continue to be important. Exactly, that's a very correct analysis of the status quo, regarding Address Autocomplete in recipient fields during composition: The real problem here is bug 558931 (and bug 445175): Autocomplete is so dull because... - we don't match "contains", we only match "begins-with" against AB fields (bug 558931), so typing - for multiple search words, instead of matching each search word separately against AB (AND-search), we match multiple search words as a single string (bug 558931 again), so typing "Doe John" will not find John Doe's card. - we probably don't match against enough fields from the AB (is there a bug for that?) Conclusions: 1) The real pain point that affects everyone is bug 558931, which should be fixed. 2) As long as we have bug 558931, fixing this bug 40046 is crucial (agree with Momoi), more so as long as we have bug 259289 (it is really weird to pre-fill for the unattended scenario, but not to pre-fill for the attended scenario, as described above in this comment); this looks easier than fixing bug 558931 as we already have this behaviour when automatically adding addresses from outgoing mails, plus we'd make the behaviour consistent between manual and automatical adding of contacts. 3) If bug 558931 were fixed, this bug becomes less important; ideally, develop a more intelligent solution for pre-filling that handles non-simple and non-Western cases better.

(In reply to Thomas D. from comment #61) > The real problem here is bug 558931 (and bug 445175): Autocomplete is so > dull because... > - we don't match "contains", we only match "begins-with" against AB fields > (bug 558931), so typing ...so typing "Smith" will not find John Doe-Smith's card, nor John Doe (Smith Ltd.)'s card, nor John Blacksmith's card.

(In reply to Thomas D. from comment #62) > (In reply to Thomas D. from comment #61) > > The real problem here is bug 558931 (and bug 445175): Autocomplete is so > > dull because... > > - we don't match "contains", we only match "begins-with" against AB fields > > (bug 558931), so typing > ...so typing "Smith" will not find John Doe-Smith's card, nor John Doe > (Smith Ltd.)'s card, nor John Blacksmith's card. ...nor JohnDoe@SmithLtd.com's card!

(In reply to Thomas D. from comment #61) > This is interesting (and inconsistent, and somewhat absurd): > The real problem here is bug 558931 (and bug 445175): Autocomplete is so > dull because... > - we don't match "contains", we only match "begins-with"... > - for multiple search words, instead of matching each search word separately... And, for perfect ux-confusion, searching from within AB behaves *differently* than composition's autocomplete: - AB search *does* have partial ("contains") matches: Search for "smith" matches "John Doe-Smith", and "John Doe (Smith Ltd.)" and "John Blacksmith", and even JohnDoe@SmithLtd.com! - ...while multiple search words are still treated as a single string (as in automplete): search for "John Smith" will *not* match any of the above examples. If we can do partial ("contains") matches for a single AB, why don't we do it for Autocomplete (bug 558931)?