User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.3) Gecko/20060426 Firefox/1.5.0.3 Build Identifier: version 1.5.0.2 (20060308) When an email contains raw non-ASCII characters in the "From:" header the "From" mail filters will not match. In my case the header is From: "Luís" O******a <l***i_2**2@yahoo.com> (the *s stand for legal alphanumeric ASCII characters). Filters of the form 'From / is / l***i_2**2@yahoo.com' or 'From / contains / l***i_2**2@yahoo.com' fail. Curiously, if I automatically create a filter from the message, tbird will generate a 'From / is / "Luís" O******a <l***i_2**2"@yahoo.com>' that works (notice the double quote before the @). All would be fine except for the problem that upon restarting tbird the condition in the filter will have disappeared and the filter is now conditionless(!!). This results in positive matching of *ALL* mails. The results are the same if I just use 'From / is / l***i_2**2"@yahoo.com'. If saved as an .eml file and then reopened with tbird, only 'Lu' will be displayed in the From field. Manually editing and replacing the 'í' with an 'i' solved this. However, I wasn't able to reimport the .eml file back into tbird to check if the filters now worked. I was also unable (lack of know-how) to forge a similar message to further verify the reproducibility. All other messages I have with non-ASCII sender names have the header properly encoded in either UTF-8 or ISO-8859-1. Maybe it is not such a widespread problem. Reproducible: Always Steps to Reproduce: 1. Create a message with a raw non-ASCII "From:" header 2. Send to self 3. Try to match the sender with a mail filter Actual Results: Unfortunately, unable to forge such messages. Expected Results: The 'From / contains / l***i_2**2@yahoo.com' filter should have found the 'l***i_2**2@yahoo.com' in the header, independently of the out-of-specs characters.

Just found out that this behavior results from the co-incidence of bugs 103658 and 182064. The first of these bugs (critical) adresses the disappearing filter problem due to an odd number of double quotes; the second bug reports misinterpretation of addresses with non-ASCII characters: tbird will add a double quote before the '@' if there are any illegal characters in the names preceding the actual address; this last report regarded misinterpretation of addresses only when sending mail, but the origin of the problem is most likely the same as in my case. This whole issue should be solved by correcting the bug that introduces the quote before the '@' since that would no longer elicit the wrong behaviour due to the odd number of quotes, and the filters would match. Anyway, sorry for not having read these bug submissions earlier (found them now, searching for this bug). Maybe this should be considered a dupe. Sorry if you feel this is a waste of time.

(In reply to comment #1) > Just found out that this behavior results from the co-incidence of > bugs 103658 and 182064. Bug 103658 has been fixed on the trunk. Bug 182064 doesn't seem to apply -- that problem was about the address field in mail being sent by Thunderbird, whereas you are talking about messages received with an illegal non-ASCII character in the Subject header. (Mail headers must be 7bit ASCII to conform.) If I try Create Filter From Message on a similar From header, I do see the extraneous double-quote thrown into the address. That is a bug, and a current one; and it does look very similar to the symptom at bug 182064. (182064 was marked WFM because the behavior was superceded by improved handling of non-ASCII characters, not because the parse method was fixed.) Is the original problem of the filter not matching *because* you created a filter using Create Filter From Message? The filter example you start the bug report with appears to be based only on the mail address and doesn't show an extraneous double-quote.

(In reply to comment #2) > Is the original problem of the filter not matching *because* you created a > filter using Create Filter From Message? The filter example you start the bug > report with appears to be based only on the mail address and doesn't show an > extraneous double-quote. Now that you pointed it out, I think I tripped over two or three separate problems: I first tried to match only the e-mail address of the sender. I found out that the filter would not match. However, I was able to match e-mail addresses from other senders; only with this one would the filter fail. This is a bug per se. I then created a filter form the message and got the above results; that shed some light as to why the first filter didn't match: because of the non-ASCII characters, the From: header is internally read with a double quote before the '@'. This is another bug, on which the previous one depends. Lastly, I tried to match just the e-mail address, adding the extra double-quote. It did work, but then I stumped over a nasty third bug in which the filter contents are forsaken upon TB restart, resulting in matching of all messages (critical?). Should these be separated into two or three different bug reports?

(In reply to comment #3) > I first tried to match only the e-mail address of the sender. I found out > that the filter would not match. However, I was able to match e-mail > addresses from other senders; only with this one would the filter fail. > This is a bug per se. [...] because of the non-ASCII characters, the From: > header is internally read with a double quote before the '@'. I would report that as a separate bug. It would be enormously helpful if you could make the report without forcing people to decode your ***** anonymization, particularly since the text strings in questions *are* the critical data. > I then created a filter form the message and got the above results... > This is another bug, on which the previous one depends. That would be *this* bug, I think. It would be interesting to know whether a correctly-quoted-and-escaped filter with the erroneous double-quote matches the incoming message which (per the situation you describe above) may also have a bogus quote in its internal representation. > Lastly, I tried to match just the e-mail address, adding the extra > double-quote. It did work, but then I stumped over a nasty third bug in which > the filter contents are forsaken upon TB restart, resulting in matching of > all messages (critical?). OK, I see this too. I think that is an additional symptom of this bug. When I tried to reproduce this with the trunk, the quotes were escaped correctly (I think); and on restarting with the trunk, there was no problem. But on restarting with branch, when I opened the Message Filters window, the filter was hosed -- the test condition simply disappeared from the rules; and on closing the Filters window, it was rewritten out broken as well.

This message has a From line in raw ISO-8859-1: From: "abcde fghij NÖ" <er@knob.orb.at> Steps to reproduce: 1) Integrate the test case into a message folder 2) Run TB or SM (trunk build preferred until bug 103658 is fixed on branch) 3) Select message; Message | Create Filter from Message; observe contents of test field in 'subject' rule 4) Add an action to filter; save filter; close Message Filters window. 5) View msgFilterRules.dat and observe text for rule. 6) Reopen same version of program; check message filter 7) Close program, open branch version; check message filter 8) Close message filter window, close program; check msgFilterRules.dat Actual results: 3) Field contains: "abcde fghij NÖ" <er"@knob.orb.at> erroneous quote right here ----------^ 5) filterRules shows this string: condition="OR (from,is,\"\\"abcde fghij NÖ\\" <er\\"@knob.orb.at>\")" 6) filter displays as previously 7) filter displays with no match criteria (may be fixed by 103658) 8) no match criteria seen Expected results: 3) Field contains: "abcde fghij NÖ" <er@knob.orb.at> no erroneous quote 7) On reopening with branch version, filter criteria persists

OK, now that bug 103658 is fixed on the branch, following the steps at comment 5, the results at 7 & 8 are as expected. (In reply to comment #4) > That would be *this* bug, I think. It would be interesting to know whether a > correctly-quoted-and-escaped filter with the erroneous double-quote matches > the incoming message which (per the situation you describe above) may also > have a bogus quote in its internal representation. I simply tried using Run Now for the filter on the folder containing the message, and the filter (Add Star) was applied. Also: I noticed that for messages with a simple, legal address, when you Create Filter From Message, only the email address appears in the criterion string; there is no name field. I also see the same thing if, instead of using CFfM from the Message menu, I right-click on the bogus address in the envelope panel and pick Create Filter From Message there. However, if I save the filter using the latter method, and then try to run it, it doesn't match on the message. And finally: I tried creating a similar message with a slightly different name string: From: "abcde NÖ fghij" <er@knob.orb.at> instead of From: "abcde fghij NÖ" <er@knob.orb.at> This doesn't cause any problem: with CFfM, only the email address appears in the filter criterion; and the filter matches the message correctly.

Regarding step 3, I do not get the full name to the From field. I even tried adding the fake user to my address book. If I manually paste the full name + address to the rule field and create the rule, the .dat file has the correct contents: condition="AND (from,is,\"\\"abcde fghij NÖ\\" <er@knob.orb.at>\")" OK to close this? Thunderbird 52.0.1 (32-bit) Windows 7 64-bit