Nelson Bolyard (seldom reads bugmail) Reporter
	Description • 17 years ago

The second filter in my large list of IMAP message filters is supposed
to match Received: headers with the (sub)string swsblss.  But it doesn't.
It used to, but not any more.

My list of custom headers is (wrapped here):

prefs.js:user_pref("mailnews.customHeaders", "Received: Sender: Content-Type: Content-Description: Reply-To");

The rule is:

name="Rcvd from swsblss (bugster) -> bugster"
enabled="yes"
type="1"
action="Move to folder"
actionValue="imap://xxxxxxx@xxxxxxxxxx.sun.com/Sun Departments/Bugster"
condition="OR (\"Received\",contains,swsblss)"

I have several messages in my IMAP inbox that clearly match that rule.
I have attached one of them to this bug.  
More messages that match that rule arrive from time to time.  They don't 
match; that is, they don't get moved into the designated folder and they 
don't show up on the filter log, (but perhaps the filter log is not a 
reliable indication of matching any more, see bug 388051).

I went into the filter dialog, selected this filter, Selected the Inbox
on which to run selected filters, and clicked on the "Run Now" button, 
and nothing happened.

	Nelson Bolyard (seldom reads bugmail) Reporter
	Comment 1 • 17 years ago

reproducible: 100% even with today's nightly trunk SeaMonkey build

	David :Bienvenu
	Comment 2 • 17 years ago

If you generate an imap protocol log, are we fetching the received: header?

Also, can you try w/o putting the ':' in the header names in the customHeaders pref? Not sure if it matters...

	Nelson Bolyard (seldom reads bugmail) Reporter
	Comment 3 • 17 years ago

This is the IMAP log file from a run in which I forced that filter 
to run on the Inbox.  The string "Received" does not appear anywhere 
in it.  I removed all the "needmore" lines from this to shorten it,
and I replaced user and server names with xuserx and xserverx.

I created this Received filter about two years ago.  
It has successfully filtered literally hundreds of email messages for 
me over the past two years, until recently when it stopped worked.

Dave, Are you suggesting that a recent change may have made custom
headers that end in a colon to stop working?

	David :Bienvenu
	Comment 4 • 17 years ago

I think running a filter after the fact may operate on the local headers now because we got so many complaints about it running on the server; that's a separate issue - a log of a session where you get new incoming mail would be more helpful.

	Magnus Melin [:mkmelin]
	Comment 5 • 17 years ago

(In reply to comment #0)
> I went into the filter dialog, selected this filter, Selected the Inbox
> on which to run selected filters, and clicked on the "Run Now" button, 
> and nothing happened.

If it's only after-the-fact and custom headers, this is bug 184490.

When you say "it used to work", what version are you talking about?
 

	Nelson Bolyard (seldom reads bugmail) Reporter
	Comment 6 • 17 years ago

It's not only after the fact.  It failed to match this message when the 
message arrived, and it failed after the fact.  This filter has worked
correctly for trunk versions for about 2 years, until recently, when it
stopped working (about the time of the switch from 1.5a to 2.0a1).

	David :Bienvenu
	Comment 8 • 17 years ago

fix checked in - that was my bug, sorry! Tomorrow's trunk build should be better, for filters run on incoming mail. Filtering after the fact is a separate, broader, issue, which I'm trying to look into.

	David :Bienvenu
	Comment 9 • 17 years ago

I'm going to re-open this for a problem my fix introduces - basically, if you have a custom header that's a substring of an other custom header, we'll only use the first. E.g., something like X-Sender and Sender. The old code wasn't removing dupes, so if you had 10 filters on SENDER, we'd send protocol to the imap server requesting SENDER 10 times. This is usually harmless, but silly; the new code that tries to remove dupes is too aggressive.

Even so, I suspect things will still be a lot better with today's build, for filters on incoming new imap mail, with custom headers.

	Wayne Mery (:wsmwk)
	Comment 10 • 17 years ago

This patch resolves "Trust spam header" bug 381589, a 1.5 to 2.0 regression which jives with comment 6. It is also broken on trunk at least as far back as 20060123, making possible culprits to be Bug 295088, Bug 266905, bug 11034.  

If the above makes sense, can we get this patch into 2.0?  Note: I don't know if this fixes bug 394244 which is also "Trust spam header".

fix range for 381589 is 2007071604 - 2007071705, eg. this bug.  

fails "Trust spam header" 
- version 1.6a1 (20060123)  [didn't test earlier builds]
- version 2 alpha 1 (20060621) 
- version 3 alpha 1 (20060303)
- version 3 alpha 1 (20070627)
- version 3.0a1pre (2007070705)
...
- version 3.0a1pre (2007071604)

works "Trust spam header" 
- version 3.0a1pre (2007071705)
- version 3.0a1pre (2007072504)

	David :Bienvenu
	Comment 11 • 17 years ago

This fix was pretty specific to the trunk - it fixed a regression on the trunk having to do with the string api changes that were trunk-only.

	Wayne Mery (:wsmwk)
	Comment 12 • 17 years ago

short summary of David and I both testing windows - his 2.0 build works but my 2.0 install fails.  I use same profile to test trunk (which now works) and 2.x which fails, so David observes "if the same profile works [for me, and his 2.0 works], that implies that it's a packaging issue."  ... with self builds working but distribution builds not.

We tested imap only.


Note: if api checkin (Bug 379070) broke trunk on 2007-05-21 but it was also broke in version 3.0a1 (20070421) (my closest prior test which failed), doesn't that imply either 
a) something fixed it between then and 2007-05-21
b) if not a) then some underlying problem still exists? 
c) Or are we still looking at a probable packaging problem?

	David :Bienvenu
	Comment 13 • 17 years ago

My 2.0 packaged build works fine as well, or rather, it does fetch the spam status headers. So I'm really not sure what's going on when it doesn't work.

	Wayne Mery (:wsmwk)
	Comment 14 • 17 years ago

adding to comment 12... per Daivd, my 2.0 is not fetching the x-spam-status header for that server

Different machine running a virgin profile and 2.0.0.6 install, to same server of course.  A customized filter on X-Spam-Flag: tagging spam as star on incoming message works.  "trust junk headers" does not work, until ... I enabled "adaptive jmc", it had previously been unchecked (but also marking a lot of stuff junk that wasn't)

That narrow it down to a profile issue causing x-spam-status from being retrieved.

	Mike Schroepfer
	Comment 15 • 17 years ago

Per discussion in m.d.a.seamonkey minusing this for 1.9 - if it is a core bug and seamonkey or tbird need this in the 1.9 timeframe please do re-nominate.

	ovidiu
	Comment 16 • 16 years ago

possible relation bug 412427?

	Nelson Bolyard (seldom reads bugmail) Reporter
	Comment 17 • 16 years ago

I've recently switched to a new mail service provider and now I have IMAPS
as the mailbox access protocol for my primary email account.  
I have huge numbers of filters that I built up for use with my old POP mail
service that I am now trying to migrate to the IMAPS server account. 
I'm now experiencing frequent issues with filtering of incoming IMAP emails.  
I think they're probably related to this problem.

Presently, my "mailnews.customHeaders" pref contains this string (wrapped here)

Content-Description: Received: Content-Type: Content-Transfer-Encoding: NNTP-Posting-Host: Path: Content-Disposition: Lines: References: 
X-Spamcatcher-Explanation: X-Bugzilla-Watch-Reason: BCC: Sender

I have numerous filters that filter incoming email on the Sender header.
I am finding that no messages ever successfully match that filter rule,
even though many should match it (that is, they match the stated filter
criteria, but the match is not recognized). 

I also have filters that filter on Body.  In find that the UI will not let
me create a filter rule for an IMAP account that filters on Body.  :(
But let's start with the issue of the Sender filter not working.

	Kent James (:rkent)
	Comment 18 • 16 years ago

(In reply to comment #17)
> I've recently switched to a new mail service provider and now I have IMAPS
> as the mailbox access protocol for my primary email account.  
...
> I have numerous filters that filter incoming email on the Sender header.
> I am finding that no messages ever successfully match that filter rule,
> even though many should match it (that is, they match the stated filter
> criteria, but the match is not recognized). 

Are you sure this isn't bug 184490

	Nelson Bolyard (seldom reads bugmail) Reporter
	Comment 19 • 16 years ago

I'm pretty sure.  I believe this is not a duplicate of bug 184490 because 
that bug is about "after the fact" filtering, and this bug is about the 
filtering that occurs when the message is first received and first processed 
("During the fact"?).  

Perhaps the "after" and "during" cases are not as different as I imagine?

	Wayne Mery (:wsmwk)
	Comment 20 • 16 years ago

kent, david, 
Nelson seems like a great testcase. Is a debug build the next best (or at least quick) step?

	Nelson Bolyard (seldom reads bugmail) Reporter
	Comment 21 • 16 years ago

I've never thought of myself as a test case before. :)

	David :Bienvenu
	Comment 22 • 16 years ago

Is this still broken? There were a few fixes having to do with Sender: header as I remember.

	Kent James (:rkent)
	Comment 23 • 16 years ago

This bug seems to drift, so it's been hard for me to make sense of. Is the bug
still that the message from attachment 272199 [details] does not match a custom Received
header, on both IMAP and after-the-fact filters?

If so, I suspect it is an aggregation issue since there are multiple headers
with the same value.

I moved the message into a local folder, and tried to reproduce. Running
filters after-the-fact, with a single criterion of Received Contains "swsblss",
the filter hits when I use "Match All" and does not hit when I select "Match
Any".

I don't know how hard it would be to fix, without special casing Received as an
aggregated header. I'm also not sure that is the real subject of this bug, and
I have not traced it out to really see what is happening.

	Nelson Bolyard (seldom reads bugmail) Reporter
	Comment 24 • 16 years ago

(In reply to comment #23)
> Is the bug still that the message from attachment 272199 [details] does not 
> match a custom Received header, on both IMAP and after-the-fact filters?

Yes.  Please ignore comment 17 if that helps.

	David :Bienvenu
	Comment 25 • 16 years ago

(In reply to comment #24)
> Yes.  Please ignore comment 17 if that helps.

Ah, thanks, yes, that threw me off. 

So we could easily treat Received as an aggregate header in nsParseMailbox.cpp since it has all the infrastructure in place for aggregate headers. Two small issues:

 this code in nsParseMailMessageState::FinalizeHeaders()

        for (PRUint32 i = 0; i < m_customDBHeaders.Length(); i++)
        {
          if (m_customDBHeaderValues[i].length)
            m_newMsgHdr->SetStringProperty(m_customDBHeaders[i].get(), m_customDBHeaderValues[i].value);
        }

would need to be taught about aggregate headers

and we would need to not break the code that uses the very first received header as the received time stamp, at the end of nsParseMailMessageState::ParseHeaders.

	Wayne Mery (:wsmwk)
	Comment 26 • 15 years ago

"received" in summary really helps

	John David Galt
	Comment 28 • 15 years ago

This sounds like the same problem I address in comment #106 of bug 16913.