Add the ability to call an external program when a filter is matched. Pipe the message to the program by stdin.

I would suggest: - to allow the filter to pass arguments to the external program, so one external program can be used to do (slightly) different things - to allow the filter to specify whether the full message or only the headers will be passed to the program, so that simple filters do not have to sift through big messages with attachments - use pseudo-headers to let the external program know about all mozilla-internal stuff like labels, which folder this comes from, etc. Likewise there should be an easy protocol for the external program to specify the desired actions, like move, mark, delete etc. Finally the external program should also be able to modify the message before it is further processed. This would allow intelligent removal of attachment and other stuff.

*** Bug 143872 has been marked as a duplicate of this bug. ***

I think this RFE mixes two different things: external program as filter criteria, and external program as filter action. Shouldn't that be separated?

*** Bug 169332 has been marked as a duplicate of this bug. ***

*** Bug 197598 has been marked as a duplicate of this bug. ***

mass re-assign.

*** Bug 205581 has been marked as a duplicate of this bug. ***

*** Bug 247723 has been marked as a duplicate of this bug. ***

*** Bug 248000 has been marked as a duplicate of this bug. ***

That would be a solution to some sound playback problems, too. Just run "esdplay" or "play" etc. instead of using the custom wav sound for notification feature. Of course, allowing/implementing protocols for the custom wav sound feature like file:/sound: for internal and shell: for external playback with given executable would be nice, too.

This has security concerns written all over it. Whoever implements this should be very careful how its done.

Or they could -- gasp -- trust the user to actually know what they're doing, which would be a welcome change for we Unix users who do (e.g. those of us who write the external programs that we'll be calling :) ). /me is still bitter from that most annoying "Are you sure that you want to do what you just said that you wanted to do?" authentication "improvement". Security is one thing, but insulting my intelligence is another. Please don't turn Mozilla into a product geared towards users who don't know what they're doing. If I want that, I'll install Windows or some desktop Linux distribution (and in either case, completely disconnect it from the network -- I hate systems wherein more than one hole per year grants someone complete system control).

(In reply to comment #11) > This has security concerns written all over it. Whoever implements this should > be very careful how its done. I get you right, that these file: or shell: URL's have such security concerns? Of course, using an external program as a filter has security concerns, roughly comparable to using a CGI script on a web server. However, from the view of the foundation, it seems sufficient not to deliver such filters and leave the concerns to the script author. Or would you suggest to stop delivering mod_cgi with Apache?

My concern is that a virus can be excecuted by an external program.

(In reply to comment #14) > My concern is that a virus can be excecuted by an external program. Evolution currently (version 1.4) implements external programs as message filters. The filter option GUI is like this: IF drop down> "Pipe Message to Shell Command" <text area for command name> drop down> "returns" "doesn't return" "returns greater than" "returns less than" <text area for numeric return value> THEN Options like "move to Folder", "Delete", etc. So, the external program reads the message from stdin and returns a number which lets Evolution know if it should perform a standard filter action. Under such a scenario, the only way an email could trigger a virus would be if the shell command you pipe into was capable of doing nefarious things based on its reading stdin. If you pass your email into a "bash -c" then you're asking for trouble, but that would be pathological behavior. As far as I can see, this would be a power user feature that one would have to go out of their way to use in a risky way.

*** Bug 253351 has been marked as a duplicate of this bug. ***

External scripting would be great. They could receive mails in some XML form and process/filter them.

*** Bug 266093 has been marked as a duplicate of this bug. ***

*** Bug 304814 has been marked as a duplicate of this bug. ***

*** Bug 312484 has been marked as a duplicate of this bug. ***

*** Bug 318909 has been marked as a duplicate of this bug. ***

(In reply to comment #15) > > Evolution currently (version 1.4) implements external programs as message > filters. The filter option GUI is like this: > ... > If you pass your email into a "bash -c" then you're asking for trouble, > but that would be pathological behavior. As far as I can see, this would be a > power user feature that one would have to go out of their way to use in a risky way. For novice users, thunderbird/mozilla-mail may ship with it a set of predefined filters, with correct command lines, and disabled at default. It can also check for existence of called programs to force disabling unavailable filters. It can even give names to those command lines, e.g., "ClamAV Virus Scanner" for "clamdscan -". Hopefully both novice and power users are satisfied this way. BTW, my feature reqest (Bug 318909) was marked a duplicate of this one. I'm grateful that my report get operated within a few days. But I wana make it clear that mine is mostly a duplicate of what comment 15 describes, quite different from the original reqeust of this bug report.

A related alternative (possibly worthy of a separate RFE, if it already does not exist) is to have a JavaScript API for message filtering.

(In reply to comment #23) > A related alternative (possibly worthy of a separate RFE, if it already does > not exist) is to have a JavaScript API for message filtering. Bug 11037

Reassign to default assignee and QA contact.

*** Bug 356451 has been marked as a duplicate of this bug. ***

*** Bug 362281 has been marked as a duplicate of this bug. ***

I am looking for just this kind of functionality, but to be really useful, there ought to be a way to read the message back in (via stdout), which would allow the command line tool to transform it. For example, it could add custom headers, filter profanity, or whatever, and the output can then be acted upon by Thunderbird after returning. This would enable integration with SpamAssassin installed locally, for example. So I propose two kinds of filters: 1) Call an executable and do nothing with the output. 2) Pipe a message through an executable to allow it to be transformed. KMail supports both of these, by the way, and power users can do some very cool things. Text-to-speech announcement of messages, SpamAssassin and anti-virus tool integration, logging, send to cell phone, etc. Neat stuff.

(In reply to comment #28) > 2) Pipe a message through an executable to allow it to be transformed. Generally speaking, we don't want to "transform" messages on receipt; we want to keep a record of what was actually sent. The only exception I know of is allowing attachments to be detached or deleted -- after the user has had a chance to see the original message. Not to say that the return value couldn't be of use, but it would need to somehow be mappable to the actions already provided in the filters.

(In reply to comment #29) > Generally speaking, we don't want to "transform" messages on receipt; we want > to keep a record of what was actually sent. Generally speaking, I'd agree. This is obviously an advanced feature that can be ignored by novices and others who don't want or need it. For many power users, though, this has great utility. SpamAssassin is a good example, and there are others.

(In reply to comment #30) > (In reply to comment #29) > > Generally speaking, we don't want to "transform" messages on receipt; we want > > to keep a record of what was actually sent. > > Generally speaking, I'd agree. This is obviously an advanced feature that can > be ignored by novices and others who don't want or need it. For many power > users, though, this has great utility. SpamAssassin is a good example, and > there are others. > I think the problem here is that we do not have a proper way to associate meta-data with emails. An external program could be used for scores of tasks, but nearly all of them would provide meta data instead of actually modifying the content of a message. Examples for filters that would provide metadata are: a better/different way to detect spam or scam emails; classifying emails as belonging to some group (assigning tags). If theses filters are based on machine learning, then we would need seperate filters for the training phase and there, metadata should record whether or not an email was already used for training. There are many good algorithms for these and similar tasks out there and a clean API for processing emails and giving back information would make it possible for them to get included in Thunderbird. I think that would help to include some truely innovative features for Thunderbird (e.g. in the form of an extension). All that is missing is some expert on the inner workings of Thunderbird to provide this API. Regarding the way how Thunderbird could exchange information with the external filter I am a bit reluctant about pipes: this nearly never works cleanly when you want to pipe both out AND in to and from a program and can lead to nasty lock-ups and other problems. The alternative is to either use temporary files or -- something I would personally prefer -- a server/client based protocol. The external filter program would be a server that accepts commands and data on a certain IP/port and passes the result back to Thunderbird in a well-defined and well documented way. If one really wants to use a pipe or file based filter program, it would be easy to wrap it in a tiny server program, thus creating a common interface for all cases. The big advantage of a client/server API for this would be that you have the flexibility to provide a single server e.g. for your entire office or family or just for your own user. BTW thank your for pointing out KMail -- I will have a look since I am working on machine learning / language technology based algorithms.

This might be overkill. A separate server process and API adds complexity and would be more difficult for non-programmers to use. And remember, there is already a mechanism for associating meta-data with e-mails: custom headers. With the approach I'm thinking of, Thunderbird would need to be enhanced to evaluate customer header values in filter criteria to handle the SpamAssassin use case. KMail is a good example, but I took a close look at Evolution last night, and I'm even more impressed by what they've done. They allow piping through a program as part of filter criteria OR as part of the action. So you can test the return code of the program or modify the message/headers or both.

(In reply to comment #32) > This might be overkill. A separate server process and API adds complexity and > would be more difficult for non-programmers to use. Hmm, my concern is this: how can one have a flexible method to invoke external functionality from Thunderbird in a *portable* way that does not put Thunderbird stability in danger? I think that a simple client/server API fulfills all those requirements. To make it possible to use pipe/file-based programs, a sample server could be part of the TB distro (e.g. written in Java). I kind of like the idea that all these programs really run seperated from the TB program. If something goes wrong, TB will just receive an error packet, a timeout, or loose the connection to the server. I think communicating with a server and handling error situations is something that is well supported in Thunderbird, so it should not be too big a problem to implement. The problem with a solution that relies only on pipes is this: unless you handle writing to the pipe and reading back from the pipe asynchronously and unless the external program promises not to buffer its output, all kind of lockup situations can happen, locking up TB in the worst case. I do not know too much about how this is done best in Windows, but this has to be portable to at least the three supported platforms. From other projects, my experience is that the additional effort of a clean solution will usually pay off many times through additional flexibility, stability and portability. > And remember, there is > already a mechanism for associating meta-data with e-mails: custom headers. Is there a clean API for doing this available in Thunderbird? Sorry about my ignorance, but I thought the problem with this is that each modification will re-write the message to the end of the mbox file. (I have no idea how this would co-operate with IMAP accounts). > With the approach I'm thinking of, Thunderbird would need to be enhanced to > evaluate customer header values in filter criteria to handle the SpamAssassin > use case. > > KMail is a good example, but I took a close look at Evolution last night, and > I'm even more impressed by what they've done. They allow piping through a > program as part of filter criteria OR as part of the action. So you can test > the return code of the program or modify the message/headers or both. > I agree: running external programs would be useful for both the criteria and the action.

(In reply to comment #33) > Hmm, my concern is this: how can one have a flexible method to invoke external > functionality from Thunderbird in a *portable* way that does not put > Thunderbird stability in danger? The program would run as a separate process, so Thunderbird stability should not be an issue. (I think extensions pose a greater risk to stability than this feature.) There is a chance that the called program might hang, but the same could be said for a server process. Plus, a server can crash, and its lifecycle needs to be managed. Regarding portability: All supported OSes provide a way to spawn external processes and Thunderbird itself already does this (e.g. launching a browser). The only potentially new part is reading from and writing to standard in/out/error streams, and all supported OSes have these concepts. I'm not saying a server is a bad idea, but it does add a layer of complexity, and I'm still not clear on what benefit it provides over the simpler alternative. > The problem with a solution that relies only on pipes is this: unless you > handle writing to the pipe and reading back from the pipe asynchronously and > unless the external program promises not to buffer its output, all kind of > lockup situations can happen, locking up TB in the worst case. Again, if the external program is launched as a separate process, it is isolated. The addition of a server layer doesn't add any new protection that I can see. > > And remember, there is > > already a mechanism for associating meta-data with e-mails: custom headers. > Is there a clean API for doing this available in Thunderbird? > Sorry about my ignorance, but I thought the problem with this is that each > modification will re-write the message to the end of the mbox file. > (I have no idea how this would co-operate with IMAP accounts). Don't know yet. I am probably more ignorant about Thunderbird internals than you. ;-)

(In reply to comment #34) > (In reply to comment #33) > > Hmm, my concern is this: how can one have a flexible method to invoke external > > functionality from Thunderbird in a *portable* way that does not put > > Thunderbird stability in danger? > > The program would run as a separate process, so Thunderbird stability should > not be an issue. (I think extensions pose a greater risk to stability than this > feature.) There is a chance that the called program might hang, but the same > could be said for a server process. Plus, a server can crash, and its lifecycle > needs to be managed. > I have programmed both -- client/server and pipe-based invokation of a program. In my experience, there is quite a difference between the problems of handling stale input pipes, handling stuffed output pipes, handling broken pipes etc. correctly and handling TCP connection problems (the former being a pain, especially on Windows). > Regarding portability: All supported OSes provide a way to spawn external > processes and Thunderbird itself already does this (e.g. launching a browser). > The only potentially new part is reading from and writing to standard > in/out/error streams, and all supported OSes have these concepts. > Again, that new part is the problem in my view. This is very hard to do correctly on either Linux or Windows, and it is even harder to get right for both. > I'm not saying a server is a bad idea, but it does add a layer of complexity, > and I'm still not clear on what benefit it provides over the simpler > alternative. > Apart from a more clean handling of problems from the TB part and a totally identical interface on all OSes, it would also make it easier to specify a flexible protocol (remember, return codes etc. are not well portable either). > > The problem with a solution that relies only on pipes is this: unless you > > handle writing to the pipe and reading back from the pipe asynchronously and > > unless the external program promises not to buffer its output, all kind of > > lockup situations can happen, locking up TB in the worst case. > > Again, if the external program is launched as a separate process, it is > isolated. The addition of a server layer doesn't add any new protection that I > can see. > The issue is the pipe: it makes it necessary to check status codes, check for zombie processes etc. This is easy to do in a wrong way that works nearly always, but only nearly. > > > And remember, there is > > > already a mechanism for associating meta-data with e-mails: custom headers. > > Is there a clean API for doing this available in Thunderbird? > > Sorry about my ignorance, but I thought the problem with this is that each > > modification will re-write the message to the end of the mbox file. > > (I have no idea how this would co-operate with IMAP accounts). > > Don't know yet. I am probably more ignorant about Thunderbird internals than > you. ;-) > Maybe somebody else can aswer this: is there an API tha abstracts from the header stuf to handling of metadata? The reason why I think this is important is that it is not a good idea to implement all kind of meta-data processing on top of a hack, instead of a meta-data API.

(In reply to comment #35) > I have programmed both -- client/server and pipe-based invokation of a program. > In my experience, there is quite a difference between the problems of handling > stale input pipes, handling stuffed output pipes, handling broken pipes etc. > correctly and handling TCP connection problems (the former being a pain, > especially on Windows). How does a server eliminate these issues? It seems to me if the server is invoking the external program, then you've moved the problem, but have not eliminated it. Unless you expect that Java will do a better job of managing the I/O...?

(In reply to comment #36) > (In reply to comment #35) > > I have programmed both -- client/server and pipe-based invokation of a program. > > In my experience, there is quite a difference between the problems of handling > > stale input pipes, handling stuffed output pipes, handling broken pipes etc. > > correctly and handling TCP connection problems (the former being a pain, > > especially on Windows). > > How does a server eliminate these issues? It seems to me if the server is > invoking the external program, then you've moved the problem, but have not > eliminated it. Unless you expect that Java will do a better job of managing the > I/O...? > The server does not necessarily eliminate these issues. It *can* because it can implement the desired function in a way that avoids these issues completely. But if the implementer of the function desires to implement it through a pipe wrapped in a server than at least the problems are at the side of the function implementer, not TB. So the main point is that these issues are, if not eliminated, moved away from TB. I also think that the in the year 2006 a pipe-based solution to providing the functionality of program A to program B is just anachronistic. Even if it is not used, a proper server/client architecture would allow to run the program on a different machine (which might run a different OS) or even on the other side of the globe. This would open up the possibility to use certain filters from a Windows server even though the filter software is only available on Linux (with a VM, this could actually run on the same machine). I just suggest to think more "year 2010" than 1980 here :) I wonder how much of this client functionality within TB could actually be implemented in an extension? Is anyone aware of a documentation that describes the API and hooks for the filter/action functionality withing TB? And BTW -- is there a way to run filters for any incoming/new mail message?

(In reply to comment #37) > So the main point is that these issues are, if not eliminated, moved away from > TB. I also think that the in the year 2006 a pipe-based solution to providing > the functionality of program A to program B is just anachronistic. Even if it > is not used, a proper server/client architecture would allow to run the program > on a different machine (which might run a different OS) or even on the other > side of the globe. > This would open up the possibility to use certain filters from a Windows server > even though the filter software is only available on Linux (with a VM, this > could actually run on the same machine). I think what you're proposing is really a different feature, beyond the scope of what this RFE describes. As a power user, I'm not interested in managing the life-cycle of a filter server or calling one on a remote machine, which has security implications anyway. I *certainly* don't want to write my own server app. All I want to do is invoke an external program on a message and maybe do something with the output. Piping is an extremely well-known and time-tested technique for portable interprocess communication. Its in heavy use in the UNIX/Linux world and still will be in 2010. Best of all, it doesn't require me, the power user, to learn anything new. I just think that introducing a server adds unnecessary complexity and potentially a different set of problems for the end-user. We may just have to agree to disagree on that. Anyone else want to weigh in?

(In reply to comment #38) > I just think that introducing a server adds unnecessary complexity and > potentially a different set of problems for the end-user. We may just have to > agree to disagree on that. Anyone else want to weigh in? I would totally agree with the above. TB is not a the kind of software whose misuse hurts others. In the worst case, filtering your mail through a mis-behaving pipe mess up all your mails, but not others'. When a power user decides to do some piping, he has the right to be responsible for himself. Running the piping filter in a separate process is another thing. TB may well do such respawning by default, and even handle common pipe lock-up errors in the child process. Power users should be happy with these, as it provides enhancements without reducing flexibility.

(In reply to comment #38) > Piping is an extremely well-known and time-tested technique for portable > interprocess communication. Its in heavy use in the UNIX/Linux world and still > will be in 2010. Best of all, it doesn't require me, the power user, to learn > anything new. > > I just think that introducing a server adds unnecessary complexity and > potentially a different set of problems for the end-user. We may just have to > agree to disagree on that. Anyone else want to weigh in? I agree completely. I just want to filter new mail with some scripts that can compare new messages with existing messages, decide where to file messages, delete them, flag them, etc. I even want the ability to munge messages. I do want my scripts to be able to tell TB (and add-ons like calendar) what to do with messages, but this can be done using pipes by adding special commands to the output. It's very simple: pipe message to external program (including special headers that tell the script where mailboxes are), read message (or response headers) back from output (or timeout if there's no response). Process special headers and decide what to do with the message.

So based on comments, there are at least three main use cases to consider: (1) User wants to pipe a message to an external program and test the value of the return code in a filter criterion (equal, not equal, less than, greater than). Any of the normal filter actions can be taken. Example: A custom script logs the message and returns non-zero if it fails. If it does fail, we want to flag the message. (2) User wants to pipe a message THROUGH an external program in order to modify it (e.g. add custom headers). The output (if any) replaces the original message, so downstream filters can run tests on it, and the user will see the modified message in the UI. Examples: SpamAssassin integration, anti-virus integration, or a profanity filter. (3) User wants to execute a program as part of a filter action, possibly passing in the message. Output should be ignored. Examples: Play a sound or text-to-speech integration. Does that cover it? Use case (1) could be handled by a "Pipe to program" filter criterion. Use cases (2) and (3) can both be handled by actions, I think. Downstream filters can be applied to modified messages because it looks like TB continues processing subsequent filters even after a message matches one. This enables use case (2). It looks like TB already allows filters to test arbitrary header values, so no enhancement is needed for that.

I think you've summarised it perfectly Rob. I agree a "server process" is a bad idea that adds unnecessary complexity and will become a support nightmare. It will also be doing some of the functionality the OS is perfectly capable of doing. Don't re-invent the wheel, just use what's already provided. This needs to be kept simple (KISS), as you've said earlier. If someone wants to run an email through a filter on a remote machine of a different OS, do this within the piped script/binary and not TB. This is easily done using the various utilities on the OS. TB shouldn't be responsible for the behaviour of the piped script; all it needs to do is pass the mail to/from the script and act accordingly. If the piped script falls over, this isn't TB's problem. TB can implement a timeout (customisable?) and simply log a message in the log (if enabled) stating the piped script fell over if it doesn't get a response or exit code within the timeout. It doesn't need to provide a reason. That's up to the piped script developer to work out. Evolution does this beautifully.

OK - can those experts who are in favor of the pipe through solution point out how this can be done in FF without the danger of a deadlock? How is the multitasking and stream capture handled so that normal operation and error handling will reliably work on all systems where TB is running? This will have to involve *reliable* multithreading. Can you give some sample code or pointers to the apropriate functions? I also strongly disagree with comment #39: "In the worst case, filtering your mail through a mis-behaving pipe mess up all your mails, but not others'." This is supposed to become an API for reliable und useful features or extensions, not some experimental stuff for power users who for some reason (?) can afford to "mess up all their mails". I do not think anything should be implemented in TB where this risk exists.

All due respect, but you're debating an implementation detail of a patch that hasn't been written yet. Why not wait for one before labeling the feature as "experimental" and "unreliable"? Judging by the replies here and features in competing products, I'd say there are a number of people who find value in adding this. I get that you're not one of them. Fine. I'd love to hack at this now, but unfortunately, my day job hasn't afforded much time.

Maybe I am wrong when it comes to TB, but I have some experience in trying to do this in other languages, so I know about the problems of reading and writing to another process at the same time. There are several issues involved here: since both the writing and the reading are buffered, they cannot happen in the same thread without the risk of a deadlock. Depending on the OS, it is hard to handle external programs that lock up or otherwise do not terminate, without leaving zombies. Since people said that these things are not issues here, I would love to learn how this can actually be done in TB, and what XPCOM services are here to support this. I would actually be happy to learn that this can be done without problems, what I was saying in my previous email that *if* the feature cannot be implemented in a stable way, I think it would be inacceptable to get included in TB. There is another issue here: how should data calculated by the filter be stored? If I understand correctly, there is, because of the way how emails are stored locally, currently no *good* way to modify a message after it has already been retrieved and added to the mbox, so simply adding a header won't work well (every modified message would have to get virtually deleted and re-appended to the folder file). Does the API currently support storing arbitrary metadata for each message? An alternative would be to run the filter when the message is being retrieved, *before* it is added to the mbox -- in that case, it can be modified without problems in any way, but that can only be done once. And I think TB does not at the moment support special "retrieval" filters (as opposed to filters that can be run at any time after retrieval).

I just released on AMO the extension "FiltaQuilla" which adds (among other things) a custom filter action to launch a file from the OS, as requested by some of the dups of this bug. It does not however pipe the email to the application, which seems to be the main request of this bug. See https://addons.mozilla.org/en-US/thunderbird/addon/10052

This was requested over EIGHT years ago, with 19 duplicate requests for this functionality, and still no movement? Is it just a bad idea? Have the Powers That Be deemed this to be unworthy of inclusion in a release? Would we get movement on this if I ponied up some cash? How much are we talking here?

Peter, most of the work that I do in filters is working to make it more accessible to extensions, so that in the future most requests for new filter actions (and new searches after bug 495519 lands) will be implemented in extensions, not in the core. Some limited ability to do the request of this bug is in FiltaQuilla. However, the piping of the message to another process is currently blocked by the fact that the Mozilla core does not have a cross-platform mechanism to pipe from process to process - and developing one is tricky because you are interacting with multiple operating systems. But I understand there is a Google SOC project to work on this, see http://jamesboston.ca/cms/?q=node/127 If and when that project provides the core capability, then I will probably be adding the mail pipe filter action requested in this bug to FiltaQuilla. I don't know about adding it to the main program yet. As for "ponied up some cash" the problem with that is that most people in OSS think $10 is "some cash". When you are willing to talk about the cash to support a developer for some weeks or months, then we can talk about "some cash". Otherwise, you are free to volunteer (as I have) to work on what you think is important to you.

I do not agree. I suggest we need simple interface for external scripting, semaless CGI-BIN: 1. at the beginning, external tool sholud be run with open pipe. 2. next thunderbird should send base email properties. all what we can test in impelemted filter. with empty line as EOF. 3. script may finish STDIN reading everywhere. thunderbird should then stop writting when external script close pipe. 4. external scirpt before exiting or closing STDIN, should print to STDOUT what should be done. simple lines is the best for me. move: folder , delete: , asterisk: on/off etc. then external script should get it and do when script will exit. it's all what we really need I think. It isn't more complicated than actually implemented internal filtering. I don't see why it should be implemented in extention.