The mail-news plain text editor inserts an extra space at the start of lines which begin with one or more spaces, when: 1 - The message is sent. 2 - The message is saved to Drafts or Templates. However, there is no problem when the body of the message is saved to a file, or when it is copied and pasted to another application. Saving to Drafts/Templates does not alter the message being composed - the problem is in the saved message. To reproduce: Compose a new message addressed to yourself. In the body, type a space and then "X". Save the message to Drafts and/or send it and see the results. There will be two spaces before the "X".

snarfing this bug for disposition. I bet this is a serializer problem, in which case it most likely won't be me who fixes it. But I'll check first.

marking this confirmed for now. I don't want it to fall off the radar.

cc'ing folks

When Joe described this it sounded similar to bug 145196, and I wondered if burpmaster's patch in that bug might also fix this one. But I just noticed that this one is plaintext, not html, so I'm less optimistic about that now.

I tried this. When I read the message, I saw a single space, but I saw there were two in the message source. Read "4.4. Space-stuffing" at http://www.rfc-editor.org/rfc/rfc2646.txt The first space is removed by format=flowed capable readers, so the correct behavior would be to put that extra space in. I think the real bug is that HTML Composer doesn't do this.

The days of having a half dozen milestones out in front of us to divide bugs between seem to be gone, though I dont know why. Lumping everything together as far out as I can. I'll pull back things that I am working on as I go.

*** Bug 162247 has been marked as a duplicate of this bug. ***

*** Bug 189147 has been marked as a duplicate of this bug. ***

By the definitions on <http://bugzilla.mozilla.org/bug_status.html#severity> and <http://bugzilla.mozilla.org/enter_bug.cgi?format=guided>, crashing and dataloss bugs are of critical or possibly higher severity. Only changing open bugs to minimize unnecessary spam. Keywords to trigger this would be crash, topcrash, topcrash+, zt4newcrash, dataloss.

editor:core

This bug still exists for me here. I use to write mails with a non-proportional font (like courier) and want to format paragraphs like this: * This is an item with one text line. * This is an item with more than one text line, which should show three blanks in front of the 2nd and all following lines. * This is another item with one text line. Sending plain text mails like this (or saving them as draft/template) always adds the damned one blank to all lines which begin with a blank, like this: * This is an item with one text line. * This is an item with more than one text line, which should show three blanks in front of the 2nd and all following lines. * This is another item with one text line. Could someone please fix it? Thank you so much in advance.

This bug is now a year old. I just want to say that this is the only reason I am still using Netscape 4.77 for browsing and mail. Once this one is fixed and propagates into the Netscape 7.x commercial version, with the spellchecker which I really need, then I will happily use that version of Netscape 7. - Robin

Yes, I also would like to use mailnews again - its IMAP support remains vastly superior to anything else which I have evaluated. But this bug is a showstopper. Tomorrow will be bug #141983's first birthday. What's up?

I should have pushed this over to the DOM to Text Conversion component long ago. But since that component hasn't had an active owner, it probably didn't matter.

I did no longer wait for this bug to be killed ... Now it does no longer exist for me here. WorksForMe since Build ID 2003042708. :-)

When sending and reading an email with Netscape 7.02 or a recent Mozilla, all looks well, but it is not. The sent mail has the extra space (if you look at the source - at the message itself) and the extra space is then ignored so it looks OK. So the original bug when sending remains and a new one has been introduced which leads to faulty display and printing of emails. I suspect this bug in displaying the message has something to do with "format=flowed" in the header, since there is no such misdisplay of a message from Netscape 4.77 which has no "format=flowed". I will investigate further and write a comprehensive bug report here ASAP. - Robin

As noted in message 15, it appears that there is no problem when reading the message on Mozilla. But this is because there is a second bug which removes one space from indented lines when the message is "format=flowed" and we are reading on screen, printing or viewing Print Preview. I have reported this as Bug 204437. The two bugs are probably separate and do not depend on each other, but one hides the other from the point of view of a Mozilla user. These bugs have been observed on Mozilla 2003050211 and Netscape 7.02 both on Win2k, but I imagine it has nothing to do with the operating system. From the user point of view, a related bug is bug 86607 concerning there being no user interface option to disable the interpretation of "format=flowed" when reading emails. I don't think there is a separate bug for the desirability of a similar user option for disabling sending with "format=flowed". My view is that both these features should be off by default and have user interface options with a good explanation so people can turn them on if they want. Both features are potentially useful but having either of them on by default means that messages will probably be seen by their recipient with different line formatting from how the sender wrote them. The reason I never found that the cause of this 1 year old bug is to do with "format=flowed" is that I never tried turning it off, because I wasn't going to use Mozilla for mail until this bug was fixed! Thus, here we have a case where an extra complication was added to the program, in a way which happened to contain a bug, and because it was turned on by default we all thought for a year that there was a basic problem in the program without having any reason to think that the bug was in the new feature. Ironically, the only clue to the bug being due to "format=flowed" when sending the message was a corresponding bug in the read code for "format=flowed" - which I would never have noticed unless it had been on be default. The workaround for both these bugs is to disable these features with the following lines in user.js: pref("mailnews.send_plaintext_flowed", false); // RFC 2646======= pref("mailnews.display.disable_format_flowed_support", true); For those who are more interested in using the program than chasing bugs, I will document how I get Mozilla and Netscape 7 to do what I want at: http://www.firstpr.com.au/web-mail/Mozilla-mail/ - Robin

This isn't a bug. See comment 5. From the format=flowed spec: 4.4. Space-Stuffing In order to allow for unquoted lines which start with ">", and to protect against systems which "From-munge" in-transit messages (modifying any line which starts with "From " to ">From "), Format=Flowed provides for space-stuffing. Space-stuffing adds a single space to the start of any line which needs protection when the message is generated. On reception, if the first character of a line is a space, it is logically deleted. This occurs after the test for a quoted line, and before the test for a flowed line. On generation, any unquoted lines which start with ">", and any lines which start with a space or "From " SHOULD be space-stuffed. Other lines MAY be space-stuffed as desired. (Note that space-stuffing is similar to dot-stuffing as specified in [SMTP].) So you see, that extra space is needed. Otherwise, readers which properly support format=flowed will show one less space than there should be. The primary goal of format=flowed is to eliminate line wrapping and quoting problems. The secondary goal is to remain as readable as possible to unsupporting readers, but format=flowed readers must still be able to piece together the original message as intended. I'm tempted to mark this as invalid right now, but I'll wait for further comments. One solution may be to space-stuff all unquoted lines, so that at least they will line up on incapable readers.

Yes, things like standards-compilance and good-looking text are nice. But the MUA should also pass a basic "is this useful" sanity test. Right now, it is not possible to use Mozilla to send patches in the body of emails. It is the _only_ MUA which has this problem. It isn't useful. Now, the preferences which Robin dug up look like they'll fix the problem (I haven't tested). Perhaps you could be persuaded to expose those preferences in the actual GUI somewhere ("don't mangle my plain text messages" checkbox). Then I expect everyone will be happy.

If, as burpmaster writes, an MUA which sends with "format=flowed" *must* add a space to indented lines (or any lines at all) such that the message is changed when read by a non "format=flowed" MUA, then I think the whole concept is nutty. That being the case, this is further argument for the default state of the MUA being not to do this - not to send "format=flowed" unless the user is fully informed that this will result in the alteration of their messages in the raw text which is sent, and therefore when being read by all MUAs which are not able to, or not configured to, properly interpret "format=flowed". Surely the most important thing is to never alter messages in any way, unless the user selects an option to do so. - Robin

<offtopic> > Surely the most important thing is to never alter messages in any way, unless > the user selects an option to do so. As we have argued before, this is not possible, For example, in plaintext, you need to munge "From " lines, and lines starting with ">" are ambiguous (quote or not?). Format=flowed solves *that*, but it can't solved everything at once while being compatibel to plaintext, because plaintext itself is too flawed. So, you see that your repeated arguments against f=f are totally biased and one-sided, and please keep them to the newsgroups. </offtopic> As stated in comment 5 (!) already and again in comment 18, this bug is indeed invalid, and I was about to mark it as such. The only thing keeping me from it is comment 19 - sending patches to non-f=f-supporting mailers (things will work fine from f=f supporting mailers to f=f supporting mailers), adjusting summary. Anybody has an idea about how to solve that, apart from space-stuffing *every* line? (I'm not sure, if that is a good idea - maybe it is, but I am sure that people will complain about that in other cases.)

Ben changed the name of this bug, which I created, from Space added to indented line when sending message to: Can't mail patches in body to non-f=f mailers (Space added to indented line when sending message) and again tried to limit me - and presumably other people - from expressing our opinion that this behaviour is a bug. I changed the name back to its original state. Ben - I think it would be best if you stated your opinion, as you do, and left it at that, rather than maligning my attempts to resolve problems with the program and rather than trying to stop us expressing our views. If, in my opinion, and in the opinion of others, the way Mozilla with "format=flowed" as the default (with no user interface option for turning it off) adds spaces to indented messages on a regular basis is a bug, then let us deal with it as a bug. If you disagree, then say so. But I think it would be best if you do not try to suppress the expressions of opinions contrary to your own. Please do not rename my bugs. You are entitled to your opinion that this aspect of "format=flowed" is of little or no real consequence and we are entitled to our view that it is. I am of the view that it is of vastly greater consequence than the one existing occasional problem with plaintext emails, the insertion of a ">" before a "From". You wrote: > For example, in plaintext, you need to munge "From " lines, and lines > starting with ">" are ambiguous (quote or not?). This is untrue. There's nothing about plaintext email which causes this - nothing in any Internet standard. The problem only occurs when the message is stored in an Mbox mailbox (one where all the messages are in one file). If Maildirs are used, there is no such problem. Maildirs have many advantages, but this is one of them. Mboxes are never, in my experience, used on serious mailservers any more. They may be used in the local mailboxes Mozilla maintains on its own computer, but I never use them - all my mailboxes are on a Maildir-based IMAP server. http://www.rfc-editor.org/rfc/rfc2646.txt does indeed confirm that this insertion of spaces into indented lines is part of the way "format=flowed" is meant to work: If a space-stuffed message is received by an agent which handles Format=Flowed, the space-stuffing is reversed and thus the message appears unchanged. An agent which is not aware of Format=Flowed will of course not undo any space-stuffing, thus Format=Flowed messages may appear with a leading space on some lines (those which start with a space, ">" which is not a quote indicator, or "From "). Since lines which require space-stuffing rarely occur, and the aesthetic consequences of unreversed space-stuffing are minimal, this is not expected to be a significant problem. I believe these people completely mistaken to think that altering someone's message like this is not a significant problem. Its not for them to judge what is not significant in other people's messages. Burpmaster, you did write about this on 30 May last year - pointing to the RFC but not mentioning "format=flowed" specifically. But I did not chase that link. If I had, then I would have seen that this was the "format=flowed" RFC. Had I, or anyone else, done this, we would have realised that this is not a bug in the serializer, editor core or DOM to text conversion, as jfrancis amongst others have been thinking. We would have realised that the problem was plain and simple: 1 - Someone, unwisely in my view, has made sending with "format=flowed" the default - and without a user interface option to disable it. 2 - None of the rest of us, you and Ben excluded, realised for a year that this adding of spaces is a functional part of "format=flowed". None of us has the moral right to foist hidden changes on people's messages - because these are people we don't know and we are not entitled to think that any such change is insignificant to them. In that case, having "format=flowed" on by default is completely out of the question. I would be interested to know what jfrancis and other full-time Mozilla developers think of this. I do not know who decides the default state of "format=flowed" for sending or for receiving, or whether these have a user option. - Robin

It's a problem that recipients that hasn't implemented RFC 2646 sees an extra space in front of lines which previously started with one or more space characters. It's a problem that quoting plain text causes too long lines or silly wrapping. While problem number two was a big problem with Netscape 4 I've heard very little about it in Mozilla. Maybe that's because Mozilla uses format=flowed. Maybe it has some other reason (me not being CC:ed on those bugs for instance). The first problem, while being irritating for those who rely on the exact layout or has a keen layout awareness, is not completely new. Mail standards has always forced modifications to mail messages. Sending QP will replace characters. There are line length limits in the SMTP standard. Some storage formats requires the message to not contain the word From in the beginning of the line. The SMTP standard doesn't allow a single period in a line. SMTP servers are allowed to strip the eigth bit when that bit is set in a message. The space stuffing is only more visible to some users than the previous limitations was. Then again, most users doesn't notice since it requires a MUA that doesn't understand RFC 2646 and that displays the text using a fixed width font. Since this is an old problem, the solution (workaround) is old: sending the mail in a presentation format instead of a content format. It could be HTML, MS Word, PostScript, PDF or even an attached text file. Turning off format=flowed sending would make some people happier, but I _think_ (by my reasoning in the beginning of the post) that it would make the situation worse for the vast and silent majority. Many of those that are irritated by the spaces are power users that are able to turn format=flowed on and off by the means that are available. Maybe those users need a power mail client and I would welcome anyone building such a client on Mozilla's back end but Mozilla shouldn't be solely built for those that are loudest. Sometimes we all have to try to place ourself in a "normal" user's shoes and try to make something good for him/her.

Robin, as for the subject of this bug, what you stated as bug is none, it's behaviour as intended, and in fact mandated by the standard. As I said and burpmaster said before, it would be invalid, but I tried to change it into something useful. You refuse that - OK. It's invalid as-is, and I'll mark it as such now. I filed bug 204478 about the patch-in-body case. As for expressing your other views, you have no right to force your f=f-propaganda down on us in every bug that is remotely related to f=f. Keep it to one place, where it's appropriate, and where it has been discussed before (incl. all your arguments mentioned), the newsgroup.

In answer to Robin's question, I do think it would be nice if users could turn off format-flowed. I don't feel I have a rich enough set of plaintext mail experience to state what the default should be, but I do note that I can't find a way to turn it off. Apparently the user.js hack is the only way. Why Ben is bent on pissing off one our most competant bug writers on mailnews and editor issues? Reading all the comments prior to the recent pissing match, I don't see much "propaganda". Robin says in one comment he finds format-flowed nutty. Hardly seems like a big deal, particularly since he has provided so much useful data in other comments. I'm unsure why this bug is being marked invalid. Does the test case in comment #11 no longer demonstrate the bug? Are other users unlikely to regard this as a bug? From my understanding of Ben's comments, it sounds more like a WONTFIX than an INVALID. Ben, can you point me at where this space stuffing is occurring, in the code? Have we got the right component and owner for this?

It's done in nsPlainTextSerializer.cpp. Search for "stuff" or "stuffing" or something like that and you'll find it. I don't think our implementation is wrong. We either has to space stuff or disable f=f. What we could do is to make it possible to restart the creation if we notice that we might need space stuffing and then do the same thing without f=f. That is if we decide that the space stuffing is a really bad part of f=f that we can't live with.

> Why Ben is bent on pissing off Because this discussion has a history. We have discussed this issue *very* long and extensively at least 2 times on Mailnews, years ago. I explained to him in detailed and long posts why some of his ideas and basic assumptions are flawed. That didn't prevent him from later writing page-long statements about how evil f=f is, in bugs not even strictly about f=f. Not one bug, but the full text in 3 bugs at once, via copy&paste, IIRC. Now he starts over here, with other words, but the same arguments. I just don't feel that this is an appropriate or workable way to discuss - if I started to counter-argue him, we'd start to have the same long discussion in 5 bugs at once. That's why I just shut this particular discussion down and try to redirect it to the newsgroups, where I think it belongs and where all the history to it is. I don't think it belongs here anyways. This is clearly standards-mandated behaviour, we have no way to fix this. If he wants to argue that the *standard* is flawed and should not be supported, then I don't think that this bug is the right place for it, because the whole issue is far broader. > I'm unsure why this bug is being marked invalid. Because the code in question (with the so-called "bug") is the format=flowed code, and the standard *requires* the code to do exactly that, what is here called a "bug". If we "fixed" this, we'd have the opposite bug, that spaces at the beginning of lines are missing when read in f=f supporting mailers, i.e. in those cases where the message is interpreted correctly.

So educate me here. Despite having looked over the f=f spec a copuple of times, there are some basic things I don't understand. Is there a way for a receiving MUA to know if the sending MUA used f=f? From what it sounds like so far, f=f and non f=f MUA's are fundamentally incompatible. They can't be smart about each other even if they tried, right?

> Is there a way for a receiving MUA to know if the sending MUA used f=f? Yes, content-type header. > From what it sounds like so far, f=f and non f=f MUA's are fundamentally > incompatible. They can't be smart about each other even if they tried, right? Not sure what exactly you are asking. The whole idea about format=flowed is to be compatible with clients which are not aware of f=f, this is its "feature" in comparison to text/enriched or HTML. It tries very, very hard to make the message appear well in such clients, but there are a few minor differences here and there, e.g. the space at the end of lines and this "bug", yes. In other words, f=f already tries very hard to be "compatible" by being "smart". What you can do is to work around this problem, minimize the visibility. For example, you could detect, if the current block of text (no empty lines) consists of single lines with less than 80 chars each and any of these lines starts with a space, in that case space-stuff them all (but only that block). (I think we don't have the problem with bulleted lists generated by Mozilla, because these will all start with a space anyways, even those with bullets.)

Ben renamed this bug and I restored the name. He opened another bug 204478 but I choose to continue the discussion here, in the original bug. Ben has closed this bug marking it as invalid - but I think it is a valid bug which should be re-opened and discussed. If it is decided that the default sending of "format=flowed" should remain on, then this bug should be marked as a WONTFIX - because this will mean the continuation of problems for users. I don't know how to re-open a bug and I would appreciate someone else doing this. Ben has repeatedly tried to suppress my contributions to this debate and in so doing has sought to inhibit the contributions of anyone else who disagrees with him about "format=flowed". I am perplexed and annoyed at the effort I and others have to go to in order to state and defend what seems obvious: that extra complexity, or anything which changes messages in any way, should not be turned on by default unless there are extraordinarily strong reasons for it. As far as I can tell, the only practical benefit of "format=flowed" sending accrues to the recipient who is using a cell-phone or PDA which has a screen narrower than 72 to 80 characters. This is a moderate usability and aesthetic benefit. These systems already cope with non-format=flowed plaintext emails - the *only* problem is that they are displayed with awkward line-breaks. I regard "format=flowed"'s ability to display and print ~72 character line messages with even longer lines as not a benefit in general at all - as seen by the bug in which people request this not happen, or that there be a narrower margin than whatever their screen or page allows. So it seems to me that "format=flowed" provides modest aesthetic and usability benefits for those with cell-phones and PDAs. There may be other benefits I can't imagine too. But the cost of having sending with "format=flowed" on by default seems totally unacceptable - the alteration of large numbers of plaintext messages in a variety of recipient and display context as previously discussed, with some new ones discussed below, such as breaking digital signatures. The whole thing revolves around the merits of sending with "format=flowed" and whether this should be on by default. This bug would be entirely resolved if the default was to Off. I suggest it be Off by default, with "format=flowed" reading On by default, and with both having a well explained user interface option to change them. This is a bug from the point of view of the user who sends a message from Mozilla to any MUA which does not handle "format=flowed" messages correctly. (This includes Mozilla in the past, because a year ago reading or re-editing its own messages would show the extra space had not been removed.) The fact that the problem is caused by compliance with a technical standard is irrelevant to the fact that from the user point of view, this is a bug. Arguments aimed at minimising our perception of how serious this problem is should not be taken too seriously. The user has a right to consider this as serious a problem as they like. For me, this extra space has been the reason I have not used Mozilla for email (and therefore most browsing) until now. (I am about to switch to N7.02.) There are line length limits in SMTP - I recall the limit is just under 1024 characters. This is only a practical problem for broken programs such as Novel Groupwise which I have observed sending paragraphs as extremely long lines in plaintext emails. Most servers cope with such lines, but I have seen one instance where the paragraph was truncated by a server. I think the line-length limit is very rarely a problem for anyone. I am not sure that it is impossible (as suggested above) to send a plain text email with a single period on the line. I have never noticed this and I just tried with N4.77, Mozilla 2003050211 and Postfix and it worked fine. The whole idea of any well designed technical standard is to preserve backward compatibility and as much simplicity as possible. Mandatory mandatory sacrifices to these goals should only be made when there is no alternative and the benefits are considered to be well worth it. I never read the full text of the "format=flowed" RFC. http://www.rfc-editor.org/rfc/rfc2646.txt It never occurred to me that an Internet standards track RFC would propose a system which altered the content of plain text emails when not read by a compliant MUA. In general, Internet standards are dedicated to backwards compatibility and avoiding any extra complications or data loss. This RFC is a glaring exception. In a year, it never occurred to other people who were watching this bug either - apart from Burpmaster. It seems that Ben did not always realise that sending with "format=flowed" involves changes to the messages - such as these additional spaces. In: http://bugzilla.mozilla.org/show_bug.cgi?id=86607#c11 in July 2001, he wrote: > Unless you can come up with a common case, where f=f messes things > up and which cannot be fixed, you'll have a hard time to argue a > UI option to disable f=f sending with me. The current bug of extra spaces being added to indented lines is a showstopper for some or many people. It alters many messages. Ben, isn't this exactly the sort of common "messing up" of messages which makes it mandatory to have a user option for sending with "format=flowed"? With that option, and with the knowledge that "format=flowed" causes such problems for many users that they cannot user Mozilla, doesn't this constitute all the arguments anyone could need to make it Off by default? There are many potential problems in adding or deleting spaces. We cannot anticipate them all so we need to be very careful in trusting our necessarily limited understanding of the problems when deciding to implement any "feature" which causes such changes. Since some examples have been discussed, here is my take on them, with some more examples. Including patches in the body of plain text emails is an important requirement in mailing lists and their web archives because the use of HTML or attachments causes far too many problems with: 1 - Archiving and searching of the archives. 2 - Problems with handling HTML emails and attachments in digests and probably various other problems, such as limiting the size of attachments, preventing virus and spam attachments in mailing lists etc. ASCII diagrams and art, including in signatures will generally be rendered completely unintelligible by these extra spaces. Since a diagram will typically rest against the left margin, some lines will be indented by one space. There are a potentially endless number of reasons why people want to use plain text email and not have any changes made to it, such as additional spaces on some or all lines. Another problem is writing a table of figures and wanting the table to be intact, including lines full of spaces, when sending it and when saving it as a draft. If I write the following, with "\" meaning newline: 100.1 30.3 20.7 13.4 10.3 \ \ 99.2 20.1 17.4 9.0 7.8 \ and I save it or send it to someone, I want that blank line of spaces to be preserved, because it makes it easier to move the cursor about and so easier to edit the table than if the line is truncated to a single newline. As I understand it, the way "format=flowed" works is that the text will sent, or saved to draft, with the line full of spaces turned into a newline (or maybe it has just one space?) and with an extra space in the lines which were indented: 100.1 30.3 20.7 13.4 10.3 \ \ 99.2 20.1 17.4 9.0 7.8 \ This is what will be received by any non "format=flowed" MUA. The table is screwed up and the line of spaces is gone. It is not good enough to justify alteration of messages like this on the basis that soon everyone will be using an MUA which properly interprets "format=flowed". Firstly, this cannot be assured. Secondly, we would need extraordinarily strong arguments to justify the mangling of peoples messages like this. Thirdly the messages are often used by systems which definitely do not have a "format=flowed" interpreter - such as mailing list digests and Web archives. An MUA which properly handles "format=flowed" will interpret this as: 100.1 30.3 20.7 13.4 10.3 \ \ 99.2 20.1 17.4 9.0 7.8 \ but this still involves changes to the sender's message. None of us are entitled to say that this is insignificant from the user's point of view, since we don't know all the users and we cannot reliably estimate the negative impact such changes have on them. Here are some other potential problems. Changing the number of bytes in the message, such as from a copy and paste composition from another document, may cause a the sender to wonder whether some important, readable part of the message has been lost. On longer messages, such as one involving a large table such as the above, they may have no other way of proving that nothing has been lost than by manually comparing the original and the version which has been mailed. What about digital signatures? I think the algorithm for collecting text to produce the hash which will be signed is designed to be insensitive to trailing whitespace - but this is just my recollection. How many signing algorithms are there? How can we be sure that removing spaces from the message will not cause the signature verification to fail? Surely any digital signature system will detect the addition of spaces. There would be something wrong with them if they didn't. So it seems that sending a signed message with one or more indented lines with "format=flowed" will cause the signature verification process to fail on any recipient MUA which does not properly reverse the "format=flowed" space stuffing. Likewise if the signed message appears with its extra spaces in a mailing list, or Web archive, etc. then this will break the ability to verify the digital signature. Just because one or more people in this discussion thinks that adding spaces, or deleting lines of them, is unimportant does not justify us making a widely-used program like Mozilla which, by default, changes hundred of millions or billions of messages in this way. We simply cannot predict the consequences of such changes and so we should make the default state of the program be not to change the messages in any such way. Ben has repeatedly tried to suppress me and therefore others from discussing the merits of "format=flowed" and whether it should be on by default. I view the purpose of bug reporting and discussion to be a vital way of discerning the needs and desires of users. To short-circuit this process by ruling that a problem for users is either unimportant or not a bug because the problem is caused by a recognised technical standard - which is what Ben has done here - is to thwart the proper process of bug reporting and discussion. Open source projects are supposedly better because of all bugs being shallow if there are enough eyeballs. Ben's attempt to stop discussion of whether "format=flowed" should be on by default etc. has the effect of inhibiting vital input from users. If developers decide that users needs and desires are unimportant, by marginilising or denying their needs or opinions based on their own (the developers') opinions, then the whole process has failed. The result of this would be important programs such as Mozilla becoming less useful to people who want or need basic, uncomplicated, transparency in their communications. This, I attest, includes most user expert and newbies alike. The problems with extra spaces and in deletion of spaces in a line which ends with multiple spaces, will *always* be a problem for some users. These changes to messages will occur whenever a message is sent by an MUA operating with "format=flowed" turned on, and when the recipient MUA, or any other place the message appears, such as mailing lists, Web pages etc., Usenet clients and web archives etc. does not properly interpret "format=flowed" messages. The designers of "format=flowed" clearly recognised that this was a cost of their new tech standard, but wrote - falsely I think - that this was not much of a problem. So we face a clear choice: 1 - Leave things as they are: Sending "format=flowed" on by default. Many messages will arrive in an altered form as described above and this will frustrate and disrupt the communications of many users all over the world. This will have the secondary effect of less people using Mozilla and/or them thinking less of open-source software in general, because we, in this open-source development project, have put narrow technical viewpoints above the basic needs of users for transparent, simple-as-possible, email communications. At best, if there was a user interface option to turn off "format=flowed" sending, then the user would be able to turn it off, but this would require them to: a - Realise that their messages are being changed - which the won't necessarily know, because they don't see what the recipient sees, and they may not be able to anticipate where their message will appear, such as in a mailing list archive. b - Somehow realising that these erroneous additions and deletions are caused by "format=flowed" being on. (They probably have no idea what "format=flowed" is and just see the spaces as a bug. We, on this bug 141983 spent a *year* thinking this was a problem with the serializer or whatever. None of us, apart from Burpmaster, had a clue it was related to "format=flowed".) c - Finding the correct user option and turning off "format=flowed" for sending. 2 - Changing the defaults for "format=flowed" to be: sending OFF and: a - reading ON (Subject to objections about how it is more trouble than it is worth on many screens, especially wide ones, and when printing on paper - so we need one or two settable margins for those situations.) or b - reading OFF and providing well-explained user interface options for changing them both. Ben, and others, what would be wrong with 2 above? I anticipate that you would argue that the default off means that many messages by default will not be sent "format=flowed" and will therefore be more difficult to read with narrow-screen devices - cell-phones and PDAs. What is wrong with my suggestion that there be address-specific options for sending with "format=flowed" when sending to people who are likely to read the message on a cell-phone etc.? These cell-phones already can read non "format=flowed" plain text messages. Its a bit rough, but it works. "Format=flowed" provides an aesthetic improvement and some extra ease of use in such situations. That's all it does, in my view, since its other use of expanding a 72 character wide message to even longer lines is rarely welcomed by users, since longer lines are harder to read. If you object to my proposal 2, then it seems to me that you are saying it is worth introducing (or rather retaining) this complexifying change to Mozilla which will functionally and aesthetically alter a very large number of plain-text messages in many situations where the user will be disadvantaged, in order to achieve limited aesthetic and usability benefits for people with cell-phones and PDAs. I don't think this is at all justifiable. Ben wrote to me a continuation of his argument that plain-text emails already suffer degradation with the conversion of "From " into ">From ". Since others may think the same way, here is my response: > BTW: mboxes and ">From " is used by sendmail to my knowledge, > and that's still the most often used mailserver. Yes, maybe > not for heavyload servers, but the likelyness is still high. > In any case, every message processed by Mozilla will have > ">From ", because Mozilla uses mboxes and is unlikely to > switch. I don't see such diatribes from you against that. Mbox is a lousy way to run a mailbox. It is slow (CPU and hard disk intensive), hard to lock, can only be accessed by one process at a time, can't be mounted on NFS, and it has this problem of having to irretrievably alter any line in the message starting with "From " because it uses this as a delimiter between messages. I would entirely support Mozilla using something smarter for the mailboxes it maintains on its own computer. It is not true to say that all messages sent by Mozilla suffer this quoting problem. Composing the message does not involve any local mailbox and neither does sending it. Only if you save the message in a local mailbox will this occur - for Drafts or in Sent mail - and in the latter case it is only your copy which has the changes. I have my Drafts, Sent and all my other mailboxes on an IMAP server which uses Maildir mailboxes (Courier IMAPD) - so I have no such problem. I have tested this. There is no quoting of lines starting with "From ". I don't know enough about Sendmail to know whether it stores messages in transit in Mbox mailboxes. I guess it delivers to Mbox mailboxes in its bog-standard implementation. But I imagine that a decreasing proportion messages go through Sendmail or Mbox mailboxes now. But even if such changes (">From ") were ubiquitous in email, this would not in any way justify wholesale changes to many more messages as the default on sending in "format=flowed" inevitably entails. No apologies for length. If Burpmaster has been a bit more prolix in his comment #5 above, mentioning "format=flowed" specifically when referring to its RFC, then I and others might have understood his point and read the RFC, thereby leading to this discussion occurring in April 2002 rather than now. - Robin

> Ben has repeatedly tried to suppress my contributions to this debate When you brought this to the newsgroups, I spent many, many hours to discuss this with you. I just hate it when you reopen the old debate at other, inappropriate places. I hope you can see why. I find it funny that you argue about a misplaced space here, or don't want to hit return when you actually want a linebreak, but find it totally acceptable when messages come across totally garbled on PDAs, my window (which is smaller than 80 chars wide and with proportional font, for better readability) or in quotes (comb), often to a level of being *unreadable*. This is *common*. That we have to cope with real plaintext anyways is no argument to stop progress into a better direction or to keep the majority of messages sent today broken. You find it inacceptable that f=f adds spaces and similar, claiming that it alters your message, while f=f gives the ability to reconstruct the message *exactly* as you composed it (with all spaces). OTOH, real plaintext cannot give you that in all cases (".", ">", "From "). The problem are mailers not understanding f=f. If all significant mailers did, then we'd have less problems of message alternation (in fact none, to my knowledge) than we have with real plaintext today. So, if you want to have your message seen unaltered, a good move would be to have f=f implemented in the other mailers. Yet you try to stop f=f, which could give hope to get messages across unaltered and unambiguous, with the argument that f=f alters your message for non-conforming mailers. > Ben, isn't this exactly the sort of common "messing up" of messages It's not clear yet that we can't work around it. For example, I made a suggestion above how to solve this bug. Maybe somebody else has an even better suggestion. A UI option to turn this on or off is offtopic here, that's another bug. One of the problems with it that almost noone would even understand it, much less being able to make an educated choice. > The whole idea of any well designed technical standard is to preserve > backward compatibility and as much simplicity as possible. > Mandatory mandatory sacrifices to these goals should only be made > when there is no alternative and the benefits are considered to be > well worth it. This is exactly what f=f is, IMHO. Go on, design a standard which * preserves soft linebreaks (paragraphs), also in quotes * unambiguously marks quotes * provides a way to keep "From ", "." and ">" etc. unaltered (as the user wrote them) on all those broken legacy systems still in widespread use * doesn't exceed the 80 char limit (required for other old mailers) * and whatnot and * also meets all your requirements. If you managed to do that, great job. Now also get IETF to approve it and the standard mailers to implement it, so that it's actually useful and not just a webpage. If you did, you have a right to complain about f=f. Note that real plaintext fails with many of these goals, and I think that's far more severe than this problem. > > BTW: mboxes and ">From " is used by sendmail to my knowledge, I wrote that to you per email, and I wrote you why, because I think it's offtopic here.

I just found how to re-open this bug - Ben had closed it as "RESOLVED INVALID". Ben, my understanding of your position is that rather than making sending with "format=flowed" being Off by default, with a user interface option to turn it on, that you believe that all Mozilla users should be forced to have their messages changed when there are indents and the MUA, web archive, mailing list etc. their message goes to does not know how to handle "format=flowed". (Unless they figure out what the cause is, and have a way of turning off sending with "format=flowed".) You mentioned workarounds for the problem of the extra space on indented lines, but I don't see any way of working around the problem, since the RFC indicates that an extra space in an indented line will be a natural consequence of this "format=flowed" algorithm. You talk up the problems supposedly caused by plain text emails such as lines starting with "From " (only if an Mbox is involved), "." (I think this is not true - please give an example) and now "," (likewise) as if these were significant problems for email users. But as far as I can see, the "From " one is only an occasional problem and it is not due to Internet standards at all - just a lousy mailbox implementation which is increasingly being replaced by better alternatives such as Maildir. You again try to limit discussion of whether "format=flowed" should be On by default and whether it should have a user option - but these questions are totally on-topic for this bug because this is the only ways to avoid the changing of messages from the user's point of view, since "format=flowed" inherently changes the message contents if there is one or more indented lines. I understand that your solution to these problems suffered by millions of Mozilla users and those who read their emails and Usenet messages is to change the entire world, beyond Mozilla, to make everything, including archiving programs and presumably digital signature software, compatible with "format=flowed". It seems that as part of your efforts to achieve this universal adoption, you are really keen to see Mozilla have "format=flowed" on for sending as a default and perhaps without any user interface option to turn it on. If this is the case, then I think you are forcing your ideas upon millions of Mozilla (and those who read their messages) to their cost - a cost none of us can fully anticipate and which none of us should regard as insignificant. It seems to me that your plan uses their difficulties with other people not being able to read their messages correctly (as well as the benefits to those who can) to increase pressure on the developers of other MUA software, archiving systems, digital signature software etc. to properly implement "format=flowed" in their systems. I think that the costs, for ordinary and expert users, and the people they are trying to communicate with, via email, Usenet, mailing list archives etc., of sending with "format=flowed" FAR outweigh the relatively minor benefits to those recipients who must use a narrow screen, or those very few who actively chose to read emails on a screen with less then 80 characters. If this summation of your position is correct, then I think your course of action will, if followed, cause millions of Mozilla users lots of difficulty, for very modest benefits. If "format=flowed" is so good, then people will choose to turn it on. If it had no costs, then I would support it being on by default. But it has serious costs to many users, so I think it should be off by default. Your suggestion that it is too hard to explain does not justify having it on by default - because that choice causes lots of trouble for users in a perplexing way, without any explanation or warning, when their messages are silently transformed after they write them in to a different form which the recipient will see, unless they are using "format=flowed" software. If you want to change the world to universal adoption of "format=flowed" then I suggest that you do this by evangelizing the concept and by making it a widely available, opt-in, option on Mozilla and every other item of software which is relevant. A good way to do this would be to have a web page explaining all the technical and user issues, problems and benefits, and then to have some code available for anyone to integrate into their programs. You, or whoever else was involved in making "format=flowed" on by default has prevented me and others from using Mozilla for mail for over a year. I regard that as a significant cost to users, and I think it carries a very high cost for the Mozilla project itself. It is supposedly the latest greatest browser and email client etc. But emails and Usenet messages sent with it are changed in many real-world instances. Until now, virtually no-one realised that this was a necessary cost of sending with "format=flowed". All this will be solved by making "format=flowed" sending Off by default, with a user option to turn it on. Please answer my question: what would be the problem with this? - Robin

In my comments above I suggested that Burpmaster in comment 5 (30 May 2002) had not mentioned "format=flowed". But he did mention it clearly in terms of the reader removing the space. Burpmaster, I apologise for misrepresenatating your actions. I wish I had read your comment more carefully and followed your link to the RFC - then I would have realised that this space was inserted as part of the "format=flowed" sending process. - Robin

> You talk up the problems supposedly caused by plain text emails These are not made up. They come from code actually in Mozilla and in 4.x, where we have to work around these problems, and from my own experience. I *know* about ">From " and know the background of it, yet I have been fooled many times by it, wondering where the bracket or quote comes from. I just try to point out that the non-alteration that you keep demanding no-matter-what is just not possible. In fact, this "bug" here exists only, because format=flowed tried to solve exactly *these* problems in plaintext. Why else do you think that the space-stuffing was introduced? You can have wrapping paragraphs with trailing spaces only, you don't need the space-stuffing for that. The latter was introduced purely to solve the existing problems with plaintext, the problems that you keep ignoring. > On by default and whether it should have a user option - but these > questions are totally on-topic for this bug These questions are far broader than this bug. There are many, many more factors to be considered. That's why they are separate bugs. It does not make sense to discuss them there and here, that's why you should not discuss them here. Makes sense, not? > including archiving programs These often use HTML in a proportional font. Good luck finding your spaces. > I think that the costs [...] FAR outweigh If I have the choice between an occasional additional space (because I use a non-supporting mailer), and regularily hard to read emails, then I take the space any day. > If this summation of your position is correct No, it was not. I just don't understand your course of action here. > If "format=flowed" is so good, then people will choose to turn it on. No, they will not. They won't know what it is, much less be aware of the problems I have while reading their mails. And most will not even care, they just want to send messages with a blue font and smilies (the yellow ones, not that :-) ). You're living in a different world than most users. Most of them couldn't care less about a missing or additional space. I believe they do care, though, if they can't read mails well on a cellphone. (That's why I mentioned it, not because it's the only reason.) > You, or whoever else was involved in making "format=flowed" on by > default has prevented me and others from using Mozilla for mail for > over a year. You know very well since 1-2 years that you can turn it off. [default off] > Please answer my question: what would be the problem with this? That we keep having all the problems that exist with plaintext. See my last comment "Go on, design a standard which...". See what happened? You dragged me into the discussion again. For the nth time. Same discussion we had on the newsgroup. Almost identical arguments. Just read the newsgroup archive and we can save us repeating it all again. This is a waste of time for everybody involved. *That's* why I shut this down and start to get angry when you bring it up *again*.

Ben Buksch said in Comment # 31: > I find it funny that you argue about a misplaced space here, > or don't want to hit return when you actually want a linebreak, Ben, I want to point your intention to the fact that this does not fix the user's problem using kind of formatted lines in mail/news as in my sample (see comment #11). When I want to indent a line, I will OF COURSE have to hit [Return] in the line before. But with format=flowed like it is implemented now, the problem of the added space seems to exist still as the [Return] is not evaluated, but the leading space in the following row is, according to rfc2646.txt, par. 4.1, 3). I believe that the spec of format=flowed is wrong in this paragraph, as far as lines with leading blanks are concerned. Format=flowed should use soft line breaks as described there in 4.1, checking for blanks at the very end of the given line length (no matter if you talk about defined line lengths with 79 or 72 or 42 characters). But for me, there is no reason why a line which begins with a blank should get another blank, neither in format=fixed, nor in format=flowed. Please stop to think about this. Thank you. As a Mozilla mail user, I simply want to be able and write paragraphs like I have shown in comment #11. This is what I entered ("\n" means hitting [Return], "." means I entered a blank myself at the beginning of a line): * This is an item with one text line.\n * This is an item with more than one text line, which should\n ...show three blanks in front of the 2nd and all following\n ...lines.\n * This is another item with one text line.\n This is what comes out: * This is an item with one text line.\n * This is an item with more than one text line, which should\n ...show three blanks in front of the 2nd and all following\n ...lines.\n * This is another item with one text line.\n The reason for this silly additional blank is the wrong spec of format=flowed in rfc2646.txt, par. 4.1, 3). +++

> the [Return] is not evaluated hm? Mozilla does insert a hard linebreak when you hit return, and it survives the sending/receivement, not? > But for me, there is no reason why a line which begins with a blank should get > another blank, neither in format=fixed, nor in format=flowed. Well, read my last comment. The space-stuffing (the leading space) is there to avoid the problems with ".", "From " and to make quotes unambiguous (">" vs. quote). In those cases, to minimise the visual effect of the alteration *in non-supporting mailers*, the spec suggests to add a space at the beginning of these problematic lines. To keep the message *unaltered* in supporting mailers, the reciepient mailer must consequently remove all leading spaces from lines. Now, we'd miss a space in supporting mailers, if the line originally started with a space (it wasn't inserted in the course of space-stuffing). To prevent that and to keep the message *unaltered*, lines with leading spaces must be space-stuffed while sending. So, the message ends up at the recipient *unaltered* (even in cases where common plaintext software would alter it), if the recipient mailer supports f=f. It ends up being *less altered* in combination with legacy software in cases which are traditionally problematic with real plaintext. It does come slightly more altered in cases where lines start with a space and the recipient does not support f=f, in that case, only a space will be added. Sound like a fair tradeoff for me, even based only on the merits of keeping this unaltered. Certainly nothing to completely disable f=f and throw away all the other advantages.

BTW: I just read that the spec does not mandate (MUST) but recommend (SHOULD) space-stuffing lines. However, because the recipient mailer will remove leading spaces, we'd lose a space with f=f supporting mailers, if we don't space-stuff lines starting with a space. So, we practically have to do it anyways, if we want to keep things as verbatim as possible.

Ben wrote, in part: > > On by default and whether it should have a user option - but these > > questions are totally on-topic for this bug > > These questions are far broader than this bug. There are many, many > more factors to be considered. That's why they are separate bugs. > It does not make sense to discuss them there and here, that's why you > should not discuss them here. Makes sense, not? As far as I know, the other bugs do not concern the question of "format=flowed" for sending being on by default. It is entirely pertinent to this bug because making it off by default is the only solution for most users . . . as long as we assume that the current implementation is the only correct way of sending "format=flowed". > > including archiving programs > > These often use HTML in a proportional font. Good luck finding > your spaces. Yes some use proportional fonts and I think its a pain. But those problems are no justification for creating problems for millions of users with "format=flowed" for sending being on by default. > > You, or whoever else was involved in making "format=flowed" on by > > default has prevented me and others from using Mozilla for mail for > > over a year. > > You know very well since 1-2 years that you can turn it off. I had no reason to turn it off, because I had no idea that these extra spaces had anything to do with "format=flowed". In this bug, for a year, until yesterday, judging by what people wrote, only Burpmaster understood the cause of these spaces. I have not had a single problem with plaintext ">From " in years, but I recognise it would be an occurrence for people with Sent and other mailboxes on the local machine. In the years when I had such problems in this way, and with an IMAP server which used Mbox, it was only something which I noticed once every few months. It never caused me any trouble, though I puzzled over it until I found out its cause. It never made an email difficult to read and this single, isolated, change to the message was no actual trouble to me, though it would have broken a digital signature. I have never seen any problem with "." or "," as you have mentioned. What are these problems? I do not accept that the average user's experience with plain text email limitations - which are solely, in my experience the ">From " and the ~1000 character line limit - are serious or any justification for introducing the very common and sometimes serious changes which "format=flowed" imposes in many situations according to the current Mozilla implementation. Ulf, I haven't tried to follow your interpretation of the "format=flowed" RFC. If you can show a way of implementing the spec without adding a space to every indented line, then this is indeed a way of solving the bug. (Although I think its a pest that it chews lines of blanks.) I have not attempted to follow it - but as I have noted above, it seems that the RFC authors anticipated that there would have to be an extra space for every indented line, as I quoted in comment 22. Here it is again, with more careful analysis. An agent which is not aware of Format=Flowed will of course not undo any space-stuffing, thus Format=Flowed messages may appear with a leading === (Ben if you are looking at this with proportional spaced fonts, then it will look like garbage. I am underlining the word "may".) space on some lines (those which start with a space, As we are discussing - indented lines. ">" which is not a quote indicator, I don't understand this, but I guess the algorithm makes some decision about what is and is not a quote, which sounds dodgy to me. or "From "). I guess they are trying to save the "From " from being quoted in the event that the message is saved in an Mbox. Since lines which require space-stuffing rarely occur, and the aesthetic consequences ============ of unreversed space-stuffing are minimal, this is not expected to be a significant problem. I have not tried to understand the "format=flowed" algorithm. At present, Mozilla adds a space to every indented line. Did the RFC authors really think that indented lines are rare? If so, then I think they are wrong. But in their conception, was a line which needs to be space-stuffed only a subset of those which start with a space? If so, then the current Mozilla implementation is probably wrong. Is the RFC confusing and/or confused/faulty? One way of approaching this is to run some tests on another MUA which implements "format=flowed" when sending. Ben, you think this should be ubiquitous in all software so hopefully you can suggest some such MUAs to test - hopefully open-source ones so we can see or even copy their algorithm, as well as testing their behavior. - Robin

Ben, I cannot follow your explanations in comment #36. Could you please give a sample of HOW a paragraph like the following one will be shown in MUA's not supporting format=flowed? Sample: 1. This is a numbered paragraph, which has been typed in Mozilla Mail with a non-proportional font in non-HTML mode. Therefore, I have to hit [Return] at each single line end and have to enter three blanks at the beginning of the following lines. To my understanding of format=flowed, this paragraph should not be changed in ANY WAY, as each single line ends with a hard line break (entered by hitting [Return]. The next sample shows that indented lines should NEVER get added or removed blanks at their beginning: My Englisch is not perfect. ^ The pointer to the wrong 'c' would point to the following 'h' like it did for a very long time now in Mozilla. I claim that this behaviour is a bug. My Englisch is not perfect. ^ If you stick to the RFC, then MUA's supporting format=flowed should even remove ALL leading blanks in these lines, resulting in: My Englisch is not perfect. ^ btw: As "f=f" could either mean "format=flowed" OR "format=fixed", this abbreviation is rather useless, isn't it. ;-)

Re Ulf's 1st example in comment 39, in a supporting reader like mozilla, the para will be shown correctly: 1. This is a numbered paragraph, which has been typed in Mozilla Mail with a non-proportional font in non-HTML mode. Therefore, I have to hit [Return] at each single line end and have to enter three blanks at the beginning of the following lines. in an MUA that doesn't support f=flowed, there will be an extra space: 1. This is a numbered paragraph, which has been typed in Mozilla Mail with a non-proportional font in non-HTML mode. Therefore, I have to hit [Return] at each single line end and have to enter three blanks at the beginning of the following lines. For the 2nd example, again, in a supporting reader, the line will look correct: My Englisch is not perfect. ^ and, again, a space will be prepended in a non-supporting MUA: My Englisch is not perfect. ^

Calum: As far as I can see, you state that I am right: The additional blank does absolutely make NO SENSE. It is added by Mozilla and makes trouble in a MUA not supporting format=flowed. If Mozilla would NOT add the blank, everything would be fine for both kind of MUA's, right?

Here is (seen in a newsgroup right now) another bull****, which has its reason in the silly space-stuffing method of format=flowed: >>2. Calamus mit dem serienmäßigen Modul Speedline. Über einen > ^^^^^^^^^ > Wer denkt sich solche Produktnamen aus? Bei welchen Gelegenheiten? Here you see a portion of quoted text. The first quoter added a blank-leaded line which wants to point to the word 'Speedline'. He than added more text: 'Wer denkt sich ...'. In Mozilla's newsreader, this stuff now looks like: | | 2. Calamus mit dem serienmäßigen Modul Speedline. Über einen | ^^^^^^^^^ | Wer denkt sich solche Produktnamen aus? Bei welchen Gelegenheiten? I am sorry, but this is freaky, whatever the mentioned rfc may want to define. PS: Ceterum censeo ...

Eudora 5.2.1's "format=flowed" behavior is identical to Mozilla's. The message I wrote: 000 111 222 333 is sent as: 000 111 222 333 This makes me think that there is nothing wrong with Mozilla's implementation of "format=flowed". Unless someone can show that "format=flowed" can be done in a way which does not introduce these extra spaces (which seems impossible to me, since without these spaces the MUA which interprets the message with "format=flowed" will produce the wrong result), then the question is whether sending with "format=flowed" should have a user interface option to control it and whether it should be on by default. - Robin

An RFE to introduce per-recipient control of f=flowed would be nice, so I can select the (very few) people I email using e.g. mailx (don't laugh), to receive f=fixed. As an aside: how many popular mail readers do not support f=flowed?

Ulf Dunkel wrote: > should not be changed in ANY WAY, as each single line ends with a hard > line break Again, hard line breaks have nothing to do with this bug, apart that they appear in the same spec. [Example for indention, already answered by Calum Mackay] Note that this spacing appears garbled on my screen *anyways*, because I use proportional fonts and refuse to use fixed-width fonts, because they are so hard to read. Your example (without format=flowed) looks on my screen like: 1. This is a numbered paragraph, which has been typed in Mozilla Mail with a non-proportional font in non-HTML > If you stick to the RFC, then MUA's supporting format=flowed should even > remove ALL leading blanks in these lines No, it doesn't say that, and Mozilla doesn't do that. > If Mozilla would NOT add the blank, everything would > be fine for both kind of MUA's, right? No, please read what I wrote in comment 36. > In Mozilla's newsreader, this stuff now looks like: huh? If the sender used f=f and you use Mozilla's newsreader, then the message should appear exactly as sent. Of course, if you disabled flowed supported for display, it won't anymore, but don't blame us for that. Robin wrote: > I guess the algorithm makes some decision about what is and is not a quote, > which sounds dodgy to me. What is dodgy about it? Mozilla *knows* what is a quote, because it created the quote itself. Easier with the HTML composer. > As far as I know, the other bugs do not concern the question of > "format=flowed" for sending being on by default. It is entirely > pertinent to this bug What? It was *you* who argued in the following places, that f=f should be off by default (and I have most likely missed some): Bug 168420, Bug 86607, Bug 88986, Bug 71110, this bug, the .mail-news newsgroup, the .editor newsgroup. Again, if you want discussion about the merits and problems of f=f, or if it should be on or off by default, read the newsgroup archive! It's all there. I have no time to repeat it all here. > I had no reason to turn it off Apart from the fact that you argue for years that you don't want it and we shouldn't give it to others as well.

Sorry, guys, but could someone please EXPLAIN the sense of the added blank in flowed format to me?

Format=flowed is a system for making plain text messages display at various widths wider and narrower than the one they were written in, together with supposedly proper handling of quotes etc. It also, apparently, is intended to overcome a problem with some limitations of plain text emails, such as how a message with a line starting with "From " has to have this changed to ">From " if it is ever to be stored in an Mbox mailbox. (This is a relatively rare and generally minor problem. The other problems Ben mentions to do with lines starting with a "." or "," are not problems as far as I am aware, and no-one here has given reason here to believe they are real.) I haven't read how it works. The spec is here: http://www.rfc-editor.org/rfc/rfc2646.txt The extra space at the start of lines which already start with a space is part of how "format=flowed" is sent. A properly programmed receiving system (such as an MUA = email client, but also, necessarily, email archivers for web sites etc. and digital signature checking software) is supposed to remove the extra space before the user sees it. The problem is that for any system which doesn't have this algorithm, we have an extra space in every indented line. Only a day or so ago did we (apart from Burpmaster) realise that this one-year-old bug was caused by the "format=flowed" send algorithm. My test with Eudora, and what is said in the RFC (see my previous comments for analysis) make me convinced that this addition of a space to each indented line is both intentional and absolutely necessary for the sending "format=flowed" algorithm to do its job. This RFC is indeed an Internet standard: http://www.rfc-editor.org/rfcxx00.html It is listed in about the middle as: The Text/Plain Format Parameter RFC 2646 I see it has only one author - from Qualcomm, in August 1999. Qualcomm is the company who produces Eudora - which is a long- established, but I think unremarkable, Windows and Mac based email client. I think it is a truly lousy Internet standard. I always thought that Internet standards were really well thought out regarding elegance, not upsetting things, not altering messages unless there was no alternative and some damn-good reasons etc. But this one mangles any message with an indented line unless it is received by a matching "format=flowed" interpreter. As far as I can see, the benefits are very minor indeed. Ben, apart from Mozilla and Eudora, which other MUAs, including Webmail programs (which are extremely widely used and generally send plain text) implement "format=flowed" when sending? - Robin

So this is the bug which has been silently screwing up my patches and ascii-art diagrams! It seems to me like format=flowed solves a non-problem, at the cost of breaking things for everyone else on the planet. Apart from Moz and Eudora, what other MUAs support format=flowed? Surely the only sane default should be to *not* silently corrupt messages? If the developers are averse to another UI option, perhaps disable format=flowed for plaintext messages, but leave it on for HTML email (where it shouldn't matter).

I think there's another sympton of this bug. Personally I like to use the "format=flowed" type but when saving the message to Draft and continuing latter some kind of <CR> of <LF> is added to the end of each line and the "flowed effect" is lost in the saved text. I hope to see these problems solved on Mozilla's next versions. Regards, Manuel

I think there's another sympton of this bug. Personally I like to use the "format=flowed" type but when saving the message to Draft and continuing latter some kind of <CR> of <LF> is added to the end of each line and the "flowed effect" is lost in the saved text. I hope to see these problems solved on Mozilla's next versions. Regards, Manuel

Manuel, yes, I guess you're using the plaintext composer and it's wrongly saving as test/plain and not f=f, or something like that. But it would be a different bug (not this one). IIRC, it's even filed already.

Ben, in Comment 45 you wrote: > Note that this spacing appears garbled on my screen *anyways*, > because I use proportional fonts and refuse to use fixed-width > fonts, because they are so hard to read. Your personal preferences for fonts etc. do not seem relevant to the question of preventing messages from having a space added to the start of indented lines. What I and other people want is for Mozilla, by default, to send emails so they are not in altered in any way - at least in the location of all printable characters - from what I see in the plain-text composer before sending it The HTML composer is not relevant to this discussion because no-one who want to send emails without visible degradation would use it. We have now established now - contrary to what you and others wrote and believed until a few days ago - that sending with "format=flowed" does alter the message, in ways more than "just" trailing spaces. All that we should be debating here is whether sending with "format=flowed" should be the default behavior and whether there should be a user option for it - because sending without "format=flowed" is the only way of fixing this bug. Instead, you mention what some text looks like with proportional fonts - which is not the question. > > I guess the algorithm makes some decision about what is and is not > > a quote, which sounds dodgy to me. > > What is dodgy about it? Mozilla *knows* what is a quote, because it > created the quote itself. Easier with the HTML composer. In plain-text emails (without any "features" or "enhancements"), the task is to convey the visible characters, whitespace and newlines exactly to the recipient in both a technical sense (byte-for-byte) and in an appearance sense - so the pattern of characters and spaces the recipient sees is precisely what the sender saw when they wrote it. In this sense it is irrelevant whether the newlines were manually or automatically inserted. In this context, there is no such thing as a "quote". There may be ">" characters in various positions which humans may have intended to represent a "quote", or which they may see as representing such quotes even if no human ever intended it, but that is absolutely irrelevant to the task of plain-text email. If you have a feature such as "format=flowed" at the sending or receiving end, or the vertical bar feature (bug 88986) then it is necessary for the algorithms to decide where there are "quotes". If you force any such algorithm on users, by it being on by default or hard or impossible to turn off, and assuming that the algorithm does something to the message in terms of bytes and/or appearance, then your algorithm will change the message according to the decisions it makes about what is and what is not a "quote". The algorithm cannot know what the human thinks are quotes. I have seen all sorts of pesky or at least non ">" quoting styles, such as: RW: blah blah RW: blah blah blah BB: blah blah the horrible AOL quotes: << Blah blah blah . . . . . blah blah blah >> and many more quote characters and styles I can't remember. I wrote that the algorithm is dodgy because it can't reliably determine what the user intended to be quotes and because it cannot tell when a ">" near the left margin is not intended by the human to be a quote. The only exception to this that I can think of - where Mozilla really does "know" what a quote is - would be when the user selects a line starting with a ">" and uses the "Rewrap" function. That shows that the user (assuming they know what they are doing) sees this line of text as a line of words and spaces, not as part of a table or diagram. Then, I think we are entitled to believe that the user interprets the ">" at the start as a quote. > > As far as I know, the other bugs do not concern the question of > > "format=flowed" for sending being on by default. It is entirely > > pertinent to this bug > > What? It was *you* who argued in the following places, that f=f > should be off by default (and I have most likely missed some): Yes, I was wrong to state that the default state of sending "format=flowed" is not relevant to other bugs. > Bug 168420, This is "We need Format=Flowed Evangelization". Clearly the question is relevant there. That "bug" is not in the software, but (purportedly) in the minds of people who through (purported) ignorance, or lack of evangelization, do not have a (purportedly) correct positive view of "format=flowed". I reckon people would have a perfectly positive experience of "format=flowed" if it was off by default with a well explained user option for turning it on. If the evangelization is false, such as the MiniFAQ's statement (about to be changed, I think) that it does not affect the message (except trailing spaces), then I think that people will have a less than positive experience with "format=flowed". A subset of people would be perfectly happy with "format=flowed" if you didn't force it on them. Instead, by forcing it on everyone, you make some people happy and you interfere with the communications of many other people. You keep avoiding this crucial point - your code in its currently default on state is stuffing up the emails and Usenet postings of thousands or millions of Mozilla users. Instead, you tend to argue that the changes should be unimportant to those people. In doing so, I think you fail to show proper respect to the needs and desires of thousands or millions of Mozilla users. > Bug 86607 This is "[RFE] UI option for disabling format=flowed". Clearly the default state is relevant to this question - except to someone who was keen to narrow the debate so as to exclude views contrary to their own. Now we know how destructive sending with "format=flowed" is, I and other people think it should be off by default, in which case this bug 86607 is irrelevant. (Of course I also argue for a user option to turn it on.) > Bug 88986 That is "Use > instead of grey bar for format=flowed quotes, optionally" I discussed it there because I see commonality between that bug, sending with "format=flowed", receiving "format=flowed" and graphic smilies - in terms of all these "features" altering the message physically or visually and in terms of them all being currently on by default. I see all these problems as resulting from a failure to exercise the basic, sensible, engineering principle of keeping things simple and adding complexity which has only marginal and occasional benefits in a way that the complexity is off by default. > Bug 71110 "need pref to set message view pane width separate from window width" Again, it is relevant. You, I think, caused most of the trouble by forcing both sending and receiving with "format=flowed" on by default. If sending with "format=flowed" was off by default, then the only "format=flowed" messages received, and therefore creating the potential problem of having lines too long, would be those sent by people who were well informed and who had made a conscious decision to request that their message have its line-lengths changed according to the recipient's display conditions. If receiving with "format=flowed" was off by default, then no-one but those who wanted to use it would be concerned about the window being too wide and therefore wanting to set something narrower. A display width setting would still be needed, but the problem of the too-long lines would be restricted to those who actively turned on reading with "format=flowed", whereas with the current situation you are forcing the problem on all Mozilla users. By making it on by default, you have caused lots of people to generate messed up messages to lasting forums with thousands of readers, such as newsgroups and mailing lists, where the readers are not using a "format=flowed" decoding algorithm and just see sloppy layout. The reader would generally figure the sender didn't care, or was not fully aware of what they were sending. Its not the senders' or the readers' fault that messages are being garbled, its your fault or the fault of whoever decided to have this stuff on by default. > this bug, the .mail-news newsgroup, the .editor newsgroup. Yes. So many problems would cease to exist if it was off by default. > Again, if you want discussion about the merits and problems of f=f, > or if it should be on or off by default, read the newsgroup archive! > It's all there. I have no time to repeat it all here. No - it is not all there. That previous discussion was based on the notion that it did not alter the text as seen by non-compliant destinations for the message. Now we know different. The only solution to this bug for most users (those who don't both know about it and can use user.js) is for sending with "format=flowed" to be off by default. There are other arguments for having it off by default, which have been debated in the past. But in this bug there is an incontrovertible new argument: As long as it is on by default, most users sending plain text messages will have their messages changed (assuming they have one or more indented lines - but read on, its worse than that) whenever their message is read verbatim, rather than interpreted by a "format=flowed" algorithm. The only way to stop the unaware user from having their messages corrupted like this is to make sending "format=flowed" off by default. Ben, you keep avoiding the question. What would be wrong with this? This is a new - and I think much more forceful - argument for it being off by default than any which were raised before a few days ago. Here's another one, which arguably should have its own bug: "Format=flowed" doesn't just add spaces in front of indented lines, it adds them in front of any line starting with a ">". This has already been mentioned. Since this is found in a very large number of messages, probably more than those with an indented line, it is a further powerful - incontrovertible I think - argument for sending with "format=flowed" being Off by default. Please answer the question Ben, since this is your code: What benefits are there to having it on by default which justify the widespread corruption of messages which have lines starting with a space or a ">"? > > I had no reason to turn it off > > Apart from the fact that you argue for years that you don't want > it and we shouldn't give it to others as well. This is a misrepresention of my position. I never argued that it should not be available - only that it should be off by default. (Actually, in an ideal world, it would be off be default for sending in all MUAs and then I think it would be OK to have it on by default in an MUA, because it would then be reasonable, or more reasonable, to think that the sender knew the consequences and made an informed positive choice to send in this format.) I had no reason to turn off "format=flowed" for either sending or receiving because I was not using Mozilla at all. The reason I was not using it, at least a year ago, was this damn bug of spaces being added. I had absolutely no idea that the cause was sending "format=flowed" until 5 May this year 4 days ago. This is in part due to your own ignorance of the workings of sending with "format=flowed". You and probably others stated that it had no visible impact on messages. I believed you. But you were wrong - and you wrote the code! I wish I had paid more attention to Burpmaster in Commment #5. - Robin

Robin, could you _please_ take this to the newsgroups, where it belongs?

Robin, I have no time to even read, much less respond, to 10+ page comments in several bugs, esp when the discussion and the arguments are old and it's all in the archives on the .mailnews newsgroup. Anybody interested in that discussion is welcome to crawl through that and maybe even respond to it, but I guess that's no harder than to crawl through endless bugzilla discussions (even several of those). I nevertheless hope that somebody can look over all this discussion and implement the workaround I mentioned in comment 29, or we'll be forever stuck with people claiming that Mozilla and f=f mess up their messages. Maybe even I implement one day.

> look over all this discussion s/look/skip/

Great attitude, that. I don't think it is good example for Moz developers to denigrate the bug reports of anyone who takes the time to articulate their position, just because said developers disagree. I wouldn't even have known that it was f=f that was screwing up my emails if it weren't for rw's bug report. Right now, f=f breaks email for a good number of mozilla users and violates the "prinicple of least surprise". I don't think that the RFC allows it to be fixed so that it doesn't corrupt email, so the only sane default seems to be off.

Ben, this bug concerns many or most messages being corrupted for all Mozilla users sending plain-text messages to systems which do not have a "format=flowed" decoder - unless the user is wise to the problem and technically sophisticated enough to fix it with a user.js. The only way of fixing this one-year old dataloss bug for the great majority of users - and so the only way of ending the burden the corruption places on their recipients (and the damage this does to the Mozilla project) is to make sending with "format=flowed" off by default, with a UI option to turn it on. Assuming you still insist that sending with "format=flowed" be on by default, please answer my question fully: Why do you think that the benefits which accrue from sending with "format=flowed" on by default, over and above those which accrue from it being off by default, justify the large-scale corruption of messages which results from it being on by default? I added bug 168420 "We need Format=Flowed Evangelization" to the list of bugs this one blocks. - Robin

> The only way of fixing this one-year old dataloss bug > [...] is to make sending with "format=flowed" off by default Wrong. See above. And IMHO, it would be dataloss to send real plaintext. We'd lose information about soft linebreaks and exact information what is a quote. IMHO more valuable than your space.

I'm pretty sure that MSN, Outlook, and Outlook Express support f=f (though I don't have these products so I can't check). I know Mozilla, NS, Eudora, and (MacOS X) Mail also do. What products are people using that do not suport f=f?

pine, mutt, emacs, netscape 4.x, kmail, sylpheed, you name it.

Quick test of the linux command-line mailers: Sent myself a flowed message composed in the plaintext editor, received on a debian woody machine, containing a few long-line paragraphs and some lists with second lines indented, e.g. 1. First item more of the first item Mutt 1.3.28i: wrapped lines as sent (72 columns, not screen width) but understood flowed: the list items lined up correctly. The non-flowing wrapping may be due to a personal mutt setting somewhere (I haven't looked because the current behavior turns out to be exactly what I want. :-) Pine 4.44, rmail in GNU Emacs 20.7.2, and rmail in xemacs 21.4: ignored flowed, wrapped lines as sent, second line of list items were one space off as described in this bug. (I don't use these regularly, so they're using the default debian settings.) What about common webmail clients, and mailing list archiver engines like mailman? Do they understand f=f? It would be nice to have things lined up on postings that are archived and available to search engines. How about the native AOL mailer?

Andrew: realize that these "you name it" products may be used by a very small portion of our userbase. It is a recurring theme in mozilla development that there is a tension between what the geeks and power-sers want, and what the typical user wants. Having said that it would be great to find a solution that worked for everyone. I don't know if that's possible here. I wonder if anyone has any hard data on what the population of recipient MUA's really looks like for mozilla senders who send in plaintext?

I fail to understand what all the fuss is about. Just stick a "don't screw up my email^W^W^W^W^Wdisable flowed format" checkbox in there and be done with it. But given that it can be turned off in prefs.js that's OK for me. As soon as mozilla gets the ability to use an external editor I might start using it again ;)

mailman 2.1 archives don't grok f=flowed. indented lines get an extra space.

akpm: the situation is nowhere as simple as you think. Seehttp://ometer.com/free-software-ui.html for one good exposition. jfrancis: you have a reasonable point, *except* that I believe we are talking specifically about *plaintext* mail. Normal users (those who do not care about sending patches) are unlikely to: o not use text/html emails o send patches o care much about badly wrapped quoting Therefore, whilst your general point is indeed important, I believe this case is one where mozilla can satisfy "geek" users without impacting the normal user (who is using HTML mail anyway). The fact is that f=f is not yet at a suitable state of interoperability in a number of mail clients that cannot be discounted out of hand.

John: Your point about typical users not using plaintext is well taken, and is why I wondered earlier what the typical target audience is for our plaintext mail senders. Meanwhile, my test message to an aol account finally worked it's way through the mail servers, and I can report that AOL 7.0 client on Windows *does not* support f=f. That's a pretty major client (unlike some of the earlier mentioned ones). I'm also wondering if my guess that MS products (MSN, Outlook, OE) support f=f is really true. Can anyone verify that? If enough major clients don't support it, then it greatly reduces the value of f=f.

When looking with Google for the amount of f=f discussions in newsgroups I saw lots of posts demanding f=f support in Outlook and OE and if I recall correctly it has been implemented at least in the latest version of one of those. If they use it only for reading or for posting too I cannot say. Someone who really uses Outlook/OE should be able to check.

An (irrelevant) question: Going from the bit of spec mentioned in comment 18 ... 1. Space-stuffing is a protection for lines that start with '>', or some other special character needing protection, right? 2. So, a line ">BLORP<" would be changed to " >BLORP<" in order to make it not appear to be a 'real' comment, right? 3. And spaces at the beginnings of lines are removed to change " >BLORP<" back into ">BLORP<", right? 4. And, since this happens, we have to pad lines beginning with a space with another one, right? Why not only apply steps 3 and 2/4 if the character immediately following the first space is '>', or one of the other special characters needing protection? -- The answer, of course, is that other f=f clients expect such a space, I guess. If the RFC were different, though... well, RFCs usually think these sorts of things through, so there's probably some other reason why that's a bad idea, anyway. I just don't see it, is all. In any case, as things are, we needs must stick with the current behavior. I'm interested in figuring out the workaround mentioned in comment 29, but have no idea where to start as far as looking through the mozilla codebase. I know a bug's not really the place to ask for pointers, but, um... well, this bug is about as blighted as it can get already!

*** Bug 275732 has been marked as a duplicate of this bug. ***

*** Bug 298485 has been marked as a duplicate of this bug. ***

Downstream bug https://bugzilla.redhat.com/show_bug.cgi?id=800891 It still affects latest thunderbird.

Windows 7 Ultimate SP1 Thunderbird/45.2.0 File user.js contains: user_pref("mailnews.display.disable_format_flowed_support", true); user_pref("mailnews.send_plaintext_flowed", false); // Disable format=flowed messages Sending a message from one of my accounts to another, I do not see this problem. Of course, that involved disabling format=flowed for both composing and receiving messages. It should be noted that there are a number of bug reports involving format=flowed. Among them are bug #448198 and bug #1051129 (which might be a duplicate of this one). It appears that a serious evaluation of the overall design and implementation of format=flowed would be appropriate.