Used 2000-02-07-08/09-M14 commercial/mozilla build: Migration does not recongize the default folders for IMAP Cyrus server From bug24715, bug23588, bug24667 & bug22349, FCC/default folders are working for Cyrus now with only "New Profile"...open this bug is tracking the problems for migration which specific for IMAP Cyrus server. Setup Info: 1)Setup the default folders from 4.7: Sent folder -- copy to sent when sending msgs Trash folder -- move to trash when deleting msgs Draft folder -- save as draft to server's draft folder Template folder -- save as template to server's template folder Steps: 1) After setup the above from 4.7. 2) Tried to do the following: Sent msgs Delete msgs Save as Draft when compose msg. Save as Template when compose msg. 3) Actual results: Msgs do not copy to Sent folder. Msgs do not move to Trash folder. Msgs do not copy to Draft folder. Msgs do not copy to Template folder. I got an error when sending msgs: "The command did not succeed. The mail server responded: Mailbox does not exist." And also after click "OK" from this error dialog, will back to compose window (compose window opened) Expected Results: Should copy msgs to the above default folders successfully without error.

Change Assign from dbragg to sspitzer, Change QA contact from gbush to me since this is IMAP Cyrus migration problem.

Karen, check out bug 26987- I had trouble migrating today if I selected from list of profiles. Automigration worked. Grace

Grace, Thanks for providing above information to me. This bug is opened specific for the IMAP Cyrus default folder of the migration problem.....for some reason, after migrate from 4.7...the account setting dialog already displayed the "Copied and Folders" for Sent, Draft, & Templates folder for the Cyrus server initially. But it just didn't implement as expected.... I need to select all the default folders again (even it already displayed initially) from the Account Setting dialog...then it will recongize the default folders under "Inbox"...Adding Account Setup & Cc: alec for this bug, too.

marking m14, I'll investigate.

Seth, don't spend too much time on it now. There are some subtle issues with this one. This invloves with namespace feature of the Imap account. Also the way we save the default folders pref in the prefs file. For example, in 4.x we can saved as "imap://test01@buggy.mcom.com/Drafts" and it will work just fine. In 5.0 we may need to saved as "imap://test01@buggy.mcom.com/INBOX/Drafts" in order to make it work. Where "INBOX/" is the personal namespace the the account. You can assign this bug to me if you wish.

moving all m14 to m15.

sliding to m16.

Triage to M18. Please let me know if this must be in beta2.

is this an IMAP interop bug? I'm not quite sure, but seems like that should be in for beta2 if it is.

yes, it's an interop bug, in a sense. It's also unclear to me if this is a migration problem, or a problem with our Cyrus support in general.

Karen - can you try this and see what happens if you were to set up a Cyrus account from scratch (ie. no migration)? Do you stil have this problem?

Today's build -- not good for migrated profile & new profiles, so I tried on 04-16-10-M16 commercial build: This bug is logged for special folders import problems from 4.0 to Netscape 6 of the IMAP Cyrus Mail account..... For new profile, user can re-define the message store on the server's special folders without problems since everything define from Netscape6. But if user imported the existing IMAP Cyrus mail account from 4.x to Netscape6, the special folders setting not been brought to Netscape6, user need to redefine again on the Netscape6 Account Setup. (ex: If users already setup on 4.x to store save as draft, save as template....etc messages on the servers' special folders. After imported to Netscape6.....even those setting display the same from Netscape6 account setup, but after try to check on servers' special folders, they are all disappear, need to redefine again on Netscape6 in order to perform the same setting & implement)

Yes. This is an interop bug with our Cyrus support in general: I remember that I talked to Jeff about this issue, the cyrus server's special folders are all under Inbox.....Netscape6 users can define well without problem if everything setup from scratch from Netscape6. But, how about the existing 4.x Cyrus IMAP mail users, it seems that there are some differences between 4.x & Netscape6 for handle Cyrus Inbox folders since those setting of special folder will be lost after imported from 4.x to Netscape6.....

Change the component from Profile Migration to Back-End. Updating the summary to make it more clear.

*** Bug 55593 has been marked as a duplicate of this bug. ***

M18 is gone, removing Target Milestone and let developers to decide the Target Milestone. Updating the current build status: Now it is displaying the error as following: "The command did not succeed. The mail server responded: Permission denied"

Karen, could you try this again in a recent Netscape daily build? This might have been fixed along with some other hierarchy delimiter problems. Thanks!

*** Bug 78265 has been marked as a duplicate of this bug. ***

No. The problem is still there......I still got an alert "The command did not succeed. The mail server responded: Permission denied" when trying to send messages from the MIGRATED Cyrus IMAP mail account...... I will suggest to fix on mozilla 0.9.2 since cyrus IMAP users definitely will see this if they use migrated profile....Ccing Scott Putterman

accepting. I did not not read about the migration part in 78265

*** Bug 91398 has been marked as a duplicate of this bug. ***

Because of bug 23625 fix (Copies & Folder point to server now), so this problem actually apply to both Migrated & New profile now....... Update the summary and the current problem as following: 1) Send Msgs: Actual Results: Msg sent but got an alert: "The current command did not succeed. The mail server responded: Permission denied" Expected Results: Shouldn't display "Permission denied" Alert when sending msgs. 2) Save as Draft: Actual Results: Cannot save as draft -> an alert displayed: "The current command did not succeed. The mail server responded: Permission denied", after select OK, it just keeps displaying "Copying Message to Drafts folder....." and never get the darf msgs to copy to Draft folder. Expected Results: Shouldn't display "Permission denied" Alert and should copy the draft msgs to Draft folder successfully. 3) Save as Template: Actual Results: Cannot save as template -> an alert displayed: "The current command did not succeed. The mail server responded: Permission denied", after select OK, it just keeps displaying "Copying Message to template folder....." and never get the template msgs to copy to Template folder. Expected Results: Shouldn't display "Permission denied" Alert and should copy the template msgs to template folder successfully. * Additional Info: Deleted msgs are now moved to Trash folder, so there is no problem for the Trash folder now.

Maybe Cavin could look at this since he knows both imap and the send/save code? Please let me know if you don't have time for this, Cavin.

OK, I'll take a look at it.

Workaround will be: After redefine the special folder (sent, draft, template) under the Inbox on the server from the Copies & folders of the account settings, then it will work without these issues....

OK, I played with the Cyrus server a bit and here is what I found from the protocol log. What I did was that I just created a new profile and set "INBOX." as the "Personal namespace" in the "Advanced IMAP Server Settings" dialog. I used the default server folders for Sent, Template and Drafts and the setting for Sent folder in prefs.js was: user_pref("mail.identity.id1.fcc_folder", "imap://test01@buggy.mcom.com/Sent"); and both Sent and Template folders exist for user test01 (I actually telnet'd to the server to verify this). It looks like we send the wrong mailbox name to the server when we try to save a copy of the (sent) msg to the Sent folder. I saw the following in the log: 10 list "" "Sent" --> server response --> 10 OK Completed 11 create "Sent" --> server response --> 11 NO Permission denied I think the IMAP command should have been: 10 list "" "INBOX.Sent" and since "INBOX.Sent" already exists the second CREATE command was not really needed. The reason why Karen's workaround works is because the Sent folder was re-configured to the one underneath the "INBOX" namespace (instead of the default one) so the prefs setting was changed to: user_pref("mail.identity.id1.fcc_folder", "imap://test01@buggy.mcom.com/INBOX/Sent"); From then on, everytime you deal with Sent folder the mailbox name sent to the server would be "INBOX.Sent" (instead of just "Sent") and this is what the server expects! If the Drafts folder has never existed and you save a msg to it then we'll have the same error because we try to do the following: 8 list "" "Drafts" --> server response --> 8 OK Completed 9 create "Drafts" --> server response --> 9 NO Permission denied Again, we're missing "INBOX." in the mailbox names! The commands should have been: 8 list "" "INBOX.Drafts" 9 create "INBOX.Drafts" I don't know what the fix will be yet but somehow we need to include "INBOX." (or whatever users enter) in the folder prefs settings for Sent, Template and Drafts when "Personal namespace:" field is changed/modified.

*** Bug 94517 has been marked as a duplicate of this bug. ***

Adding bug 26439 for this bug dependence. Cavin, Are you talking about bug 26439 which I logged long time ago?......

No, Karen. I meant that we need to include INBOX in the url when we try to operate on an imap folder. So intead of using "imap://test01@buggy.mcom.com/Sent" we should be using "imap://test01@buggy.mcom.com/INBOX/Sent" if "Personal namespace:" field is set to "INBOX.".

*** Bug 90980 has been marked as a duplicate of this bug. ***

I probably need to release notes this bug for this release..... Can I know what's this bug target milestone?

I am nominating this bug "nsbeta1" for next release since without this fix, users will see "Permission denied" alert all the time when just sending msgs, save as draft & save as template for the cyrus mail account....unless users redefine the FCC/default folders again.....

*** Bug 113525 has been marked as a duplicate of this bug. ***

Actually, having read through all the comments and having redone the workaround recently (read tonight)...just make sure you set your 'Personal Namespace' to "INBOX." and 'Other Users' to "user.". Note the periods, they are absolutely necessary. Also, in "Copies and Folders" if you explicitly choose the folders (ie, choose 'other' for each option) it works just fine. While I'm sure that this may not have been how it's supposed to work that is the ONLY way I've ever been able to get it to work with Cyrus.

In Comment #34, Galen Johnson wrote: > make sure you set your 'Personal Namespace' to "INBOX." I can confirm that in my case, it is and always has been "INBOX." (it was indeed automatically set when I setup the account, since Courier-IMAP told Moz it had to be there), but Moz appears to be ignoring that when trying to file in Sent, Drafts or Templates.

There is another bug relating to this which they claim is fixed, and I say it's not. They don't seem to be listening to me. :-( The bug ID is http://bugzilla.mozilla.org/show_bug.cgi?id=105662 Since the severity of this bug is major, what is the difficulty with this bug? Bear in mind that is't not just migration, but even with a completely new account, this has the same problem. Can't you just insert the personal namespace into the folder name? Surely that should be done for all servers, not just Courier/Cyrus servers. Warmest regards, Brian

*** Bug 115976 has been marked as a duplicate of this bug. ***

This bug is a result of an IMAP interop bug -- basically, IMAP puts no restriction on where personal folders live vs. where shared folders live in the IMAP folder hierarchy. The fix for that IMAP interop bug was RFC 2342: http://www.innosoft.com/rfc/rfc2342.html which allows an IMAP client to discover the necessary prefix for folders like "Sent" and "Drafts" so the prefix doesn't have to be manually configured by the user.

The Personal namespace for Cyrus is already correctly identified. And a UI for the user to change it if required is already provided. The problem is that this prefix is not added to the Sent/Trash folders that are set by default. Therefore new IMAP accounts on servers that use anything other than root as the personal namespace trigger this bug. The other related bug is 105385, which is also a problem with creation of IMAP accounts on these server types. Developers requiring a test account for resolving this can get one at http://fastmail.fm , which uses the Cyrus IMAP server.

*** Bug 135631 has been marked as a duplicate of this bug. ***

CC to myself.

*** Bug 112880 has been marked as a duplicate of this bug. ***

I am running into a very similiar problem (the same?) with the Mirapoint email appliance. You end up setting up the sent/draft/template folders as subfolders of the Inbox. It appears that mozilla is ignoring the user setting and is trying to create a new "Sent" folder at the top level which then generates a permission denied error. The same setup works fine with netscape 4.7. This error also occurs in Netscape 6.2.2 "release" for Solaris.

*** Bug 141729 has been marked as a duplicate of this bug. ***

After adding a user.js entry to work around 105385, by removing the trailling '/', I notice that the "Sent" folder is now created correctly. Therefore I believe that fixing 105385 will also fix this bug.

Nominating nsbeta1 since users will complain that they cannot send mail/save as draft, save as template without knowing they need to reset those under Inbox from account settings again.....

*** Bug 117917 has been marked as a duplicate of this bug. ***

Make the path that is operated namespaced. :)

Comment on attachment 84597 [details] [diff] [review] patch for review I tested this patch with a Courier server, a Cyrus server and two UW servers. But I found a mistake with the test of Cyrus server. I modified the preferences that led to a successful test result. So this patch is more suitable for bug 90494 than for this one. I obsolete now. With this patch and a workaround as followings, this bug 27002 can be solved. I'll also find a better way for this bug. Workaround: 1. Open "Mail & Newsgroup Account Settings..." 2. Select "Server Settings" of your Cyrus server 3. Click "Advanced..." 4. In the namespace area, clear the one that is just "". In my test, this is the Public(shared) namespace. 5. Clear the option "Allow server to override these namespaces"

This is the protocol log from a Cyrus server. * OK crash.fce.vutbr.cz Cyrus IMAP4 v2.0.16 server ready a OK User logged in * NAMESPACE (("INBOX." ".")) (("user." ".")) (("" ".")) a OK Completed This is the protocol log from a Courier Server. * OK Courier-IMAP ready. Copyright 1998-2001 Double Precision, Inc. See COPYING for distribution information. a OK LOGIN Ok. * NAMESPACE (("INBOX." ".")) NIL (("shared." ".")) a OK NAMESPACE completed.

Great to see the activity on this bug--thanks Henry! Please note that you can not rely on Cyrus providing any particular delimiter or namespaces--they are configurable at run-time using imapd.conf, using the new Cyrus 2.1 series. Here are the relevent lines of imapd.conf(5): ---- unixhierarchysep: no Use the UNIX separator character '/' for delimiting levels of mailbox hierar- chy. The default is to use the netnews separator character '.'. altnamespace: no Use the alternate IMAP namespace, where personal folders reside at the same level in the hierarchy as INBOX. ---- BTW, will this patch also resolve the related bug 105385 ?

Comment on attachment 84597 [details] [diff] [review] patch for review Thanks for Jeremy Howard's quick feedback and the information. I re-thinked the bug and the patch. Now, I'm sure that it is not my mistake in the test of Cyrus server. From the information related namespace, the server says it has the shared namespace "", but when handling the directory "Sent", etc, it gives out the error information. So the patch should be correct. I clear the "obsolete" flag for this patch(attachment 84597 [details] [diff] [review]). The workaround in comment #49 also should work for the enviroment above. Another thing, this patch is not for the bug 105385 and can't solve it. Sorry. :< But I'll try to solve it.

Now, mozilla may dynamically read the information such as delimiter from the IMAP server just like before. Hi, Jeremy Howard, if the patch can really solve the bug and is checked in, please help me to verify it on your server. Thanks.

Following is from the discussion between Jeremy Howard and me by mail. Jeremy Howard: Cyrus sending "" as the public namespace is not an error--this is correct; Cyrus by default treats the top level of the hierarchy as the public namespace. It is important that Mozilla can work OOTB without manual configuration with this namespace. Mulberry ( http://www.cyrusoft.com) is a good example of an IMAP client that handles IMAP NAMESPACE particularly well. Henry: As to the issue that Cyrus send "" as the public namespace, I just mean it should not my patch's mistake in the comments. You are right it's also not Cyrus' error. The real reason is that you can just set the directory without namespace type in mozilla's preference to be the 'Send to, Draft, Trash, etc' directory. Then when mozilla locate this directory, it try to match a namespace. The result is that it matches the public namespace and indeed there is not a directory named 'Sent' or something like that there and there is no permission to create such a directory. So mozilla gives out the error message retruned from the server. I'll try to file a new bug to let mozilla to set namespace type with the directory name to support namespace more perfectly. BTW: the new bug I just filed is bug 146418.

Discussed in mail news bug meeting. Decided to minus this bug.

Mozilla should use the personal namespace for special folders (Sent, Trash, etc) by default. This namespace is for a user's personal mailboxes--other namespaces are not appropriate for storing user special folders.

But with this bug you can not send mail OOTB with 2 of the 3 main open source IMAP servers (UW works, Courier and Cyrus don't). And fixing it simply requires prepending the personal namespace, which is already correctly identified by Moz, to the folder name. So why minus this bug?

I agree with Jeremy,that only personal namespaces are appropiate for sent, draft etc. folders. But remember though that IMAP does allows you to have multiple personal namespaces each of which can have their own draft, sent etc. folder theoretically. So a combobox would be nice to have which namespace to use for the account.

Good point Michael. However, such a UI is not necessary to resolve this bug--Mozilla already correctly identifies an appropriate Personal Namespace and this is shown in the existing namespace preference UI. Using this namespace will resolve this bug correctly. A UI for choosing between multiple personal namespaces is a further enhancement that deserves it's own bug. BTW, I'm not aware of any servers that support multiple personal namespaces (although that doesn't mean they do not or should not exist!)

Yes, personal namespaces are appropiate for sent, drafts, etc, folders, but we should not forbid user to use such a called directory under public namespaces. In some way, we can think that Cyrus server has a nested namespace, the personal namespace("Inbox.") is a sub namespace of the public namespace(""). So I think this patch plus a "IMAP server directory" setting or this patch plus a workaround(comment #49) is the solution for this bug. The patch is trying to find a appreciate namespace first, if it can't find one, it then tries to use the default personal namespace. This patch works well with Courier IMAP server. The "IMAP server directory" is for setting of the root directory with namespace called "". If this is set, the directories in it will be shown as root directories. This setting is not the same as personal namespace.

Since "IMAP Server Directory" and personal namespace are not the same things, they should not have the dependency relationship. So, clear the "depends on 105385". Solving bug 105385 is not the solution for this bug. On one hand, after you use "IMAP Server Directory", you'll no longer see the directories under the "" namespace. For Cyrus server, you'll no longer see the directories under the public namespace. On the other hand, they are not the same things. It's occasional that after you delete the last character '/' after the "IMAP Server Directory", you can send and save mails, etc. But they have the different concepts. We should solve this bug using namespace related methods.

The main aim of this patch is the following: The server directory is left as it is and the delimiter for the server directory is always a /. To achieve this, the following is modified: If we go under the assumption that the server directory delimiter is always a /, then nsImapUrl::AllocateCanonicalPath contains a bug: (original version, not patch) int len = PL_strlen(onlineDir); if (!PL_strncmp(onlineDir, currentPath, len)) The problem here was that onlinedir used the / as a delimiter and currentPath uses whatever the server uses. The patch converts the currentPath to a canonical form before the comparison is done instead of at the end as in the original version. If we continue the assumption that the server directory should use the / as a delimiter, then nsImapUrl::AllocateServerPath to reconstruct the server path from the canonical name also contains a problem: AddOnlineDirectoryIfNecessary which adds the server directory in front of the folder name has to be executed while the folder name is still in a canonical form. The patch just rearranges the calls a little to ensure that the conversion into canonical form is done after the AddOnlineDirectoryIfNecessary is called. I think this is a good and proper fix. I have tested this and it works properly. It does though show the same symptoms of difficulty of finding a trash folder as it did when we just forced the server subdirectory to be "INBOX." in the prefs.js or user.js. The trash issue is a general problem when using a server directory. For this I will open a new bug now.

Sorry, can an admin move this patch and the last comment from me to #105385 ? I accidentally posted it to the wrong bug.

Comment on attachment 85493 [details] [diff] [review] Change to IMAP URL obsoleting

Michael, I've obsoleted your patch. Can you attach a -uw diff to the other bug instead of this patch?

As this bug discussed namespace issues, bug #147995 may be of interest which I have just entered which discusses difficulties finding the trash folder. I believ it to be of interest to many of the same people that also have an interest in this bug. Sorry for the spam ;-)

will the patch also fix the problem that occurs when you have the pref: mail.check_all_imap_folders_for_new set to true and use a cyrus IMAP server. Then I get the following prefs set: user_pref ("mail.server.server1.namespace.personal", "\"INBOX.\",\"INBOX.\",\"INBOX.\",\"I NBOX.\",\"INBOX.\""); user_pref("mail.server.server1.namespace.public", "\"\",\"\",\"\",\"\",\"\""); I also get and "Mailbox does not exist" alert. To reproduce: 1. add mail.check_all_imap_folders_for_new pref 2. just add an account that uses an cyrus IMAP server 3. errors and incorrect prefs written to prefs.js the weird thing is that I can reproduce this 100% on my machine at work but not at home. My PC at work is a slow 350Mz P2 and at Home 800Mhz P3 both running Windows 2000. Could it be something to with timing?

Hi, Henrik Gemal, I tested as what you said, but everything is ok, including the preference. The solution for this bug should be the patch plus a workaround as followings. Workaround: 1. Open "Mail & Newsgroup Account Settings..." 2. Select "Server Settings" of your Cyrus server 3. Click "Advanced..." 4. In the namespace area, clear the one that is just "". In my test, this is the Public(shared) namespace. 5. Clear the option "Allow server to override these namespaces" 6. Restart mozilla Would you please try it again for you PC at work? Thanks. Henry

Since we can set the specific directories "Sent", "Drafts" and "Templates" to any other directoies and under different namespaces("Mail & Newsgroups Account Settings" -> "Copies & Folders" -> "Place a copy in" -> "Other"), we can defaultly use the personal namespace's for these pure directories. Change a little to the old patch, just add a check. Now, no need for the workaround, this patch can solve this bug completely.

Henrik, just yesterday I reported the problem you are seeing - see bug 148043.

Comments on patch #85588 - The online server directory is only still used if there is no personal namespace. Is this a good thing, or is it even an issue? I don't really think so as the name space is configurable. - The special folder names are hardcoded. Something I do not really like at all. Can we at least convert them to #defines so they can later be changed to variables for localisation or whatever? I would prefer to keep special folder matching totally out of nsImapUrl.cpp. Your line of code checking for Drafts, Templates is the only instance in the nsImapURL where the names occur. Checking which olders to use for these special purposes stay a matter of nsImapURL. - Only these special folders will be under the same level as the inbox, other folders will not be unless a server directory is set. - This patch assumes that the server directory is in non-canonical namespace and thus adds dependency on how bug #105385 is fixed. It is the task of the nsImapURL functions to just to do the conversion from canonical internal namespace to server namespace. Folder discovery logic, no matter how primitive, should not belong to it. Taking this as a given, the patch is a hack and not a fix. Sorry Henry. So how do we go about fixing this? Basically I only see a fix in the way we use canonical namespace. Add a top level canonical directory: /personal /users /shared Using this, /personal/Drafts has to be converted to whatever the personal namespace prefix is and Drafts added on the end. Likewise: /shared/News from Webmaster is given the prefix of the shared namespace. If multiple namespaces exist for either of these groups, a further subdirectory may be required. But this is something for the future. We already have a gui and automatic detection of the 3 namespaces. Adding these into the canonical path would be a good fix in my opinion. If we add this, then we should also either totally get rid of the concept of a server directory, or have 3 server directories: one for personal, one for folders from other users and one for shared folders.

yes, we shouldn't hardcode those folder names, or do this here. I was thinking this should be done in the code that's trying to find/create the templates/sent/drafts folder, not in the core imap code.

Question: Would it be a good idea to have an external file divided into several sections: one for possible trash folder names, one for possible sent items names, one for possible draft names,..... Upon initial folder discovering, having such a list could be very very useful. For example "Gesendete Objekte" could also be added to the sent items folder list. This doesn't change the fact though that folder discovery at present is not working properly because of the name space issue which needs to be solved first. Using my suggestion of adding a top level directory to the canonical names describing which type of namespace they come from, this bug can be fixed and anything beyond that would be a cool enhancement for future work. If any of you think such an extra top level folder is a good idea, then I'll work on a fix which uses this idea. I'll keep the server directory, but only make it apply to the "/personal" hierarchy of folders. I'm actually thinking of setting the server directory automatically if there is only one private namespace and it isn't defined yet. Problems I see with this before coding is just the downloaded e-mail stored on the hard disk. The canonical names are used to construct the directories in which the files on the hard disk live which contain the bodies. Adding a migration fix shouldn't be too difficult. I guess with Moz 1.0 coming out this is necessary if we really do change the canonical name structure and divide the tree into a few more parts ("/personal" "/users" "/shared"). I really do feel this would fix the bug and add more functionality at the same time. Thoughts?

Michael, while this patch isn't particularly neat, as you say, it *does* fix this bug (or my limited testing suggests it does, anyway). For the 1.0 release, I think patch 85588 should go in, since without it most IMAP servers will not work OOTB with Cyrus. Then, we can concentrate on a "real fix", which IMHO will involve removing "Server Directory" altogether and using only namespaces as we have discussed. But this would be a major change and should wait until after 1.0.

Changes: get ride of the hardcodes for specific directories Comments: 1. The online server directory is used if there is no appreciate namespace instead of no personal namespace. 2. Check the three specific directories is because they are set in the preference, by default, they should be under the personal namespace. 3. I think we should handle the namespace related issues in the code of nsImapUrl, because here is the central place for this issue, just like what we did for the online server directory. I even think we should put all the handling of these code here instead of other places. This can give us a good maintance of the whole code. 4. About the canonical format of the directories, I think we can use the format "<namespace>//<directory>/<directory>...". This can give us flexibility for namespace handling in the future without any information losing. P.S. the delimiter between directories and namespace is '//' instead of '/'.

Jeremy, the 1.0 train is missed. Let's try 1.0.1.

Tom, have we entirely missed NS 7?

Oh, I got it. "Netscape 7.0 RTM will be based on the mozilla 1.0.1 milestone.". We still have some chance to catch NS 7.0 train. Let's try.

I personally think we should get a fix out, at the same time fix #105385 and concentrate later on fixing this properly. Henry, could you comment on whether it is possible to get the folders from the prefs instead of defining them Henry, you say: "3. I think we should handle the namespace related issues in the code of nsImapUrl, because here is the central place for this issue, just like what we did for the online server directory. I even think we should put all the handling of these code here instead of other places. This can give us a good maintance of the whole code." This is exactly the purpose of the whole nsImapUrl class. This is the place to handle namespace issues. You are right there. "4. About the canonical format of the directories, I think we can use the format "<namespace>//<directory>/<directory>...". This can give us flexibility for namespace handling in the future without any information losing. P.S. the delimiter between directories and namespace is '//' instead of '/'." For the discussion of this, I have opened a new enhancement bug #148460.

I just tested the last patch, and I noticed something: - while sending a mail works OOTB now because the folder now can be created, it does not get flagged as a sent mail folder. This has two consequences: The message list when viewing the sent folder only displays the sender (me) and not the recipient and no special icons appear for it. The same is true for the icons for the other special folders apart from Trash. Anybody else seeing this? (No server directory is in use)

I see the same thing when I manually set IMAP folders for Drafts, Sent, etc. I opened bug 148062 to report it a few days ago.

Well, with Henry's patch for bug 105385 now in trunk (thanks Henry!), we're now half way to having Cyrus/Courier work OOTB. bug 27002 is the one outstanding one (there's lots of smaller issues of course, but in terms of at least being able to send and receive email without errors OOTB, these are the only issues). Are there any outstanding issues before attachment 85723 [details] [diff] [review] gets reviewed?

This is a new patch which doesn't change the core URL code. It can give us better performance than the previous one. One thing is that the icons for the specific folders are not right. See comments #80 and comments #81. But I think it should belong to another issue. As for that, we should fix bug 148062. mscott & David, please r= & sr=. Thx.

Cavin, please r=. Thx.

Comment on attachment 91320 [details] [diff] [review] patch without changing the core URL code, better performance r=cavin. But I think you can use: nsXPIDLCString perfectFolderURI; in the call to imapServer->InsertNamespacePrefixIfNecessary(); then you don't have to worry about freeing the space. Use perfectFolderURI.IsEmpty() to check if you get something back.

Henry, this fix is the right approach. I have some comments and questions, however. + void insertNamespacePrefixIfNecessary(in long namespaceType, in string originalUri, out string convertedUri); this should be something like: string getUriWithNamespacePrefixIfNecessary(in long namespaceType, in string originalUri); because you're returning a string - it's better idl style. I also think you might as well always return a uri here (instead of sometimes not) - then the caller doesn't have to check. Either that, or add a comment to the effect that this will return the empty string if the uri already had the correct personal namespace. I don't understand what this means: + PRBool isSpecificFolder/* = PR_FALSE*/) Who ever asks for an unspecific folder? What does that mean? + if (namespacePrefix.Length() > 0) should be if (!namespacePrefix.IsEmpty()) + char *perfectFolderURI = nsnull; this variable name is not very descriptive. you could call it folderUriWithNamespace or something like that.

Thx, Cavin & David. I give a new patch according to your suggestions. 1. use nsXPIDLCString instead of char** for folderUriWithNamespace 2. use string getUriWithNamespacePrefixIfNecessary(in long namespaceType, in string originalUri); instead of string getUriWithNamespacePrefixIfNecessary(in long namespaceType, in string originalUri); in the idl 3. yes, no one use an unspecific folder. delete PRBool isSpecificFolder/* = PR_FALSE*/. Cavin & David, please r= & sr= again. mscott, please also r= if you have some time. This is the same patch for bug 90494.

Henry, looks better. We're almost there. I think the following code can be rewritten to be a bit cleaner, readable, and more efficient, now that you're using an nsCAutoString. In particular, you can just insert the namespace directly into the nsCAutostring using the Insert method, without the need for all the temp variables and string construction. Merely find out if the namespace is where it should be (after the '/'), and if not, insert it directly into uri (which you could call resultUri). Here's what I have in mind. (I haven't compiled or run it, but it should give you the general idea.) Does this make sense? + if (!namespacePrefix.IsEmpty()) + { + nsCAutoString resultUri(originalUri); + PRInt32 index = resultUri.Find("//"); + namespacePrefix.ReplaceChar(ns->GetDelimiter(), '/'); // use canonical format + index = resultUri.Find('/', index + 1); // find '/' after scheme + if (resultUri.Find(namespacePrefix.get(), index) != 0) // Necessary to insert namespace prefix + resultUri.Insert(index, namespacePrefix); // namespace prefix + *convertedUri = nsCRT::strdup(resultUri.get()); + }

Here is the new patch according to David's suggestion. Pls r= & sr=, thx.

Comment on attachment 91593 [details] [diff] [review] new patch without changing the core URL code, better performance sr=bienvenu, thx.

Comment on attachment 91593 [details] [diff] [review] new patch without changing the core URL code, better performance Cavin, pls r= again. Thx.

Comment on attachment 91593 [details] [diff] [review] new patch without changing the core URL code, better performance r=cavin.

Comment on attachment 91593 [details] [diff] [review] new patch without changing the core URL code, better performance a=chofmann for 1.1b

Fixed in trunk.

Removing the item for this bug from the release notes for 1.1beta and beyond.

Build 2002071808 on Linux: It doesnt work again Cyrus imapd server still. Traditional error message displays after send email. "The current command did not succeed. The server responded: Permission denied." Any ideas?

Kristof Petr, Can you make sure the folder 'INBOX.Sent' does exist? Thx. Henry

No. Folder 'INBOX.Sent' doesnt exist. It is fresh account. Mozilla should will create it by itself, I hope. [...after some more testing...] Grrr. I created 'INBOX.Sent' by hands and _now_ it is working good. Mail is sent OK and copy is stored in 'INBOX.Sent'. Is not the another bug, that Mozilla doesnt create system folders, if they are not exist?

Yes, Mozilla is supposed to create INBOX.Sent if it doesn't exist. Does your patch not handle that, Henry? What's supposed to happen is nsMsgCopy::CreateIfMissing() calls nsImapMailFolder::CreateStorageIfMissing(), which creates the folder if it doesn't exist. It might be a hieararchy delimiter problem, or still the namespace problem.

OK. My patch doesn't handle that. I'll work on to make the precess smooth. :)

Patch for 'CreateIfNecessary'. I've tested it on fastmail.fm (Cyrus) & backup2.canaan.co.il (Courier). cavin & bienvenu, please r=/sr=. Thx.

Comment on attachment 92883 [details] [diff] [review] patch for the smooth handling r=cavin.

Henry, this patch makes it so we create the INBOX.Sent on the server? I just want to make sure - it looks like it will create the INBOX.sent resource locally, but I want to verify that it also causes it to be created and subscribed to on the server.

Yes, David. After we make 'msgFolder' namespaced, in 'CreateIfNecessary', we'll use this to create the necessary folder, including 'subscribe'. I tested this on a Cyrus IMAP server (fastmail.fm ) & a Courier IMAP server (backup2.canaan.co.il).

Could it be that this patch is causing problems with folder discovery when using a server directory? What I mean is, I deleted my Cyrus IMAP Account (fastmail), deleted all the local directories, then added it again. When using a server directory (which now works, see: bug #105385) I get my folders all neatly alligned under the Inbox - BUT(!) they are all appearing under the inbox too, which isn't right. This is especially annoying because when there is new mail in any folder other than the inbox, the inbox becomes bold because a subfolder of it has new mail. I didn't find a bug for this, I could create a new one, but I am pretty certain that it has to do with this fix, that's why I am putting the comment here.

Michael Klose, Nice to hear your voice. :-) I tested as you said on fastmail, but everything was ok. Here are my steps. 1. use one of my account that I ever used to send a mail 2. everything is OK. 3. delete and unsubscribe 'INBOX.Sent' folder 4. send a mail, mozilla create the 'INBOX.Sent' folder and send successfully 5. Set the 'Imap Server Directory' advanced setting as 'INBOX' 6. refresh the folder in mozilla 7. folders like 'INBOX.Sent', 'INBOX.Drafts', etc show parallelly with 'INBOX' as 'Sent', 'Drafts', etc, no one appears under 'INBOX' as you said. 8. Send a mail, everything is ok 1. use my another account on fastmail that I have never used (I want to use it for shared namespace and public namespace test), there is no 'INBOX.Sent', etc 2. send a mail, mozilla create the folder 'INBOX.Sent' and send successfully 3. Set the 'IMAP Server Directory' advanced setting as 'INBOX' 4. refresh mailnews folder, mozilla shows as we expect and no one shows under 'INBOX' 5. send a mail, mozilla send successfully Michael Klose, which patch did you apply? What are the steps you produce this issue? Another interesting thing, also for your comment #80, when you set the 'IMAP Server Directory', you'll see the correct icons. :-) Also, you can do this by set the 'Copies & Folders' to see the correct icons. I've marked bug 148602 as new and try to fix it. Regards. Henry Jeremy, do you have any time to join this discussion.

Sorry for the typo. I've marked bug 148602 as new and try to fix it. should be I've marked bug 148062 as new and try to fix it.

Let take care of this bug. I think it can be solved perfectly. Add Cavin to CC list.

Henry, actually, I am just using a trunk build. Didn't have time for much hacking on Mozilla that much recently. I was using 2002072408, but have now tried it again with 2002073019, both times Win98SE, and it is still happening. But, I know why it is happening now. I specified "Inbox" instead of "INBOX" in some absent state of mind. Changing it to "INBOX" maked it work. I just checked the RFC to see if this is a bug: RFC2060: "5.1. Mailbox Naming The interpretation of mailbox names is implementation-dependent. However, the case-insensitive mailbox name INBOX is a special name reserved to mean "the primary mailbox for this user on this server". " I interpret this to mean that INBOX has to be case insensitive, every other folder as case sensitive. So a bug yes, but a pretty minor one. By the way, the exacts steps I took to reproduce this: 1. Edit/Mail-News-Pref 2. Add account 3. Enter details, click finish 4. when back in prefs, set server directory to "Inbox" 5. Click on Inbox. I see every folder twice - once "parallel" to the inbox, and once under the inbox. Maybe this comment should better be under #105385.

As the main mailbox, 'inbox' is case insensitive. As the 'IMAP Server Directory', it should be case sensitive. But I also think it's a bug about what you have seen. I can verify it. For the incorrect Input "Inbox", mozilla should not show the folders parallelly with 'INBOX'. I think this bug is caused by mozilla always uses 'Inbox' instead of other format like 'inbox', etc to show the main folder and what you input is 'Inbox'. It should be a new bug. I'll take a deeper look at this issue.

I've filed a Bug 160643 (case sensitive issue of the special 'IMAP Server Directory' setting - 'INBOX') for the issue of comment #109. I think for some time about if we should use the special 'IMAP Server Directory' setting 'INBOX' case sensitive or case insensitive. Now, I think we should handle it case insensitive. It's 'IMAP Server Directory', so it should abide the specification of IMAP folder, in which the special 'INBOX' should be case insensitive.

Comment on attachment 92883 [details] [diff] [review] patch for the smooth handling David, please sr=

Comment on attachment 92883 [details] [diff] [review] patch for the smooth handling sr=bienvenu

Landed on trunk. Marking resolved.

There is a logical problem in the first patch. What's more, make the comparasion with 'INBOX/' case insensitive.

Cavin & David, please r=/sr=.

The logical problem is we should use if (resultUri.Find(namespacePrefix.get(), PR_FALSE, index+1) != 0) instead of if (resultUri.Find(namespacePrefix.get()) != 0) // Necessary to insert namespace prefix

I think this last patch can be much shorter - you're really just trying to see if the path starts out with "INBOX/" in a case insensitive way, right? So, it would be something like this: PRBool nameSpaceIsInbox = namespacePrefix.EqualsIgnoreCase("INBOX/", 6); if (resultUri.Find(namespacePrefix.get(), nameSpaceIsInbox /* ignore case */) != 0) resultUri.Insert(namespacePrefix, index+1); // insert namespace prefix I'm not sure if those are the correct latest and greatest string functions to use, but the general idea should be fine. We should use the power of our string classes - they can make the code much cleaner than the old C char ptr style of programming.

I guess my proposal should be more like: if (resultUri.Find(namespacePrefix.get(), nameSpaceIsInbox /* ignore case */, index + 1) != 0)

Here is the new patch according to David's suggestion.

Henry, what was wrong with the two lines I attached? I thought that would be sufficient.

According to RFC 2043, namespace is Namespace = nil / "(" 1*( "(" string SP (<"> QUOTED_CHAR <"> / nil) *(Namespace_Response_Extension) ")" ) ")" and according to RFC 2060, string is A string is in one of two forms: literal and quoted string. The literal form is the general form of string. The quoted string form is an alternative that avoids the overhead of processing a literal at the cost of limitations of characters that can be used in a quoted string. A literal is a sequence of zero or more octets (including CR and LF), prefix-quoted with an octet count in the form of an open brace ("{"), the number of octets, close brace ("}"), and CRLF. In the case of literals transmitted from server to client, the CRLF is immediately followed by the octet data. In the case of literals transmitted from client to server, the client MUST wait to receive a command continuation request (described later in this document) before sending the octet data (and the remainder of the command). A quoted string is a sequence of zero or more 7-bit characters, excluding CR and LF, with double quote (<">) characters at each end. The empty string is represented as either "" (a quoted string with zero characters between double quotes) or as {0} followed by CRLF (a literal with an octet count of 0). Note: Even if the octet count is 0, a client transmitting a literal MUST wait to receive a command continuation request. So, I think though I haven't seen the namespace prefix like 'INBOX.a.', it is allowed and only 'INBOX.' part is case insensitive. Right? If I'm not right, please correct me. Thx. You worked longer than me in this area and have more experience. :-)

well, you're right that the namespace can be more than the INBOX, but I don't see any evidence in the namespace RFC that the INBOX comparison should be case-insensitive when comparing namespaces - the server, it seems, should report the correct case for Namespaces. Are you making the case change because there's a server that's returning a namespace of "Inbox." when it should really be returning "INBOX." ? Or is this just because of the separate problem where the user-specified server directory is specified by the user as "Inbox." when it should be "INBOX."? Does this namespace code get invoked for the user-specified server directory too?

I think we should fix the logic error, since that's totally breaking some people, and worry about the INBOX case problem, if it really is a problem, after that.

Agreed. Please fix the logic error ASAP, and worry about the details later. This is causing me problems for my private email so please don't hold back the fix any longer than you have to.

I'm going to check this fix in, so that people can start using cyrus again. this is sr=sspitzer - note that the check for where we find the uri needs to be with index + 1, not 0.

marking fixed - the case-sensitivity issues can be moved to another bug, and we can decide if they're valid or not.

Apply for a= for OEM branch.

In OEM branch.

*** Bug 161424 has been marked as a duplicate of this bug. ***

Looks like it maybe isn't fixed yet, at least for me: 2002090314. relevant prefs: user_pref("mail.account.account9.identities", "id8"); user_pref("mail.account.account9.server", "server9"); user_pref("mail.identity.id8.fcc_folder", "imap://mklose@www.fastmail.fm/Sent Items"); user_pref("mail.identity.id8.fcc_folder_picker_mode", "1"); user_pref("mail.server.server9.hostname", "www.fastmail.fm"); user_pref("mail.server.server9.namespace.other_users", "\"user.\""); user_pref("mail.server.server9.namespace.personal", "\"INBOX.\""); user_pref("mail.server.server9.namespace.public", "\"\""); user_pref("mail.server.server9.server_sub_directory", "INBOX"); OK, every time I try and send email, the email is sent fine, but I get an error that it is not possible to upload it to the sent items folder which is no surprise when you look at the protocol log: -1706107[c4eb840]: www.fastmail.fm:NA:CreateNewLineFromSocket: * OK IMAP4 Ready www.fastmail.fm -1706107[c4eb840]: www.fastmail.fm:NA:SendData: Logging suppressed for this command (it probably contained authentication information) -1706107[c4eb840]: www.fastmail.fm:NA:CreateNewLineFromSocket: 1 OK You are so in -1706107[c4eb840]: www.fastmail.fm:A:SendData: 2 append "INBOX.INBOX^^INBOX^^Sent Items" (\Seen) {398} -1706107[c4eb840]: www.fastmail.fm:A:CreateNewLineFromSocket: 2 NO Mailbox does not exist -1706107[c4eb840]: www.fastmail.fm:A:SendData: Message-ID: <3D7A8BB6.7010903@klose.us> Date: Sun, 08 Sep 2002 01:28:54 +0200 From: Michael Klose <michael@klose.us> User-Agent: Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.1b) Gecko/20020903 X-Accept-Language: en-us, en MIME-Version: 1.0 To: test@klose.us Subject: Sent Items folder test Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Test!!! -1706107[c4eb840]: www.fastmail.fm:A:SendData: -1706107[c4eb840]: www.fastmail.fm:A:CreateNewLineFromSocket: Message-ID: BAD Unrecognized command I'm going to try this on a new build tomorrow, but since my build is only 3 days old and I saw no fixes going in in the last few days concerning this, I guess it won't make much difference. OK: Summary: Copying to Sent items folder does not work if using a server directory. Something is seriously going wrong with the conversion to non canonical name (INBOX^INBOX^INBOX^...) The other thing which I think is weird is that it is even trying to send the mail when the server has already replied with an error that the mailbox doesn't exist.

Is bug 166603 related?

No, it isn't related because you say in bug #166603: "With this setup Mozilla (1.1 on Red Hat Linux 7.3) correctly saves messages to INBOX.Sys.Drafts when I do "save as draft". However when I go to the Drafts folder, the message is displayed same as if it was an ordinary folder and there is no way to continue editing the draft (other then "Edit as new")!" That means in your case the email is actually reaching the target folder just the \DRAFT flag does not get set. In my case it isn't even reaching the Sent Items folder eventhough I have selected it from the GUI.

Comment #131 referred to my Win98 Laptop which unfortunately does not work anymore since this afternoon (cable between chassis and screen is broken, only works at a very specific angle, at any other the screen goes blank) so I can't check a recent build. But it DOES work fine on my desktop running Win2K and build 2002090908. Once (and if) the laptop is repaired, I'll try a recent build on that and see if the error still occurs on that. Just wanted to know that the error I reported in comment #131 WFM on Win2K on the desktop.

20020913 trunk builds on WinNT and Win2K. Tested with Cyrus IMAP. Sending mail got Alert saying that The current command did not succeed. The mail server responded: Permission denied. Clicking OK button to dismiss the Alert. 'Sending Messages' dialog appeared and never goes away. Clicking Cancel button to dismiss it. Compose window remained there. I found out later that the mail was actually sent out. Reopen the bug.

yulian@netscape.com: I tested with the mine build of 2002.09.03. I tested using fastmail.fm. Here are the steps. 1. Compose a mail with 'IMAP Server Directory' not set 2. Save the mail as a draft. It's OK. 3. Save the mail as a template. It's OK. 4. Send the mail. It's OK. 5. Compose a mail with 'IMAP Server Directory' set to 'inbox' 6. Save the mail as a draft. It's OK. 7. Save the mail as a template. It's OK. 8. Send the mail. It's OK. Not sure why you see the alert. There are two reasons for what you see. One is you don't have the right permission on the server. The other is that mozilla give out the wrong command to the server. So, would you please, 1. check the permissions on the server, 2. give out the mozilla imap log, 3. give out your test enviroment, such as the 'IMAP Server Directory' setting? If this is a real bug, let's try to fix it. Thx. Henry

Michael, it looks like you've set the server sub-directory, BUT your server supports the namespace extension, so we're using both, ending up with double INBOX. You could try removing the INBOX. server sub-directory setting from your imap server settings.

*** Bug 155532 has been marked as a duplicate of this bug. ***

My bug was marked a duplicate of this bug, and I believe this is true. I've been told to try the latest 1.2alpha, and it worked for me alright. Thank you very much for your dedicated work!

is the situation still the same with a newer build? (bug cleaning)

should this be closed? Perhaps we should open a new bug for the remaining issue, if any.

Agree.

ok, I'm marking fixed.

Just wanted to note that I still find intermittent problems sending to Sent with Cyrus, particularly when there's packet loss between me and the server (e.g. when I'm using WiFi.) The problems are that Mozilla becomes unstable and often hangs. The <sending to sent items> window is cancellable, but then the compose window becomes non-responsive (no buttons work); sometimes closing it provides an opportunity to save the email to Drafts. With not really reproducible bugs like this, I thought a comment here before opening bugs might make sense. Sorry for the spam. Seems like missing timeout/error checking or a memory leak - just guessing. Haven't seen any open bugs addressing these problems.

I think i have just hit this bug with mozilla v1.6: http://bugzilla.mozilla.org/show_bug.cgi?id=234219