Unable to finish folder discovery on account; get multiple .msf files generated for one folder Mac OS 9 2001-05-25-08 commercial trunk build. Win32 does not have this problem. 1. Create new profile 2. Start the product 3. Cancel out of activation dlg 4. Browser appears 5. Click Buddy List tab and set up for IM screenname 6. Exit and restart. (note: step 5,6 are necessary due to another bug on commercial build where this IM dlg will cause some other problems if not set up for IM). 7. Browser appears. Tasks | Mail 8. Set up for my IMAP netscape.com account 9. Log into IMAP 10. The status bar will eventually read "looking for folders..." 11. After a short while, the throbber stops moving and my system is locked up. 12. Must force reboot. 13. Now, when I look into the Finder into my IMAPMail folder, I see many (up to 33!) .msf files created for one folder I have. Note: This does not have to do with the folder name because I've renamed the folder and then I see this problem with another folder name. In my conversation with David, he writes: LisaChiang (6:12:56 PM): interesting note. when I crashed, the next time I started on the Mac during "looking for folders" I see the following: LisaChiang (6:13:08 PM): I have a folder called archive (it's at the top level). LisaChiang (6:13:51 PM): I see folders: Archive, Archive-1 all the way through a very high number (ie. Archive-30) or more. What do you think would cause these other archive folders to show up? dbienvenu (6:13:59 PM): oh. LisaChiang (6:14:32 PM): It's very strange. dbienvenu (6:14:38 PM): the -1, -2, etc is an attempt to create unique database names LisaChiang (6:14:43 PM): I think I now have tons of those msf files. LisaChiang (6:14:56 PM): why would we do that when I just have one archive folder. dbienvenu (6:15:07 PM): so what happens is we discover a folder foo and we find a database called foo.msf, but it doesn't seem like the same folder to us, for some reason. dbienvenu (6:15:12 PM): so we create foo-1.msf LisaChiang (6:15:21 PM): I think that is where I am hanging on the mac. dbienvenu (6:15:33 PM): so the problem is that we don't think the database is the right db for the folder. dbienvenu (6:15:44 PM): Seth and Cavin made some changes in this area for migration. LisaChiang (6:15:48 PM): this is on a new profile, first time logging in. dbienvenu (6:15:53 PM): right. LisaChiang (6:16:01 PM): I will file bug then for them to debug this. dbienvenu (6:16:08 PM): it creates all those .msf files for just one run? Note2: I tried another IMAP account with folders in it and it works fine. Able to get through folder discovery and loading folders. If you need my account to debug this, pls let me know.

cc: seth and cavin.

get this on the 0.9.1 radar. david, I've got a mac build here at home, I can look into it.

Seth - if you want to debug this further, give me a page and, unless I'm asleep, I will call you with my mail password. Thanks.

Seth graciously offered to look at this.

updating status.

Reproduced on more than one system. Will work with Seth to reproduce this on this Mac build.

so far, here's what it looks like. we aren't properly handling long file names on the mac. lisa has some folders on her imap server that exceed the minimum. there is code to hash long file names. my guess (so far) is that we aren't using it in all places. specifically, looking at my panacea.dat (where we store paths to the summary files) we aren't using the hashed names. still investigating...

getting somewhere, I think. nsMsgDBFolder::CreatePlatformLeafNameForDisk() needs to do some magic hashing. (NS_MsgHashIfNecessary()). that gets called on folder discovery [by nsImapMailFolder::CreateClientSubfolderInfo()] I'll start with that problem and see how much further it gets me. (there might be more places we need the magic hashing code)

the following patch is the start of the fix. it fixes the problem that lisa found on the mac. (note, that patch also has a change to allow me to test the fix on win32 / linux, which I won't check in.) three things need to be done: 1) I need to add back the code that will make the .msf unique if it already exists (I removed that) 2) move the "illegal char substitution" code into NS_MsgHashIfNecessary(), and if the file doesn't get hashed, I need to make the substitution because... 3) NS_MsgHashIfNecessary() is really NS_MakeFilenameSafeForOS(). so I'll rename it. I'll work on these three things tomorrow, and have something to land tomorrow or thursday.

changing qa contact and assigning it to myself

commercial buildid: 2001052508 Mac-OS-9.0.4 I tried to recreate this problem on my mac machine by doing the following: -On my win98 machine I created the folders below in my IMAP account -I created the two folders which apparently was causing this problem. 1st Level folder name: ! Me - To Do and created a sub folder ->nscp6_problems_to_investigate. -I copied few messages to both the folders I followed the steps described from 1 to 10 in the initial description of the bug and saw different results: 1.The application did not hang and created the folder structure and downloaded some 1500 messages in my mail box. 2.I checked my IMAPMail folders and did not see multiple .msf files created for one folder 3.I noticed that the subfolder 'nscp6_problems_to_investigate' was no where to be found in the 3 pane. Folder ! Me - TO DO did not have the twisty to indicate the subfolder at all. 4.I closed the application and relaunched it again. I still could not see the subfolder 5. I checked ImapMail Folder on my hard drive and I see '! Me -To Do.sbd'. When I open the twisty I should see 'nscp6_problems_to_investigate.msf' file but I don't see it. Could this behavior be related to this problem?

that sounds related. when I test my final fix on the mac, we can test that scenario. here comes a cleaned up fix...

yikes. Looks OK, except for the K&R braces style creeping in :-)

some comments: 1) I tried to keep this patch clean and minimal. (trying to reduce the risk so close to 0.9.1 & beta). [That's the reason I didn't rename NS_MsgHashIfNecessary()] 2) the illegal chars / length code [which was not turned on] that was in nsMsgDBFolder.cpp is now in NS_MsgHashIfNecessary(). If the suggested name is too long, we do the same old hashing. If (and this is new) it contains illegal characters, I simply hash the whole string. (hash results are always safe). this should prevent collision and always make it so we hash to a unique file name. the worry is that "foo_bar" and "foo:bar" will map to the same summary file ("foo_bar.msf"). but by hashing completely, we avoid this edge case 3) CreatePlatformLeafNameFromDisk() is now CreateFileSpecForDB(), and we return the file spec. I've moved all the "unique" magic in CreateFileSpecForDB (). that, plus the hashing code, makes us very safe. 4) the existing callers of NS_MsgHasIfNecessary() are safe. OS/2 now hashes when the name is > 8 (they have 8.3 limits). imap also uses this, but I'm was broken (and using it consistently will fix it.) news is safe, since the "illegal characters" are not valid characters in newsgroup or host names. 5) I've added comments in the code to explain other changes

I tried a bunch of tests on win32, new profile, existing profile, deleting panacea.dat, all worked fine. I did see a problem on a new profile when selecting the inbox failed the first time, because of Sera flakiness, but I don't think that had to do with this patch.

thanks for testing. an extra note: since I'm hashing (instead of simply replacing illegal chars with '_'), this can cause some .msf (for imap folders only) to get ignored the first time someone runs with a build with this patch. according to bienvenu, losing (and rebuilding) the .msf file one time is acceptable.

tested on the mac. looking good. can I get an official r/sr?

r/sr=bienvenu, whichever you need.

With your patch builds, does my IMAP account successfully get set up with all folders discovered and loading? If you want, perhaps we can try this on the Mac tomorrow too on your debug builds (if we don't have release builds in time with your patches in it).

I'm at home, and my mac is outside the firewall so I can't test your problem. but, I tested original "proof of concept" patch inside the firewall yesterday on your account and it fixed the problem. outside the firewall, I've tested on my mac against an imap account I have with long folder names. the problems I was seeing are gone. (I also tested some edge cases on win32 by setting the max len to 5 and adding common characters to the illegal char list, to test out that code) I hope to land this patch tonight.

sr=mscott

updating.

a=leaf, we must protect the users from weak filesystems!

fixed. note, another side effect of this bug was you could end up with a file named ".msf" in the application directory. lisa / sheela, can you test again with tomorrow? I don't think you should have to do anything special (like remove any files or create a new profile.) it should just work out of the box. (famous last words...)

Creating an .msf file at a higher directory has always been there on the Mac, I think: http://bugzilla.mozilla.org/show_bug.cgi?id=6402

lisa: not "<server>.msf", literally ".msf" in the same folder as "prefs.js"

buildid: 2001053108 My problem seemed to have been fixed which I mentioned in comments 2001-05-30 15:36. I checked for the folders that I had created previously on existing profile and also on a new profile. I was able to see the subfolder and messages in that folder. I also did not see any .msf created in the same directory as perf.js In verifying this bug I created some more folders with long names and with different characters and they all seemed to work fine. Checked this on imap account. Still need to check this for pop and will update the bug after I do so. Also, I will verify this bug after checking with lisa on her mac machine.

I am going to verify this bug since it fixed my problem as mentioned in my comments below. I will log a new bug for local folders since creation of a new folder with long name created a .msf file in application directory and as well does not create a folder at all. Seth and I created lisa's imap account on my mac machine and were able to successfully discover all the folders on the imap server. However, this problem still exists on lisa's machine which could be a different problem. I will log a new bug for that specific problem too. Hmmm....Also multiple .msf files getting created for folders has also been resovled by this fix.

I logged #84008 to cover the problem we saw on esther's and lisa's machine after this fix landed.

Actually, this bug is hitting part of bug 20324 problems, so that's when I didn't verify for bug 20324 yet. I am adding this bug in bug 20324 for future reference.....