Closed Bug 65086 Opened 24 years ago Closed 2 years ago

sometimes mork doesn't automatically compress commit. causes slow db access, high memory. panacea.dat can get large. In addressbook deleting contacts doesn't make .mab file smaller.

Categories

(MailNews Core :: Database, defect)

Product:
MailNews Core
Component:
Database
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED INVALID

People

(Reporter: sspitzer, Unassigned)

References

(Blocks 1 open bug,
URL
)

Details

(Keywords: memory-leak, perf, Whiteboard: [gs]) 

I deleted about half the entries in my collected address book (the multiple
delete repainting bug gets us there, too) and then quit.

my history.mab file did not shrink.

-rw-rw-r--   1 seth     seth      1897535 Jan 11 09:58 history.mab

I thought I remembered bienvenu fixing this for mork (for .msf files) so you'd
think .mab files would get the same fix.

assigning to bienvenu for now, it might be a duplicate of another bug.
Using today commercial linux build 2001-011108-mtrunk build, my history file has 
the same permissions that Seth noted.  When I delete entries from the Collected 
address book, the entries where removed but the history.mab size did not 
decrease.  In my case, it increased!  After re-launching seamonkey, I just 
openned the collected address book then quit.  I looked at the history.mab file. 
 The file was increased a few bytes.  This doesn't look good.

Nominating for nsbeta1.
Keywords: nsbeta1
the problem is that the mork code that determines whether or not a client should
do a compress commit is fundamentally and conceptually broken. I fixed history
to use a different logic for determining whether a compress commit should be
done. I doubt I'm going to be able to fix the broken mork code, so realistically
I suspect this fix will need to be done in the address book code.
Dr. B:  how about for .msf files?  do we need to fix that code too?

yes, .msf files have the same problem.
marking nsbeta1+ and moving to mozilla0.9
Priority: -- → P2
Whiteboard: [nsbeta1+]
Target Milestone: --- → mozilla0.9
QA-assign-to fenella.
QA Contact: pmock → fenella
marking nsbeta1- and moving to future milestone.
Keywords: nsbeta1 → nsbeta1-
Whiteboard: [nsbeta1+] → [nsbeta1+ 2/21]
Target Milestone: mozilla0.9 → Future
bringing back into nsbeta1+.  I'm pretty sure David fixed this on the perfBranch.
Keywords: nsbeta1- → nsbeta1
Whiteboard: [nsbeta1+ 2/21] → [nsbeta1+]
Target Milestone: Future → mozilla0.9
I only partially fixed this - the address book code was never calling compress
commit, so it never had any chance of working. I fixed that on the branch, but I
have not fixed the mork code that determines if you should call compress commit.
 It will probably work in some situations (e.g., it did shrink my 1.4MB CAB to
60K), so it will at least be better than before, but fixing this for the general
case in the mork code is hard. 
David,

It looks like this bug was mostly for making the .mab file smaller.  It sounds
like you did this.  Are the other cases you mentioned related to the address
book or is it for mork in general?  If it's the latter then why don't we mark
this fixed and then we can open another bug to make mork do this by itself.
Only in some situations will the .mab file get smaller - the mork code that
determines the % wasted space is pretty broken - it only sometimes works.
moving to mozilla0.9.1
Target Milestone: mozilla0.9 → mozilla0.9.1
David, is the mork code something you think you can fix or should we punt on this?
moving to 0.9.2
Target Milestone: mozilla0.9.1 → mozilla0.9.2
moving to 0.9.3
Target Milestone: mozilla0.9.2 → mozilla0.9.3
Target Milestone: mozilla0.9.3 → mozilla0.9.4
Mail news triage meeting --> .9.5
Target Milestone: mozilla0.9.4 → mozilla0.9.5
QA Contact: fenella → nbaca
moving to 1.0 - not sure when this will get done. I need time to sift through 
mork, which requires a big chunk of time.
Target Milestone: mozilla0.9.5 → mozilla1.0
Blocks: 104166
Keywords: nsbeta1+
Whiteboard: [nsbeta1+]
Blocks: 122274
Status: NEW → ASSIGNED
Keywords: nsbeta1+ → nsbeta1-
Target Milestone: mozilla1.0 → mozilla1.2
No longer blocks: 122274
*** Bug 118385 has been marked as a duplicate of this bug. ***
Marking nsbeta1 because deleting entries should reflect a change in the *.mab
file size.
Keywords: nsbeta1- → nsbeta1
Mail triage team: nsbeta1-
Keywords: nsbeta1-
Keywords: nsbeta1
*** Bug 238617 has been marked as a duplicate of this bug. ***
Product: Browser → Seamonkey
for Thunderbird see Bug 279998
Hardware: PC → All
Target Milestone: mozilla1.2alpha → ---
Component: Address Book → MailNews: Address Book
Product: Mozilla Application Suite → Core
QA Contact: nbaca → addressbook
*** Bug 279998 has been marked as a duplicate of this bug. ***
After years without a fix wouldn't be the new storage system aka sqlite a better solution? 
(In reply to comment #24)
> After years without a fix wouldn't be the new storage system aka sqlite a
> better solution? 

Yes it may be a better solution, there is another bug on doing that transfer. Until that is fixed however, this bug is still valid.


(In reply to comment #25)
> Yes it may be a better solution, there is another bug on doing that transfer.
> Until that is fixed however, this bug is still valid.

Right, the implementation of the sqlite backend is covered by bug 382876.

a simple fix in the ab would be to make commits after a card is deleted automatically do a compress commit, or use a threshhold, like 10 deleted cards.
Or, do what I did for history, and do some simple math based on the number of cards and the size of the .msf file, and do compress commit if the math indicates that the ratio of .msf file size/num cards in the db is large.
Mmh if I have a bunch of cards selected for deletion and hit delete, why the compress cannot be automatically invoked after all selected cards were removed? Same would apply if cards are moved into another address book. Wouldn't it be easier to implement as doing a lot of math?
what if I deleted a single card at a time, from a very large address book? With your approach, that would involve writing out a large file over and over again, once for each delete.
That's true. And we don't know how many deleted cards are laying around? So your approach will be reasonable.
Product: Core → MailNews Core
Depends on: 453975
No longer depends on: 453975
May I suggest that the compress commit be included with the option that auto-compresses the mail during the Tb close procedures. Or another option of the AB having a "Has Deleted" flag set that if true does the commit during Tb close.

Further I suggest this goes into the Tb 2.0 branch prior to offering the 3.0 upgrade notification as a version migration cleanup. Then when the final storage solution is available there will be a cleaner AB to migrate.
Assignee: bienvenu → nobody
Priority: P2 → --
Status: ASSIGNED → NEW
http://forums.mozillazine.org/viewtopic.php?p=10283825 has 950MB panacea.dat
URL: http://forums.mozillazine.org/viewtop...
Severity: normal → major
Component: Address Book → Database
Keywords: perf
QA Contact: address-book → database
Summary: mork / addressbook: deleteing entries doesn't make the .mab file smaller → client compress commit is broken, makes db access slow -- addressbook: deleteing entries doesn't make the .mab file smaller// panacea.dat gets large
Summary: client compress commit is broken, makes db access slow -- addressbook: deleteing entries doesn't make the .mab file smaller// panacea.dat gets large → sometimes mork doesn't automatically compress commit, makes db access slow -- addressbook: deleteing entries doesn't make the .mab file smaller// panacea.dat gets large
examples of large/extreme panacea.dat files (recap)

* http://getsatisfaction.com/mozilla_messaging/topics/thunderbird_has_started_using_3gb_of_ram_at_start_up#reply_3563429 (weidenrinde: 90mb)
* http://getsatisfaction.com/mozilla_messaging/topics/thunderbird_3_1_6_3_1_7_very_slow_startup#reply_4708282 (Graham: 950MB
* http://forums.mozillazine.org/viewtopic.php?p=10283825 (Hiraeth: 950MB, wheelz, 1.7MB)
* (possibly) http://getsatisfaction.com/mozilla_messaging/topics/tb_crashes_at_start_up_after_update_of_an_add_on#reply_4738337


in talking with bienvenu about the size of panacea.dat, we might use a rough rule of thumb of 300bytes per folder (news, imap, local, rss), or 300KB per 1,000 folders.  That's in it's idle compressed state, after a compress commit happens at shut down.  panacea grows while thunderbird is running, and can double, triple, etc in size.
URL: http://forums.mozillazine.org/viewtop... → http://getsatisfaction.com/mozilla_me...
Whiteboard: [gs]
This is also a major security problem.
While searching for a lost address book for a client I found all his confidential deleted addresses...

Workaround: 
 - export all address books to LDIF (for example) 
 - close programme and delete all abook*.mab files
 - re-import from LDIF
 - move imported addresses to now really empty address books
Keywords: mlk
Summary: sometimes mork doesn't automatically compress commit, makes db access slow -- addressbook: deleteing entries doesn't make the .mab file smaller// panacea.dat gets large → sometimes mork doesn't automatically compress commit. causes slow db access, high memory. panacea.dat can get large. In addressbook deleting contacts doesn't make .mab file smaller.
Blocks: 1084276
Blocks: tb-startupperf
No longer blocks: 1084276
Is this still a problem in addressbook? There seems to be attempt to compressCommit at http://mxr.mozilla.org/comm-central/source/mailnews/addrbook/src/nsAddrDatabase.cpp#640
Which bug is related to the mxr citation?  Bienvenu did some work for case of commit after large scale deletion of contacts trying to lower commit threshold. IIRC.

What we still have with abook mabs is retention of deleted contacts.  I still have contacts in my abook.mab years after contacts were removed from viewing. Those files are not undergoing any cleanup for many users.  It is not so much a size issue as one of security, as comment 35 points out.

Comment 2 points out that the mork code is broken and has been worked around to resolve other bugs. What ever happens, panacea.dat use of mork should be a focus since it effects performance.  the abook issues probably will benefit from spillover.
(In reply to Ronald Killmer from comment #37)
> Which bug is related to the mxr citation?  Bienvenu did some work for case
> of commit after large scale deletion of contacts trying to lower commit
> threshold. IIRC.
> 

> What ever happens, panacea.dat use of mork should be
> a focus since it effects performance.

The biggest single obstacle to replacing mork so far has been that, because it is memory resident, it is really fast. The earlier replacement effort of mork in panacea.dat was backed out because of a performance regression.
(In reply to :aceman from comment #36)
> Is this still a problem in addressbook? There seems to be attempt to
> compressCommit at
> http://mxr.mozilla.org/comm-central/source/mailnews/addrbook/src/
> nsAddrDatabase.cpp#640

impossible to know what's happening in the field.  a job for telmetry? 

(eg. I just dealt with a user with a huge panacea.dat, and attendant performance issues - i.e. non-functional after startup)

(In reply to Ronald Killmer from comment #37)
> Which bug is related to the mxr citation?  

I don't see one.

> Bienvenu did some work for case
> of commit after large scale deletion of contacts trying to lower commit
> threshold. IIRC.

A citation would be great. I can't find the bug report.  Can you find it or recall more details?
Flags: needinfo?(killjay)
(In reply to Wayne Mery (:wsmwk) from comment #39)
> (In reply to :aceman from comment #36)
> > Is this still a problem in addressbook? There seems to be attempt to
> > compressCommit at
> > http://mxr.mozilla.org/comm-central/source/mailnews/addrbook/src/
> > nsAddrDatabase.cpp#640
> 
> impossible to know what's happening in the field.  a job for telmetry? 
> 
> (eg. I just dealt with a user with a huge panacea.dat, and attendant
> performance issues - i.e. non-functional after startup)
> 
> (In reply to Ronald Killmer from comment #37)
> > Which bug is related to the mxr citation?  
> 
> I don't see one.
> 
> > Bienvenu did some work for case
> > of commit after large scale deletion of contacts trying to lower commit
> > threshold. IIRC.
> 
> A citation would be great. I can't find the bug report.  Can you find it or
> recall more details?

Reading back through this bug, comments 27 through 31 are the dialog I remember reading in my bug mail.
I did a search of the moz.dev.apps.tb newsgroup archive and it was not the source of my recollection.

The Dev comments in the mxr citation of Aceman define the compress commit threshold is 30% or greater file size reduction. Comment 9 states the abook code was never calling morks compress commit and comment 11 declares the mork code broken, only sometimes working.

I looked at my panacea.dat in notepad and it still has much outdated data, being the file is now 7 years old. This hints that mork is not triggering compress commit to clean up the file.

If we follow the pattern described in this bugs comments, the panacea code might not be a calling for a compress commit by mork. However, if we stay with mork, work is needed to fix the brokenness for the general case which should aid any module needing services to clean up mork indexes.
(In reply to Ronald Killmer from comment #40)
> I looked at my panacea.dat in notepad and it still has much outdated data,
> being the file is now 7 years old. This hints that mork is not triggering
> compress commit to clean up the file.

yes. What is the size of your panacea.dat? 


> If we follow the pattern described in this bugs comments, the panacea code
> might not be a calling for a compress commit by mork. However, if we stay
> with mork, work is needed to fix the brokenness for the general case which
> should aid any module needing services to clean up mork indexes.

Bug 418551 would help make it unneccessary to fix this. for panacea.dat anyway.
Depends on: 418551
(In reply to Wayne Mery (:wsmwk) from comment #41)
> (In reply to Ronald Killmer from comment #40)
> > I looked at my panacea.dat in notepad and it still has much outdated data,
> > being the file is now 7 years old. This hints that mork is not triggering
> > compress commit to clean up the file.
> 
> yes. What is the size of your panacea.dat? 
> 
Stable in range 489 to 497 KB.  Accounts: 1 POP3 and 2 NNTP.
Flags: needinfo?(killjay)

This should no longer be an issue now that Bug 418551 was fixed in version 93.

Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → INVALID
You need to log in before you can comment on or make changes to this bug.