User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.2) Gecko/20090803 Fedora/3.5.2-2.fc11 Firefox/3.5.2 Build Identifier: Bugzilla 3.4.1 I noticed that jobqueue.pl was consuming the most memory on one of my bugzilla servers: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3910 wwwrun 16 0 256m 225m 2372 R 14 5.6 40:30.92 jobqueue.pl After restarting it: 6003 wwwrun 16 0 65752 33m 2324 S 0 0.8 0:00.48 jobqueue.pl After a few days, it is now up to 65m: 6003 wwwrun 16 0 95612 63m 2336 S 0 1.6 3:09.49 jobqueue.pl Reproducible: Always Steps to Reproduce: 1. Start jobqueue.pl as a non-root user 2. Use 'top' sorted by memory usage 3. Wait a week or two 4. Goto 2.

Cool, thanks for filing. Could you attach (as an attachment) the current output of your checksetup.pl --check-modules?

Do you have any idea how fast it's growing? I'd like to get some idea of how many email-sending iterations I have to run to see the same growth you're seeing.

> Do you have any idea how fast it's growing? Eclipse's Bugzilla sends out about 7000-10000 emails/day during the week, and according to comment 0, jobqueue.pl doubled in size after a few days.

We are currently restarting each day to as our jobqueue.pl is growing quite large after 24 hours of use even with no jobs left to process: Some info After a restart top - 14:54:26 up 4 days, 20:34, 5 users, load average: 1.53, 1.55, 1.50 28825 root 15 0 193m 41m 2900 S 0.0 0.5 1:03.52 jobqueue.pl After running for 24 hours top - 08:44:07 up 5 days, 14:24, 2 users, load average: 1.44, 1.72, 1.99 28825 root 15 0 2351m 2.1g 2900 S 0.0 27.5 8:39.13 jobqueue.pl Here is some more info. [bhowmick@deathstar ~]$ cat jobqueue.txt After a restart top - 14:54:26 up 4 days, 20:34, 5 users, load average: 1.53, 1.55, 1.50 28825 root 15 0 193m 41m 2900 S 0.0 0.5 1:03.52 jobqueue.pl After running for 24 hours top - 08:44:07 up 5 days, 14:24, 2 users, load average: 1.44, 1.72, 1.99 28825 root 15 0 2351m 2.1g 2900 S 0.0 27.5 8:39.13 jobqueue.pl top - 08:19:41 up 6 days, 13:59, 2 users, load average: 1.74, 1.76, 1.79 18365 root 15 0 2354m 2.2g 2924 S 0.0 27.6 7:29.40 /var/www/html/bugzilla/jobqueue.pl top - 08:19:54 up 42 days, 15:51, 1 user, load average: 2.21, 2.11, 2.17 30641 root 15 0 1180m 1.0g 2880 S 0.0 12.9 7:07.45 /var/www/html/bugzilla/jobqueue.pl

Putting this on our radar for a fix, or at least an investigation.

Okay, I can confirm that each message sent through jobqueue.pl increases its memory size by a very small amount--maybe 4K or 8K. I'll investigate the cause of the problem.

It's not each message, it seems somewhat random; the process increases by 4K or 8K after a seemingly random number of messages are sent.

My investigations didn't get me anywhere. If anybody wants to help find the memory leak here, I'd be really grateful.

Bugzilla 3.4 is now restricted to security bugs. We will retarget this bug (to 3.6 or later) when it's fixed.

Not a hard blocker. We will take it when it's ready.

I have found it!!! $email->walk_parts leaks.

For those curious, here is now I instrumented jobqueue.pl to detect the memory leak. Most of my time was spent polishing up Bugzilla::Leak to work on arrays--after that. Once Bugzilla::Leak actually worked, the leaking object was somewhat obvious. I also had to modify TheSchwartz itself to sleep if the job's $class->is_sleeping returned true, or it would kill the job before it was done dumping, and try to do other work.

I've filed an upstream bug here: https://rt.cpan.org/Ticket/Display.html?id=59581

Excellent news Max. So it is a matter of waiting til a newer fixed version is available from CPAN or is it possible to subclass Email::MIME and fix the issue on Bugzilla end? We will volunteer to help by re-enablingthis feature when a fix is available and report back. Dave

Well, rjbs (the maintainer of Email::MIME) said that he'd get right on it. So I'm going to wait to see if he can fix it relatively soon, and then we can just up our requirement.

Okay, so Email::MIME 1.904 fixes this problem. Anybody upgrading to that version will immediately have this issue fixed for them. Since this is such a significant upgrade to, essentially, a cutting-edge version of a module, I don't want to check it in on the 3.6 branch--I don't like changing requirements that drastically for stable branches. However, we can tell anybody who's hit by this that all they have to do is update Email::MIME to 1.904 and they're good to go.

We should wait a few days to check this in, because 1.904 won't show up on CPAN mirrors for a little while.

Comment on attachment 472227 [details] [diff] [review] v1 You also have to remove Email::MIME::Encodings and Email::MIME::Modifier from docs/en/xml/installation.xml around line 360. Otherwise looks good. This will also fix bug 65477 comment 83.

Ah, right, thanks for catching that. Here's a fixed patch.

Comment on attachment 472577 [details] [diff] [review] v2 r=LpSolit

Committing to: bzr+ssh://bzr.mozilla.org/bugzilla/trunk/ modified Bugzilla/Mailer.pm modified Bugzilla/Install/Requirements.pm modified docs/en/xml/installation.xml Committed revision 7464. Committing to: bzr+ssh://bzr.mozilla.org/bugzilla/4.0/ modified Bugzilla/Mailer.pm modified Bugzilla/Install/Requirements.pm modified docs/en/xml/installation.xml Committed revision 7395.