looks like maybe the cron job didn't run, or we ran out of disk space or something. the confidential url report did get produced and placed in its usual location under cm-fs01://data so some parts of the cron job seem to be working.

The server that runs this cron stopped responding around 3 AM and was rebooted at 4 AM. We suspect that this is the first bottleneck we've seen from having Fx4 with throttling off (bug 643661). The hypothesis is that the script is using up all available memory. jabba set up one of the processors to run this job instead, we're going to test it out today. This should help us test the hypothesis, and if it correct I will do some profiling and code inspection to see if we can make the script use less memory.

This failed yesterday as well, so jabba is running it on a different box while I am collecting some basic data (grabbing memory info every few seconds, to pinpoint which script(s) are the hogs). The original box (sp-admin01) is slated to replaced, and has 8GB of RAM. The processor we've taken over to run this (sp-processor10) has 24GB of RAM. The processor is not doing anything else (the processor service has been stopped).

I'm rerunning the cron manually on the processor box for yesterday and today and will set it up to cron at midnight tonight like normal. I'll still need to manually scp the data over to people.m.o after I'm finished with the manual crons (could take a few hours).

Let's spin the profiling work out into another bug and have it block bug 635098.

(In reply to comment #4) > Let's spin the profiling work out into another bug and have it block bug > 635098. Done (bug 645704)

We re-ran the cron jobs and copied the files over directly. Tomorrow's cron should run correctly, but if not, we can re-push the files again manually.