Nagios is warning about this: ***** Nagios ***** Notification Type: PROBLEM Service: disk - /mnt/netapp/breakpad Host: dm-symbolpush01 Address: 10.2.74.40 State: WARNING Date/Time: 08-22-2010 15:15:23 Additional Info: DISK WARNING - free space: /mnt/netapp/breakpad 107076 MB (10% inode=94%): We have cleanup scripts that should be keeping things in check, but we've added a few more directories lately and maybe they're not being cleaned up. We need to sanity check that things are being cleaned up and then assess what our disk usage looks like. With the additional symbols being uploaded from things like Ubuntu it's possible we'll need to bump up the storage a bit. Nick: were you the one that managed the cleanup scripts in the past?

The script that does the work lives at http://hg.mozilla.org/build/tools/file/default/buildfarm/breakpad/cleanup-breakpad-symbols.py Aravind has been maintaining the cron jobs that call it, not sure where. Aki suspects symbols_mob is not being cleaned up - there's 1742 txt files there, compared to 1575 in symbols_ffx. Note that we enabled symbols for Firefox mac64 a week or so ago, so symbols_ffx will be using a bit more space than it did.

(In reply to comment #1) > The script that does the work lives at > http://hg.mozilla.org/build/tools/file/default/buildfarm/breakpad/cleanup-breakpad-symbols.py > > Aravind has been maintaining the cron jobs that call it, not sure where. From this, comments in email before this bug was filed, and now comments in irc by nthomas about RelEng not having access, I'm pushing this bug to ServerOps. If that is not correct component, please reassign this bug back or redirect. Sorry for the bouncing around - I'm still trying to figure out who owns this. > Aki suspects symbols_mob is not being cleaned up - there's 1742 txt files there, > compared to 1575 in symbols_ffx. Note that we enabled symbols for Firefox mac64 > a week or so ago, so symbols_ffx will be using a bit more space than it did. Sounds plausible.

The cron jobs that control the cleanup are all running on surf. Here is the list of current jobs. Please let us know if we need to add/remove stuff. [root@surf cron]# grep symbol * calbld:# Clean up symbols every night. calbld:10 16 * * * /usr/local/bin/clean_symbols.py /mnt/breakpad_symbols/symbols_sbrd ffxbld:# Clean up symbols every night. ffxbld:10 22 * * * /usr/local/bin/clean_symbols.py /mnt/breakpad_symbols/symbols_ffx seabld:# Clean up symbols every night. seabld:10 01 * * * /usr/local/bin/clean_symbols.py /mnt/breakpad_symbols/symbols_sea tbirdbld:# Clean up symbols every night. tbirdbld:10 0 * * * /usr/local/bin/clean_symbols.py /mnt/breakpad_symbols/symbols_tbrd xrbld:10 23 * * * /usr/local/bin/clean_symbols.py /mnt/breakpad_symbols/symbols_xr [root@surf cron]# @joduinn: This should go to Ted for the final word.

So we're not cleaning up the following directories: symbols_camino symbols_mob Camino has txt files going back to May 1 2009, but a dry run doesn't actually want to remove anything. mobile goes back to May 14 this year, and dry run reports files that would save 80G of space. We're also not cleaning these, but there's not much in there symbols_fedora symbols_opensuse symbols_ubuntu symbols_penelope symbols_solaris and this isn't really touchable symbols_os

Sounds like we should get cleanup jobs set on basically everything there, so we can avoid repeats. symbols_mob seems like the obvious first candidate. I expect symbols_ubuntu will start to grow since they're starting to actively upload symbols now. symbols_os is slowly growing (I have a nightly script that tries to fetch missing Windows symbols from the Microsoft symbol server), but it's not something we can easily clean up, since figuring out what we're actually using in there is probably hard. It may be ratcheting up our baseline usage a bit, as may the addition of these other directories like Ubuntu. If we get cleanup jobs started for these other dirs, and our usage is still high, we may just need to add some more space.

Oh, also, every time we add a project branch (like JaegerMonkey), we bump our disk usage up a little bit. If we need to set a new baseline we should be able to look at our number of active branches and do the math to figure out how much space should be required.

Will the manifests for the Linux distros be named correctly for the script ? I'm seeing lots of linux style package versioning in the opensuse dir and no use of 'pre'. Likely to be just release versions ?

Wolfgang Rosenaur is the contact for OpenSuSE, and Chris Coulson is the contact for Ubuntu.

(In reply to comment #7) > Will the manifests for the Linux distros be named correctly for the script ? > I'm seeing lots of linux style package versioning in the opensuse dir and no > use of 'pre'. Likely to be just release versions ? Currently there are only released versions in the symbolstore. I'm going to add the FF4beta stuff soon if possible. Let me know if you need explanations or changes to the current setup. Some notes: We currently have every released version for several distributions (1110, 1120, 1130) and we can have several revisions for versions (e.g. with different patchset).

(In reply to comment #3) > The cron jobs that control the cleanup are all running on surf. Here is the > list of current jobs. Please let us know if we need to add/remove stuff. > > [root@surf cron]# grep symbol * > calbld:# Clean up symbols every night. > calbld:10 16 * * * /usr/local/bin/clean_symbols.py > /mnt/breakpad_symbols/symbols_sbrd > ffxbld:# Clean up symbols every night. > ffxbld:10 22 * * * /usr/local/bin/clean_symbols.py > /mnt/breakpad_symbols/symbols_ffx > seabld:# Clean up symbols every night. > seabld:10 01 * * * /usr/local/bin/clean_symbols.py > /mnt/breakpad_symbols/symbols_sea > tbirdbld:# Clean up symbols every night. > tbirdbld:10 0 * * * /usr/local/bin/clean_symbols.py > /mnt/breakpad_symbols/symbols_tbrd > xrbld:10 23 * * * /usr/local/bin/clean_symbols.py > /mnt/breakpad_symbols/symbols_xr > [root@surf cron]# > > @joduinn: This should go to Ted for the final word. Ted: can you drive this, please?

It's hard for me to do anything other than offer advice. I didn't setup the cleanup jobs, and I don't have root access on that box to do so. We should be running a cleanup job on every directory on that mount. Once we have that accomplished, we can reassess our baseline usage.

I added two new clean up scripts that should run tonight. One of them (camino) probably won't clean up much (per nick in comment 4). Here are the current cleanup scripts. calbld:# Clean up symbols every night. calbld:10 16 * * * /usr/local/bin/clean_symbols.py /mnt/breakpad_symbols/symbols_sbrd caminobld:10 19 * * * /usr/local/bin/clean_symbols.py /mnt/breakpad_symbols/symbols_camino ffxbld:# Clean up symbols every night. ffxbld:10 22 * * * /usr/local/bin/clean_symbols.py /mnt/breakpad_symbols/symbols_ffx ffxbld:10 21 * * * /usr/local/bin/clean_symbols.py /mnt/breakpad_symbols/symbols_mob seabld:# Clean up symbols every night. seabld:10 01 * * * /usr/local/bin/clean_symbols.py /mnt/breakpad_symbols/symbols_sea tbirdbld:# Clean up symbols every night. tbirdbld:10 0 * * * /usr/local/bin/clean_symbols.py /mnt/breakpad_symbols/symbols_tbrd xrbld:10 23 * * * /usr/local/bin/clean_symbols.py /mnt/breakpad_symbols/symbols_xr What other directories need clean up scripts?

We'll need to figure out symbols_{fedora,opensuse,ubuntu}, but they may not have the right files in place for cleanup right now. We can move that to a separate bug if that would make life easier. What's our disk usage look like with those in place?

bug 598757 has more action going on.