symbols_camino on the symbol server is using 41GB of space. I'm not sure anything has even been cleaned up there, since there are 1,994 -symbols.txt files. (In comparison, Firefox has 1,569 across all branches and platforms.) I think, since the -symbols.txt files are not named the same as the ones for other apps, that our existing cleanup script will not quite work: http://hg.mozilla.org/build/tools/file/tip/buildfarm/breakpad/cleanup-breakpad-symbols.py However, it could probably be easily adapted to work. I think this needs to be fixed by a cleanup cron job, and not manually, since otherwise we'll just wind up back here again in a few months.

> I think, since the -symbols.txt files are not named the same as the ones for > other apps, that our existing cleanup script will not quite work Do you know what CL that format was changed in? I could merge that into our copy of the symbol generation script.

(In reply to comment #0) > anything has even been cleaned up there, since there are 1,994 -symbols.txt > files. (In comparison, Firefox has 1,569 across all branches and platforms.) Note that the -symbols.txt file count will be 2x the number of complete sets of symbols stored, since we have to do a 2-pass symbol-generation to make things work with Xcode- and make-built binaries and then have to upload 2 single-arch archives. That means we have 997 symbol-sets across three branches, which is still not awesome, but not as bad as just comparing -symbols.txt file counts might make it seem ;) (In reply to comment #1) > Do you know what CL that format was changed in? I could merge that into our > copy of the symbol generation script. The symbol archive filename is set in our Makefile (and we'll never be able to make it match exactly, because we have to do the 2-pass thing and have to use CPU arch in the filenames to not clobber them, whereas OS_ARCH for Firefox et al. means "Darwin"). We could, however, re-order the filename parts to put CPU arch where Core has OS_ARCH, and the cron job could just set OS_ARCH for Camino to "ppc,i386" when it calls the cleanup script. It would mean that the filenames don't sort by date any more, but since the only thing that sees the filenames is this cleanup script and my wiki links, I don't think that's probably a problem.

(In reply to comment #2) > We could, however, re-order the filename parts to put CPU arch where Core has > OS_ARCH, and the cron job could just set OS_ARCH for Camino to "ppc,i386" when > it calls the cleanup script. Ted, if the cron job can be set up to call the script with values for the OS_ARCH (not sure if the cron job is in VCS anywhere for examination), then I've opened bug 599578 on our end to make it easier going forward.

The cron job just runs the script linked in comment 0, passing it the directory to clean up. The script mostly doesn't care what the actual values are in the OS_ARCH fields etc, it just uses them as distinct values to put sets of builds into bins.

So as long as we get the buildid to be the fourth value in the filename, the script will be happy and just work, at least going forward? (My python is pretty useless, but I think that's what I'm seeing.)

That sounds right, yes.

OK, I landed bug 599578 on both our active branches, so tomorrow's nightlies will start using app-version-arch-buildid-symbols.txt for the filename structure; going forward any cron run should just work when passed the symbols_camino directory. To clean up our existing 999 symbol-sets, someone will need to do a one-time run with a modified version of cleanup-breakpad-symbols.py that looks for the buildid in the third position instead of the fourth, and I guess then another one-time run 30 days later to clean up the last 30 days of nightlies using the old filename.

Could you instead do a one-off mass-rename to get the files in the right format, so then the script would work as intended? Something like this should do the trick: for x in *-i386-symbols.txt *-ppc-symbols.txt; do mv $x `echo $x | awk -F- '{ print $1"-"$2"-"$4"-"$3"-"$5 }'`; done (I'd run it myself, but I don't have write permission to symbols_camino.)

(In reply to comment #8) > Could you instead do a one-off mass-rename to get the files in the right > format, so then the script would work as intended? Done; thanks for the command :) Back to default owner for the cron part.

Aravind, is there a cron job for cleaning up symbols_camino? (probably in caminobld's cron)

Yup, runs every evening around 7:10 PM. 10 19 * * * /usr/local/bin/clean_symbols.py /mnt/breakpad_symbols/symbols_camino

Nothing left to do here, then.