Right now we backup the whole (releases) snippet store each time we want to push out more release snippets, but it takes around 50 minutes to pull tens of thousands of small files in a deep dir structure from the NFS mount. We should investigate doing this smarter, eg making a daily full backup and teaching backupsnip to only backup the directories/snippets that are going to change. This should be muuuuuch master, and may allow us to enforce that backupsnip is run before every snippet push (eg 3.5.2 followed by 3.0.13 followed by 3.0.13 MU).

In addition to the smart-backup strategy of only backing up things we're changing, we could also start excluding old snippets. For example, the most recent backup has 232968 directories and files, and 56% of those are for the discontinued versions - Firefox 1.5 & 2.0, plus Thunderbird 1.5 (most of that is from the Fx2 partner nightmare). So there's a better than 2x speed up just from making a one-off backup and then excluding those directories in backupsnip. Comments ?

(In reply to comment #1) > In addition to the smart-backup strategy of only backing up things we're > changing, we could also start excluding old snippets. For example, the most > recent backup has 232968 directories and files, and 56% of those are for the > discontinued versions - Firefox 1.5 & 2.0, plus Thunderbird 1.5 (most of that > is from the Fx2 partner nightmare). So there's a better than 2x speed up just > from making a one-off backup and then excluding those directories in > backupsnip. Comments ? r=bhearsum on excluding snippets for any unsupported products

The old snippets are still present, and are backed up at /opt/aus2/snippets/backup/20091129-FX15-FX20-TB15-FINAL-BACKUP.tar.bz2 but we won't include them in any future backupsnip's. I tried defining EXCLUDES="--exclude='Firefox/1.5* ..." but the shell was breaking the quoting so I gave it away. Also adds 'time' since we do that frequently. Checking in backupsnip; /cvsroot/mozilla/tools/release/bin/backupsnip,v <-- backupsnip new revision: 1.3; previous revision: 1.2 done aus2-staging and staging-stage updated.

37 minutes for a backupsnip run today, although it does depend a bit on how busy the nfs share is.

Status: some speedup achieved. There are basically three big chunks of data there now - Fx3.0, Fx3.5, and Tb2.0. The approach of comment #0 would maybe get a factor of 3 improvement.

This patch generates a list of all directories at depth 3 in the staging directory. This corresponds to the '20100115-Firefox-3.6rc2/Firefox/3.6b1/WINNT_x86-msvc', etc. directories. Directories that don't exist in the live snippets directory are filtered out of this list. We then tar up all these directories from the live snippets directory. Switched to using gz instead of bz2 since it's faster as well. runtime is a few minutes for 3.6 snippets, and 8 minutes for 3.5 snippets using this method. We'd have to generate daily backups, and document recovery procedure as well.

Comment on attachment 422236 [details] [diff] [review] only backup directories that have changed Looks good. We'll not save modification timestamps on Firefox/ and Firefox/$version but that seems unlikely to bite us in a recovery situation, which I imagine will be disable all/some updates, restore nightly backup somewhere, apply any updates you want to keep, swap out live directory.

Comment on attachment 422341 [details] Nightly backup script Did you decide to stick with bz2 here ?

(In reply to comment #9) > (From update of attachment 422341 [details]) > Did you decide to stick with bz2 here ? Yes, the speed here isn't as critical, and we get a small space savings.

Comment on attachment 422236 [details] [diff] [review] only backup directories that have changed Checking in backupsnip; /cvsroot/mozilla/tools/release/bin/backupsnip,v <-- backupsnip new revision: 1.4; previous revision: 1.3 done

Comment on attachment 422341 [details] Nightly backup script RCS file: /cvsroot/mozilla/tools/release/bin/backupsnip-nightly,v done Checking in backupsnip-nightly; /cvsroot/mozilla/tools/release/bin/backupsnip-nightly,v <-- backupsnip-nightly initial revision: 1.1 done

backupsnip for 3.5.8 took 17 minutes compared to 42 minutes for 3.5.7.