We talked about how to improve the push to mirrors push time in the post mortem yesterday. Two ideas we came up with: 1) Symlink all the files we care about from releases -> candidates in the "push to mirrors builder". Replace them with real copies of the files in "postrelease". If we do this, we need to take care to do it atomically. 2) Hardlink files from releases -> candidates. This is fast and saves disk space. There's some leg work if we do it though, because these directories aren't on the same filesystem at the moment.

Either of these options will require changes to the pushToMirrors method of the stage-tasks.py script: https://github.com/mozilla/build-tools/blob/master/scripts/release/stage-tasks.py#L101. Basically, one will need to replace the "rsync" command with a "find" command that filters out what we don't want and does the linking. Something like: find /pub/mozilla.org/firefox/candidates/17.0b1 -not -name blah -and -not -name ack [...] -exec ln -s {} /pub/mozilla.org/firefox/releases/17.0b1/{}

This would be a really nice to have, but not a priority.

Bug 988288 has put firefox/candidates back on the same partition as firefox/releases, and mobile, seamonkey, thunderbird, and xulrunner are in that state too, so hardlinks can be done now. An example rsync command argument is rsync -av --link-dest=/path/to/candidates/29.0b5-candidates/build1/ \ /path/to/candidates/29.0b5-candidates/build1/ /path/to/releases/29.0b5/

Morphing to chose hardlinks as the solution. We should add cleanup of candidates later, since hardlinks allow for accidental modification of releases/ by futzing around in candidates/ (unusual, but not unheard of). We have to stop using old candidates dirs when making new releases first though.

With the ever-onward growth of releases bumping this vol up, this bug may need to be bumped in priority a touch. And/or we might need a tangential plan for fragging off some of the older releases, sometime in the next year. Not a crisis.

Still need to be tested in staging, but the overall idea worked running the commands from shell. 1) copy candidates/xxx/buildN to candidates/xxx/.buildN using "cp -al" 2) delete not needed files by "rsync ... --delete-excluded" 3) move the "hidden" directory to its final destination

I ran the commands manually in staging and they took 2-3 seconds. \o/

Comment on attachment 8484490 [details] [diff] [review] hardlinks-tools.diff Another incoming

http://dev-master1.srv.releng.scl3.mozilla.com:8444/builders/release-mozilla-beta-push_to_mirrors/builds/6 Elapsed 38 secs

the log output

WCPGW?!! sent 51883 bytes received 4442 bytes 112650.00 bytes/sec total size is 32794695723 speedup is 582240.49 real 0m0.100s user 0m0.014s sys 0m0.033s

Comment on attachment 8484578 [details] [diff] [review] hardlinks2-tools.diff Looks good to me. You could test it out on http://dev-stage01.build.mozilla.org/pub/mozilla.org/firefox/candidates/31.0.0esr-candidates/

Comment on attachment 8484578 [details] [diff] [review] hardlinks2-tools.diff https://hg.mozilla.org/build/tools/rev/e44e9e285864 worked as expected in staging.

http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/33.0b2-candidates/build1/logs/release-mozilla-beta-push_to_mirrors-bm84-build1-build13.txt.gz Elapsed 50 sec Win! \o/