In July last year the test zips for debug linux64 builds were 107MB. Today we're up to 261MB. e.g. http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux64-debug/1374551088/firefox-25.0a1.en-US.linux-x86_64.tests.zip vs http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux64-debug/1390451117/firefox-29.0a1.en-US.linux-x86_64.tests.zip The larger these are, the more time we spend uploading/downloading/extracting them in our automation. Can we make an effort to shrink these down?

The increase is probably mostly from bug 811404, see bug 811404 comment 15. However, supposedly we fixed that in bug 889078, bug 889078 comment 7 indicates the zip size was back to normal. Did we regress this at some point?

Bug 920055 made some changes to how the stripping is done. Maybe we're missing some flags?

IIRC the files were as big as 450MB when the striping wasn't working.

So apparently bug 920055 regressed this. Compare: http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux64-debug/1380330154/ (from 79cba73c77ba, a few changesets before that patch) [ ] firefox-27.0a1.en-US.linux-x86_64.tests.zip 28-Sep-2013 02:32 111M and: http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux64-debug/1380333677/ (c868eb8d83fd, last changeset in that push) [ ] firefox-27.0a1.en-US.linux-x86_64.tests.zip 28-Sep-2013 03:23 253M

bug 963651 should shrink them back down by about 100MB on debug Linux. I think splitting them into per-testsuite packages is the next logical step but requires a bit more work.

ipsec issues resolved without needing to implement this, no longer blocks

While working on some code for bug 1059943, I discovered we have quite a bit of duplicate data in the test zips. e.g. https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1416229704/firefox-36.0a1.en-US.win32.tests.zip is 120MB. Contained within is 30MB of duplicate data. If we remove these duplicate files and re-compress we get the zip down to 107MB. There are 4,414 files that are duplicated within this archive. I'll attach a report shortly.

The format of the file is <number of duplicate bytes> [list of duplicate files] The number of duplicate bytes is sizeof(files[0]) * (len(files) - 1)

S3 all of the things and transfer a manifest of the files?

That is exactly bug 1059943 :)