On OS X we currently package the .dSYM files with the Breakpad symbols and upload them to the symbol server. Unfortunately right now the .dSYM files are universal for our universal builds (they contain both ppc and x86), but we upload two copies of each. This is lame. We should instead be running "dsymutil --arch=ARCH".

This is causing us to use 616MB(!) per symbol upload on the symbol store. (Each copy of XUL.dSYM.tar.bz2 is 254MB.)

This is responsible for about 300GB of extra disk usage, I estimate.

(In reply to comment #0) > universal for our universal builds (they contain both ppc and x86), but we > upload two copies of each. This is lame. We should instead be running "dsymutil > --arch=ARCH". http://mxr.mozilla.org/mozilla-central/source/Makefile.in#151 http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/symbolstore.py#710 makes it look to me like you're passing in two archs and translating -a foo to the --arch=foo syntax dsymutil is expecting. What am I missing? Or are you actually running symbolstore.py against each thin binary (I thought that was only something we had to do in Camino to work around assumptions in the build system, and other Gecko apps acted on the fat binary)?

We pass it all the arches, so we call "dsymutil --arch=ppc --arch=i386 whatever", thus getting a universal .dSYM. And then we copy it to the symbol dir twice, because we loop over each arch and handle copy_debug inside there: http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/symbolstore.py#554

(In reply to Ted Mielczarek [:ted] from comment #1) > This is causing us to use 616MB(!) per symbol upload on the symbol store. > (Each copy of XUL.dSYM.tar.bz2 is 254MB.) It's almost 1GB, now.

Okay, I've run this past tryserver and it seems to work. The meat of this patch is just changing ProcessFile to use enumerate for looping over archs, and only calling CopyDebug for the first architecture. The rest is mostly tweaks so the unit tests work properly.

Comment on attachment 634103 [details] [diff] [review] stop uploading duplicate dSYM files in the symbol package >diff --git a/toolkit/crashreporter/tools/symbolstore.py b/toolkit/crashreporter/tools/symbolstore.py > print >> sys.stderr, "Processing file: %s" % file >+ sys.stderr.flush() Oh yes! >- if self.copy_debug: >+ # only copy debug the first time if we have multiple architectures >+ if self.copy_debug and arch_num == 0: > self.CopyDebug(file, debug_file, guid) Was hoping we could short circuit processing both XUL files for a big time win, but see they yield different sym files. Oh well. >diff --git a/toolkit/crashreporter/tools/unit-symbolstore.py b/toolkit/crashreporter/tools/unit-symbolstore.py All the stubbing out made my brain hurt, so I'm relying on that try run for this.

Right, since it's a universal binary we really do need to dump both architectures. This was the least invasive way I found to accomplish what I wanted to do. Thanks for the reviews!

Looks like a nice savings (~400MB per build): $ ~/totalsize.py ./firefox-16.0a1-Darwin-20120620030535-x86-macosx64-symbols.txt total: 1069482711 bytes (1019.94MB, 1.00GB) $ ~/totalsize.py ./firefox-16.0a1-Darwin-20120621030536-x86-macosx64-symbols.txt total: 620038212 bytes (591.31MB, 0.58GB)

Tiny followup because I lost the bit where we redirected the stdout of dsymutil to /dev/null: http://hg.mozilla.org/integration/mozilla-inbound/rev/32a058c88ce8