We have trouble aggregating bugs just based on addresses, as those change between versions of OS X. For example, bug 1176404.

Ted, I understand you know the steps that need to be completed for this to be done? Are they written down somewhere we can follow? (I admit I may have asked this before, and you may have answered, but I can't find it :)

I did some work in bug 951229: https://github.com/luser/breakpad-mac-update-symbols That repository contains a Dockerfile for building a Docker image that can run Reposado to mirror Apple update packages, and then run some scripts on them to dump symbols out of the system libraries in them. I ran it in Taskcluster once, but I didn't get a good setup to make it runnable as a cron job. I think it would be fairly straightforward to do that, I just haven't found time.

I'm calling this a blocker as it's a dependency for a series of topcrashes which has been affecting at least 7% of our Mac users for a few releases.

I'm switching this to the General component since it's not an infra issue.

Alright made some good progress on this today with some help from :ted. Yosemite + changed the update Payload form to 'pbzx' format which we just failed on and it got lost in the multi-hour multi-megabyte log file and explains why I wasn't getting new symbols. We've merged 2 PR to fix issues and the new one should be merged soon: https://github.com/luser/breakpad-mac-update-symbols/pull/3 I'm going to re-run the script which will take several hours. We will know if there's anything else that's not working. We might also use this chance to have a hardcoded 'processed-packages' which will allow bootstrapping the process and save several hours to the next person to run this script.

I've got 52MB of missing symbols. How do I submit them to crash stats?

If you put the zip file somewhere I can get to it I will upload them for you.

I've emailed you a link.

I uploaded the symbols that Benoit dumped. Here's the list of files.

Very cool! This means I will eventually be able to see crashes like this one: https://crash-stats.mozilla.com/report/index/fdb283cf-8f1d-4417-ad27-ed4dc2160906 with actual symbols instead of @ addresses? Is there a way to reprocess some of the crash reports for that to happen, or does it only work on the new ones?

If you are logged in (and maybe if you have crash dump access?) then there is a "reprocess" tab, at least for one-off you can do it yourself. I requested it for the one you linked. I think larger scale reprocessing requires making a request of the Socorro people.

Yes, that particular crash report should have symbols now. If mccr8 reprocessed it it should show up with symbols modulo caching in the web app. There's a planned Socorro feature for "reprocess a set of crashes", which hasn't been done yet, but peterbe of the Socorro team also has a little Python script that uses the Socorro API to reprocess a set of crashes: https://github.com/peterbe/reprocess-supersearch I used this to reprocess some Win64 crashes not long ago and it worked well.

(The report/index page is cached for 1 hour, FYI.)

I filed bug 1301471 on getting that script running regularly.

Let me know if there's anything missing after it's reprocessed.

The signature is now [@ shutdownhang | __psynch_cvwait | <name omitted> | nsThread::ProcessNextEvent | NS_ProcessNextEvent | nsThread::Shutdown | nsThreadManager::Shutdown ] so it looks like the missing symbol is there now. It doesn't actually look that helpful but maybe others will be. :)

Andrew, do you know why this crash: https://crash-stats.mozilla.com/report/index/15616700-1274-4886-8393-89e392161023 still shows up as libsystem_kernel.dylib@0x16286 instead of with symbols?

I don't know anything about how these symbols work. Maybe Ted has some ideas. Though if I had to guess, maybe it is because the script mentioned in comment 14 hasn't been run in a while. :)

FYI bug 1301471 covers running this on a regular basis.