Another report related to AMD graphics and bug 844156/bug 772330. Currently we check correlations when we see a topcrash. But I suspect that we may actually have these crashes in other builds, but they are affecting code locations which run less often, or cause crashes where the signature is more random. What I'd like to do is have a report for each beta and release which spits out the signature of any crash signature which correlates highly with AMD drivers, whether or not it is a topcrash. Limiting factor: * windows-only * For now, correlate only on AdapterVendorID: this is in postgres in the app_notes field, or directly in the raw annotations in hbase There is no point in running this report for nightly or Aurora: the signatures are likely to change on every build, and the populations there aren't likely large enough to give us good statistics for single days. This does need to be a report specific to a single build. I suggest we run it: 1 week after each beta is released, over the data from the release date (1 week) 1, 2, and 4 weeks after a release, over the data from the past week

Oh sorry, important datapoint: for the AMD crashes in question, the AdapterVendorID is usually "0x1002".

Working on this today. I have a set of queries from the last correlations I worked on that can mostly be pressed into service. Should have some data by about noon PT today.

A little analysis on the data I gathered signature F1398665248_____________________________ really jumped out. selena=# select signature, product_version,total_crashes, crash_by_signature_and_vendorid, "P(sig, vendid) / (P(sig) * P(vendid))" from crashes_by_amd where crash_by_signature_and_vendorid > 100 order by "P(sig, vendid) / (P(sig) * P(vendid))" desc; -[ RECORD 1 ]-------------------------+--------------------------------------------------------------------------------------------------- signature | F1650855128_____________________________ product_version | 20.0 total_crashes | 12803 crash_by_signature_and_vendorid | 108 P(sig, vendid) / (P(sig) * P(vendid)) | 4.667517317 -[ RECORD 2 ]-------------------------+--------------------------------------------------------------------------------------------------- signature | @0x0 | __RtlUserThreadStart | _RtlUserThreadStart product_version | 20.0 total_crashes | 12803 crash_by_signature_and_vendorid | 106 P(sig, vendid) / (P(sig) * P(vendid)) | 1.492998807 -[ RECORD 3 ]-------------------------+--------------------------------------------------------------------------------------------------- signature | @0x0 | __RtlUserThreadStart | _RtlUserThreadStart product_version | 19.0 total_crashes | 33303 crash_by_signature_and_vendorid | 169 P(sig, vendid) / (P(sig) * P(vendid)) | 1.232001035 -[ RECORD 4 ]-------------------------+--------------------------------------------------------------------------------------------------- signature | F1398665248_____________________________ product_version | 18.0.2 total_crashes | 4175 crash_by_signature_and_vendorid | 464 P(sig, vendid) / (P(sig) * P(vendid)) | 1.162656008 -[ RECORD 5 ]-------------------------+--------------------------------------------------------------------------------------------------- signature | F1531689212_______________________________________________________________________________________ product_version | 19.0 total_crashes | 33303 crash_by_signature_and_vendorid | 133 P(sig, vendid) / (P(sig) * P(vendid)) | 1.143797300 -[ RECORD 6 ]-------------------------+--------------------------------------------------------------------------------------------------- signature | F1398665248_____________________________ product_version | 19.0.1 total_crashes | 756 crash_by_signature_and_vendorid | 120 P(sig, vendid) / (P(sig) * P(vendid)) | 1.122744487 -[ RECORD 7 ]-------------------------+--------------------------------------------------------------------------------------------------- signature | F_11935219________________________________________________ product_version | 19.0 total_crashes | 33303 crash_by_signature_and_vendorid | 165 P(sig, vendid) / (P(sig) * P(vendid)) | 1.075617654 -[ RECORD 8 ]-------------------------+--------------------------------------------------------------------------------------------------- signature | F1398665248_____________________________ product_version | 19.0 total_crashes | 33303 crash_by_signature_and_vendorid | 3071 P(sig, vendid) / (P(sig) * P(vendid)) | 1.067251971 -[ RECORD 9 ]-------------------------+--------------------------------------------------------------------------------------------------- signature | F2005414273____________________________________________________________________________________ product_version | 19.0 total_crashes | 33303 crash_by_signature_and_vendorid | 192 P(sig, vendid) / (P(sig) * P(vendid)) | 1.026301914 -[ RECORD 10 ]------------------------+--------------------------------------------------------------------------------------------------- signature | F1398665248_____________________________ product_version | 20.0 total_crashes | 12803 crash_by_signature_and_vendorid | 921 P(sig, vendid) / (P(sig) * P(vendid)) | 0.977953676 -[ RECORD 11 ]------------------------+--------------------------------------------------------------------------------------------------- signature | F_855855466_____________________ product_version | 19.0 total_crashes | 33303 crash_by_signature_and_vendorid | 186 P(sig, vendid) / (P(sig) * P(vendid)) | 0.903953422

I'm attaching the quick and dirty script I used to limit the time spent on the secondary server. These queries are quite lengthy when run together, but are ok if limited to one version at a time. Looking forward to having a reporting server where long running queries don't set off alarms!!

Selena, the F* signatures are Flash crashes, which are mostly nothing we ourselves can do anything about, so while they are an interesting data point, I think Benjamin also wants to know about others. Also, why are the P() values larger than 1?

(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #7) > why are the P() values larger than 1? See https://bugzilla.mozilla.org/show_bug.cgi?id=797068#c19 for a detailed explanation of the proability ratio.

* There's absolutely no need to run this on multiple versions: we'll know externally what release or beta we're trying to test. * This is primarily for browser crashes, not plugin crashes, so I'm going to remove the limiters. Have you committed these scripts to any repos? I'll play with them and stick them in socorro-toolbox unless you have a better place for them.

Back to me, using https://github.com/bsmedberg/socorro-toolbox/compare/correlationscripts to generate better data now.

I can run these as-needed.