We observed the following crash signature during testing. [@ nsCOMPtr<imgIContainer>::nsCOMPtr | mozilla::dom::quota::QuotaManager::AbortCloseStoragesForProcess | mozilla::dom::ContentParent::ShutDownProcess | mozilla::dom::ContentParent::ActorDestroy ] Cafbot will upload the decoded minidump and extra file. This crash was produced during stability tests which involves monkey testing for several hours and there is no clear STR for this. If we are not able to identify the issue using provided logs then please feel free to provide us a debug patch with additional logging to identify the issue.

Observed on: Device: msm8909 Gonk Version: AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.123 Moz BuildID: 20150405002503 Manifest: https://www.codeaurora.org/cgit/quic/lf/b2g/manifest/tree/caf_AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.123.xml?h=release Gecko Version: 37.0 Gaia: http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=a6351e1197d54f8624523c2db9ba1418f2aa046f Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=6bb2afcce9872a7cbc65b4a58f752e2d5ac02345 Patches: bug 1145724, bug 1143694, bug 1146987, bug 1133398, bug 1152095, bug 1150924, bug 1133147, bug 1150271, bug 1150916

Hi Andrew, Please have Olli or someone else on the DOM team pick this up. This issue is directly impacting our ability to meet our MTBF goal per CAF's testing. Thanks, Mike

Probably the same issue as bug 1150023 or bug 1109363.

Yeah, something about QuotaManager. (I'm not familiar with that at all.)

triage: impacting MTBF

Andrew, do you know who can own this bug and take a look? This is a bug in CAF list with higher priority. Thank you.

I can take this.

Hey Jan, so you have an eta on the patch here? This is one of the critical blockers for FXOS and hence the eta would be helpful. Please note for crashes like this, debugging patches that can collect more information helpful for investigation are more than helpful.

I think this should be now fixed by patch in bug 1109363. Do you still see the crash with the patch landed ?

Observed on: Device: msm8909 Gonk Version: AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.129 Moz BuildID: 20150409002503 Manifest: https://www.codeaurora.org/cgit/quic/lf/b2g/manifest/tree/caf_AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.129.xml?h=release Gecko Version: 37.0 Gaia: http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=ea735c21bfb0d78333213ff0376fce1eac89ead6 Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=ac12d858a58574b2db482b5eed61750a3e210040 Patches: bug 1150923, bug 1133147, bug 1142806, bug 1148641, bug 1150924

(In reply to Jan Varga [:janv] from comment #13) > I think this should be now fixed by patch in bug 1109363. > Do you still see the crash with the patch landed ? actually, I meant bug 1150023 (In reply to cafbot (PoC: ggrisco) from comment #14) > Observed on: > > Device: msm8909 > Gonk Version: AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.129 > Moz BuildID: 20150409002503 I think that build doesn't contain the fix. The fix landed on m-c on 2015-04-13.

(In reply to Jan Varga [:janv] from comment #17) > (In reply to Jan Varga [:janv] from comment #13) > > I think this should be now fixed by patch in bug 1109363. > > Do you still see the crash with the patch landed ? > > actually, I meant bug 1150023 Janv, CAF is using this : https://hg.mozilla.org/releases/mozilla-b2g37_v2_2 as the gecko code base, so the patch needs to land there. Can you please request b2g37 approval on this and make sure it applies on this codebase? > > (In reply to cafbot (PoC: ggrisco) from comment #14) > > Observed on: > > > > Device: msm8909 > > Gonk Version: AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.129 > > Moz BuildID: 20150409002503 > > I think that build doesn't contain the fix. > The fix landed on m-c on 2015-04-13.

Observed on: Device: msm8909 Gonk Version: AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.134 Moz BuildID: 20150417002504 Manifest: https://www.codeaurora.org/cgit/quic/lf/b2g/manifest/tree/caf_AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.134.xml?h=release Gecko Version: 37.0 Gaia: http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=d50b8a3919a7b4d8d289f150d3b9bed704ebafa9 Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=87e2ee9470e81fafdd63d340c2770479b3152ccc Patches: bug 1154439, bug 1133147, bug 1152439, bug 1142806, bug 1150924

I submitted a backport patch in bug 1150023 along with the approval request.

Observed on: Device: msm8909 Gonk Version: AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.135 Moz BuildID: 20150420002502 Manifest: https://www.codeaurora.org/cgit/quic/lf/b2g/manifest/tree/caf_AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.135.xml?h=release Gecko Version: 37.0 Gaia: http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=c15a2b6d3a783813959c2b3bffd2a131f4270b9e Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=ed7fd6f380d24ed2c7cd918c0406d91ca6edc23f Patches: bug 1154439, bug 1154411, bug 1150924, bug 1133147, bug 1142806, bug 1152439

(In reply to cafbot (PoC: ggrisco) from comment #25) > Created attachment 8595335 [details] > decoded minidump - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.135 Hi! Greg, Just want to clarify, Is the patch of bug 1150023 in AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.135? -- Keven

Observed on: Device: msm8909 Gonk Version: AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.132 Moz BuildID: 20150413002502 Manifest: https://www.codeaurora.org/cgit/quic/lf/b2g/manifest/tree/caf_AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.132.xml?h=release Gecko Version: 37.0 Gaia: http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=cec00d643f517ffd96cde559cd3bbd43ab85816c Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=a47c1908cca4b834ae92e9a0507f09da46993f8b Patches: bug 1154439, bug 1133147, bug 1152439, bug 1142806, bug 1154411

Sorry, I can't tell if RyanVM's landing of Jan's backport in bug 1150023 is in the build referenced in comment 28.

(In reply to Keven Kuo [:kkuo] from comment #26) > (In reply to cafbot (PoC: ggrisco) from comment #25) > > Created attachment 8595335 [details] > > decoded minidump - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.135 > > Hi! Greg, > > Just want to clarify, > Is the patch of bug 1150023 in AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.135? > > -- > Keven I looked at the Gecko link from comment 23 and didn't see the change there yet, so I don't think we have it in AU 135 (which should also answer Andrew's question in comment 31).

Observed on: Device: msm8909 Gonk Version: AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.138 Moz BuildID: 20150423002502 Manifest: https://www.codeaurora.org/cgit/quic/lf/b2g/manifest/tree/caf_AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.138.xml?h=release Gecko Version: 37.0 Gaia: http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=b838d0e7c163e66660dcb6e387d8339944a7a30e Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=5f087089472477031462d7cc4cd5de1f90139258 Patches: bug 1154439, bug 1133147, bug 1152439, bug 1156503, bug 1154411

It looks like bug 1150023 is built into AU 138 where we still see the crash.

Observed on: Device: msm8909 Gonk Version: AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.139 Moz BuildID: 20150423002502 Manifest: https://www.codeaurora.org/cgit/quic/lf/b2g/manifest/tree/caf_AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.139.xml?h=release Gecko Version: 37.0 Gaia: http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=b838d0e7c163e66660dcb6e387d8339944a7a30e Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=5f087089472477031462d7cc4cd5de1f90139258 Patches: bug 1154439, bug 1133147, bug 1152439, bug 1156503, bug 1154411

Observed on: Device: msm8909 Gonk Version: AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.137 Moz BuildID: 20150420002502 Manifest: https://www.codeaurora.org/cgit/quic/lf/b2g/manifest/tree/caf_AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.137.xml?h=release Gecko Version: 37.0 Gaia: http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=c15a2b6d3a783813959c2b3bffd2a131f4270b9e Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=ed7fd6f380d24ed2c7cd918c0406d91ca6edc23f Patches: bug 1154439, bug 1133147, bug 1152439, bug 1156503, bug 1154411

Observed on: Device: msm8909 Gonk Version: AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.140 Moz BuildID: 20150423002502 Manifest: https://www.codeaurora.org/cgit/quic/lf/b2g/manifest/tree/caf_AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.140.xml?h=release Gecko Version: 37.0 Gaia: http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=b838d0e7c163e66660dcb6e387d8339944a7a30e Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=5f087089472477031462d7cc4cd5de1f90139258 Patches: bug 1154439, bug 1147763, bug 1154411, bug 1133147, bug 1152439, bug 1156503

Jan, can you provide an update on the activity here? From comment 37, bug 1150023 hasn't resolved the crash.

I think we should try to backport patches from bug 1156063 and bug 1157029.

Due to time constraints I submitted the patches here. They are intended for the b2g37_v2_2 branch. I only tested on mac by running: mach xpcshell-test dom/indexedDB mach mochitest-plain dom/indexedDB

Observed on: Device: msm8909 Gonk Version: AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.141 Moz BuildID: 20150427002504 Manifest: https://www.codeaurora.org/cgit/quic/lf/b2g/manifest/tree/caf_AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.141.xml?h=release Gecko Version: 37.0 Gaia: http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=265ca0bc9408c21fc4b25a259fcee7fb642cd06b Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=0f5d4b8958d8388ac5e8ea6753ce98e0f9d396d4 Patches: bug 1154439, bug 1057091, bug 1157029, bug 1154411, bug 1133147, bug 1156063, bug 1152439, bug 1156503, bug 1147763

Greg, can you confirm that attachment 8599522 [details] [diff] [review] and attachment 8599523 [details] [diff] [review] were applied to the build that failed in comment 52?

Silly me, I checked the code myself. Those patches are definitely not applied to the revision that failed in comment 52. Please retest with those applied?

The change went into AU144 last night. Comment 52 was from last build’s report. We should get report on the new build in a day or two.

How were the results over the weekend with those two patches applied?

(In reply to Dylan Oliver [:doliver] from comment #58) > How were the results over the weekend with those two patches applied? Another change is causing high instability, preventing us from determining whether these patches are working. We need to wait a day or two now before giving feedback here.

So far we haven't seen this reproduce on AU 148, but we still need to complete the test cycle. Should know better tomorrow. Adding ni? for myself to report back.

We had enough testing now to feel that this problem is fixed with the two patches applied. Can we proceed to land those patches?

(In reply to Greg Grisco from comment #61) > We had enough testing now to feel that this problem is fixed with the two > patches applied. Can we proceed to land those patches? We already tried to land them, but they caused a problem on ASAN builds. See https://bugzilla.mozilla.org/show_bug.cgi?id=1157029#c30

I relanded them, these patches weren't the problem. I recommend you apply: https://hg.mozilla.org/releases/mozilla-aurora/rev/720b046ff520 https://hg.mozilla.org/releases/mozilla-aurora/rev/7270d0935640

Thanks Jan & Ben. If these are relanded, can we then close this bug out? (dupe it to one of those patches or worksforme?)

It looks like Josh made a branch-specific patch at https://bugzilla.mozilla.org/show_bug.cgi?id=1156063#c38. Is that the same as the two referenced by Ben here?

Hi Ben, The patch in bug 1156063 seems similar to https://hg.mozilla.org/releases/mozilla-aurora/rev/720b046ff520 but not include https://hg.mozilla.org/releases/mozilla-aurora/rev/7270d0935640. Could you please help to confirm this? Thanks!

(In reply to Josh Cheng [:josh] from comment #66) > Hi Ben, > The patch in bug 1156063 seems similar to > https://hg.mozilla.org/releases/mozilla-aurora/rev/720b046ff520 this patch alone is not sufficient > but not include > https://hg.mozilla.org/releases/mozilla-aurora/rev/7270d0935640. you also need this one to fix the crash properly > > Could you please help to confirm this? > Thanks!

Hi Jan, Are fix in bug 1156063 same as https://hg.mozilla.org/releases/mozilla-aurora/rev/720b046ff520? Do we need to backout fix in bug 1156063 and land these 2? or just go and land https://hg.mozilla.org/releases/mozilla-aurora/rev/7270d0935640 will do?

Josh, I'm not sure what you're asking exactly. The two patches in comment 63 are the ones that I recommend you backport.

We're going in circles here but it looks to me like both of those patches are landed and the bugs are marked as status-b2g-v2.2:fixed. Can we close this out? https://bugzilla.mozilla.org/show_bug.cgi?id=1156063#c39 https://bugzilla.mozilla.org/show_bug.cgi?id=1157029#c34

It looks that way to me too.