We observed the following crash signature during testing. [@ mozilla::MediaStreamGraphImpl::AddBlockingRelatedStreamsToSet | mozilla::MediaStreamGraphImpl::AddBlockingRelatedStreamsToSet | mozilla::MediaStreamGraphImpl::AddBlockingRelatedStreamsToSet | mozilla::MediaStreamGraphImpl::RecomputeBlocking ] Cafbot will upload the decoded minidump and extra file.

Johnny & Andrew, Can you have someone(s) on the DOM team familiar with the media codebase look into this critical fxOS 2.2 crash asap. It's currently blocking CAF and is directly contributing to their inability to reach our MTBF goal. Thanks, Mike

Observed on: Device: msm8909 Gonk Version: AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.119 Moz BuildID: 20150330002503 Manifest: https://www.codeaurora.org/cgit/quic/lf/b2g/manifest/tree/caf_AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.119.xml?h=release Gecko Version: 37.0 Gaia: http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=473cd63f53c855299b719285d9b95e3f2910782f Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=e44e28d2a37258fe0c908c59f2e91e5287777b40 Patches: bug 1133398, bug 1143694, bug 1146987, bug 1145724, bug 1147646, bug 1133147, bug 1150271, bug 1142770

Hi roc, Could you please check this problem? I checked out b2g-2.2. It crashes at the first line of MediaStreamGraphImpl::AddBlockingRelatedStreamsToSet, "if(aStream->mInBlockingSet)". It looks like aStream has beeb deleted.

Bug 1113927 crashes at the same place.

ni padenot, too.

This also looks like an UAF (like 1150918). Having steps to reproduce would make this actionable. From the stacks, we can see that this is a video-only application (there is no audio flowing in the graph), but it's not clear to me what it is?

This crash was produced during stability tests which involves monkey testing for several hours and there is no clear STR for this. If we are not able to identify the issue using provided logs then please feel free to provide us a debug patch with additional logging to identify the issue.

(In reply to StevenLee[:slee] from comment #8) > Bug 1113927 crashes at the same place. Unfortunately we only have two crashes reported there, both in December last year. I've tried running dom/media/tests many times locally on desktop Linux, with chaos mode and --shuffle, with no success. (In reply to StevenLee[:slee] from comment #7) > Could you please check this problem? I checked out b2g-2.2. It crashes at > the first line of MediaStreamGraphImpl::AddBlockingRelatedStreamsToSet, > "if(aStream->mInBlockingSet)". It looks like aStream has beeb deleted. Can you explain the reasoning that lead to that conclusion? Can you tell from the disassembly and the crash dump that aStream is a non-null pointer to unmapped memory? Is this crash occurring frequently enough that we can add some logging to get some additional information?

(In reply to Inder from comment #11) > This crash was produced during stability tests which involves monkey testing > for several hours and there is no clear STR for this. You mean tests that drive the interface of a phone? So Gaia application code involved in triggering this?

(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #12) > (In reply to StevenLee[:slee] from comment #7) > > Could you please check this problem? I checked out b2g-2.2. It crashes at > > the first line of MediaStreamGraphImpl::AddBlockingRelatedStreamsToSet, > > "if(aStream->mInBlockingSet)". It looks like aStream has beeb deleted. > Can you explain the reasoning that lead to that conclusion? Can you tell > from the disassembly and the crash dump that aStream is a non-null pointer > to unmapped memory? I was guessing. From the crash dump below, it crashed at [1]. I assumed r1 and r2 are the arguments, nsTArray<MediaStream*>* aStreams and MediaStream* aStream. The offset from r2 to crash address, 0x2d6f665a, is 0xF6. I checked the class definition if MediaStream and it looks like the address should be located in the object. So that I think the memory address pointed by aStream should be deleted in other places. Crash address: 0x2d6f665a 0 libxul.so!mozilla::MediaStreamGraphImpl::AddBlockingRelatedStreamsToSet [MediaStreamGraph.cpp : 793 + 0x0] r0 = 0xb1f01600 r1 = 0xaef8fb60 r2 = 0x2d6f6564 r3 = 0xb1f5b920 1 libxul.so!mozilla::MediaStreamGraphImpl::AddBlockingRelatedStreamsToSet [MediaStreamGraph.cpp : 800 + 0x9] r0 = 0xb1f01600 r1 = 0xaef8fb60 r2 = 0x2d6f6564 r4 = 0x0000008e [1] http://git.mozilla.org/?p=integration/gecko-dev.git;a=blob;f=dom/media/MediaStreamGraph.cpp;h=4a50ea3185be2f8f01ea44d2e52444e119c2e11a;hb=643f37b68305d90ecfa148f81db63891c30e4620#l793 > > Is this crash occurring frequently enough that we can add some logging to > get some additional information? Hi Inder, Can you tell us the frequency of the problem happening?

This patch will log the addresses of destroyed MediaStreams. It should help us tell if a MediaStream is being referenced after it has been destroyed.

Steven, can you help to find someone to do the testing based on the patch provided by :roc and provide the logs again? Thanks.

Nicholas, since you're the reporter, can you also test it with the patch from :roc? Thanks.

Hi Kevin, Since we have no any information about how this bug happens, we may need Nicholas to test this.

(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #13) > (In reply to Inder from comment #11) > > This crash was produced during stability tests which involves monkey testing > > for several hours and there is no clear STR for this. > > You mean tests that drive the interface of a phone? So Gaia application code > involved in triggering this? Testing is done for several hours with monkey testing so no assumptions can be made about how this crash was triggered. For all we know the crash could happen if we just leave the phone on for several hours with no UI interaction. All we know is that there was a crash, and we have collected as much as we could at the time of the crash. I have added the logging patch. It will be included in either the very next AU or the one after. Thanks!

(In reply to StevenLee[:slee] from comment #14) > (In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #12) > > (In reply to StevenLee[:slee] from comment #7) > > > Could you please check this problem? I checked out b2g-2.2. It crashes at > > > the first line of MediaStreamGraphImpl::AddBlockingRelatedStreamsToSet, > > > "if(aStream->mInBlockingSet)". It looks like aStream has beeb deleted. > > Can you explain the reasoning that lead to that conclusion? Can you tell > > from the disassembly and the crash dump that aStream is a non-null pointer > > to unmapped memory? > > I was guessing. From the crash dump below, it crashed at [1]. I assumed r1 > and r2 are the arguments, nsTArray<MediaStream*>* aStreams and MediaStream* > aStream. The offset from r2 to crash address, 0x2d6f665a, is 0xF6. I checked > the class definition if MediaStream and it looks like the address should be > located in the object. So that I think the memory address pointed by aStream > should be deleted in other places. > > Crash address: 0x2d6f665a > 0 libxul.so!mozilla::MediaStreamGraphImpl::AddBlockingRelatedStreamsToSet > [MediaStreamGraph.cpp : 793 + 0x0] > r0 = 0xb1f01600 r1 = 0xaef8fb60 r2 = 0x2d6f6564 r3 = 0xb1f5b920 > 1 libxul.so!mozilla::MediaStreamGraphImpl::AddBlockingRelatedStreamsToSet > [MediaStreamGraph.cpp : 800 + 0x9] > r0 = 0xb1f01600 r1 = 0xaef8fb60 r2 = 0x2d6f6564 r4 = 0x0000008e > > [1] > http://git.mozilla.org/?p=integration/gecko-dev.git;a=blob;f=dom/media/ > MediaStreamGraph.cpp;h=4a50ea3185be2f8f01ea44d2e52444e119c2e11a; > hb=643f37b68305d90ecfa148f81db63891c30e4620#l793 > > > > > Is this crash occurring frequently enough that we can add some logging to > > get some additional information? > Hi Inder, > Can you tell us the frequency of the problem happening? We saw this crash twice in a single run on AU 119

(In reply to Nicholas Troast [:ntroast] from comment #19) > Testing is done for several hours with monkey testing so no assumptions can > be made about how this crash was triggered. For all we know the crash could > happen if we just leave the phone on for several hours with no UI > interaction. All we know is that there was a crash, and we have collected as > much as we could at the time of the crash. Right, but just to be clear: this testing doesn't run specific Mozilla test code, e.g. our mochitests?

Also, do these builds have #ifdef DEBUG enabled (e.g. NS_ASSERTIONs)?

(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #21) > (In reply to Nicholas Troast [:ntroast] from comment #19) > > Testing is done for several hours with monkey testing so no assumptions can > > be made about how this crash was triggered. For all we know the crash could > > happen if we just leave the phone on for several hours with no UI > > interaction. All we know is that there was a crash, and we have collected as > > much as we could at the time of the crash. > > Right, but just to be clear: this testing doesn't run specific Mozilla test > code, e.g. our mochitests? Right, this testing does not run specific Mozilla test code

(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #22) > Also, do these builds have #ifdef DEBUG enabled (e.g. NS_ASSERTIONs)? These builds do not have DEBUG enabled

I removed attachment 8589477 [details] [diff] [review] from our build due to a conflict with the logging patch from bug 1152439. I believe the patch from bug 1152439 supersedes this one.

"Closing issue which has not been seen since 04/01/15 18:39"

See comment 26.