STR: 1. Run testt which will change the call subscription and make calls. 2. Pair a phone via BT. 3. Pair a car kit via BT. 4. After few minutes, do BT on and off multiple times. mini dumps are generated in the phone.

Hi Eric, Could you find the right person to solve this? Thanks.

Jamin will take a first look.

buildid = 20140331000202 Observed on following gaia/gecko SHA1s: http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=4c3b2f57f4229c5f36f0d8fd399e65f4db88f104 http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=d02b9250ef7fedafc6c709dfcc899844b8624ab6

Frame 0 in the stack trace looks very strange, and impossible, Bluetooth did not use AsyncLatencyLogger. 0 libxul.so!mozilla::StaticRefPtr<AsyncLatencyLogger>::AssignAssumingAddRef(AsyncLatencyLogger*) [StaticPtr.h : 158 + 0x0] r4 = 0xb10d2900 r5 = 0x00000000 r6 = 0xbeb7a6fc r7 = 0xbeb7a72f r8 = 0xb6b02510 r9 = 0x00000001 r10 = 0x00000000 fp = 0x00000000 sp = 0xbeb7a6d8 lr = 0xb54c3025 pc = 0xb4e5a6b6 Found by: given as instruction pointer in context Maybe this required to have NOOPT build to get the correct crash location.

Shawn What do we need from QC? Why is it not part of the build if we're seeing it on 3/31 build? per comment 4

(In reply to Preeti Raghunath(:Preeti) from comment #6) > What do we need from QC? > Why is it not part of the build if we're seeing it on 3/31 build? per > comment 4 Default gecko build is optimized, so sometimes it shows incorrect stack trace, and it may mislead us. NOOPT build means optimization disabled, it usually gives reliable stack trace. Jamin is working on this bug. So I would not comment further.

(In reply to Shawn Huang [:shuang] [:shawnjohnjr] from comment #7) > (In reply to Preeti Raghunath(:Preeti) from comment #6) > > What do we need from QC? > > Why is it not part of the build if we're seeing it on 3/31 build? per > > comment 4 > Default gecko build is optimized, so sometimes it shows incorrect stack > trace, and it may mislead us. NOOPT build means optimization disabled, it > usually gives reliable stack trace. Jamin is working on this bug. So I would > not comment further. Shawn, can you be more specific as to what should be enabled in the gecko build?

Moving NI to Jamin who is working on the bug.

(In reply to bhargavg1 from comment #8) > Shawn, can you be more specific as to what should be enabled in the gecko > build? Hi bhargavg1, If you want to disable the optimizer for gecko build, you have to create a .userconfig file in your B2G project and add "export B2G_NOOPT=1" into the file. You would get a NOOPT build after you rebuild the B2G project. Please refer to https://developer.mozilla.org/en-US/Firefox_OS/Customization_with_the_.userconfig_file. P.S. "export B2G_DEBUG=1" is also recommended. NOOPT build would provide better stack trace, however, the performance would be much slower.

I can't reproduce this bug in my test. As Shawn mentioned in comment #5, the last two frames of stack trace is unexpected to me. > 0 libxul.so!mozilla::StaticRefPtr<AsyncLatencyLogger>::AssignAssumingAddRef(AsyncLatencyLogger*) > 1 libxul.so!mozilla::dom::bluetooth::BluetoothRequestParent::ReplyRunnable::Run() 'frame 0' could be hit only if we use gAsyncLogger which was declared in content/media/Latency.cpp. I put log_print for few functions which would use gAsyncLogger and observe them during my BT test. As far as I know, the BT on/off procedure wouldn't trigger the assignment of AsyncLatencyLogger. It makes no sense to me that it only took one frame from ReplyRunnable::Run() to AssignAssumingAddRef(AsyncLatencyLogger*). I assume the 'frame 0' might be incorrect due to compile optimization. The risk of segmentation fault in ReplyRunnable::Run() could be incorrect pointer usage of mRequest and mReply. Thus, I add a sanity check for them. In addition, mReplyRunnable of BluetoothRequestParent should be automatically revoked by its design, however, it can't be revoked correctly since the variable mEvent has been cleaned up. Please refer to http://dxr.mozilla.org/mozilla-central/source/xpcom/glue/nsThreadUtils.h?from=nsThreadUtils.h&case=true#504. Attachment #8412544 [details] [diff] want to avoid to call Forget() before Revoke() did.

(In reply to Jamin Liu [:jaliu] from comment #10) > (In reply to bhargavg1 from comment #8) > > Shawn, can you be more specific as to what should be enabled in the gecko > > build? > > Hi bhargavg1, > If you want to disable the optimizer for gecko build, you have to create a > .userconfig file in your B2G project and add "export B2G_NOOPT=1" into the > file. You would get a NOOPT build after you rebuild the B2G project. > Please refer to > https://developer.mozilla.org/en-US/Firefox_OS/Customization_with_the_. > userconfig_file. > P.S. "export B2G_DEBUG=1" is also recommended. > > NOOPT build would provide better stack trace, however, the performance would > be much slower. I am trying to reproduce this issue with this build. I will update here soon

Comment on attachment 8412544 [details] [diff] [review] Avoid to call Forget to early and add an additional proctection for mReply. (v1) I don't quite understand what this patch is attempting to do. The actual crash looks to me like memory corruption (ReplyRunnable::mRequest is non-null but somehow a bad pointer). Checking 'mReply' seems useless since it is already asserted above. I don't think this patch can really fix anything, I just think it might hide the problem or move it elsewhere.

For comment #10: Enabling B2G_NOOPT causes b2g to be hang during enabling bluetooth: STR: 1) Build gecko with B2G_NOOPT=1 2) Enable Bluetooth. 3) b2g crashes with stack trace. b2g keeps rebooting again and again . This is blocker to reproduce bug 997962 with B2G_NOOPT=1.

(In reply to ben turner [:bent] (use the needinfo? flag!) from comment #14) > Comment on attachment 8412544 [details] [diff] [review] Hi Ben, Thank you for your feedback. I agree with you, the most acceptable explanation to me is the access to ReplyRunnable::mRequest which is a bad pointer. However, I can't figure how it have a chance to be a invalid pointer. mRequest was initialed in member initialization list by the "this" pointer of BluetoothRequestParent() in its constructor and it seems safe to me. I'll try to figure it out with further test. Thanks.

(In reply to Tapas Kumar Kundu from comment #15) Hi Tapas, Thank you for your time. I may have found few potential risks. BluetoothProfileController could act incorrectly when user connect and disconnect Hand-free device rapidly by toggling BT switch. And the ReplyRunnable might cause a small leak when BluetoothRequestParent replys to some specific requestes with Bluedroid stack. I'll check whether they are relative to this bug and make a patch for them. I got a problem during B2G booting with NOOPT build today. I would try to reproduce this bug without compile optimization if I could handle the issue on NOOPT build.

I create few bugs which cause stability issues on Bluetooth component. The STR of this bug 997962 could hit this following bugs and might lead to unexpected errors. Bug 1003657 - The memory of ReplyRunnable for DefaultAdapterPathRequest wouldn't be released and becomes a leak. Bug 1003658 - The memory of ReplyRunnable for ConnectRequest wouldn't be released and becomes a leak. Bug 1003661 - HfpManager and A2dpManager hit MOZ_ASSERT(aController && !mController) in Connect() and Disconnect() function during BT connection. Bug 1003662 - The number of BT profile controller increases incorrectly when user connect and disconnect Hand-free device rapidly by toggling BT switch. It would lead to abnormal connection behaviour. Bug 1003663 - BluetoothProfileController would act incorrectly and fail to connect when user connect and disconnect Hand-free device rapidly by toggling BT switch. Bug 988171 - Setting app would create mozBluetooth when app launch but never release it. I've made fixes for these bugs and the patches are under review. When the review is done, I think it would be nice if QA or reporter test again to find out whether this bug could be reproduced under these stability fixes. P.S. I can't reproduce this bug in my experiment. In the same way, I would try to dig some things from NOOPT build.

:jalui, nice find!

Based on the dependencies, looks like the ETA should be updated to 5/8 or later. Do we have an updated ETA about this bug? Thanks.

Hi Tapas, The patches of blocking bugs were landed. (The patch of Bug 1003661, 1003662, 1003663 is in b2g-inbound right now and will be merged into master and 1.4.) I can't reproduce this bug in my experiment. I wonder if you could test this bug again by following the STR you provided. It would be really helpful. Let's see if this bug can be reproduce under these stability fixes. Thank you very much.

(In reply to Jamin Liu [:jaliu] from comment #21) > I can't reproduce this bug in my experiment. > I wonder if you could test this bug again by following the STR you provided. > It would be really helpful. > Let's see if this bug can be reproduce under these stability fixes. > > Thank you very much. Thank you so much for your work . I will update here soon.

Hi Tapas, any update for this? Thank you.

howie -- No update yet. We should have an update in couple of days, stability tests with all the patches are ongoing.

Marking the bug resolved. We haven't seen any BT issues show up in recent stability tests. We will reopen/create new bug if it resurfaces.

(In reply to Inder from comment #25) > Marking the bug resolved. We haven't seen any BT issues show up in recent > stability tests. We will reopen/create new bug if it resurfaces. Thanks for the update.

(In reply to Tapas Kumar Kundu from comment #22) > (In reply to Jamin Liu [:jaliu] from comment #21) > Thank you so much for your work . I will update here soon. Thank you for your time and effort on this issue.

Crash is again seen on following gaia/gecko: gaia: https://www.codeaurora.org/cgit/quic/lf/b2g/mozilla/gaia/commit/?h=mozilla/v1.4&id=17fb44880e95bc7ae363a609d811bf5a9a067b5b gecko: https://www.codeaurora.org/cgit/quic/lf/b2g/mozilla/gecko/commit/?h=mozilla/v1.4&id=2f11e3aba98eb785ec24504fe9988ab61a03b64d Please suggest

stack trace for #comment 28

(In reply to Tapas Kumar Kundu from comment #28) > Created attachment 8423354 [details] > crash is again reproduced in latest build which has all fixes Thanks for your report. I'm investigating it to see if I can provide a fix. Could you describe in detail about the steps to reproduce ? Is it hit by monkey test or manual test ? What's the last action user attempt to do ? (Turn on BT? Pair with hand-free device? Connect with hand-free device?) What's kind of BT device has been paired with ? One or multiple devices involved in this test?

(In reply to Jamin Liu [:jaliu] from comment #30) > (In reply to Tapas Kumar Kundu from comment #28) > > Created attachment 8423354 [details] > > crash is again reproduced in latest build which has all fixes > > Thanks for your report. > I'm investigating it to see if I can provide a fix. > > Could you describe in detail about the steps to reproduce ? > Is it hit by monkey test or manual test ? STR: 1. Receive MT SMS continuosuly. 2. Play Music in repreat mode. 3. Open camera and take pictures continuously. 4. BT on/off multiple times. 5. Mini dumps are generated on the phone. Please note that this issue comes during stability testing which runs for 24 hours. It is not possible to reproduce it manually. Please let me know if you want me to add additional traces for next stability run. > What's the last action user attempt to do ? (Turn on BT? Pair with hand-free > device? Connect with hand-free device?) From logcat, i can see "Turn on BT" activity just before crash [1] I attached full logcat for more details . Exact timestamp when this issue happened is "Jan 03 1970 08:16:03" . You can see logs (#comment 28 and also this attachment) to find out BT activity just before/after this. > What's kind of BT device has been paired with ? > One or multiple devices involved in this test? I asked my colleague about this . I will update her soon. [1] "01-03 08:16:02.409 226 696 I GeckoBluetooth: AdapterStateChangeCallback: BT_STATE 1"

Please let me know if you want me to add additional traces for next stability run.

Crash observed on: Device: msm8226 Gonk Version: AU_LINUX_GECKO_B2G_KK_3.5.01.04.00.113.097 Moz BuildID: 20140511000204 B2G Version: 1.4 Gecko Version: 30.0 Gaia: http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=17fb44880e95bc7ae363a609d811bf5a9a067b5b Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=2f11e3aba98eb785ec24504fe9988ab61a03b64d

Hi Tapas, Thank you for the information you provided. It's helpful. In my observation, it might crash under the specific extreme condition during the inter-process communications when setting app send a 'ConnectRequest' to chrome process. In my test, it can't cause the crash manually unless I put a busy loop to delay the IPC and mess up the sequence of request. However, it seems like it could happen in stability script test. I'm working on the fix of this issue.

(In reply to Jamin Liu [:jaliu] from comment #30) > What's kind of BT device has been paired with ? > One or multiple devices involved in this test? We are not using any BT device to pair with FFOS device. We are simply doing bluetooth on/off here. Another set of log is in : https://drive.google.com/file/d/0B1cSMS8_GuAES0VVQTR3RDR1eE0/edit?usp=sharing Crash observed on: Device: msm8226 gaia: https://www.codeaurora.org/cgit/quic/lf/b2g/mozilla/gaia/tag/?h=0a25eff60fcc699687e45ba2ac8b9a3ab3782672&id=AU_LINUX_GECKO_B2G_KK_3.5.01.04.00.113.102 gecko: https://www.codeaurora.org/cgit/quic/lf/b2g/mozilla/gecko/tag/?h=0b83d5e6a54df9bfccdbb5ba662527bf657f5381&id=AU_LINUX_GECKO_B2G_KK_3.5.01.04.00.113.102 Exact Crash timestamp is: 05-19 14:16:52 and bluetooth activity before crash is : "05-19 14:16:51.101 202 774 I GeckoBluetooth: AdapterStateChangeCallback: BT_STATE 1" I think that this is same error which we observed in #comment 31 .

Crash observed on: Device: msm8226 Gonk Version: AU_LINUX_GECKO_B2G_KK_3.5.01.04.00.113.102 Moz BuildID: 20140515000202 B2G Version: 1.4 Gecko Version: 30.0 Gaia: http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=2e97bee6bb79d3577dba1bf2a1bbfcba64ee99ab Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=35f27a8e9b3f651748aa22095553024556272de8

Hi Tapas, I uploaded an experimental patch. Would it be possible for you to pick this patch and run the stability test? I want to check the affect to the stability, it would be really helpful. Thank you very much.

Hi Tapas, I spent some time on investigating this bug as well. I suspect the root cause would be the racing problem of using those static nsTArrays on different threads. Here is a hotfix which may be the answer of this issue. May not be the final patch, but the main idea will be the same. Please try to apply either my patch or Jamin's patch for the next test run and see if they work. Thank you.

(In reply to Eric Chou [:ericchou] [:echou] from comment #38) > Created attachment 8425477 [details] [diff] [review] > WIP-Bug-997962-Hotfix-doing-all-callbacks-on-main-th.patch > > Hi Tapas, > > I spent some time on investigating this bug as well. I suspect the root > cause would be the racing problem of using those static nsTArrays on > different threads. Here is a hotfix which may be the answer of this issue. > May not be the final patch, but the main idea will be the same. > > Please try to apply either my patch or Jamin's patch for the next test run > and see if they work. Thank you. Thanks a lot for your work. I will update here soon with my test results

(In reply to Eric Chou [:ericchou] [:echou] from comment #38) > Created attachment 8425477 [details] [diff] [review] > WIP-Bug-997962-Hotfix-doing-all-callbacks-on-main-th.patch > Please try to apply either my patch or Jamin's patch for the next test run > and see if they work. I am trying your patch only.

(In reply to Eric Chou [:ericchou] [:echou] from comment #38) > Created attachment 8425477 [details] [diff] [review] > WIP-Bug-997962-Hotfix-doing-all-callbacks-on-main-th.patch > Please try to apply either my patch or Jamin's patch for the next test run > and see if they work. Thank you. Please ignore my earlier #comment 40. Patch from #comment 38 does not apply cleanly. So I picked up picked up patch from #comment 37 for next stability run. Please re-base and post a new patch for v1.4 if needed.

(In reply to Tapas Kumar Kundu from comment #41) > (In reply to Eric Chou [:ericchou] [:echou] from comment #38) > > Created attachment 8425477 [details] [diff] [review] > > WIP-Bug-997962-Hotfix-doing-all-callbacks-on-main-th.patch > > > Please try to apply either my patch or Jamin's patch for the next test run > > and see if they work. Thank you. > > Please ignore my earlier #comment 40. Patch from #comment 38 does not apply > cleanly. So I picked up picked up patch from #comment 37 for next stability > run. > > Please re-base and post a new patch for v1.4 if needed. oh, sorry about that. I'll rebase today.

Hi Tapas, Please use this one for 1.4 codebase. Thank you.

* updated. This is for 1.4.

Hi Eric, I am picking up your patch from #comment 44. Please ignore my #comment 41. I am asking my colleague to test it for next stability run. Thanks for your help :)

(In reply to Tapas Kumar Kundu from comment #45) > Hi Eric, > I am picking up your patch from #comment 44. Please ignore my #comment 41. > > I am asking my colleague to test it for next stability run. > > Thanks for your help :) Thank you, too, Tapas. Please note that this is a hotfix of fixing crash issue but may cause some UI issues. I'll provide a more formal one once the patch has been proven to work.

(In reply to Eric Chou [:ericchou] [:echou] from comment #46) > (In reply to Tapas Kumar Kundu from comment #45) > > Hi Eric, > > I am picking up your patch from #comment 44. Please ignore my #comment 41. > > > > I am asking my colleague to test it for next stability run. > > > > Thanks for your help :) > > Thank you, too, Tapas. > > Please note that this is a hotfix of fixing crash issue but may cause some > UI issues. I'll provide a more formal one once the patch has been proven to > work. What UI issues?

(In reply to Michael Vines [:m1] [:evilmachines] from comment #47) > (In reply to Eric Chou [:ericchou] [:echou] from comment #46) > > (In reply to Tapas Kumar Kundu from comment #45) > > > Hi Eric, > > > I am picking up your patch from #comment 44. Please ignore my #comment 41. > > > > > > I am asking my colleague to test it for next stability run. > > > > > > Thanks for your help :) > > > > Thank you, too, Tapas. > > > > Please note that this is a hotfix of fixing crash issue but may cause some > > UI issues. I'll provide a more formal one once the patch has been proven to > > work. > > What UI issues? The name of paired devices may not be seen in Settings app sometimes.

Adding NI flag on me as I have to update results from next stability run

Crash observed on: Device: msm8226 Gonk Version: AU_LINUX_GECKO_B2G_KK_3.5.01.04.00.113.106 Moz BuildID: 20140519000201 B2G Version: 1.4 Gecko Version: 30.0 Gaia: http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=defd0650fb9d30c6515d50a89e72d8fb74ce7e62 Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=49653a2e9c8709028640af39c919f1f8f4c53806

(In reply to cafbot (PoC: ggrisco) from comment #50) > Crash observed on: > > Device: msm8226 > Gonk Version: AU_LINUX_GECKO_B2G_KK_3.5.01.04.00.113.106 > Moz BuildID: 20140519000201 > B2G Version: 1.4 > Gecko Version: 30.0 > Gaia: > http://git.mozilla.org/?p=releases/gaia.git;a=commit; > h=defd0650fb9d30c6515d50a89e72d8fb74ce7e62 > Gecko: > http://git.mozilla.org/?p=releases/gecko.git;a=commit; > h=49653a2e9c8709028640af39c919f1f8f4c53806 Is my patch or Jamin's patch applied onto this Gecko?

Greg, Can we please try the patch on gecko per comment 51?

Eric, no it didn't make it into that build. It went in last night's build and we should have the results in a day or so. Thanks.

Crash observed on: Device: msm8226 Gonk Version: AU_LINUX_GECKO_B2G_KK_3.5.01.04.00.113.106 Moz BuildID: 20140519000201 B2G Version: 1.4 Gecko Version: 30.0 Gaia: http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=defd0650fb9d30c6515d50a89e72d8fb74ce7e62 Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=49653a2e9c8709028640af39c919f1f8f4c53806

Crash observed on: Device: msm8226 Gonk Version: AU_LINUX_GECKO_B2G_KK_3.5.01.04.00.113.106 Moz BuildID: 20140519000201 B2G Version: 1.4 Gecko Version: 30.0 Gaia: http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=defd0650fb9d30c6515d50a89e72d8fb74ce7e62 Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=49653a2e9c8709028640af39c919f1f8f4c53806

sorry for the spam, please ignore comments 54, 55.

Crash observed on: Device: msm8226 Gonk Version: AU_LINUX_GECKO_B2G_KK_3.5.01.04.00.113.106 Moz BuildID: 20140519000201 B2G Version: 1.4 Gecko Version: 30.0 Gaia: http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=defd0650fb9d30c6515d50a89e72d8fb74ce7e62 Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=49653a2e9c8709028640af39c919f1f8f4c53806

Hi Tapas, I would like to know the test result after applying our patches. Would you mind giving some feedback here?

Crash observed on: Device: msm8226 Gonk Version: AU_LINUX_GECKO_B2G_KK_3.5.01.04.00.113.106 Moz BuildID: 20140519000201 B2G Version: 1.4 Gecko Version: 30.0 Gaia: http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=defd0650fb9d30c6515d50a89e72d8fb74ce7e62 Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=49653a2e9c8709028640af39c919f1f8f4c53806

(In reply to Eric Chou [:ericchou] [:echou] from comment #58) > Hi Tapas, > > I would like to know the test result after applying our patches. Would you > mind giving some feedback here We have not seen this crash after applying your fix so far. But we want to take some more time before confirming it as fixed. Stability test is still going on with your patch. I will update asap.

(In reply to Tapas Kumar Kundu from comment #60) > (In reply to Eric Chou [:ericchou] [:echou] from comment #58) > > Hi Tapas, > > > > I would like to know the test result after applying our patches. Would you > > mind giving some feedback here > > We have not seen this crash after applying your fix so far. But we want to > take some more time before confirming it as fixed. Stability test is still > going on with your patch. I will update asap. Good to know. I just merged my final patch into b2g-inbound (bug 1015826). I'll request for uplift to 1.4 and we'll see if the crash will still be seen in later builds. Thanks for your update.

Quick update: bug 1015826 has been uplifted to 1.4. Manually applying my patch should not be needed anymore.

(In reply to Eric Chou [:ericchou] [:echou] (GSMA MAE @ Shanghai, 6/10 ~ 6/13) from comment #62) > Quick update: bug 1015826 has been uplifted to 1.4. Manually applying my > patch should not be needed anymore. We are not seeing this issue anymore with your fix. Thanks for your work.

This will need to be coverd in new test case

Automated. No test case added in moztrap.