+++ This bug was initially created as a clone of Bug #1061993 +++ When using the patch from Bug 1057220 to catch all invalid fds being closed, we see a crash when operating the WiFi module You can see the point of crash in the logs as, 10-15 01:35:55.218 202 871 E libnoselect: -1 = close(194) failed. errno=9

[Blocking Requested - why for this release]:

complete logcats is in [1]. [1] https://drive.google.com/file/d/0B1cSMS8_GuAENXR5U2pyR0F1YUk/view?usp=sharing Please note crash timestamps is : "10-15 01:35:55.218 202 871 E libnoselect: -1 = close(194) failed. errno=9"

Observed on: Device: msm8226 Gonk Version: AU_LINUX_GECKO_B2G_KK_3.6.01.04.00.000.076 Moz BuildID: 20140829000203 B2G Version: 2.1 Gecko Version: 32.0 Gaia: http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=5bf3b8cdea15e62ce7bf77a15085a18e24e33c44 Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=f2ab70e9ab18439dae25e6eb21f186708b2888de Patches: bug 1050751

(In reply to cafbot (PoC: ggrisco) from comment #4) > Observed on: > > Device: msm8226 > Gonk Version: AU_LINUX_GECKO_B2G_KK_3.6.01.04.00.000.076 > Moz BuildID: 20140829000203 > B2G Version: 2.1 > Gecko Version: 32.0 > Gaia: > http://git.mozilla.org/?p=releases/gaia.git;a=commit; > h=5bf3b8cdea15e62ce7bf77a15085a18e24e33c44 > Gecko: > http://git.mozilla.org/?p=releases/gecko.git;a=commit; > h=f2ab70e9ab18439dae25e6eb21f186708b2888de > Patches: bug 1050751 The patch for Bug 1061993 seems not in the gecko you use [1]. [1] http://git.mozilla.org/?p=releases/gecko.git;a=blob;f=dom/network/src/NetUtils.cpp;h=87151b361e3f7f3f3b295e69f1e073183e98fea9;hb=f2ab70e9ab18439dae25e6eb21f186708b2888de#l95

Hi Tapas, can you check for comment 5 ? Thanks.

crash + FC and CC blocker = 2.1+

(In reply to howie [:howie] from comment #6) > Hi Tapas, can you check for comment 5 ? Thanks. We have also seen it in https://www.codeaurora.org/cgit/quic/lf/b2g/mozilla/gaia/commit/?h=mozilla/v2.1&id=f5d4ff60ffed8961f7d0380ada9d0facfdfd56b1 https://www.codeaurora.org/cgit/quic/lf/b2g/mozilla/gecko/commit/?h=mozilla/v2.1&id=db7fce920e7d782d9f601384dc95924abcdaeeb8 We raised this bug for v2.1 FFOS. Comment 4 is not applicable for v2.1 FFOS.

(In reply to Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me) from comment #8) > https://www.codeaurora.org/cgit/quic/lf/b2g/mozilla/gaia/commit/?h=mozilla/ > v2.1&id=f5d4ff60ffed8961f7d0380ada9d0facfdfd56b1 > https://www.codeaurora.org/cgit/quic/lf/b2g/mozilla/gecko/commit/?h=mozilla/ > v2.1&id=db7fce920e7d782d9f601384dc95924abcdaeeb8 > The patch for Bug 1061993 has already landed in ^^ .

For more clarifications: logs in Comment 0, Comment 2 and Comment 3 are collected on v2.1 FFOS build [1] which already has fix from bug 1061993 [1] See Comment 8 for exact v2.1 gaia/gecko SHA1 .

(In reply to Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me) from comment #10) > For more clarifications: > > logs in Comment 0, Comment 2 and Comment 3 are collected on v2.1 FFOS build > [1] which already has fix from bug 1061993 > > > [1] See Comment 8 for exact v2.1 gaia/gecko SHA1 . Thanks for clarifying. Lets get Henry's input here.

Observed on: Device: msm8226 Gonk Version: AU_LINUX_GECKO_B2G_KK_2.0.01.04.00.114.109 Moz BuildID: 20141011000201 B2G Version: 2.1 Gecko Version: 34.0a2 Gaia: http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=f5d4ff60ffed8961f7d0380ada9d0facfdfd56b1 Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=db7fce920e7d782d9f601384dc95924abcdaeeb8 Patches: bug 1075077, bug 1070431, bug 1082234, bug 1076327, bug 1081528, bug 1074419

Observed on: Device: msm8226 Gonk Version: AU_LINUX_GECKO_B2G_KK_2.0.01.04.00.114.109 Moz BuildID: 20141011000201 B2G Version: 2.1 Gecko Version: 34.0a2 Gaia: http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=f5d4ff60ffed8961f7d0380ada9d0facfdfd56b1 Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=db7fce920e7d782d9f601384dc95924abcdaeeb8 Patches: bug 1075077, bug 1070431, bug 1082234, bug 1076327, bug 1081528, bug 1074419

Henry is working on this.

Probably ifc_enable/ifc_disable needs to be protected as well. It's worthy of giving a try. Thansk!

Any update?

(In reply to Ken Chang[:ken] from comment #16) > Any update? The patch from attachment 8506657 [details] [diff] [review] has been landed internally and we're awaiting feedback from the test team

Comment on attachment 8506657 [details] [diff] [review] Bug1083449.patch Review of attachment 8506657 [details] [diff] [review]: ----------------------------------------------------------------- Looks good.

henry, can we please land this asap so this can get uplifted, given the r+ ?

Observed on: Device: msm8226 Gonk Version: AU_LINUX_GECKO_B2G_KK_2.0.01.04.00.114.121 Moz BuildID: 20141026001201 B2G Version: 2.1 Gecko Version: 34.0 Gaia: http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=c97463d61f45513a2123b19610386ddbfc916819 Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=9d2ff8f6528e1bf1e18252a479e2c3ccf06201df Patches: bug 1070431, bug 1083449

(In reply to Danylo Hirnyj from comment #17) > (In reply to Ken Chang[:ken] from comment #16) > > Any update? > > The patch from attachment 8506657 [details] [diff] [review] has been landed > internally and we're awaiting feedback from the test team We have seen this issue again with fix from attachment 8506657 [details] [diff] [review] Stack trace is same and crash timestamp is : 10-28 09:33:41.311 207 207 E GeckoConsole: Content JS LOG at dummy file:599 in GaiaDataLayer.disableWiFi/<: wifi enabled status: true Full logcat logs : https://drive.google.com/file/d/0B1cSMS8_GuAEUFZUUEVBaTdNMzQ/view?usp=sharing

stacktrace for comment 24

NI :henry to check this asap.

(In reply to Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me) from comment #24) > Created attachment 8512895 [details] > .extra file logs > > (In reply to Danylo Hirnyj from comment #17) > > (In reply to Ken Chang[:ken] from comment #16) > > > Any update? > > > > The patch from attachment 8506657 [details] [diff] [review] has been landed > > internally and we're awaiting feedback from the test team > > We have seen this issue again with fix from attachment 8506657 [details] [diff] [review] > [diff] [review] > > Stack trace is same and crash timestamp is : > > 10-28 09:33:41.311 207 207 E GeckoConsole: Content JS LOG at dummy > file:599 in GaiaDataLayer.disableWiFi/<: wifi enabled status: true > > > Full logcat logs : > https://drive.google.com/file/d/0B1cSMS8_GuAEUFZUUEVBaTdNMzQ/view?usp=sharing Hmmm, were you doing gaia-ui-test? (from "GaiaDataLayer.disableWiFi")? If yes, I would expect we can reproduce it and gdb it! It would be helpful!

Oops how stupid I was. There would be 2 instances of NetUtils but not 1. Those instances should share the same mutex ... [1] http://hg.mozilla.org/releases/mozilla-b2g34_v2_1/file/318019f80a8e/dom/wifi/WifiUtils.cpp#l390 [2] http://hg.mozilla.org/releases/mozilla-b2g34_v2_1/file/318019f80a8e/dom/system/gonk/NetworkUtils.cpp#l1017

(In reply to Henry Chang [:henry] from comment #28) > Oops how stupid I was. There would be 2 instances of NetUtils but not 1. > Those instances should share the same mutex ... > > [1] > http://hg.mozilla.org/releases/mozilla-b2g34_v2_1/file/318019f80a8e/dom/wifi/ > WifiUtils.cpp#l390 > [2] > http://hg.mozilla.org/releases/mozilla-b2g34_v2_1/file/318019f80a8e/dom/ > system/gonk/NetworkUtils.cpp#l1017 which means patch for Bug 1061993 fixed nothing. Will attach another patch to use a shared mutex instead of a per instance mutex.

Comment on attachment 8513256 [details] [diff] [review] Bug 1083449 (the REAL patch! for v2.1) Ask vchang as well as chuck for review to prevent my making another stupid mistake ... Thanks!

Comment on attachment 8513256 [details] [diff] [review] Bug 1083449 (the REAL patch! for v2.1) Review of attachment 8513256 [details] [diff] [review]: ----------------------------------------------------------------- Nice Job!

(In reply to Henry Chang [:henry] from comment #27) > (In reply to Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me) from > ..... > Hmmm, were you doing gaia-ui-test? (from "GaiaDataLayer.disableWiFi")? > If yes, I would expect we can reproduce it and gdb it! It would be helpful! I wish I can do that . But its not easy to do this in our current stability setup. But I am landing your patch from Comment 31 and I will confirm you asap.

(In reply to Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me) from comment #33) > (In reply to Henry Chang [:henry] from comment #27) > > (In reply to Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me) from > > ..... > > Hmmm, were you doing gaia-ui-test? (from "GaiaDataLayer.disableWiFi")? > > If yes, I would expect we can reproduce it and gdb it! It would be helpful! > > I wish I can do that . But its not easy to do this in our current stability > setup. But I am landing your patch from Comment 31 and I will confirm you > asap. The patch has gotten reviewed. I am going to flag checkin-needed first since it's definitely a necessary patch no matter it fixes this crash or not. Thanks :)

The most recent checkin-needed is for attachment 8513305 [details] [diff] [review]. The try run result can be found in comment 36.

Please request b2g34 approval on whatever needs to land on v2.1 here.

We are not seeing this issue anymore in FFOS 2.1 after landing fix from Comment 31

Comment on attachment 8513256 [details] [diff] [review] Bug 1083449 (the REAL patch! for v2.1) NOTE: Please see https://wiki.mozilla.org/Release_Management/B2G_Landing to better understand the B2G approval process and landings. [Approval Request Comment] Bug caused by (feature/regressing bug #): Wifi User impact if declined: B2G will crash while repeatedly enable/disable wifi Testing completed: Yes Risk to taking this patch (and alternatives if risky): No String or UUID changes made by this patch: No

Backed out for B2G debug cpptest segfaults. https://hg.mozilla.org/releases/mozilla-b2g34_v2_1/rev/dc11764d6c3f https://treeherder.mozilla.org/ui/logviewer.html#?job_id=24883&repo=mozilla-b2g34_v2_1

(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #45) > Backed out for B2G debug cpptest segfaults. > https://hg.mozilla.org/releases/mozilla-b2g34_v2_1/rev/dc11764d6c3f > > https://treeherder.mozilla.org/ui/logviewer.html#?job_id=24883&repo=mozilla- > b2g34_v2_1 Hi Ryan, Was the segfaults caused by only this patch or a bunch of patches? Thanks!

Yours was the only one that needed backing out for them to go away.

Unable to verify this bug and its dependency as it is a back-end issue.

According to the following warning message: 09:34:37 INFO - [1093] WARNING: XPCOM objects created/destroyed from static ctor/dtor: file ../../../gecko/xpcom/base/nsTraceRefcnt.cpp, line 148 09:34:37 INFO - Segmentation fault I suspect it's due to WifiProxyService or NetworkWorker is created in the static ctor, which caused uninitialized mutex to be referenced. Figuring out how to run cpp unit test locally to verify my guess.

(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #45) > Backed out for B2G debug cpptest segfaults. > https://hg.mozilla.org/releases/mozilla-b2g34_v2_1/rev/dc11764d6c3f > > https://treeherder.mozilla.org/ui/logviewer.html#?job_id=24883&repo=mozilla- > b2g34_v2_1 We are again hitting same issue after backing our this change :(

(In reply to Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me) from comment #50) > (In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #45) > > Backed out for B2G debug cpptest segfaults. > > https://hg.mozilla.org/releases/mozilla-b2g34_v2_1/rev/dc11764d6c3f > > > > https://treeherder.mozilla.org/ui/logviewer.html#?job_id=24883&repo=mozilla- > > b2g34_v2_1 > > We are again hitting same issue after backing our this change :( It's totally expected :( I am striving on getting the patch pass the cpp unit test on 2.1 to land it back. Please waiting for my good news! Thanks!

Observed on: Device: msm8226 Gonk Version: AU_LINUX_GECKO_B2G_KK_2.0.01.04.00.114.129 Moz BuildID: 20141105001204 B2G Version: 2.1 Gecko Version: 34.0 Gaia: http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=154da5e17029a51002d5d9b7df39563d509edde6 Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=3b31ab3658aee0c3dc459f25bc10cb28826f8876 Patches: bug 1089706, bug 1070431, bug 1093278

New findings: Have run cpp unit test locally and found it crashed on destroying static/global object, which is very likely the static mozilla::Mutex. But I cannot explain why it doesn't occur on release build and m-c. I/DEBUG ( 37): #00 pc 003a2358 /data/local/tests/cppunittests/b/libxul.so I/DEBUG ( 37): #01 pc 003a2410 /data/local/tests/cppunittests/b/libxul.so I/DEBUG ( 37): #02 pc 0033e8dc /data/local/tests/cppunittests/b/libxul.so I/DEBUG ( 37): #03 pc 0001ec02 /system/lib/libc.so (__cxa_finalize) I/DEBUG ( 37): #04 pc 0001ef9c /system/lib/libc.so (exit) I/DEBUG ( 37): #05 pc 0001ef9c /system/lib/libc.so (exit) I/DEBUG ( 37): #06 pc 0001ef9c /system/lib/libc.so (exit) I/DEBUG ( 37): #07 pc 0001ef9c /system/lib/libc.so (exit)

This patch provides basic idea of how to fix the cppunittest crash. Just attach first not fully tested yet.

Hi Vincent, Fabrice, Could you please review this patch regarding using a static bare PRLock instead of static mozilla::Mutex? I found the destruction of static mozilla::Mutex would cause the cppunittest to crash because of the XPCOM shutdown timing issue. I am not sure if my current approach is the best one but it does fix the and does protect potential concurrent ifc_* function calls. Do you have any idea? Thanks!

Hi Tapas, Could you please try the attachment 8520467 [details] [diff] [review] out? Thanks!

(In reply to Henry Chang [:henry] from comment #57) > Hi Tapas, > > Could you please try the attachment 8520467 [details] [diff] [review] out? > Thanks! We have asked our test team to confirm. We will update asap. Thanks for your help

(In reply to Henry Chang [:henry] from comment #57) > Hi Tapas, > > Could you please try the attachment 8520467 [details] [diff] [review] out? > Thanks! Is this patch also applicable for v2.0 ?

Hi Ryan, How can I have a try run on v2_1? Is it okay to simply "push -f try"? Thanks!

Yes, any tests that would normally run on b2g34_v2_1 should run fine on Try. Your simplest option is probably just going with |try: -b do -p emulator,emulator-jb,emulator-kk -u all -t none|, which I would expect to work fine.

(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #61) > Yes, any tests that would normally run on b2g34_v2_1 should run fine on Try. > Your simplest option is probably just going with |try: -b do -p > emulator,emulator-jb,emulator-kk -u all -t none|, which I would expect to > work fine. Thanks Ryan! I was afraid of pushing too many commits to try at first :p I will give it try. Thanks!

(In reply to Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me) from comment #59) > (In reply to Henry Chang [:henry] from comment #57) > > Hi Tapas, > > > > Could you please try the attachment 8520467 [details] [diff] [review] out? > > Thanks! > > Is this patch also applicable for v2.0 ? Not directly applicable since the location of source files are different.

The cpp unit test passed! https://tbpl.mozilla.org/?tree=Try&rev=16843bbd1967

Comment on attachment 8520467 [details] [diff] [review] Bug1083449.patch (to fix cppunittest crash, for 2.1) Review of attachment 8520467 [details] [diff] [review]: ----------------------------------------------------------------- Looks good, thank you.

(In reply to Henry Chang [:henry] from comment #63) > (In reply to Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me) from > comment #59) > > (In reply to Henry Chang [:henry] from comment #57) > > > Hi Tapas, > > > > > > Could you please try the attachment 8520467 [details] [diff] [review] out? > > > Thanks! > > > > Is this patch also applicable for v2.0 ? > > Not directly applicable since the location of source files are different. Do you think that this issue can also happen in v2.0 because of this bug ? If so then please land a patch for v2.0 too . Thanks a lot for your help

(In reply to Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me) from comment #66) > (In reply to Henry Chang [:henry] from comment #63) > > (In reply to Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me) from > > comment #59) > > > (In reply to Henry Chang [:henry] from comment #57) > > > > Hi Tapas, > > > > > > > > Could you please try the attachment 8520467 [details] [diff] [review] out? > > > > Thanks! > > > > > > Is this patch also applicable for v2.0 ? > > > > Not directly applicable since the location of source files are different. > > Do you think that this issue can also happen in v2.0 because of this bug ? > If so then please land a patch for v2.0 too . > > Thanks a lot for your help I think it will. I will prepare a patch first and land it until your latest test result. Thanks!

Hi Tapas, Do you have any update? I will land the patch (both 2.0/2.1) once upon it works fine from your side. Thanks!

(In reply to Henry Chang [:henry] from comment #69) > Hi Tapas, > > Do you have any update? I will land the patch (both 2.0/2.1) > once upon it works fine from your side. Thanks! Hi Henry, We are still waiting for update from test team.

(In reply to Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me) from comment #70) > (In reply to Henry Chang [:henry] from comment #69) > > Hi Tapas, > > > > Do you have any update? I will land the patch (both 2.0/2.1) > > once upon it works fine from your side. Thanks! > > Hi Henry, > > We are still waiting for update from test team. We still waiting updates from Test team. Please land this patch for both v2.0 and v2.1 if you are "OK" with it. We will create new bug if we see this issue again .

Comment on attachment 8520467 [details] [diff] [review] Bug1083449.patch (to fix cppunittest crash, for 2.1) NOTE: Please see https://wiki.mozilla.org/Release_Management/B2G_Landing to better understand the B2G approval process and landings. [Approval Request Comment] Bug caused by (feature/regressing bug #): Wifi User impact if declined: Toggling wifi would cause b2g to crash. Testing completed: Yes Risk to taking this patch (and alternatives if risky): No String or UUID changes made by this patch: No

(In reply to Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me) from comment #71) > (In reply to Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me) from > comment #70) > > (In reply to Henry Chang [:henry] from comment #69) > > > Hi Tapas, > > > > > > Do you have any update? I will land the patch (both 2.0/2.1) > > > once upon it works fine from your side. Thanks! > > > > Hi Henry, > > > > We are still waiting for update from test team. > > We still waiting updates from Test team. Please land this patch for both > v2.0 and v2.1 if you are "OK" with it. > > We will create new bug if we see this issue again . Got it! By the way, are you also testing the patch on 2.0? Thanks.

(In reply to Henry Chang [:henry] from comment #73) > (In reply to Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me) from > comment #71) > > (In reply to Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me) from > > comment #70) > > > (In reply to Henry Chang [:henry] from comment #69) > > > > Hi Tapas, > > > > > > > > Do you have any update? I will land the patch (both 2.0/2.1) > > > > once upon it works fine from your side. Thanks! > > > > > > Hi Henry, > > > > > > We are still waiting for update from test team. > > > > We still waiting updates from Test team. Please land this patch for both > > v2.0 and v2.1 if you are "OK" with it. > > > > We will create new bug if we see this issue again . > > Got it! By the way, are you also testing the patch on 2.0? > Thanks. No . Please also land this patch in v2.0 if you are "OK" with it .

Henry this needs to land on central still, so NI you to help with that.

(In reply to bhavana bajaj [:bajaj] from comment #75) > Henry this needs to land on central still, so NI you to help with that. Since the threading model has changed on central, this patch is not needed anymore and only applicable for 2.0/2.1. Bug 1098192 is tracking this. Thanks!

checkin-needed for Attachment 8520467 [details] [diff] (patch for 2.1). 2.0 is still under testing. Thanks!

Comment on attachment 8522786 [details] [diff] [review] Bug1083449.patch (to fix cppunittest crash, for 2.0) NOTE: Please see https://wiki.mozilla.org/Release_Management/B2G_Landing to better understand the B2G approval process and landings. [Approval Request Comment] Bug caused by (feature/regressing bug #): Wifi User impact if declined: Toggling Wifi would cause b2g to crash Testing completed: Yes Risk to taking this patch (and alternatives if risky): No String or UUID changes made by this patch: No

Comment on attachment 8522786 [details] [diff] [review] Bug1083449.patch (to fix cppunittest crash, for 2.0) [Triage Comment] Switching the approval to b2g32.

*Please* don't reopen bugs for release branch backouts. We have uplift bug queries that depend on bug resolution and doing so causes uplifts to be missed (as happened here). Status flags are the correct way to track uplifts.

https://hg.mozilla.org/releases/mozilla-b2g34_v2_1/rev/c5a3f8d101d4 https://hg.mozilla.org/releases/mozilla-b2g32_v2_0/rev/c756bd8bf3c3

(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #80) > *Please* don't reopen bugs for release branch backouts. We have uplift bug > queries that depend on bug resolution and doing so causes uplifts to be > missed (as happened here). Status flags are the correct way to track uplifts. Thanks for clarifying :)