[Blocking Requested - why for this release]: This is a re-open of bug 1033618 which is still happening.

hi Kevin, We are again hitting this issue after 48 hour stability testing. STR will be difficult. Observations: 1) |adb shell b2g-info| shows that homescreen process is not running. 2) its not a memleak. Could you please provide us a patch which will log more details during homescreen launching and will help us to understand why it is not launched again after killed ?

This could be a dupe of bug 1038854.

Observed on: Device: msm8610 Gonk Version: AU_LINUX_GECKO_B2G_KK_3.6.01.04.00.000.043 Moz BuildID: 20140721000201 B2G Version: 2.0 Gecko Version: 32.0a2 Gaia: http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=8cb1a949f2e9650bb2c5598e78a6f24a58bbaf97 Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=5f27d3ee3ccf01ac91a3efacb5e3e22ea62fd73c

Thanks for the info Tapas. It sounds like we should wait for a patch in bug 1038854 and check with that patch.

Blocks a blocker.

DOes this fall under your plate?

According to comment 2 and 4, the patch in bug 1038854 was just landed on 8/7. We need to retest with that patch. Hi Greg, could you please help?

I have no comment before getting test result of the bug 1038854. Please ni me, again, if bug 1038854 does not fix the issue.

We have been running tests with the fixes in bug 1038854. We will update if it shows up again.

(In reply to Inder from comment #9) > We have been running tests with the fixes in bug 1038854. We will update if > it shows up again. The fix for bug 1038854 has already landed and has been built into AUs since AU51.

We've ruled out anything Nuwa or process launching related. The homescreen is definitely running. Still investigating why it doesn't paint.

(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #12) > We've ruled out anything Nuwa or process launching related. The homescreen > is definitely running. Still investigating why it doesn't paint. As discussed, here is the layer dump for both affected and unaffected device

Hi Kyle, As discussed , here is gecko memory report from unaffected device.

The homescreen is missing most of its DOM nodes, so this is a Gaia bug until proven otherwise.

Hi Greg, Kyle, Can you provide any of: 1 - Screenshot the problem. 2 - Video of getting into this state. 3 - Dump of the dom nodes of the homescreen (according to comment 15).

4 - Also a logcat of the homescreen would be useful, didn't see one here.

Passing this to Kevin as he's looking into this issue.

> 3 - Dump of the dom nodes of the homescreen (according to comment 15). The about:memory reports have that (via the CC dumps). I can show you how to read it tomorrow. Tapas has #1 and #4.

(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #19) > > 3 - Dump of the dom nodes of the homescreen (according to comment 15). > > The about:memory reports have that (via the CC dumps). I can show you how > to read it tomorrow. > > Tapas has #1 and #4. Here is the link to download #1 and #4 from device which we debugged together for this issue. https://drive.google.com/file/d/0B1cSMS8_GuAETF93SlpGVkM0OHc/edit?usp=sharing

I see some lines in the logcat regarding dragdrop. I don't think this actually impacts this issue, but removing the line will help us debug it, and maybe even fix it. Bug 1051061 will fix this.

Tapas - we are going to try to get the patch from bug 1051061 reviewed and landed soon, but in the meantime would you want to try to test with this patch? Also - the line numbers in the log do not match up. Could you provide us the homescreen zip file for us to debug this. The file should be located at: /system/b2g/webapps/verticalhome.gaiamobile.org/application.zip

(In reply to Kevin Grandon :kgrandon from comment #23) > Tapas - we are going to try to get the patch from bug 1051061 reviewed and > landed soon, but in the meantime would you want to try to test with this > patch? > Thanks . I asked out internal test team to test it. > Also - the line numbers in the log do not match up. As we discussed in #gaia IRC, here is the exact gaia/gecko SHA1s which we picked up for logs in Comment 20 https://www.codeaurora.org/cgit/quic/lf/b2g/mozilla/gaia/commit/?h=mozilla/v2.0&id=8cc28fd31905a0ea2b2e15d13e80a0eab2feb1ba https://www.codeaurora.org/cgit/quic/lf/b2g/mozilla/gecko/commit/?h=mozilla/v2.0&id=f7bd772b1e42774708a4ede13b149a1706a59b25 Please note that we also cherry-picked following patches on top of above gaia/gecko: gaia patch: bug 1050751 attachment 8470312 [details] , bug 1050423 gecko patch: bug 1008791, bug 1030112, bug 1050423, bug 1028532, bug 1047149, bug 1044322, bug 1047645, bug 1049806 attachment 8469653 [details] [diff] [review] > Could you provide us the > homescreen zip file for us to debug this. > The file should be located at: > /system/b2g/webapps/verticalhome.gaiamobile.org/application.zip Yes. I attached it here.

Thanks Tapas. These log lines do match up and I will see if there is additional information I can pull out of the logs. Are you by chance able to provide a remote debug session to the device in this state with the app manager enabled for certified applications? These are the device preferences necessary to access the app manager: https://github.com/mozilla-b2g/gaia/blob/master/build/preferences.js#L256 I am essentially looking for the HTML content of the body, as well as the content within the javascript object located at: app.grid.getItems().

(In reply to Kevin Grandon :kgrandon from comment #25) > Thanks Tapas. These log lines do match up and I will see if there is > additional information I can pull out of the logs. > > Are you by chance able to provide a remote debug session to the device in > this state with the app manager enabled for certified applications? > > These are the device preferences necessary to access the app manager: > https://github.com/mozilla-b2g/gaia/blob/master/build/preferences.js#L256 > > I am essentially looking for the HTML content of the body, as well as the > content within the javascript object located at: app.grid.getItems(). Sorry, device is rebooted but i can arrange another remote gdb session with app manager enabled. Thanks for the ideas.

(In reply to Kevin Grandon :kgrandon from comment #25) > Thanks Tapas. These log lines do match up and I will see if there is > additional information I can pull out of the logs. > > Are you by chance able to provide a remote debug session to the device in > this state with the app manager enabled for certified applications? > > These are the device preferences necessary to access the app manager: > https://github.com/mozilla-b2g/gaia/blob/master/build/preferences.js#L256 > > I am essentially looking for the HTML content of the body, as well as the > content within the javascript object located at: app.grid.getItems(). Please note that it is always fast if we can do analysis using logcat logs. So please give us some patch for this if possible.

Tapas - here is a patch which will add a bunch of output to the vertical homescreen. It's not nearly as large as debugging IPC logging, but it should still provide significant logging. Please try this patch out if possible and send me the updated logs if you have any. https://github.com/mozilla-b2g/gaia/pull/22805

We got device ready with gdb access with this issue reproduced. Tapas is going to get in touch with Kevin for debug.

(In reply to Inder from comment #29) > We got device ready with gdb access with this issue reproduced. Tapas is > going to get in touch with Kevin for debug. Sorry, nevermind -- wrong bug :(. We are still not able to reproduce this issue with additional logs.

New test case needs to be added. There is no existing test case.

Test case added in moztrap: https://moztrap.mozilla.org/manage/case/14321/

Not reproducing anymore, closing. Will reopen if it comes back again.

it is reproduced again and It has following logs: 1) Screenshot of affected device after pressing home key 2) memory report from affected device after pressing home key 3) kernel and logcat logs We also hav gdb acess to affected device . Please make a NI on me if you want to debug it remotely using gdb or not

NI :kgrandon here to get started on the investigation.

Kyle, I attached another memory report from same device and it contains gc-cc logs for visualizing dom tree nodes I remember that gecko memory report and dumping layer tree (using gdb trick) didn't help us here last time and you concluded that homescreen does not have layers for icon Please let me know how I can help you to debug further

In the latest screenshots there is zero chrome from the homescreen as all. Tracking as a system issue for now.

Updating the screenshot based on latest findings. I don't think it's the same as the originally reported issues, so clearing out legacy attachments as well.

I agree with Kevin that this appears to be different since the search box is not painting. The layer tree dump might prove useful after all.

Although the DOM does look pretty much the same in both the system app and the homescreen compared to last time.

(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #39) > I agree with Kevin that this appears to be different since the search box is > not painting. The layer tree dump might prove useful after all. here is the layer tree dump in logcat after pressing homekey on affected device.

The layer tree dump indicates that we're scrolled down 56 pixels on the home screen, which is probably why we don't see the search box. Other than that this appears to be identical to the last time we looked at this.

just as an update: We have both marionette and gdb access to affected device. Please let us know if you want to try something on device for debugging.

(In reply to Tapas Kumar Kundu from comment #43) > just as an update: We have both marionette and gdb access to affected > device. Please let us know if you want to try something on device for > debugging. I did check with kyle sometime back and he had sufficient information to investigate and we discussed we'll do additional follow-up once he has some ideas from his debugging.

We noticed this in the logs which seems weird: 08-20 18:31:27.137 32331 32331 E GeckoConsole: [JavaScript Error: "TypeError: manifest.entry_points is undefined" {file: "app://verticalhome.gaiamobile.org/gaia_build_defer_index.js" line: 417}] Tapas - this patch will add a small safeguard and adds some logging when we can't find an entrypoint. Can you apply this patch and try to get into this scenario again? Thanks!

I've added a second commit to the pull request with more logging.

(In reply to Kevin Grandon :kgrandon from comment #46) > I've added a second commit to the pull request with more logging. I am asking our internal team to reproduce with this log. I will update here asap

Issue is reproduced on device around "08-21 13:55:00"

Kevin has this under control for now.

This is a good hint: 08-21 13:55:07.076 9786 9786 E GeckoConsole: [JavaScript Error: "TypeError: cyclic object value" {file: "app://verticalhome.gaiamobile.org/gaia_build_defer_index.js" line: 330}] Going to see if I can reproduce this on a phone, otherwise I will provide another patch which should flush this out a bit more.

Tapas - I am investigating to see if I can reproduce the situation locally. In the meantime, I've refreshed the attached patch to change the logging. Would you mind applying it again and reporting the logs? Thank you. https://github.com/mozilla-b2g/gaia/pull/23115

(In reply to Kevin Grandon :kgrandon from comment #51) > Tapas - I am investigating to see if I can reproduce the situation locally. > In the meantime, I've refreshed the attached patch to change the logging. > Would you mind applying it again and reporting the logs? Thank you. > > https://github.com/mozilla-b2g/gaia/pull/23115 Thanks for helping . I will give it a try.

Is there any progress on here? There hasn't been any update for five days. Thanks

We thought that this issue is fixed as it was not coming stability runs. But we have seen this issue again in latest stability run today. I will update here as soon as I get logs from our test team .

(In reply to Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me) from comment #54) > We thought that this issue is fixed as it was not coming stability runs. > But we have seen this issue again in latest stability run today. I will > update here as soon as I get logs from our test team . Inder/Tapas, any logs on this yet ?

(In reply to bhavana bajaj [:bajaj] from comment #55) > (In reply to Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me) from > comment #54) > > We thought that this issue is fixed as it was not coming stability runs. > > But we have seen this issue again in latest stability run today. I will > > update here as soon as I get logs from our test team . > > Inder/Tapas, any logs on this yet ? We tried to reproduce it in test build with logging patch from Comment 52 but that test build didn't reproduce this issue. But we are seeing same issue in another test build.

Hi Kevin, Could you please take a look in logs of Comment 56 ?

After inspecting the patch, it appears that the issue might not reproduce after applying it because we are catching the exception. It seems absolutely impossible to get into this case, so I'm not sure how it's happening, unless you have some test app that is doing weird things, or you are installing some malformed apps. If you are not seeing additional weirdness with the homescreen it seems that we can land this guard and it may fix it. Though it seems this may be an issue that the user will never see. Cristian - Could you review this patch? Thanks!

Comment on attachment 8480998 [details] Github pull request LGTM, thanks

In master: https://github.com/mozilla-b2g/gaia/commit/4419091ae760328a52606920e335331a16fcb448 v2.0: https://github.com/mozilla-b2g/gaia/commit/aa7bcde4a751aaf65dd8c57bdfa6cbd75eabaa77 Tapas - please let us know if you run into this issue again or see something similar. Thanks!

Tapas -- please monitor this fix in the stability runs. Keeping a ni on you.

(In reply to Inder from comment #61) > Tapas -- please monitor this fix in the stability runs. Keeping a ni on you. I will update here if this really fixes this bug :)

We are not seeing this issue for a long time. It seems to be fixed !