User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:34.0) Gecko/20100101 Firefox/34.0 (Beta/Release) Build ID: 20140831030206 Steps to reproduce: 1. Launch Games/Sudoku from homescreen (or Cupcakes, Paddle Game) 2. Leave it for 1 or 2 min. 3. Rotate your device (portrait => horizontal or vice versa) (If lock screen or display timeout is set, unlock them, and then confirm your device) ==> Lockup occurs. (100% reproducible) Actual results: Lockup -All processes are alive normally but never reply from input. Expected results: No lockup

This is a log file that is captured when lockup occurs. The last gecko log message is printed at 09-05 06:25:01.177 and 09-05 06:25:11.957. After that time, any input never be processed.

It looks like a rendering issue and a deadlock issue according to the callstack from the locked device. Both b2g process and sudoku app process are waiting for each other. - B2g process is waiting for the reply for mozilla::layers::PLayerTransactionChild::SendUpdate(). - Sudoku app process is waiting for the completion of mozilla::gl::SharedSurface_Gralloc::WaitForBufferOwnership() while processing nsViewManager::ProcessPendingUpdates(). It needs a specialist for rendering to analyze this issue.

Can you make sure this happens on Flame and provide the SW information for reproducing this?

Yes, It's certain to happen on flame. My gaia/gecko version - gaia : (v2.0) d056733f8a7a1a152f5458b323f092c47dbffa48 - gecko : (v2.0) 408b724952cfb0409c7367d0052037d91d335e79

Youngwoo, can you please also provide the game you find? Danny, apparently the lock up is in Gecko, caused by this game. When you have enough information can you please take a look?

Games folder in homescreen includes Sudoku, Cupcakes and Paddle Game. They are included in gaia v2.0. And you can open Sudoku app with http://sudoku.tresensa.com on a browser, because it's a general html5 webapp. I saw that the same lock up happens also in a browser.

FYI, The Game folder is not consistent for everyone, it is a live search as you open.

(In reply to Youngwoo Jo from comment #2) > It looks like a rendering issue and a deadlock issue according to the > callstack from the locked device. > > Both b2g process and sudoku app process are waiting for each other. > - B2g process is waiting for the reply for > mozilla::layers::PLayerTransactionChild::SendUpdate(). > - Sudoku app process is waiting for the completion of > mozilla::gl::SharedSurface_Gralloc::WaitForBufferOwnership() while > processing nsViewManager::ProcessPendingUpdates(). > > It needs a specialist for rendering to analyze this issue. Dear. Sotaro can you advice to us about comment #2

Hi Youngwoo, Jeremy - I download the Sudoku game you mentioned, but I cannot reproduce here in Taipei, could you provide a video showing the symptoms? Thanks Vance

This is a video file to show how to reproduce the issue. In the video, I made the device portrait for about 40s, and then made it horizontal. After that, it never respond.

Thanks for the video Jeremy, however oddly enough i still cannot reproduce this issue here with the newest 2.0 build. Please refer to my video: https://www.youtube.com/watch?v=asmdRvp5z9k&feature=youtu.be Could you try with the newest 2.0 build on flame as well? or could you let me what i did wrong on my video? Thanks for your help Vance

I also can not reproduce the problem on latest v2.0 flame. But it might be related to CanvasLayer destruction and re-creation.

A possible fix.

Youngwoo, jeremy can you confirm if attachment 8490040 [details] [diff] [review] fix the problem. And if it does not fix the problem, can you provide actual call stack?

I'm sorry for the late response. I found the latest v2.0 flame works fine. Since the commit from Bug 1049251 has been applied, flame is OK. However, my own device still has the same problem. So I will attach the callstack retrieved from the previous version of flame. It shows that b2g and sudoku(browser) are in deadlock.

(In reply to Youngwoo Jo from comment #15) > I'm sorry for the late response. I found the latest v2.0 flame works fine. > Since the commit from Bug 1049251 has been applied, flame is OK. It just disable SkiaGL depend on memory and screen size. It does not fix the SkiaGL and SurfaceStream's problem. The problem could happen on another device and another application.

FYI: canvas 2d related class diagram is the following. https://github.com/sotaroikeda/firefox-diagrams/blob/master/dom/dom_canvas_CanvasRenderingContext2D_FirefoxOS_2_1.pdf?raw=true

I locally confirmed the problem by disabling Bug 1049251.

[Blocking Requested - why for this release]: This bug could make application freeze.

QA Wanted for branch checks.

(In reply to Sotaro Ikeda [:sotaro] from comment #18) > I locally confirmed the problem by disabling Bug 1049251. I locally confirmed that attachment 8490040 [details] [diff] [review] fix the problem in Comment 18.

Need to make clear why the problem happens.

(In reply to Sotaro Ikeda [:sotaro] from comment #21) > (In reply to Sotaro Ikeda [:sotaro] from comment #18) > > I locally confirmed the problem by disabling Bug 1049251. > > I locally confirmed that attachment 8490040 [details] [diff] [review] fix > the problem in Comment 18. From the debugging attachment 8490040 [details] [diff] [review] seem not related to the problem.

Compositor thread in b2g process seems freezing when the problem happens. Then b2g main thread and applicaiton's main thread seem to freeze.

The freeze seems to happen by waitng AcquireFence Complete.

The AcquireFence is created in SharedSurface_Gralloc::Fence() in an applicaiton process. Then delivered to b2g process. http://mxr.mozilla.org/mozilla-central/source/gfx/gl/SharedSurfaceGralloc.cpp#170

The patch add logout that used for debugging. And it also re-enable the freeze in flame.

attachment 8490276 [details] has a lot of the following error message > W/Adreno-GSL( 1302): <gsl_ldd_control:397>: ioctl fd 93 code 0xc0140910 (IOCTL_KGSL_RINGBUFFER_ISSUEIBCMDS) failed: errno 35 Resource deadlock would occur

Comment 29 seems like driver's internal problem.

(In reply to Sotaro Ikeda [:sotaro] from comment #30) > Comment 29 seems like driver's internal problem. It seems that acquire fence does not complete because of the above error.

(In reply to Sotaro Ikeda [:sotaro] from comment #29) > attachment 8490276 [details] has a lot of the following error message > > > W/Adreno-GSL( 1302): <gsl_ldd_control:397>: ioctl fd 93 code 0xc0140910 (IOCTL_KGSL_RINGBUFFER_ISSUEIBCMDS) failed: errno 35 Resource deadlock would occur From attachment 8490276 [details], the first AcquireFence since display timeout and unlock did not complete.

Can you also share the kernel logs (adb shell cat /proc/kmsg), when the issue occurs?

Actually same to dmesg log.

In attachment 8490301 [details], the following seems related to gpu. > <3>[ 297.951654] kgsl kgsl-3d0: |adreno_ft_detect| Proc Browser, ctxt_id 4 ts 873used GPU for 2050 ms long ib detected on global ts 13642 > <3>[ 297.963314] kgsl kgsl-3d0: |_adreno_check_long_ib| IB ran too long, invalidate ctxt > <3>[ 297.970332] kgsl kgsl-3d0: |_adreno_ft| Bad context commands failed > <3>[ 297.981096] kgsl kgsl-3d0: |adreno_ft| policy 0x6 status 0x0

Sushil, can you investigate more about the freeze on your side?

I am not familiar with KGSL. I need to check with graphics team.

Thanks :-)

Sotaro, can you also test by disabling HWC (from Settings) to force GPU Composition ? I want to make sure it is not related to HWC.

I just confirmed that the freeze happened also by disabling HWC.

Please raise an SR for this issue.

thanks all, following as Comment #36, we will fill a SR in internally.

Un-assign from this bug based on Comment 42 and Comment 43.

I could not reproduce this issue on the latest builds when attempting to branch check. If desired I can do checks on builds from around the date of the build in comment 0, but I was unable to find that exact build. Issue does not occur on the latest Flame KK 2.2, Flame KK 2.1, Flame KK 2.0, Flame 2.0 on the v123 JB base. Environmental Variables: Device: Flame 2.2 BuildID: 20140916133056 Gaia: e2d70bee03b5380ac327a145e5d694fb2443f018 Gecko: 55b46de164d8 Version: 35.0a1 (2.2) Firmware Version: v165 User Agent: Mozilla/5.0 (Mobile; rv:35.0) Gecko/35.0 Firefox/35.0 Environmental Variables: Device: Flame 2.1 BuildID: 20140917073957 Gaia: 47939f4c41d0c941e5047e5d1af74a79b7d8e0d5 Gecko: d7ad9b5167d8 Version: 34.0a2 (2.1) Firmware Version: v165 User Agent: Mozilla/5.0 (Mobile; rv:34.0) Gecko/34.0 Firefox/34.0 Device: Flame 2.0 BuildID: 20140917073857 Gaia: 31434a3949556171f3565ca47ac2b44e810e95e6 Gecko: e02fe140c0d5 Version: 32.0 (2.0) Firmware Version: v165 User Agent: Mozilla/5.0 (Mobile; rv:32.0) Gecko/32.0 Firefox/32.0 Environmental Variables: Device: Flame 2.0 BuildID: 20140917073857 Gaia: 31434a3949556171f3565ca47ac2b44e810e95e6 Gecko: e02fe140c0d5 Version: 32.0 (2.0) Firmware Version: v123 User Agent: Mozilla/5.0 (Mobile; rv:32.0) Gecko/32.0 Firefox/32.0 Issue did occur on Central 2.1 from 8-31. Environmental Variables: Device: Flame 2.1 BuildID: 20140831181816 Gaia: e7d31f0e9b6b19d9b484eeec8fb980718bc40d79 Gecko: ef40d40dd8c9 Version: 34.0a1 (2.1) Firmware Version: v123 User Agent: Mozilla/5.0 (Mobile; rv:34.0) Gecko/34.0 Firefox/34.0

(In reply to Jayme Mercado [:JMercado] from comment #45) > I could not reproduce this issue on the latest builds when attempting to > branch check. If desired I can do checks on builds from around the date of > the build in comment 0, but I was unable to find that exact build. > > Issue does not occur on the latest Flame KK 2.2, Flame KK 2.1, Flame KK 2.0, > Flame 2.0 on the v123 JB base. > Environmental Variables: The bug is just masked by Bug 1049251. It change some size of canvas 2d from SkiaGL to Skia. But the cause of the problem still exist.

(In reply to jeremy.kim.leo from comment #43) > thanks all, > following as Comment #36, we will fill a SR in internally. A result of asking partner(in through SR), <3>[ 220.981666] kgsl kgsl-3d0: |adreno_ft_detect| Proc Browser, ctxt_id 3 ts 901used GPU for 2050 ms long ib detected on global ts 11420 <3>[ 220.993244] kgsl kgsl-3d0: |_adreno_check_long_ib| IB ran too long, invalidate ctxt <3>[ 221.000345] kgsl kgsl-3d0: |_adreno_ft| Bad context commands failed <3>[ 221.010964] kgsl kgsl-3d0: |adreno_ft| policy 0x6 status 0x0 this error is raised from GPU, because one application renders too heavy scenario or there are so many drawcalls with a eglSwapBuffers. This safe guard is necessary and a normal exceptional procedure to avoid overhead on GPU side. in case firefox browser on android, if it has this situation, just kill application to avoid hangup by framework(view manager???) in b2g case, if b2g has this situation, just waiting forever between b2g and app process. How is it that such a case is terminated app process like android?

I tested the same case on android firefox browser and android chrome browser. In android firefox, I observed the same issue or the black screens. But, in android chrome, It had no problem and updated the screen very smoothly, on rotated. According to this result, I suspect that this issue is caused by b2g, not by the specific app. So it needs to analyze the root cause from gecko, not from the app side and the GPU side.

(In reply to Youngwoo Jo from comment #48) > > According to this result, > I suspect that this issue is caused by b2g, not by the specific app. > So it needs to analyze the root cause from gecko, not from the app side and > the GPU side Just compare the result does not say gecko has a problem. Android and Firefox OS uses OpenGL differently. Such difference could make the symptom different.

Direct cause of freeze is made by adreno driver. Analysis of it need to be done first. Otherwise, no one know about an possibility of the workaround by use side.

Bhavana, Michael, this should be tagged as codeaurora issue, rather than core/graphics. How do we do that?

There's nothing we can do over here AFAICT. The commands getting sent to the GPU are too complex and it's protecting itself. The fix needs to come from the framework or the app itself, to reduce the GPU load.

The below comment is from SR issue, ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ There are three options. If it's a known scenario and frequent usage case, please guide them to follow the option 3 to fix the issue. 1) No change : There are some PC web sites which cause the same problem due to very heavy rendering in mobile. In this case, it looks like a hang and other tasks are also get stuck if we don't the timeout. So, we usually negotiate it with a report for similar cases. 2) timeout optimization : Each OEM have a different standard to decide how much waiting could be fine for long ib cases. If it's more than 5~6 seconds, some people can think that a phone is totally locked up and some people are okay for more than 10 secs. Because of that, we are saying OEMs to increase the number if OEMs are not happy with the default timeout number. Please think how long is acceptable waiting for your OS. 3) Fix in applications : As it's verified that it's a long ib issue, we can fix it if application is changed to avoid the situation. If it's a native application or frequent usage from your platform, we strongly recommend to change the application to divide into multiple short ibs instead of making heavy rendering cases. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ I tested some timeout values according to 2)timeout optimization, The current(default) timeout value is 2 seconds, and this value makes the lock up. => In case of 3 seconds, lock up occurs. => In case of 4 seconds, no lock up I observed 4 seconds timeout is OK in my device. However, even if this timeout optimization is applied, all the exceptional cases from the random web content can not be handled. When a user is being on the web, firefox os phone can become totally locked up. I think it's a very serious problem. It might be just a web content issue, but in our case, it makes the phone locked up totally. And when the lock up occurs, if we kill the blocked app process, the phone gets recovered. So the root cause still exists but this exception handling is also needed in gecko-side, because we can't handle all the app-side misuses. Sotaro, could you take this issue? I think you are the best person for this issue. And also I want you to analyze and fix the root cause.

Triage - 2.0+ given the observed symptom and reproducibility

The most biggest problem is an virtually invalid AcquireFence is delivered to the Compositor in b2g process. The AcquireFence is never signaled because, gl context is already invalidated by adreno driver. I am not sure why adreno OpenGL module permit this situation. If open gl context is already invalid stage, it should not allocate acquire fence. And if an OpneGL context that is related to AcquireFence becomes invalid, the AcquireFence should be signaled state.

The following candidates seems to exist for the workarounds. - [1] adreno OpenGL module/diriver implemet Comment 55 - [2] The application detect that OpenGL context becomes invalid. - [3] The compositor uses timeout to do fencing.

[3] seems easier to implement.

This is a regression caused by bug 1001417.

(In reply to Sotaro Ikeda [:sotaro] from comment #56) > - [3] The compositor uses timeout to do fencing. On andorid, the timeout is 1sec. It could be shorter on b2g. http://androidxref.com/4.4.4_r1/xref/frameworks/native/libs/gui/GLConsumer.cpp#687

Locally confirmed that attachment 8493880 [details] [diff] [review] fixed the problem on flame.

Youngwoo, can you confirm that attachment 8493880 [details] [diff] [review] fixed the problem?

Comment on attachment 8493880 [details] [diff] [review] patch - Set timeout to fClientWaitSync() Review of attachment 8493880 [details] [diff] [review]: ----------------------------------------------------------------- ::: gfx/layers/opengl/TextureHostOGL.cpp @@ +265,5 @@ > > EGLint status = sEGLLibrary.fClientWaitSync(EGL_DISPLAY(), > sync, > 0, > + 400000000 /*400 usec*/); Please add a comment that briefly explains why we need to time out and links to this bug.

I confirmed that lock up never happen. However, I found one issue that it can not complete painting in the below case, with the same GPU errors. Steps to reproduce 1. Launch sudoku app in vertical mode. 2. Wait until LCD is automatically off and don't rotate horizontally until LCD off 3. Turn on LCD. 4. Rotate horizontally. In this case, painting is incomplete and many gpu error logs are printed. However, the incomplete painting does not happen if the landscape mode is painted once by rotating before LCD timeout. I think the lock up issue is resolved by Sotaro's patch. However, there are still GPU timeout(2 seconds) errors and it seems to affect the painting in an app.

Youngwoo, thanks for the confirmation. The problem is fixed by the patch. A problem in comment 64 is a different problem. It should be handle in a different bug. I am going to create a bug for it.

(In reply to Youngwoo Jo from comment #53) > The below comment is from SR issue, > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > There are three options. > If it's a known scenario and frequent usage case, please guide them to > follow the option 3 to fix the issue. > > 1) No change > : There are some PC web sites which cause the same problem due to very heavy > rendering in mobile. > In this case, it looks like a hang and other tasks are also get stuck if we > don't the timeout. > So, we usually negotiate it with a report for similar cases. I do not think this problem is caused by "very heavy rendering". The adreno driver seems falsely detect it as doing "very heavy rendering" in the STR.

Please request b2g32 and aurora approval on this patch when you get a chance.

Comment on attachment 8493880 [details] [diff] [review] patch - Set timeout to fClientWaitSync() NOTE: Please see https://wiki.mozilla.org/Release_Management/B2G_Landing to better understand the B2G approval process and landings. [Approval Request Comment] Bug caused by (feature/regressing bug #): Bug 1001417. User impact if declined: User could face UI and whole system freeze. Testing completed: locally tested. Risk to taking this patch (and alternatives if risky): low risk. String or UUID changes made by this patch: none.

Comment on attachment 8493880 [details] [diff] [review] patch - Set timeout to fClientWaitSync() Adding verifyme for help with verification once this lands.

https://hg.mozilla.org/releases/mozilla-aurora/rev/cb2848634e33 https://hg.mozilla.org/releases/mozilla-b2g32_v2_0/rev/61d78a6855fb

[Blocking Requested - why for this release]: The freeze could happen also on b2g-v1.4.

Please nominate this patch for b2g30 approval when you get a chance :)

Comment on attachment 8493880 [details] [diff] [review] patch - Set timeout to fClientWaitSync() NOTE: Please see https://wiki.mozilla.org/Release_Management/B2G_Landing to better understand the B2G approval process and landings. [Approval Request Comment] Bug caused by (feature/regressing bug #): Bug 1001417. User impact if declined: User could face UI and whole system freeze. Testing completed: locally tested. Risk to taking this patch (and alternatives if risky): low risk. String or UUID changes made by this patch: none

Issue is verified fixed in Flame 2.0, 2.1, 2.2 (Full Flash, nightly). Actual Result: Third party app (Cupcakes and Veggies) does not crash device after idling. Device: Flame 2.0 Build ID: 20141024000201 Gaia: 86d83f4b4111ca45ebc92ca779348cc966f43cff Gecko: f8432250efb7 Version: 32.0 (2.0) Firmware Version: v188 User Agent: Mozilla/5.0 (Mobile; rv:32.0) Gecko/32.0 Firefox/32.0 Device: Flame 2.1 Build ID: 20141024001204 Gaia: 0f76e0baac733cca56d0140e954c5f446ebc061f Gecko: 7d78ff7d25b6 Version: 34.0 (2.1) Firmware Version: v188 User Agent: Mozilla/5.0 (Mobile; rv:34.0) Gecko/34.0 Firefox/34.0 (Test with 319 and 512 MB memory) Device: Flame Master Build ID: 20141024040202 Gaia: d893a9b971a0f3ee48e5a57dca516837d92cf52b Gecko: a5ee2769eb27 Version: 36.0a1 (Master) Firmware Version: v188 User Agent: Mozilla/5.0 (Mobile; rv:36.0) Gecko/36.0 Firefox/36.0

Issue is verified fixed in Flame 1.4 build (Full Flash, nightly). Actual Results: Third party app (Cupcakes and Veggies) does not crash device after idling. Device: Flame 1.4 Build ID: 20141027000203 Gaia: bb76c81f83e1e4acc2d2972a451db2bce78c8f34 Gecko: 1bde54b2e7b0 Version: 30.0 (1.4) Firmware Version: v123 User Agent: Mozilla/5.0 (Mobile; rv:30.0) Gecko/30.0 Firefox/30.0

Verify passed, this issue can't be repro on Woodduck 2.0. Test with Cupcakes. Attached: Verify_Woodduck_Game.MP4 Reproducing rate: 0/5 Build version: Gaia-Rev 7bc7f75d712ccef535fd371bfcc7fe61dcdcf874 Gecko-Rev 8f21e6d8abf8ee01d8da066495d8febf3138375a Build-ID 20141113050313 Version 32.0