Closed Bug 1027592 Opened 10 years ago Closed 10 years ago

[Flame with 273MB only] Camera app receives sigkill after 15mins 480p recording in 256MB devices

Categories

(Firefox OS Graveyard :: Gaia::Camera, defect)

Product:
Firefox OS Graveyard
Component:
Gaia::Camera
Platform:
ARM
Gonk (Firefox OS)
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

(blocking-b2g:2.0+, b2g-v2.0 wontfix, b2g-v2.1 wontfix)

Status:
RESOLVED FIXED
Project Flags:
blocking-b2g 2.0+
Tracking Flags:
Tracking Status
b2g-v2.0 --- wontfix
b2g-v2.1 --- wontfix

People

(Reporter: poojas, Assigned: mikeh)

References

(
URL
)

Details

(Keywords: smoketest, Whiteboard: [caf priority: p2][CR 680340] [273MB-Flame-Support])

Attachments

(4 files)

dmesg after the crash
(deleted), text/plain
Details
memlimit - script to set memory limit on Flame
(deleted), application/x-shellscript
Details
Bisection of memory limit and its affect on video recording
(deleted), text/plain
Details
Pull Request
(deleted), text/x-github-pull-request
wilsonpage
: feedback+
jsmith
: feedback-
Details 
Device: Flame devices (msm8610) with 256MB memory
Pre condtion: 
Keep a moving/rotating colorful object in background so that consecutive frames will not have same data.

STR:
1. Open Camera app 
2. Set Video Resolution to 480p
3. Switch to camcorder, start recording and wait for 15 minutes.

Actual behavior:
Camera app got killer by LMK.
Also for lower resolution like cif (352x288) camera gets killed after 25 minutes recording.

Expeced Behaviour:
Camera should be able to record <15 min

Not reproduced in 512MB devices 

kernel logs:
<6>[  665.028111] send sigkill to 8808 (Camera), adj 134, size 2870
Logcat:
01-01 23:30:02.310   215   215 E OomLogger: [Kill]: send sigkill to 1294 (Camera), adj 667, size 2865
Whiteboard: [CR 680340] → [caf priority: p2][CR 680340]
Can you periodically grab about:memory reports over the course of this test?
Flags: needinfo?(poojas)
Can you provide "adb shell b2g-procrank --oom" data before LMK here?
Sam, do you meet the same issue on dolphin?
Flags: needinfo?(ying.xu)
Flags: needinfo?(sam.hua)
Hema

Please check if this can be taken in 1.4?

QA,

Please check on 1.3 with lower memory.
Flags: needinfo?(hkoka)
Keywords: qawanted
QA-Wanted: We can't test this with 256 MB Flame due to Bug 1008050 blocking us; let's try to test this on 1.4 Buri and if we can repro move on the 1.3 Buri to do the 1.3 check
Unable to reproduce this issue on 1.4 Buri. I filmed a 16 min video without crashing or any issues.

Unable to reproduce this issue on 1.4 Flame with 288mb mem (please reference https://bugzilla.mozilla.org/show_bug.cgi?id=1024692#c12 for this mem decision). I filmed a 17+ min video without crashing. I wasn't able to set video resolution to 480p (there's no option available) so I filmed with the default settings (1280x720)

The above tests were done with camera looking at this gif the entire time http://seamac.info/cupboard/cinema/gif/animoshuns/saverlab/plasma3.gif

Tested on:
(no repro)
Device: Flame
Build ID: 20140619000201
Gaia: 233da2865de3a2c73329e04c6deb21766f0d09d4
Gecko: 7978e6b470f2
Version: 30.0 (1.4) 
Firmware Version: v121-2

(no repro)
Device: Buri
Build ID: 20140619000201
Gaia: 233da2865de3a2c73329e04c6deb21766f0d09d4
Gecko: 7978e6b470f2
Version: 30.0 (1.4) 
Firmware Version: v1.2device.cfg
QA Whiteboard: [QAnalyst-Triage?]
Flags: needinfo?(jmitchell)
Keywords: qawanted
Flags: needinfo?(jmitchell)
we need a way to reproduce it (see comment 4/5)
Flags: needinfo?(hkoka)
QA Whiteboard: [QAnalyst-Triage?] → [QAnalyst-Triage+][lead-review+]
Is this happening on dolphin? QA, please help reproduce.

Thanks
Hema
More info being tracked at, Bug 1029856 on possible memory leak
Wayne

Please follow up with SPRD if issue is seen on Dolphin
Flags: needinfo?(wchang)
Minusing this since no reproduction has been seen or informed about by partner for almost 2 weeks.
blocking-b2g: 1.4? → ---
Flags: needinfo?(wchang)
As per commnet#8 iam removing needinfo from myself :)
Flags: needinfo?(poojas)
This issue also occurs on Flame 2.1 and 2.0 with 273mb memory. When the user starts recording a video, within 10 seconds the devices restarts or displays an empty homescreen.

Flame 2.1

Environmental Variables:
Device: Flame Master
BuildID: 20140709040203
Gaia: 0f9f11d0a6dadb3ea27160204bbe911c1ad69a6f
Gecko: 196d05832e12
Version: 33.0a1 (Master) 
Firmware Version: v122
User Agent: Mozilla/5.0 (Mobile; rv:33.0) Gecko/33.0 Firefox/33.0

Flame 2.0

Environmental Variables:
Device: Flame 2.0
BuildID: 20140708000322
Gaia: e935f4ff190b76c70d9b2af8856c542a6e4a7546
Gecko: 3f9d7a3a0b7b
Version: 32.0a2 (2.0) 
Firmware Version: v122
User Agent: Mozilla/5.0 (Mobile; rv:32.0) Gecko/32.0 Firefox/32.0

The issue does NOT occur on devices with 512mb memory.
QA Whiteboard: [QAnalyst-Triage+][lead-review+] → [QAnalyst-Triage?][lead-review+]
status-b2g-v2.0: --- → affected
status-b2g-v2.1: --- → affected
Flags: needinfo?(ktucker)
Attached file dmesg after the crash (deleted) — 
I can confirm that with |fastboot oem mem 273| as per comment 12, the Camera app crashes almost right away when starting recording. Highlights from the kernel message dump:

<6>[   41.124629] send sigkill to 970 (Built-in Keyboa), adj 667, size 2802
<4>[   41.124814] Built-in Keyboa used greatest stack depth: 4708 bytes left
  ...
<6>[   71.843742] send sigkill to 1116 (Smart Collectio), adj 667, size 1979
<3>[   73.015112] msm_dsi_isr_handler: isr=3100003 1000000
  ...
<6>[   90.901800] goodix_ts_suspend i2c_transfer send sigkill to 1061 (FTU), adj 667, size 3558
<4>[   91.225341] Chrome_ChildThr used greatest stack depth: 4588 bytes left
  ...
<6>[  484.060150] send sigkill to 1060 (Homescreen), adj 534, size 2692
<4>[  484.071236] Chrome_ChildThr used greatest stack depth: 4340 bytes left
  ...
<6>[  499.836014] zram: Error allocating memory for compressed page: 17084, size=4096
<1>[  499.836036] Write-error on swap-device (253:0:136672)
  ...
<6>[  500.927403] zram: Error allocating memory for compressed page: 20531, size=4096
<1>[  500.927427] Write-error on swap-device (253:0:164248)
  ...
<6>[  503.698433] send sigkill to 1119 (Camera), adj 134, size 2914
<6>[  503.947668] msm_vidc: 4: Closed video instance: ce9fc000

It looks like 273MB is just not enough memory to run the video recorder on Flame.
The script I've been using to set the memory limit on Flame. Usage example:

# memlimit 512

WARNING: this script has no error checking other than that provided by the utilities it calls.
Results obtained using the following build:

Gonk      v122
Gaia      44d283dbe6ad28268e70008272f72d78cb7858c9
Gecko     https://hg.mozilla.org/integration/b2g-inbound/rev/4ad5fe9c00ba
BuildID   20140709111232
Version   33.0a1
ro.build.version.incremental=108
ro.build.date=Tue Jun 10 19:40:40 CST 2014

The thresholds below are very likely to shift with different builds:

mem >= 319 MiB:        normal
338 <= mem <= 318 MiB: one process, usually the Keyboard, gets sigkilled[1]
303 <= mem <= 317 MiB: two processes get sigkilled: Keyboard and Homescreen[2]
mem <= 302 MiB:        three process eventually get sigkilled, including the Camera[3]

1. also observed the Homescreen getting sigkilled, 1/4 times
2. occasionally one process, always the Keyboard, gets sigkilled before I press the "start recording" button
3. it took over two minutes for the Camera app to eventually get sigkilled

Also observed while processes are getting sigkilled: the viewfinder becomes mildly to extremely janky, depending on the degree of memory pressure.

Based on the above data, it looks like Flame can't support video recording with less than ~303 to 319MiB of memory.
Clarifying note 3 in comment 15: it took over two minutes for the Camera app to get sigkilled when the mem limit is set to 302 MiB. With the mem limit set to 301 MiB, the Camera app gets killed within 10 to 30s.
blocking-b2g: --- → 2.0+
A potential solution may be to drop the resolution of the recorded video.
QA Whiteboard: [QAnalyst-Triage?][lead-review+] → [QAnalyst-Triage+][lead-review+]
Flags: needinfo?(ktucker)
Keywords: smoketest
poojas, per comment 8, can you still reproduce this problem? If so, what build are you running?
Flags: needinfo?(poojas)
FYI: Mike's tests from comment 15/16 are on 1080p. 

Pooja, 

Can you please check to see if you can run your tests again and reproduce this, given https://bugzilla.mozilla.org/show_bug.cgi?id=1029856 is fixed.

Thanks
Hema
Assignee: nobody → dmarcos
(In reply to Mike Habicher [:mikeh] from comment #18)
> poojas, per comment 8, can you still reproduce this problem? If so, what
> build are you running?

Mike, this issue is reloved now.
Flags: needinfo?(poojas)
Attached file Pull Request (deleted) — 
It introduces a low tier HW configuration profile. To build the app on low end configuration we need to set the env. variable DEVICE_TIER=low. e.g:

DEVICE_TIER=low make APP=camera

This solves the following problems:

1. Out of memory issues in very low end hardware. We will have a camera profile that can disable certain features or turn down certain settings like picture or video resolutions.
2. We keep a default profile for mid/high end devices so customers can enjoy the hardware they paid for.
Attachment #8454293 - Flags: feedback?(yurenju.mozilla)
Attachment #8454293 - Flags: feedback?(wilsonpage)
(In reply to Diego Marcos [:dmarcos] from comment #21)
> It introduces a low tier HW configuration profile. To build the app on low
> end configuration we need to set the env. variable DEVICE_TIER=low. e.g:
> 
> DEVICE_TIER=low make APP=camera
> 

Diego, wondering if we are introducing a new flag here. Can we come have common solution for this bug as well as Bug 1024692 specifically https://bugzilla.mozilla.org/show_bug.cgi?id=1024692#c67 from :evilemachines
Comment on attachment 8454293 [details]
Pull Request

Deigo, we have GAIA_MEMORY_PROFILE[1] to build gaia for low-end device, we can reuse this flag for camera app.

for the pull request, you can just rename DEVICE_TIER to GAIA_MEMORY_PROFILE and it should work magically.

[1] https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/make_options_reference#Low_memory_profile_build
Attachment #8454293 - Flags: feedback?(yurenju.mozilla)
Comment on attachment 8454293 [details]
Pull Request

Nice one Diego, impressed you got your head around the all the confusing build system!

My only point of feedback is to make the config-lowend.js profile only contain the properties that differ from config-default.js. Otherwise I fear we may risk the two files getting out of sync. It's quite likely that someone will forget to update the properties in the profile config files when they are changed in the default file.

This would require doing a deep-mix of the given profile config into the default config. I can't advise how to do this in bash :(

For maintainability, I think this is quite important, so this is a conditional f+ :)
Attachment #8454293 - Flags: feedback?(wilsonpage) → feedback+
Comment on attachment 8454293 [details]
Pull Request

This will not work for any testing workflow on Flame. Flame allows dynamic setting of memory via fastboot on a certain build to test certain workflows under a certain memory setting. Asking to spin another build with this setting will be extremely expensive for rel eng, so I don't think we should relying on a build configuration setting. As such, this should be dynamically determined at runtime, rather than controlled at build time.
Attachment #8454293 - Flags: feedback-
(In reply to Jason Smith [:jsmith] from comment #25)
> Comment on attachment 8454293 [details]
> Pull Request
> 
> This will not work for any testing workflow on Flame. Flame allows dynamic
> setting of memory via fastboot on a certain build to test certain workflows
> under a certain memory setting. Asking to spin another build with this
> setting will be extremely expensive for rel eng, so I don't think we should
> relying on a build configuration setting. As such, this should be
> dynamically determined at runtime, rather than controlled at build time.

Doing this kind of thing at runtime will regress startup time, especially if we have to do costly settings requests to determine device capabilities.
FYI, Gecko side can get total system memory size by using hal::GetTotalSystemMemory(). It is used to configure Skia cache size in Bug 920160.
(In reply to Sotaro Ikeda [:sotaro] from comment #27)
> FYI, Gecko side can get total system memory size by using
> hal::GetTotalSystemMemory(). It is used to configure Skia cache size in Bug
> 920160.

Any idea if there is a quick way to get memory size on the client?
I don't want to pollute gaia with low level HW specific code. This is opening a can of worms for maintainability and complexity and goes against what the cross platform nature of the web. As we add more devices it's going to cause a combinatorial explosion of code paths in the app. I won't be almost impossible to guarantee stability. Adding a low memory build for flame is way way simpler.
Flags: needinfo?(jsmith)
Sorry, I clicked save changes too early and I had some errors on my writing. This is the corrected version of comment #29:

I don't want to pollute gaia with low level HW specific code. This is opening a can of worms for maintainability and complexity and goes against the cross platform nature of the web. Those aspects should not be handled at the gaia level. As we add more devices it's going to cause a combinatorial explosion of code paths in the app. I will be almost impossible to guarantee stability. Adding a low memory build for flame is way way simpler.
Flags: needinfo?(wilsonpage)
(In reply to Diego Marcos [:dmarcos] from comment #30)
> Sorry, I clicked save changes too early and I had some errors on my writing.
> This is the corrected version of comment #29:
> 
> I don't want to pollute gaia with low level HW specific code. This is
> opening a can of worms for maintainability and complexity and goes against
> the cross platform nature of the web. Those aspects should not be handled at
> the gaia level. As we add more devices it's going to cause a combinatorial
> explosion of code paths in the app. I will be almost impossible to guarantee
> stability. Adding a low memory build for flame is way way simpler.

I'd argue the opposite on the cross-platform concern here. The fact that the app is handling memory settings via the build means that it's not cross-platform, since the app cannot magically ported to another platform easily without forcing a build configuration change. The device increase concern just trades off the complexity for builds, which isn't cheap since this will force rel eng to generate multiple daily builds for a single device, which is an extremely expensive hosting cost. The whole point of the fastboot workflow was to allow for an easy switch of an existing build to test in different memory settings easily without introducing a new build.

Lawrence - What's your take on this in regards to the rel eng hosting cost?
Flags: needinfo?(jsmith) → needinfo?(lmandel)
We don't want to handle memory settings at build time in camera. We are just introducing a configuration profile for low end hardware (low memory is just one the characteristics of low tier devices) that will be selected at build time.

Each profile is an application state we guarantee it works. We will just have to maintain and test two discrete states of the app (default and lowend). We won't have to verify every single device with every single combination of characteristics (memory, CPU, video, image resolutions...). The explosion in the number of devices is one of the biggest challenges we're dealing with in camera at the moment. The configuration profile classifies all devices in two groups. Partners will be able to create custom profiles it they wish but they will have to deal with derived bugs.

If we pollute the code with if statements all over the place: checking memory, cpu, camera HW characteristics, resolutions... several things will happen:

1. All those checks won't be free introducing a performance penalty as Wilson mentioned.
2. We increase the different states that the application can end up. This will slow down the addition of features to a halt as we grow the number of devices. It will also compromise stability making almost impossible to guarantee that we don't introduce any regressions. 

I'm confident that the cost of introducing a new build is going to be much much lower than letting a web application micromanage hardware details at run time.
Blocks: 1037638
Diego, the issue here is that you can't classify the flame (or any other 256MB device) as a low end hardware.

It seems you don't like this idea, but a short term solution would be to use navigator.getFeature("hardware.memory");
I don't have any device where I can reproduce this on. comment #20 indicates that this have been solved. Can anyone confirm this?
Flags: needinfo?(poojas)
Flags: needinfo?(jsmith)
Flags: needinfo?(hkoka)
Mike. Is there any findings on the gecko side?
Flags: needinfo?(mhabicher)
(In reply to Diego Marcos [:dmarcos] from comment #34)
> I don't have any device where I can reproduce this on. comment #20 indicates
> that this have been solved. Can anyone confirm this?

This reproduced in today's smoketest.
Flags: needinfo?(lmandel)
Flags: needinfo?(jsmith)
comment #0 mentions that the app is also killed after recording 25 mins of cif (352x288) video in 2.0

comment #5 mentions that the problem doesn't reproduce on 1.4 recording video at 720p (1280x720) and OEM build v121-2

We still need to understand what's the root of the issue. Is v122 OEM build introducing any changes on how we handle video? Why was 256MB enough to record 720p video on 1.4 (v121-2) and on 2.0 we cannot even handle 480p?
Can hamachi with 256 MB record 25 mins of cif (352x288) video?

It hamachi can do it flame should be able to do it as well.
(In reply to Diego Marcos [:dmarcos] from comment #38)

> Can hamachi with 256 MB record 25 mins of cif (352x288) video?
> 
> It hamachi can do it flame should be able to do it as well.

It's worth noting that a 256MB hamachi has ~13MB more memory available than a 273MB flame:

Flame (aurora):

Gonk      v122
Gaia      ca022f811bcbbda0f89086094a9e92bb220fea18
Gecko     https://hg.mozilla.org/releases/mozilla-aurora/rev/b3f6dbd37993
BuildID   20140712000201
Version   32.0a2
ro.build.version.incremental=108
ro.build.date=Tue Jun 10 19:40:40 CST 2014

root@flame:/ # cat /proc/meminfo                                               
MemTotal:         171852 kB

Hamachi (aurora):

Gaia      ca022f811bcbbda0f89086094a9e92bb220fea18
Gecko     https://hg.mozilla.org/releases/mozilla-aurora/rev/b3f6dbd37993
BuildID   20140712000201
Version   32.0a2
ro.build.version.incremental=eng.cltbld.20140122.035944
ro.build.date=Wed Jan 22 04:22:01 EST 2014

root@android:/ # cat /proc/meminfo
MemTotal:         184764 kB

Des anyone have a flame flashed with v121-2 handy? If so, can you load the above flame/aurora build and run |adb shell cat /proc/meminfo| and paste he results here? Maybe the newer base image has a bigger kernel.
Flags: needinfo?(mhabicher)
With flame loaded with a v121-2 base build and mem limit set to 273MB:

root@flame:/ # getprop ro.build.date
Tue Jun 10 19:40:40 CST 2014
root@flame:/ # cat /proc/meminfo
MemTotal:         171852 kB

...the same as v122.
Flags: needinfo?(wilsonpage)
We need to make a decision on this bug. It's definitively a regression as comment #5 mentions but it seems to only affect flame on low mem configuration. Buri with 256 MB doesn't seem to have any issue recording video. Does Tarako have any problem?

Are there any other devices out there with 256 MB, other than Flame on low mem configuration, and capable of recording 480p / 720p video?

I wonder if we are stressing out on a bug that nobody is going to hit because there's no real device with such a HW configuration.

I would reconsider blocking on this if there's no device in the market affected by the problem.
Flags: needinfo?(mhabicher)
(In reply to Diego Marcos [:dmarcos] from comment #41)
> We need to make a decision on this bug. It's definitively a regression as
> comment #5 mentions but it seems to only affect flame on low mem
> configuration. Buri with 256 MB doesn't seem to have any issue recording
> video. Does Tarako have any problem?

Buri is not an accurate comparison - it does not have the same chipset that the qrd reference device uses.

> 
> Are there any other devices out there with 256 MB, other than Flame on low
> mem configuration, and capable of recording 480p / 720p video?

Note - Flame under low memory should be recording under the same conditions that the qrd has, as that matches the min specs we need to support for 2.0.

> 
> I wonder if we are stressing out on a bug that nobody is going to hit
> because there's no real device with such a HW configuration.
> 
> I would reconsider blocking on this if there's no device in the market
> affected by the problem.

Flame 273 MB was deemed as a close representation to the qrd being used for cert testing, so we must support that configuration in the 2.0 timeframe.
Does the issue reproduce on the qrd device?
Flags: needinfo?(jsmith)
Can qrd record 480p and 720p video?
Can we have qrd devices?
I can't answer that question. I think you should ask a CAF person for that info.
Flags: needinfo?(jsmith)
Hema, Who should ping to answer those questions?
Flags: needinfo?(sam.hua)
Pooja,

You mentioned in comment 20 that you are not able to reproduce this after bug 1029856 was fixed? Can you clarify if that is the case for both 480 and 720p on QRD.

Thanks
Hema
Flags: needinfo?(hkoka)
(In reply to Diego Marcos [:dmarcos] from comment #45)
> Can we have qrd devices?

Diego: They are limited, we have a couple, will check with folks who have it and try to send one to you for testing. In the meantime, NI Pooja/Inder for help with testing on QRD.

Thanks
Hema
(In reply to Hema Koka [:hema] from comment #48)
> Pooja,
> 
> You mentioned in comment 20 that you are not able to reproduce this after
> bug 1029856 was fixed? Can you clarify if that is the case for both 480 and
> 720p on QRD.

Due to memory restriction  we are limited only upto 480p Resolution and 720p doesn't supported.On 480p its not reproducible.
Flags: needinfo?(poojas)
We obviously have to fix the issue on flame but given that it doesn't reproduce on QRD, Should we still be blocking on 2.0?
Flags: needinfo?(hkoka)
(In reply to Diego Marcos [:dmarcos] from comment #51)
> We obviously have to fix the issue on flame but given that it doesn't
> reproduce on QRD, Should we still be blocking on 2.0?

Sounds like we have a UI bug (blocker?) where we should not be allowing 720p choice of recording on 256mb devices?  Do we have that bug open?
(In reply to Milan Sreckovic [:milan] from comment #52)
> (In reply to Diego Marcos [:dmarcos] from comment #51)
> > We obviously have to fix the issue on flame but given that it doesn't
> > reproduce on QRD, Should we still be blocking on 2.0?
> 
> Sounds like we have a UI bug (blocker?) where we should not be allowing 720p
> choice of recording on 256mb devices?  Do we have that bug open?

Never mind, I misinterpreted the comments above.
Diego/Mike,

While the CAF priority on this is low and is not a blocker on QRD, we still need to figure out the root cause of this issue on Flame. Because Flame (with low mem configuration) is used as our reference for testing to certify release quality and readiness (smoketests are run using this device). The fact that recording fails even for cif and 480p it looks like there is some fundamental issue with Flame configured with low memory. 

Comment #5 says we are unable to reproduce on 1.4 Flame with 288 MB and v121-2. On 2.0 Flame with 273MB and V122 base build, this is still an issue. Lets continue investigating and figuring out the root of the problem -- perhaps it is an issue with v122 base build (Creating low memory profile to enable/disable some features with Diego's patch in comment 21 will not address this particular issue, because video recording just fails in any resolution on Flame with 273MB)

Thanks
Hema
Flags: needinfo?(hkoka)
Assigning to Mike to help investigate the video recording root problem on Flame. 

Diego, lets track the low mem config profile in bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1037638
Assignee: dmarcos → mhabicher
Data point: a 273MB-flame running v121-2/v1.4 LMK's the Camera app when video recording:

18:11:14 ➜  B2G-flash-tool git:(master) memtotal
MemTotal:         171852 kB
18:11:20 ➜  B2G-flash-tool git:(master) adb shell b2g-ps
APPLICATION      USER     PID   PPID  VSIZE  RSS     WCHAN    PC         NAME
b2g              root      290   1     187844 57340 ffffffff b6eb363c S /system/b2g/b2g
(Nuwa)           root      900   290   53496  5060  ffffffff b6ec263c S /system/b2g/plugin-container
Homescreen       u0_a1040  1040  900   71028  24576 ffffffff b6ec263c S /system/b2g/plugin-container
18:11:25 ➜  B2G-flash-tool git:(master) adb shell b2g-ps
APPLICATION      USER     PID   PPID  VSIZE  RSS     WCHAN    PC         NAME
b2g              root      290   1     213724 32564 ffffffff b6eb363c S /system/b2g/b2g
(Nuwa)           root      900   290   53496  4084  ffffffff b6ec263c S /system/b2g/plugin-container
Homescreen       u0_a1040  1040  900   73292  8624  ffffffff b6ec263c S /system/b2g/plugin-container
Camera           u0_a1400  1400  900   94192  20872 ffffffff b6ec263c S /system/b2g/plugin-container
18:12:02 ➜  B2G-flash-tool git:(master) adb shell dmesg | grep sigkill
<6>[   19.338916] send sigkill to 959 ((Preallocated a), adj 667, size 1852
<6>[   33.903355] send sigkill to 1108 ((Preallocated a), adj 667, size 3551
<6>[  206.187931] send sigkill to 1494 ((Preallocated a), adj 667, size 3555
<6>[  221.850032] send sigkill to 1040 (Homescreen), adj 534, size 1609
<6>[  222.628535] send sigkill to 1400 (Camera), adj 134, size 892
Flags: needinfo?(mhabicher)
With a 288MB-flame w/ v121-2/v1.4 starting recording causes LMKs:

18:12:33 ➜  B2G-flash-tool git:(master) memlimit 288
Setting memory limit to 288 MiB
< waiting for device >
...
(bootloader) 	capcity:288
(bootloader) 	Device adjusted mem: 288m
OKAY [  0.005s]
finished. total time: 0.005s
rebooting...

finished. total time: 0.005s
18:14:20 ➜  B2G-flash-tool git:(master) memtotal
MemTotal:         187104 kB
18:14:51 ➜  B2G-flash-tool git:(master) adb shell b2g-ps
APPLICATION      USER     PID   PPID  VSIZE  RSS     WCHAN    PC         NAME
b2g              root      326   1     192560 64356 ffffffff b6f0a63c S /system/b2g/b2g
(Nuwa)           root      898   326   53496  6912  ffffffff b6f3563c S /system/b2g/plugin-container
Homescreen       u0_a1020  1020  898   83192  28188 ffffffff b6f3563c S /system/b2g/plugin-container
(Preallocated a  u0_a1116  1116  898   60732  14572 ffffffff b63de040 R /system/b2g/plugin-container
18:15:03 ➜  B2G-flash-tool git:(master) adb shell b2g-ps
APPLICATION      USER     PID   PPID  VSIZE  RSS     WCHAN    PC         NAME
b2g              root      326   1     194188 37420 ffffffff b6f0a63c S /system/b2g/b2g
(Nuwa)           root      898   326   53496  5472  ffffffff b6f3563c S /system/b2g/plugin-container
Homescreen       u0_a1020  1020  898   72052  17036 ffffffff b6f3563c S /system/b2g/plugin-container
Camera           u0_a1116  1116  898   90224  20144 ffffffff b6f3563c S /system/b2g/plugin-container
(Preallocated a  u0_a1294  1294  898   60668  15368 ffffffff b6f3563c S /system/b2g/plugin-container
18:16:23 ➜  B2G-flash-tool git:(master) adb shell b2g-ps
APPLICATION      USER     PID   PPID  VSIZE  RSS     WCHAN    PC         NAME
b2g              root      326   1     181688 27492 ffffffff b6f0a63c S /system/b2g/b2g
(Nuwa)           root      898   326   53496  1032  ffffffff b6f3563c S /system/b2g/plugin-container
Camera           u0_a1116  1116  898   104940 10480 ffffffff b6f3563c S /system/b2g/plugin-container
18:16:30 ➜  B2G-flash-tool git:(master) adb shell dmesg | grep sigkill
<6>[   19.236459] send sigkill to 934 ((Preallocated a), adj 667, size 3708
<6>[   96.395341] send sigkill to 1294 ((Preallocated a), adj 667, size 3763
<6>[  113.180293] send sigkill to 1020 (Homescreen), adj 534, size 681
With a 288MB-flame w/ v121-2/v2.1:

18:23:25 ➜  ~  memtotal
MemTotal:         187104 kB
18:23:27 ➜  ~  adb shell b2g-ps
APPLICATION      USER     PID   PPID  VSIZE  RSS     WCHAN    PC         NAME
b2g              root      298   1     206248 61908 ffffffff b6ec863c S /system/b2g/b2g
(Nuwa)           root      913   298   55304  3444  ffffffff b6e8863c S /system/b2g/plugin-container
Homescreen       u0_a1132  1132  913   122236 33140 ffffffff b6e8863c S /system/b2g/plugin-container
(Preallocated a  u0_a1183  1183  913   61456  5768  ffffffff b6e8863c S /system/b2g/plugin-container
18:23:30 ➜  ~  adb shell b2g-ps
APPLICATION      USER     PID   PPID  VSIZE  RSS     WCHAN    PC         NAME
b2g              root      298   1     212732 26704 ffffffff b6ec863c S /system/b2g/b2g
(Nuwa)           root      913   298   55304  2652  ffffffff b6e8863c S /system/b2g/plugin-container
Homescreen       u0_a1132  1132  913   107452 10212 ffffffff b6e8863c S /system/b2g/plugin-container
Camera           u0_a1183  1183  913   94436  19972 ffffffff b6e8863c S /system/b2g/plugin-container
(Preallocated a  u0_a1662  1662  913   60432  11232 ffffffff b6e8863c S /system/b2g/plugin-container
18:24:21 ➜  ~  adb shell b2g-ps
APPLICATION      USER     PID   PPID  VSIZE  RSS     WCHAN    PC         NAME
b2g              root      298   1     201108 50180 ffffffff b6ec863c S /system/b2g/b2g
(Nuwa)           root      913   298   55304  940   ffffffff b6e8863c S /system/b2g/plugin-container
Homescreen       u0_a1662  1662  913   161984 53224 ffffffff b6e88808 R /system/b2g/plugin-container
(Preallocated a  u0_a1853  1853  913   58380  3860  ffffffff b62ed12e R /system/b2g/plugin-container
18:24:49 ➜  ~  adb shell dmesg | grep sigkill
<6>[   46.795372] send sigkill to 987 (Built-in Keyboa), adj 667, size 3784
<6>[   74.393637] send sigkill to 1184 (Smart Collectio), adj 667, size 1905
<6>[  161.147733] send sigkill to 1132 (Homescreen), adj 534, size 2394
<6>[  171.639694] send sigkill to 1183 (Camera), adj 134, size 2427

A significant difference in v2.1 is that RSS numbers are down for all processes listed, except the Camera app which has an RSS of nearly twice the size as in v1.4; and the Homescreen, which is _significantly_ larger in v2.1 vs v1.4.
Further to comment 57 and comment 58, a 288MB-flame running v1.4 has just enough memory available to record a video with all other background processes getting killed. v2.1, however, seems to add enough new features to the Camera app to push it over the edge as well.
Whiteboard: [caf priority: p2][CR 680340] → [caf priority: p2][CR 680340] [273MB-Flame-Support]
A few questions:

*What's the time threshold where the LMK doesn't kill the camera app?  
*I'm assuming at 512MB, we run into this same issue, but at a longer time threshold (maybe 30+ mins?)
*Do we have data around what 1.1 or 1.3 offered time wise at 256MB?  I'd like to understand at what point this bug is considered a regression?
Summary: Camera app receives sigkill after 15mins 480p recording in 256MB devices → [Flame with 273MB only] Camera app receives sigkill after 15mins 480p recording in 256MB devices
(In reply to Chris Lee [:clee] from comment #60)
> A few questions:
> 
> *What's the time threshold where the LMK doesn't kill the camera app?

At 309MB, the Camera app survives all of the other background processes getting killed.

At 319MB, the other background processes survive as well.

> *I'm assuming at 512MB, we run into this same issue, but at a longer time
> threshold (maybe 30+ mins?)

Not that I've observed.

> *Do we have data around what 1.1 or 1.3 offered time wise at 256MB?  I'd
> like to understand at what point this bug is considered a regression?

It's been a while since I've tested 1.1 or 1.3, but I don't recall there ever being an issue with recording length on Unagi, Inari, or the like.

fabrice, some left-field thinking here, but can you think of any process on b2g and/or flame that gets triggered every 15 minutes or so that might cause an already-memory-constrained device to go OoM and cause the camera app to get killed?
Flags: needinfo?(fabrice)
(In reply to Mike Habicher [:mikeh] from comment #61)

> fabrice, some left-field thinking here, but can you think of any process on
> b2g and/or flame that gets triggered every 15 minutes or so that might cause
> an already-memory-constrained device to go OoM and cause the camera app to
> get killed?

I'm not sure how often the calendar app is waking itself up but it's worth checking. On the gecko side, we only have once per day timers for update checks.
Flags: needinfo?(fabrice)
Mike will investigate if there is any impact from calendar waking up every 15 mins

Note: This issue is only on Flame 273MB
Bug 1039883 might affect to this.
I have some more information on this, from a 273MB Flame obtained running:

Gonk      v122
Gaia      7bf9b6a6c262a8ff656a8d415c9f6cfef5c23e54
Gecko     https://hg.mozilla.org/integration/b2g-inbound/rev/03c4e6074f63
BuildID   20140721074816
Version   33.0a1
ro.build.version.incremental=109
ro.build.date=Mon Jun 16 16:51:29 CST 2014

...with the following patch applied to restrict recording to 480p:

# git diff
diff --git a/apps/camera/js/config/config.js b/apps/camera/js/config/config.js
index 52278b6..c206dc6 100644
--- a/apps/camera/js/config/config.js
+++ b/apps/camera/js/config/config.js
@@ -206,7 +206,7 @@ module.exports = {
     header: 'video-resolution-header',
     icon: 'icon-video-size',
     options: [],
-    exclude: ['high', '1080p'],
+    exclude: ['high', '1080p', '720p'],
     persistent: true,
     optionsLocalizable: false,
   },

Furthermore, the Flame was configured with a WiFi connection, and the Calendar app was configured to sync every 15 minutes with my Mozilla calendar using CalDav.

Observations:

1. Unlike with 720p recording, background processes don't get killed right away in this configuration; but eventually, most background processes get killed:

# adb shell dmesg | grep sigkill
<6>[ 6229.371288] gtp_init_ext_watchdog i2c_transfer send sigkill to 9934 (Smart Collectio), adj 734, size 2350
<6>[ 6299.375213] send sigkill to 9811 (Homescreen), adj 534, size 2429
<6>[ 6477.462492] send sigkill to 10913 (Calendar), adj 667, size 7569
<6>[ 6576.809965] send sigkill to 10745 (Homescreen), adj 534, size 1164

2. Approximately 17 minutes into the recording, the kernel log shows a new instance of the Calendar app getting killed (the Camera app survived):

# adb shell dmesg | grep sigkill
<6>[ 7283.049865] send sigkill to 11483 (Calendar), adj 667, size 1690

(Perhaps this is the app waking up in response to the previous sync alarm expiring, and getting killed before it can set a new one, as neither the kernel log nor b2g-ps show further evidence of the Calendar app.)

3. Approximately 50 to 60 minutes into the recording, the Camera app gets killed:

# adb shell dmesg | grep sigkill
<6>[ 9872.870120] send sigkill to 11011 (Camera), adj 134, size 2854

...a b2g-ps run prior to the kill shows the Camera app with VSIZE and RSS values similar to those reported by a new instance of the app:

# adb shell b2g-ps
APPLICATION      USER     PID   PPID  VSIZE  RSS     WCHAN    PC         NAME
b2g              root      9686  1     175772 22756 ffffffff b5c10dfc R /system/b2g/b2g
(Nuwa)           root      9746  9686  54412  348   ffffffff b6f2363c S /system/b2g/plugin-container
Camera           u0_a11011 11011 9746  96876  11460 ffffffff b66471b0 D /system/b2g/plugin-container
(Preallocated a  u0_a12098 12098 9746  60308  3648  ffffffff b63bbc40 R /system/b2g/plugin-container

4. At no point does the Camera viewfinder ever stop being janky. Throughout the hour-long test run, it kept freezing and jumping.
And in an attempt to repeat, the Camera app died a few minutes into a 480p recording without the Calendar app getting (re)started.

Based on this and comment 65, I think we must conclude (going back to comment 15) that 273 MB is just not enough memory to record video on a Flame.
So my understanding after talking with Hema and reading comment 66, it sounds like this is a WONTFIX, since it's simply not realistic that we can support camera recording on Flame 273 MB. We could downgrade the resolution, but that wouldn't match the QRD anymore, which does not seem to have value to pursue.

Do you agree on a WONTFIX decision on this bug?
Flags: needinfo?(mhabicher)
Also notice that the issue doesn't reproduce on QRD devices:

https://bugzilla.mozilla.org/show_bug.cgi?id=1027592#c50

The problem is flame specific
(In reply to Jason Smith [:jsmith] from comment #67)

> Do you agree on a WONTFIX decision on this bug?

Agreed.
Status: NEW → RESOLVED
Closed: 10 years ago
Flags: needinfo?(mhabicher)
Resolution: --- → WONTFIX
More information--with the camera configured to 480p on Flame, with the following build:
- gonk:  v122
- gecko: b2g-inbound:195513:0fb3c6714933
- gaia:  master:77e73c20c180fdf7f37247fa60a3d1886cdc726e

With various memory limits:
mem >= 286(?) MB:
   - recording at 480p starts and stops properly
   - no basic processes killed
   - viewfinder gets briefly janky after stopping recording (likely due to thumbnail generation)
284 <= mem <= 285(?) MB:
   - recording at 480p starts and stops properly
   - no basic processes killed
   - viewfinder gets janky after starting and stopping recording
279 <= mem <= 283 MB:
   - recording at 480p starts and stops properly
   - Homescreen occasionally gets killed (size in sigkill log 1136..1263)
   - viewfinder gets janky
274 <= mem <= 276 MB:
   - Homescreen occasionally gets killed
   - Camera occasionally gets killed

Some other observations:
mem ~273 MB:
   - Homescreen -always- gets killed when stopping recording
   - viewfinder is minimally janky
mem ~270 MB:
   - Homescreen -always- gets killed when stopping recording
   - viewfinder is -very- janky
mem ~265 MB:
   - Homescreen and Camera -always- get killed when stopping recording
   - viewfinder is -very- janky

Follow-up: testing recording length at a mem limit of 286 MB.
Resolution: WONTFIX → FIXED
Resolution: FIXED → WONTFIX
Flags: needinfo?(ying.xu)
Test case is valid because the bug only needs 15 minutes to be recorded and the test case requires recording 1 hour of video.

https://bugzilla.mozilla.org/show_bug.cgi?id=1027592
QA Whiteboard: [QAnalyst-Triage+][lead-review+] → [QAnalyst-Triage?][lead-review+]
Flags: needinfo?(ktucker)
Flags: in-moztrap?(rmead)
No test case added since this was resolved as WNF and we are currently using 319mb of memory.
QA Whiteboard: [QAnalyst-Triage?][lead-review+] → [QAnalyst-Triage+][lead-review+]
Flags: needinfo?(ktucker)
Flags: in-moztrap?(rmead)
Flags: in-moztrap-
We already have a test case in moztrap to cover recording long videos:

https://moztrap.mozilla.org/manage/case/8471/
URL: https://moztrap.mozilla.org/manage/ca...
Flags: in-moztrap- → in-moztrap+
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: