There has been memory regressions in camera/camcorder app. On 256 MB devices with v2.2 camcorder quits as soon as recording is started! Camera and Camcorder lag/crash is quite evident on Flame too with memory restrictions.

Who can look at this? Thanks!

Shawn is having someone on it. -- Keven

Vincent: Please have a check on this issue first. Also please update n5-l camcoder memory usage here too.

(In reply to shawn ku [:sku] from comment #3) > Vincent: > Please have a check on this issue first. > Also please update n5-l camcoder memory usage here too. Sure.

(In reply to Inder from comment #0) > There has been memory regressions in camera/camcorder app. On 256 MB devices > with v2.2 camcorder quits as soon as recording is started! > 1. Do you have more detailed way to reproduce it? > Camera and Camcorder lag/crash is quite evident on Flame too with memory > restrictions. 2. What environment you used? KK or L? From your sentence, it seems we are looking into it on KK based. Can you explain more? 3. Do you have more log for the issue?

> > 1. Do you have more detailed way to reproduce it? Here are the steps to reproduce on Flame: 1. Reduce the RAM of Flame close to 256 MB (~283MB) 2. Start camera and switch to camcorder 3. Start recording 4. You can see the lag in preview when you move your phone 5. Stop the recording and notice the camera killed due to memory pressure. Camcorder works fine with v2.1 with the same RAM size. Also, increase the RAM on v2.2 and notice the issue go away. > > > Camera and Camcorder lag/crash is quite evident on Flame too with memory > > restrictions. > > 2. What environment you used? KK or L? From your sentence, it seems we are > looking into it on KK based. Can you explain more? I experimented with KK although it should happen on L also. On L camcorder is currently broken on our QRD. See bug 1133147 > > 3. Do you have more log for the issue? Attached

Inder, does this work if Flame is set to 319MB? That was our minimum supported level for Flame/KK.

On v2.1 we had our 8x10 QRD (KK based) set to 271MB and camcorder works fine but with the same RAM and exact same gonk on v2.2 camcorder does NOT work. So, it has certainly bloated.

Hema, Can you have someone on the Camera team look into this issue? There appears to be a memory regression in Camera. As per Inder's comment #9 this is a problem in 2.2 that didn't occur in 2.1 . Thanks, Mike

No-Jun, Could you please test Camcorder recording on Flame with KK and 319 MB config on 2.2 and let us know if you are seeing broken recording function and memory regressions (from 2.1 same device config) (Flame with 319 MB has been our reference config that we have been using since 2.1 release to compare with 256MB production devices) MikeH, Leaving an NI for you to keep an eye on this bug for No-Jun's test results and any optimizations required. Thanks Hema

Tested 2.1 and 2.2, In 2.1, under 283MB, it fires up the app fine and records video without issue. In 2.2, under 283MB, it took noticeably long time to fire up the app, and when video recording started, it OOMed. I attached both the zipped up memory folder and json export. they were taken when video recording was in progress.

Version Info: 2.2 Gaia-Rev da509caa7395d3d090ce973e8de082b4680a590d Gecko-Rev https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/96da179a7d3a Build-ID 20150218002515 Version 37.0a2 Device-Name flame FW-Release 4.4.2 FW-Incremental eng.cltbld.20150218.041956 FW-Date Wed Feb 18 04:20:07 EST 2015 Bootloader L1TC000118D0 2.1 Gaia-Rev 0d4b3c63d5cfb01f3312675f85c5ee43a0836d6b Gecko-Rev https://hg.mozilla.org/releases/mozilla-b2g34_v2_1/rev/f61986c6df4d Build-ID 20150218002207 Version 34.0 Device-Name flame FW-Release 4.4.2 FW-Incremental eng.cltbld.20150218.035837 FW-Date Wed Feb 18 03:58:48 EST 2015 Bootloader L1TC000118D0

For 319MB, there is no issue with video recording, (above attached memory reports are all taken with 319 config). The log seems to indicate that there is an issue with homescreen. Mikeh to confirm.

Just to clarify, I am highlighting the memory bloat in camera between v2.1 and v2.2 and not really discussing about 283 vs 319MB configuration. The memory bloat is quite evident from No-Jun's experiments too.

Thanks for the memory reports, No-Jun. An about:memory diff shows that the 2.2 Camera app is using ~2.2MB of more memory than the 2.1 version. The numbers below are deltas (positive numbers indicate an increase in memory from 2.1 to 2.2, negative numbers indicate a decrease). 2.80 MB (100.0%) -- explicit ├──1.31 MB (46.80%) -- js-non-window │ ├──0.70 MB (25.03%) -- runtime │ │ ├──0.35 MB (12.47%) ++ gc │ │ ├──0.25 MB (08.94%) ++ code │ │ ├──0.07 MB (02.46%) ── script-data │ │ ├──-0.03 MB (-1.12%) ── atoms-table │ │ ├──0.03 MB (01.12%) ── runtime-object │ │ ├──0.03 MB (01.12%) ── temporary │ │ └──0.00 MB (00.04%) ++ (2 tiny) │ ├──0.34 MB (11.99%) ++ zones/zone(0xNNN) │ └──0.27 MB (09.78%) ++ gc-heap ├──0.79 MB (28.22%) ── xpti-working-set [+] ├──0.47 MB (16.85%) ++ window-objects/top(app://camera.gaiamobile.org/index.html, id=NNN) ├──0.10 MB (03.68%) ++ heap-overhead ├──0.15 MB (05.22%) ++ gfx ├──-0.12 MB (-4.19%) ++ dom/memory-file-data/large ├──0.05 MB (01.83%) ── heap-unclassified ├──0.03 MB (01.00%) ── telemetry └──0.02 MB (00.60%) ++ (8 tiny) Inder, is this consistent with what you're observing? If not, can you please grab memory reports for 2.1 and 2.2 and attach them to this bug? The command will look something like this: $B2G # python tools/get_about_memory.py --product flame --no-gc-cc-log --no-auto-open --minimize --gecko-objdir /home/mikeh/dev/mozilla/b2g/058_flame-kk/objdir-gecko-b2g-inbound Diego, did we add any new features to the Camera app from 2.1 to 2.2 that would account for the above memory increase?

Also, looking at the b2g-procrank for v2.1: APPLICATION PID Vss Rss Pss Uss cmdline Camera 1227 75168K 17544K 10943K 7988K /system/b2g/b2g ...and for v2.2: APPLICATION PID Vss Rss Pss Uss cmdline Camera 1935 114436K 12140K 6655K 5228K /system/b2g/plugin-container Vss is certainly higher, but this number is meaningless, as it only reflects allocated address space, not memory that is actually being used. Please refer to: http://stackoverflow.com/questions/22372960/is-this-explanation-about-vss-rss-pss-uss-accurately The two numbers of interest are Pss and Uss, both of which are (significantly) smaller in v2.2.

Inder/No-Jun, one further consideration: is the reported VSIZE the same when the Camera app starts vs after a few minutes of regular/rapid picture taking or video recording? If it starts our considerably smaller (e.g. closer to the v2.1 value), then gradually increases, that could indicate a memory leak; if not, then it's just v2.2 reserving more address space than in v2.1 -- address space that the RSS/PSS values indicate isn't being used, and thus doesn't contribute to overall RAM usage. I notice that in general, all v2.2 apps report a higher VSIZE.

Wilson has spent more time than me measuring memory and will have a better insight than me.

So this is b2g-procrank when I just started Camera app, without taking any pictures: APPLICATION PID Vss Rss Pss Uss cmdline b2g 216 256860K 61476K 52884K 48884K /system/b2g/b2g Camera 4709 93604K 26828K 17805K 13628K /system/b2g/b2g Homescreen 1167 99132K 23024K 15572K 12636K /system/b2g/b2g (Preallocated a 7012 72624K 13964K 8364K 6380K /system/b2g/b2g (Nuwa) 324 66364K 2404K 703K 300K /system/b2g/b2g ------ ------ ------ 107003K 92760K TOTAL And after a few shots (photo + Video and preview): APPLICATION PID Vss Rss Pss Uss cmdline b2g 216 259496K 58536K 49601K 45436K /system/b2g/b2g Camera 3035 93924K 27704K 18450K 14080K /system/b2g/b2g Homescreen 1167 99132K 22948K 15325K 12292K /system/b2g/b2g (Preallocated a 4709 73584K 12152K 7205K 5600K /system/b2g/b2g (Nuwa) 324 66364K 4016K 1034K 296K /system/b2g/b2g There are some increases in Rss and Pss (876K and 452K, respectively). Is the increase significant?

above result was from Master, and from 2.2, I see: After initial bootup: mozilla@ubuntu:~/B2G/tools/about-memory-0$ more b2g-procrank APPLICATION PID Vss Rss Pss Uss cmdline b2g 207 253188K 54632K 49646K 47880K /system/b2g/b2g Homescreen 947 90280K 15548K 10928K 9692K /system/b2g/b2g Camera 1304 108064K 14188K 9220K 7748K /system/b2g/b2g Smart Collectio 1232 82204K 11836K 8856K 8236K /system/b2g/plugin- container Find My Device 1234 72728K 10544K 6460K 5444K /system/b2g/b2g (Preallocated a 1413 71616K 7976K 4740K 4056K /system/b2g/b2g (Nuwa) 454 66388K 1336K 185K 4K /system/b2g/b2g ------ ------ ------ 97344K 89732K TOTAL After taking a few shots: mozilla@ubuntu:~/B2G/tools/about-memory-0$ more ../about-memory-1/b2g-procrank APPLICATION PID Vss Rss Pss Uss cmdline b2g 207 253788K 48416K 42156K 39324K /system/b2g/b2g Camera 1304 117836K 22768K 16495K 13932K /system/b2g/b2g Homescreen 947 89320K 13204K 8536K 7360K /system/b2g/b2g Smart Collectio 1232 82204K 11640K 8358K 7640K /system/b2g/plugin- container Find My Device 1234 72728K 9128K 4740K 3664K /system/b2g/b2g (Preallocated a 1413 71616K 7496K 3893K 3116K /system/b2g/b2g (Nuwa) 454 66388K 1300K 180K 4K /system/b2g/b2g ------ ------ ------ 92895K 82912K TOTAL

No-Jun, if you keep taking photos, do PSS and USS keep rising? Either way, can you please grab memory reports for 2.2, one immediately after starting the Camera app, and the other after taking "a few" shots?

in 2.1, when I do roughly the similar thing, I see: mozilla@ubuntu:~/B2G/tools$ more about-memory-2/b2g-procrank APPLICATION PID Vss Rss Pss Uss cmdline b2g 208 229540K 61236K 49522K 46684K /system/b2g/b2g Camera 1561 86436K 25096K 13261K 10616K /system/b2g/b2g Homescreen 1167 76036K 24188K 12432K 9860K /system/b2g/b2g Smart Collectio 1256 67396K 20012K 8979K 6804K /system/b2g/b2g (Preallocated a 1897 62980K 16504K 7288K 5684K /system/b2g/b2g OperatorVariant 945 64068K 16436K 6310K 4460K /system/b2g/b2g (Nuwa) 426 54692K 8240K 1450K 300K /system/b2g/b2g ------ ------ ------ 115610K 99696K TOTAL mozilla@ubuntu:~/B2G/tools$ more about-memory-3/b2g-procrank APPLICATION PID Vss Rss Pss Uss cmdline b2g 208 228060K 51016K 42138K 39408K /system/b2g/b2g Camera 1561 96112K 28812K 19867K 17264K /system/b2g/b2g Homescreen 1167 75076K 18708K 10677K 9060K /system/b2g/b2g Smart Collectio 1256 67460K 16016K 8254K 6740K /system/b2g/b2g OperatorVariant 945 64132K 12912K 5597K 4252K /system/b2g/b2g (Preallocated a 1897 62980K 11988K 5114K 3892K /system/b2g/b2g (Nuwa) 426 54692K 5572K 783K 4K /system/b2g/b2g ------ ------ ------ 111621K 98872K TOTAL

Hi Mike, I made a tarball consisting of 4 tarballs (wasn't planned): https://www.dropbox.com/s/jzwio45v8ahpw8y/memory_report_22_21.tar.gz?dl=0 contains 2.1 and 2.2 memory reports of after startup, and after taking few shots (including AF, video, preview)

Thanks, No-Jun. A diff of the just-launched and after-a-few-pictures memory reports shows that the extra space is mostly being taken up by files (pictures, I'm guessing): 7.39 MB (100.0%) -- explicit ├──3.50 MB (47.32%) -- dom/memory-file-data │ ├──3.48 MB (47.09%) -- large │ │ ├──0.80 MB (10.89%) ── file(length=843334, sha1=48f7056bf06bdc5c1690b8813466910950ba2392) [+] │ │ ├──0.62 MB (08.40%) ── file(length=649126, sha1=1a3b42983cdd644b259bbc2afc0d83ef8bc7a969) [+] │ │ ├──0.58 MB (07.82%) ── file(length=605966, sha1=b12af7b8de6391afcd7af4db5dee3493cf892c0e) [+] │ │ ├──0.52 MB (07.08%) ── file(length=548352, sha1=dea2f44a617972d12813967ea8e37a61d9d0057c) [+] │ │ ├──0.46 MB (06.18%) ── file(length=479077, sha1=d2f0559eb2b6eb3b0206afbc5ab37b401b8c2ad7) [+] │ │ ├──0.43 MB (05.76%) ── file(length=445250, sha1=8e65ea96572c0ec98b2da5abe0718d2ac758a474) [+] │ │ └──0.07 MB (00.95%) ++ (2 tiny) │ └──0.02 MB (00.22%) ── small ├──1.44 MB (19.43%) -- images/content/raster │ ├──1.20 MB (16.25%) -- used │ │ ├──1.20 MB (16.28%) -- image(blob:app://camera.gaiamobile.org/4a75a6e0-8da0-4481-bebd-5128ecb127f3) │ │ │ ├──1.17 MB (15.86%) ── decoded-nonheap [+] │ │ │ └──0.03 MB (00.42%) ++ (2 tiny) │ │ └──-0.00 MB (-0.03%) ++ (2 tiny) │ └──0.23 MB (03.18%) ++ unused ├──0.73 MB (09.89%) ++ window-objects/top(app://camera.gaiamobile.org/index.html, id=NNN) ├──0.70 MB (09.40%) ++ js-non-window ├──0.64 MB (08.71%) ── heap-unclassified ├──0.55 MB (07.48%) -- media │ ├──0.30 MB (04.02%) ── libogg │ └──0.26 MB (03.46%) ── resources ├──-0.26 MB (-3.48%) ++ heap-overhead ├──0.13 MB (01.77%) ++ gfx └──-0.04 MB (-0.51%) ++ (4 tiny) No-Jun, can you look at the file sizes of the last six images you took using the 2.2 build, to see if they're approximately the 'length' values shown above? djf, could this be the Camera's preview screen holding onto the last few taken photos? If so, is there an upper limit to how many are help? (I looked, but couldn't see anything obvious in the Camera app.)

Yup, they match the last 6 pictures I've taken, although they're not in the same time order

https://www.dropbox.com/s/i34xz8p98n6yen0/22_memory_report.zip?dl=0 Above linked file contains two folders. About-memory-4 was captured after the camera app startup with --minimize parameter, and About-memory-5 was captureed after taking 195 shots in succession, also with --minimize parameter.

Thanks, No-Jun. Diffing these reports shows that before taking 195 photos, and after, additional memory used by the Camera app is a little over a megabyte -- most of which is in heap-management. 1.13 MB (100.0%) -- explicit ├──0.58 MB (50.92%) -- heap-overhead │ ├──0.48 MB (42.57%) ── bin-unused │ └──0.09 MB (08.35%) ── bookkeeping ├──0.42 MB (36.65%) ── heap-unclassified ├──-0.27 MB (-23.97%) ++ js-non-window ├──0.27 MB (23.74%) ++ window-objects/top(app://camera.gaiamobile.org/index.html, id=NNN) ├──0.07 MB (05.93%) ++ media ├──0.07 MB (05.80%) ── freetype └──0.01 MB (00.92%) ++ (6 tiny) So the 6 images in comment 33 above are properly collected by the GC/CC (removing ni? djf). Inder, it looks like everything is working properly.

> Inder, it looks like everything is working properly. Not sure i follow. How do we address the issue mentioned in the description where the camcorder is quitting right after recording is started with mem pressure on v2.2?

(In reply to Inder from comment #37) > Not sure i follow. How do we address the issue mentioned in the description > where the camcorder is quitting right after recording is started with mem > pressure on v2.2? Inder, let's see what your QRD thinks is going on: please attach about:memory reports for your 2.1 and 2.2 builds. A sample command line is in comment 24. You can even load the reports into Firefox yourself for analysis: go to the about:memory URL and select "Load and diff..." -- you'll be prompted to open each of the two 'memory-reports' file. Note that if you don't see this button, you'll need to try a newer version of Firefox.

Update some findings in camera recording by co-work with vliu: 1. Camera app was killed due to cache is less than 6MB <6>[ 162.061011] Killing 'Camera' (1398), adj 134, <6>[ 162.061016] to free 5252kB on behalf of 'kswapd0' (97) because <6>[ 162.061020] cache 5944kB is below limit 6144kB for oom_score_adj 117 2. With 1G memory, we found system need extra 20MB memory on v2.2 if no any memory pressure. Compare with v2.1, b2g consumed extra 12.8MB on v2.2 and camera consumed extra 3.9MB on v2.2 You can reference following google doc in detail: http://goo.gl/bdNcYe 3. With 283MB memory, we use more SWAP space due extra 20MB memory usage on v2.2, it caused bad performance.

Will update memory report on both v2.1 and v2.2 to clarify extra 20MB memory

I collected the about:memory reports for camcorder when it's recording and I am seeing ~1.5MB diff between v2.2 and v2.1: Camera (pid NNN) Explicit Allocations 1.56 MB (100.0%) -- explicit ├──0.86 MB (54.95%) -- js-non-window │ ├──0.39 MB (24.69%) -- runtime But, b2g-procrank shows a completely different picture as mentioned in comment 39: APPLICATION PID Vss Rss Pss Uss cmdline v2.2 Camera 1382 137168K 47196K 25866K 19976K /system/b2g/plugin-container v2.1 Camera 1229 103372K 33376K 17877K 13316K /system/b2g/b2g Let me know if I still need to attach the reports but looks like we are seeing the same symptoms.

Thanks, Inder. Can you please attach your about_memory reports to this bug? Also, can you please attach the smaps of the Camera apps from v2.1 and v2.2, after carrying out whatever STR you're using to get the results in comment 41? (What are they, anyway? as in comment 7? or something else?) The procedure is: # adb shell b2g-ps --> get the PID of the Camera app # adb shell cat /proc/$PID/smaps > smaps.txt The smaps will show the break down of Pss and Uss by module, so that we can see where the extra memory is getting taken up.

(In reply to Mike Habicher [:mikeh] from comment #42) > Thanks, Inder. Can you please attach your about_memory reports to this bug? > done > Also, can you please attach the smaps of the Camera apps from v2.1 and v2.2, > after carrying out whatever STR you're using to get the results in comment > 41? (What are they, anyway? as in comment 7? or something else?) > The steps are the same as in comment 7. This issue has gotten worse since filing of the bug and right now the camera crashes after few seconds of starting the recording. I had to remove the 271MB restriction we have in place to generate this data. > The procedure is: > > # adb shell b2g-ps > --> get the PID of the Camera app > # adb shell cat /proc/$PID/smaps > smaps.txt > > The smaps will show the break down of Pss and Uss by module, so that we can > see where the extra memory is getting taken up. MikeH, are you not able to reproduce this issue on your Flame?

Takes stdin from /proc/#/smaps and emits a list of memory regions in the map by module, and their total PSS (from the 'Pss' field) and USS (from the 'Private_Dirty' field) usage.

From attachment 8570217 [details]: cat smaps_v21.txt | smaps.py | sort

From attachment 8570219 [details]: cat smaps_v22.txt | smaps.py | sort > smaps_v21_sorted.txt

Same output as before: # cat /proc/#/smaps | smaps.py ...also: # cat /proc/#/smaps > smaps.txt # smaps.py smaps.txt Now has a diff mode: # smaps.py smaps_before.txt smaps_after.txt Output looks like this: | /system/lib/libstlport.so +21 +12 - /system/lib/libsuspend.so -1 +0 | /system/lib/libsync.so +8 +8 | /system/lib/libsysutils.so +9 +8 | /system/lib/libui.so +3 +4 | /system/lib/libusbhost.so +8 +8 | /system/lib/libutils.so +7 +8 | /system/lib/libvorbisidec.so +9 +8 | /system/lib/libwpa_client.so +7 +8 | /system/lib/libz.so +8 +8 + /system/vendor/lib/libdiag.so +8 +8 The first character indicates whether the module is present in both smaps reports ("|"), only smaps_before.txt ("-"), or only smaps_after.txt ("+"). The next field is the name of the module; the next field is the Pss delta (after minus before); and the final field is the Uss delta. Pss and Uss deltas are in the same units as emitted by proc/#/smaps (i.e. kB), and if both deltas are 0, the module is omitted from the report.

This filtered report shows all of the modules that are loaded into the Camera app in v2.2 that aren't loaded in v2.1. These additional modules are responsible for the following increases: - Pss: 1851 kB - Uss: 1068 kB

This filtered report shows all of the modules that are loaded into the Camera app both in v2.1 and in v2.2. These additional modules are responsible for the following increases: - Pss: 6278 kB - Uss: 6008 kB libxul.so alone has grown by ~3.3MB/2.5MB. Anonymous regions have also grown, as have regions in jemalloc.

(In reply to Mike Habicher [:mikeh] from comment #53) > Created attachment 8570324 [details] > smaps.py 2.1/smaps_v21.txt 2.2/smaps_v22.txt | grep "^\+" > > This filtered report shows all of the modules that are loaded into the > Camera app in v2.2 that aren't loaded in v2.1. > > These additional modules are responsible for the following increases: > - Pss: 1851 kB > - Uss: 1068 kB That's pretty interesting. The new modules are all main b2g process things that have no business existing in a content process. This feels like something Nuwa-y related to me.

(In reply to Michael Vines [:m1] [:evilmachines] from comment #55) > That's pretty interesting. The new modules are all main b2g process things > that have no business existing in a content process. This feels like > something Nuwa-y related to me. Thinker, can you confirm whether or not this is a Nuwa problem? (Or at least part of the problem?)

ni? cyu as well -- khuey pointed out in IRC that in the 2.2 b2g-ps, all of the content processes are forked from b2g, while in 2.1 they are forked from Nuwa. This would explain our memory regression, but it also means the problem probably isn't limited to the Camera app.

I think the regression result mainly from that on 2.1 the camera app is forked from Nuwa, while on 2.2 it isn't. This is supposed to be fixed in bug 1053634. I suggest we redo the test without the Nuwa variable to see whether there is really memory regression. If not, then we may safely dupe this bug to 1053634.

Attached about memory on v2.1 with 1G memory when camere recording.

Attached about memory on v2.2 with 1G memory when camere recording.

I got about memory report on v2.1/v2.2, the main differences as following: Main Process (pid NNN) Explicit Allocations 10.54 MB (100.0%) -- explicit ├───3.58 MB (33.92%) ++ images/content/raster/used ├───3.43 MB (32.50%) ++ js-non-window ├───1.17 MB (11.11%) ++ workers ├───1.48 MB (14.00%) ── heap-unclassified ├──-0.05 MB (-0.52%) ++ window-objects ├──-0.77 MB (-7.34%) ── xpti-working-set [-] ├───0.62 MB (05.87%) ++ heap-overhead ├───0.64 MB (06.07%) ++ dmd ├───0.36 MB (03.46%) ++ gfx ├──-0.02 MB (-0.17%) ++ atom-tables ├───0.13 MB (01.22%) ── freetype └──-0.01 MB (-0.12%) ++ (13 tiny) System Other Measurements 18.23 MB (100.0%) -- kgsl-memory └──18.23 MB (100.0%) -- b2g (pid=NNN) ├──15.69 MB (86.03%) ── egl_image [53] ├───1.80 MB (09.85%) ── texture [14] ├───0.88 MB (04.80%) ── any(0) [35] └──-0.13 MB (-0.69%) ── command [18] It looks like egl_image consume more memory, is there anyone have any idea on egl_image?

memory report with patch of bug-1053634

With patch of bug-1053634 on v2.2, we saw camera app is forked from Nuwa and it saved ~5.2MB. V2.2 w/ patch of bug-1053634: | megabytes | NAME PID PPID CPU(s) NICE USS PSS RSS SWAP VSIZE OOM_ADJ USER b2g 208 1 50.6 0 59.3 64.4 82.8 0.0 257.5 0 root (Nuwa) 447 208 1.1 0 4.9 8.5 23.8 0.0 63.7 -16 root Homescreen 980 447 5.7 0 11.5 14.6 30.3 0.0 84.2 8 u0_a980 Built-in Keyboa 1170 447 2.9 0 8.3 10.9 25.3 0.0 73.3 3 u0_a1170 Find My Device 1191 447 1.7 0 6.8 9.7 24.7 0.0 70.3 10 u0_a1191 Camera 1865 447 14.5 0 13.6 17.5 34.6 0.0 108.3 2 u0_a1865 (Preallocated a 1975 447 0.6 0 5.4 7.4 18.6 0.0 67.9 1 u0_a1975 V2.2 w/o any patch: | megabytes | NAME PID PPID CPU(s) NICE USS PSS RSS SWAP VSIZE OOM_ADJ USER b2g 209 1 46.9 0 58.2 63.9 84.8 0.0 261.5 0 root (Nuwa) 436 209 1.1 0 5.2 8.6 24.0 0.0 63.4 -16 root Homescreen 975 436 5.1 0 11.6 15.0 30.4 0.0 80.9 8 u0_a975 Built-in Keyboa 1171 209 3.3 0 12.8 16.5 31.9 0.0 77.9 11 u0_a1171 Find My Device 1278 436 1.7 0 6.9 10.0 24.4 0.0 71.0 10 u0_a1278 Camera 1456 209 13.6 0 17.9 22.7 39.8 0.0 114.6 2 u0_a1456 (Preallocated a 1529 436 0.6 0 5.3 7.6 18.4 0.0 67.7 1 u0_a1529 Camera app forked from Nuwa can reduce some memory, but there is still kgsl-memory problem. Hi Becker, do you have any idea on memory leak of kgsl-mem? or you can find proper owner to check on this?

Danny, I'm going to hand this one over to you.

Also, do we have any idea what change triggered this regression?

About egl_image, I have more study. 1. v2.1 1.1 Keep in Homescreen after reboot device. ───3.83 MB (31.73%) ── egl_image [9] 1.2 Launch camera app and keep in preview. ───5.19 MB (37.72%) ── egl_image [4] 1.3 Do camera recording. ───6.13 MB (40.97%) ── egl_image [5] 2. v2.2 2.1 Keep in Homescreen after reboot device. ──14.50 MB (59.23%) ── egl_image [26] 2.2 Launch camera app and keep in preview. ──16.52 MB (60.05%) ── egl_image [44] 2.3 Do camera recording. ──20.68 MB (64.25%) ── egl_image [47] From the above results, egl_image increases about 10~11MB comparing between v2.1 and v2.2 no matter whether Camera app is launched or not.

Hi Inder, About the memory report, the |gralloc| field also reply difference between v2.1 and v2.2. The data was captured when camera recording. ===> v2.1 16.43 MB (100.0%) -- gralloc ├──11.69 MB (71.20%) -- Camera (pid=2183) │ ├───5.44 MB (33.11%) ── buffer(width=720, height=480, bpp=-22, stride=720) [11] │ ├───3.13 MB (19.04%) ── buffer(width=480, height=854, bpp=4, stride=480) [2] │ ├───3.07 MB (18.71%) ── buffer(width=480, height=839, bpp=4, stride=480) [2] │ └───0.06 MB (00.34%) ++ (2 tiny) └───4.73 MB (28.80%) -- b2g (pid=232) ├──3.13 MB (19.04%) ── buffer(width=480, height=854, bpp=4, stride=480) [2] ├──1.48 MB (09.03%) ── buffer(width=1440, height=135, bpp=4, stride=1440) [2] └──0.12 MB (00.73%) ++ (2 tiny) ===> v2.2 20.22 MB (100.0%) -- gralloc ├──11.78 MB (58.28%) -- Camera (pid=1281) │ ├───6.25 MB (30.93%) ── buffer(width=480, height=854, bpp=4, stride=480) [4] │ ├───5.44 MB (26.89%) ── buffer(width=720, height=480, bpp=-22, stride=720) [11] │ └───0.09 MB (00.45%) ── buffer(width=150, height=75, bpp=4, stride=160) [2] └───8.44 MB (41.72%) ── b2g (pid=208)/buffer(width=288, height=256, bpp=4, stride=288) [30] From this report, there is no big difference consumed in Camera app. Instead, b2g in v2.2 consumes two times bigger than in v2.1. [30] means the number of tiling buffers v2.2 used. It means when Camera recording is running, v2.2 consumes more memory than v2.1. Furthermore, from profiling through layerscope, the number of layers v2.2 consumes is bigger than in v2.1. This result lead to v2.2 is rendered by GPU while v2.1 is render by HWC. I think it may be part of reason why egl_image in v2.2 got 10MB larger than in v2.1. I tried to limited RAM size to 283MB on v2.2 flame and did a test. 1. Go to Settings->Developer. Disabling Tiling. 2. Enable "Draw tile borders". 3. stop/start b2g. Use tile borders to check tiling is enable or not after stop/start b2g. If you can still see tiling, keep doing stop/start b2g until you make sure tiling is disable. 4. Disable "Draw tile borders" 5. Run camera recording. I did camera recording many times and I didn't see any crash happens. Could you please also have this trail in your side? Thanks

Hi! Peter, Need your comment on this case. Thanks -- Keven

I tried to grab an unexpected layer from layerscope. This layer was generated when Camera recording was in progress on v2.2 Flame. Please see the attached file. This layer indicates that even when camera is recording, some other app still draw this layer in the background. Actually I don't see this layer before v2.1.

For 2.2 or master, we can't enter HWC now and it looks like we got lots of tile layers. I think it would help after finding the fix of bug 1139541.

One thing blocking HWC is that I found system app created a dom element called 'AppChrome3', and caused a very small size layer which couldn't use gralloc buffer. alive, do you know anything about 'appchrome3'? D/HWComposer( 2188): PaintedLayerComposite Layer 0xad2b7c00 doesn't have a gralloc buffer D/HWComposer( 2188): Render aborted. Nothing was drawn to the screen After deleting this element, the HWC was still not using because HAL returns gpu composite. Need to check more detail. D/HWComposer( 2188): use gpuComposite idx 4 D/HWComposer( 2188): H/W Composition failed alive, do you know anything about 'appchrome3'?

(In reply to peter chang[:pchang][:peter] from comment #71) > One thing blocking HWC is that I found system app created a dom element > called 'AppChrome3', and caused a very small size layer which couldn't use > gralloc buffer. > > alive, do you know anything about 'appchrome3'? > > D/HWComposer( 2188): PaintedLayerComposite Layer 0xad2b7c00 doesn't have a > gralloc buffer > D/HWComposer( 2188): Render aborted. Nothing was drawn to the screen > > > After deleting this element, the HWC was still not using because HAL returns > gpu composite. > Need to check more detail. > > D/HWComposer( 2188): use gpuComposite idx 4 > D/HWComposer( 2188): H/W Composition failed > > > alive, do you know anything about 'appchrome3'? It's created with the app to display some chrome UI like back/forward buttons. I think it's also being used to put the <Search..> input for each app.

(In reply to Alive Kuo [:alive][NEEDINFO!] from comment #72) > (In reply to peter chang[:pchang][:peter] from comment #71) > > One thing blocking HWC is that I found system app created a dom element > > called 'AppChrome3', and caused a very small size layer which couldn't use > > gralloc buffer. > > > > alive, do you know anything about 'appchrome3'? > > > > D/HWComposer( 2188): PaintedLayerComposite Layer 0xad2b7c00 doesn't have a > > gralloc buffer > > D/HWComposer( 2188): Render aborted. Nothing was drawn to the screen > > > > > > After deleting this element, the HWC was still not using because HAL returns > > gpu composite. > > Need to check more detail. > > > > D/HWComposer( 2188): use gpuComposite idx 4 > > D/HWComposer( 2188): H/W Composition failed > > > > > > alive, do you know anything about 'appchrome3'? > > It's created with the app to display some chrome UI like back/forward > buttons. > I think it's also being used to put the <Search..> input for each app. I think bug 1139541 comment 17 explain this element.

Based on the data in comment 67, I dump the display list of b2g process under camera preview screen. ===> v2.2 20.22 MB (100.0%) -- gralloc ├──11.78 MB (58.28%) -- Camera (pid=1281) │ ├───6.25 MB (30.93%) ── buffer(width=480, height=854, bpp=4, stride=480) [4] │ ├───5.44 MB (26.89%) ── buffer(width=720, height=480, bpp=-22, stride=720) [11] │ └───0.09 MB (00.45%) ── buffer(width=150, height=75, bpp=4, stride=160) [2] └───8.44 MB (41.72%) ── b2g (pid=208)/buffer(width=288, height=256, bpp=4, stride=288) [30] From the display list dump in [1], you can see there are four layers with full screen size that are allocated in b2g process. And they are related to lockScreenWindow, lockscreen-canvas, lockscreen-icon, AppChrome3. Tim, could you have someone to check why above elements still exists for camera preview?: For the appchrome3, I think this is related to gesture detection. [1]https://pastebin.mozilla.org/8824567

I am not proper owner due to it looks like camera/graphic issue.(In reply to Mike Habicher [:mikeh] from comment #64) > Danny, I'm going to hand this one over to you. I am not proper owner due to it looks like camera/graphic issue.

(In reply to Vincent Liu[:vliu] from comment #67) > 1. Go to Settings->Developer. Disabling Tiling. > 2. Enable "Draw tile borders". > 3. stop/start b2g. Use tile borders to check tiling is enable or not after > stop/start b2g. If you can still see tiling, keep doing stop/start b2g until > you make sure tiling is disable. > 4. Disable "Draw tile borders" > 5. Run camera recording. > > I did camera recording many times and I didn't see any crash happens. Could > you please also have this trail in your side? Thanks Vincent -- i followed these steps and it doesn't seem to be helping for me. I did see the Developer setting itself crash couple of times after disabling Tiling and when i start camcorder recording it crashes every time with memory pressure.

Peter, I am confused with your comment here. (In reply to peter chang[:pchang][:peter] from comment #74) > Based on the data in comment 67, I dump the display list of b2g process > under camera preview screen. > > ===> v2.2 > 20.22 MB (100.0%) -- gralloc > ├──11.78 MB (58.28%) -- Camera (pid=1281) > │ ├───6.25 MB (30.93%) ── buffer(width=480, height=854, bpp=4, stride=480) > [4] > │ ├───5.44 MB (26.89%) ── buffer(width=720, height=480, bpp=-22, > stride=720) [11] > │ └───0.09 MB (00.45%) ── buffer(width=150, height=75, bpp=4, stride=160) > [2] > └───8.44 MB (41.72%) ── b2g (pid=208)/buffer(width=288, height=256, bpp=4, > stride=288) [30] > > From the display list dump in [1], you can see there are four layers with > full screen size that are allocated in b2g process. > And they are related to lockScreenWindow, lockscreen-canvas, > lockscreen-icon, AppChrome3. The tree shows 3 layers under camera and one in b2g. Are you clipping the wrong part of the tree here? I can't find the full information from [1] either. > Tim, could you have someone to check why above elements still exists for > camera preview?: > For the appchrome3, I think this is related to gesture detection. I am needinfo'ing Greg and Etienne here. Please be aware of regression here in System app -- we are holding three invisible layers in the System app there if Peter's interpretation is correct. > [1]https://pastebin.mozilla.org/8824567

(In reply to Tim Guan-tin Chien [:timdream] (slow response; please ni? to queue) from comment #77) > Peter, I am confused with your comment here. > > (In reply to peter chang[:pchang][:peter] from comment #74) > > Based on the data in comment 67, I dump the display list of b2g process > > under camera preview screen. > > > > ===> v2.2 > > 20.22 MB (100.0%) -- gralloc > > ├──11.78 MB (58.28%) -- Camera (pid=1281) > > │ ├───6.25 MB (30.93%) ── buffer(width=480, height=854, bpp=4, stride=480) > > [4] > > │ ├───5.44 MB (26.89%) ── buffer(width=720, height=480, bpp=-22, > > stride=720) [11] > > │ └───0.09 MB (00.45%) ── buffer(width=150, height=75, bpp=4, stride=160) > > [2] > > └───8.44 MB (41.72%) ── b2g (pid=208)/buffer(width=288, height=256, bpp=4, > > stride=288) [30] > > > > From the display list dump in [1], you can see there are four layers with > > full screen size that are allocated in b2g process. > > And they are related to lockScreenWindow, lockscreen-canvas, > > lockscreen-icon, AppChrome3. > > The tree shows 3 layers under camera and one in b2g. Are you clipping the > wrong part of the tree here? I can't find the full information from [1] > either. Tim, we are talking about the increase of b2g memory between 2.1 and 2.2, please refer comment 67 for detail. And there are 30 (tiles)layers created in b2g process. > > └───8.44 MB (41.72%) ── b2g (pid=208)/buffer(width=288, height=256, bpp=4, > > stride=288) [30] > > > Tim, could you have someone to check why above elements still exists for > > camera preview?: > > For the appchrome3, I think this is related to gesture detection. > > I am needinfo'ing Greg and Etienne here. Please be aware of regression here > in System app -- we are holding three invisible layers in the System app > there if Peter's interpretation is correct. > > > [1]https://pastebin.mozilla.org/8824567 Therefore, I tried to dump the display items of b2g in above link to see which components in b2g process are still required memory.

Discussed with Peter. The 3 layers are irrelevant with the tree slice. They're exactly in the b2g process from the log. And I would: 0. manually hide the elements in WebIDE 1. ./tools/get_about_memory.py -m 2. check if the used memory reduced 3. if so, debug with JS & CSS 4. NI Peter to get the detailed report again, so that we can make sure the patch works

(In reply to peter chang[:pchang][:peter] from comment #74) > Tim, could you have someone to check why above elements still exists for > camera preview?: > For the appchrome3, I think this is related to gesture detection. During camera preview you can still: - swipe from the left to get to your previous app - swipe from the right to get to your next app - swipe from the top to reveal the statusbar (another swipe will let you access the utility tray) And we have 3 small divs on each side of the screen to handle this. If this is causing issues once layerized I'll be happy to help workaround it on the gaia side but I'm not sure I understand the problem (is this triggering a fullscreen-sized layer?).

Peter: on my Flame with the current master build it shows b2g process uses only few memory than your report, at least when it's idle. The process would even be allocated without any memory for layers. I don't know whether I profiled it rightly, or this make sense. I would attach the profiling result here, and try to get the result from v2.2 to see if we have a 'good' regression fixed the issue on v3.0. --- The Flame: Serial: e47cd8b0 (State: device) Build ID 20150309160227 Gaia Revision b84adf4acfc53915ec33cf169298c2e019398b28 Gaia Date 2015-03-11 22:32:47 Gecko Revision https://hg.mozilla.org/mozilla-central/rev/23f1f0369df5 Gecko Version 39.0a1 Device Name flame Firmware(Release) 4.4.2 Firmware(Incremental) 65 Firmware Date Mon Dec 15 18:51:29 CST 2014 Bootloader L1TC000118D0

Clear the NI because I may need to do more tests first.

(In reply to Etienne Segonzac (:etienne) from comment #80) > (In reply to peter chang[:pchang][:peter] from comment #74) > > Tim, could you have someone to check why above elements still exists for > > camera preview?: > > For the appchrome3, I think this is related to gesture detection. > > During camera preview you can still: > - swipe from the left to get to your previous app > - swipe from the right to get to your next app > - swipe from the top to reveal the statusbar (another swipe will let you > access the utility tray) > And we have 3 small divs on each side of the screen to handle this. > > If this is causing issues once layerized I'll be happy to help workaround it > on the gaia side but I'm not sure I understand the problem (is this > triggering a fullscreen-sized layer?). Etienne, the following are the display items dump for these three small divs. They required almost a full screen memory (w=480, h=825) and are visible in the top area (x=0,y=0,w=480,h=1). I guess the visible area is related to 'AppChrome3' element. Are you able to make it as invisible? ClientPaintedLayer (0xaf932170) [bounds=(x=0, y=-31, w=480, h=825)] [visible=< (x=0, y=0, w=480, h=1); >] { hitregion=< (x=0, y=0, w=480, h=30); (x=0, y=90, w=30, h=704); (x=450, y=90, w=30, h=704); > dispatchtocontentregion=< (x=0, y=0, w=480, h=30); (x=0, y=90, w=30, h=704); (x=450, y=90, w=30, h=704); >} [valid=< (x=0, y=0, w=480, h=1); >] nsDisplayTransform p=0xac69c670 f=0xaefa32b8(Block(div)(0)) z=5 bounds(0,-1228,19200,1229) layerBounds(0,-1228,19200,1229) visible(0,0,19200,1) componentAlpha(0,0,0,0) clip(0,0,19200,34160) AppChrome3 chrome (opaque 0,0,19200,1)[ 1 0; 0 0.682; 0 -30.69; ] layer=0xaf932170 BackgroundColor p=0xac69d230 f=0xb14e3458(Block(div)(105)) z=4093 bounds(0,3600,1200,28160) layerBounds(0,3600,1200,28160) visible(0,3600,1200,28160) componentAlpha(0,0,0,0) clip(0,3600,1200,28160 [0,3600,1200,28160 corners 0,0,1200,1200,1200,1200,0,0]) uniform left-panel gesture-panel (rgba 1,0,0,0) layer=0xaf932170 BackgroundColor p=0xac69d3a8 f=0xb14e36f0(Block(div)(107)) z=4093 bounds(18000,3600,1200,28160) layerBounds(18000,3600,1200,28160) visible(18000,3600,1200,28160) componentAlpha(0,0,0,0) clip(18000,3600,1200,28160 [18000,3600,1200,28160 corners 1200,1200,0,0,0,0,1200,1200]) uniform right-panel gesture-panel (rgba 0,0.501961,0,0) layer=0xaf932170

Peter: now it's really strange: I've hidden and even deleted all LockScreen related DIVs, but it not works. I'll attach the log.

OK I've change the CSS rules to make sure the layers really get hidden and it reduce about 2MB. I now attach the result of about:memory.

Comment on attachment 8576498 [details] [gaia] snowmantw:bug1133144 > mozilla-b2g:master Tim: I've change the CSS to hide it. It could be verified with the get_about_memory tool, and the result is as what I attached. Of course Peter's verification is necessary, but I don't know whether I should set review or not, so I set feedback first.

BTW: The v2.2 patch is exactly the same patch of master.

Comment on attachment 8576498 [details] [gaia] snowmantw:bug1133144 > mozilla-b2g:master Thanks for identifying the problem!

Comment on attachment 8576498 [details] [gaia] snowmantw:bug1133144 > mozilla-b2g:master With this patch, I can't find the lock screen related display items in b2g process. And it also reduces the b2g memory.

(In reply to peter chang[:pchang][:peter] from comment #84) > (In reply to Etienne Segonzac (:etienne) from comment #80) > > (In reply to peter chang[:pchang][:peter] from comment #74) > > > Tim, could you have someone to check why above elements still exists for > > > camera preview?: > > > For the appchrome3, I think this is related to gesture detection. > > > > During camera preview you can still: > > - swipe from the left to get to your previous app > > - swipe from the right to get to your next app > > - swipe from the top to reveal the statusbar (another swipe will let you > > access the utility tray) > > And we have 3 small divs on each side of the screen to handle this. > > > > If this is causing issues once layerized I'll be happy to help workaround it > > on the gaia side but I'm not sure I understand the problem (is this > > triggering a fullscreen-sized layer?). > > Etienne, the following are the display items dump for these three small > divs. They required almost a full screen memory (w=480, h=825) and are > visible in the top area (x=0,y=0,w=480,h=1). I guess the visible area is > related to 'AppChrome3' element. Are you able to make it as invisible? After checking greg's patch, I couldn't see display item regarding to 'AppChrome3' anymore. Therefore, I would suggest to verify this issue after 2.2 PVT build is ready. > > ClientPaintedLayer (0xaf932170) [bounds=(x=0, y=-31, w=480, h=825)] > [visible=< (x=0, y=0, w=480, h=1); >] { hitregion=< (x=0, y=0, w=480, h=30); > (x=0, y=90, w=30, h=704); (x=450, y=90, w=30, h=704); > > dispatchtocontentregion=< (x=0, y=0, w=480, h=30); (x=0, y=90, w=30, h=704); > (x=450, y=90, w=30, h=704); >} [valid=< (x=0, y=0, w=480, h=1); >] > nsDisplayTransform p=0xac69c670 f=0xaefa32b8(Block(div)(0)) z=5 > bounds(0,-1228,19200,1229) layerBounds(0,-1228,19200,1229) > visible(0,0,19200,1) componentAlpha(0,0,0,0) clip(0,0,19200,34160) > AppChrome3 chrome (opaque 0,0,19200,1)[ 1 0; 0 0.682; 0 -30.69; ] > layer=0xaf932170 > BackgroundColor p=0xac69d230 f=0xb14e3458(Block(div)(105)) > z=4093 bounds(0,3600,1200,28160) layerBounds(0,3600,1200,28160) > visible(0,3600,1200,28160) componentAlpha(0,0,0,0) clip(0,3600,1200,28160 > [0,3600,1200,28160 corners 0,0,1200,1200,1200,1200,0,0]) uniform left-panel > gesture-panel (rgba 1,0,0,0) layer=0xaf932170 > BackgroundColor p=0xac69d3a8 f=0xb14e36f0(Block(div)(107)) > z=4093 bounds(18000,3600,1200,28160) layerBounds(18000,3600,1200,28160) > visible(18000,3600,1200,28160) componentAlpha(0,0,0,0) > clip(18000,3600,1200,28160 [18000,3600,1200,28160 corners > 1200,1200,0,0,0,0,1200,1200]) uniform right-panel gesture-panel (rgba > 0,0.501961,0,0) layer=0xaf932170

Hi Greg, mind putting assignee as you? Thanks.

OK. Done.

Comment on attachment 8576499 [details] [gaia] snowmantw:bug1133144-2.2 > mozilla-b2g:v2.2 [Approval Request Comment] [Bug caused by] (feature/regressing bug #): DOM structure got changed but CSS left the same [User impact] if declined: Bug occurs [Testing completed]: Try result (green): https://treeherder.mozilla.org/#/jobs?repo=gaia-try&revision=1c80244b8ec0 [Risk to taking this patch] (and alternatives if risky): No [String changes made]: No

Master: https://github.com/mozilla-b2g/gaia/commit/958272d2984a3dac54be5094e55c6feb5e71f9db v2.2: https://github.com/mozilla-b2g/gaia/commit/a791a06bb3528c407a346ab56e01c3b019131ea7