All of our Tinderbox builds from mozilla-central for Hamachi are failing to flash, and seem to fill up the device; here's a sample job: http://selenium.qa.mtv2.mozilla.com:8080/view/B2G%20Master/job/b2g.hamachi.mozilla-central.perf.fps.bug958036/37/console Its output, with strategic snips: We start off with: 16:42:25 Showing free space 16:42:25 Filesystem Size Used Free Blksize 16:42:25 /dev 90M 48K 90M 4096 16:42:25 /mnt/asec 90M 0K 90M 4096 16:42:25 /mnt/obb 90M 0K 90M 4096 16:42:25 /system 200M 173M 26M 4096 16:42:25 /data 161M 1M 159M 4096 16:42:25 /persist 4M 908K 3M 4096 16:42:25 /cache 40M 1M 38M 4096 then we push Gaia/webapps: ... 6:42:45 push: shallowflashgaia_temp/gaia/profile/webapps/webapps.json -> /system/b2g/webapps/webapps.json 16:42:45 117 files pushed. 0 files skipped. 16:42:45 3556 KB/s (63873179 bytes in 17.537s) 16:42:45 79 KB/s (3547 bytes in 0.043s) 16:42:46 3980 KB/s (229257 bytes in 0.056s) Pushing Gecko: 16:42:46 Pushing Gecko 16:42:46 push: shallowflashgecko_temp/b2g/dictionaries/en-US.dic -> /system/b2g/dictionaries/en-US.dic 16:42:46 push: shallowflashgecko_temp/b2g/dictionaries/en-US.aff -> /system/b2g/dictionaries/en-US.aff 16:42:46 push: shallowflashgecko_temp/b2g/Throbber-small.gif -> /system/b2g/Throbber-small.gif 16:42:46 push: shallowflashgecko_temp/b2g/ua-update.json -> /system/b2g/ua-update.json 16:42:46 push: shallowflashgecko_temp/b2g/plugin-container -> /system/b2g/plugin-container 16:42:46 push: shallowflashgecko_temp/b2g/run-mozilla.sh -> /system/b2g/run-mozilla.sh 16:42:46 push: shallowflashgecko_temp/b2g/libnss3.so -> /system/b2g/libnss3.so 16:42:47 push: shallowflashgecko_temp/b2g/application.ini -> /system/b2g/application.ini 16:42:47 push: shallowflashgecko_temp/b2g/crashreporter.ini -> /system/b2g/crashreporter.ini 16:42:47 push: shallowflashgecko_temp/b2g/dependentlibs.list -> /system/b2g/dependentlibs.list 16:42:47 push: shallowflashgecko_temp/b2g/updater -> /system/b2g/updater 16:42:47 push: shallowflashgecko_temp/b2g/libxul.so -> /system/b2g/libxul.so 16:42:55 failed to copy 'shallowflashgecko_temp/b2g/libxul.so' to '/system/b2g/libxul.so': No space left on device The odd thing is that 1.3-flashing jobs "recover" the device, so it might be a build-only/flash-time only issue, which would be nice... Still, this is critical to our automation on mozilla-central (Gecko)/master (Gaia)

Here's what I saw after we failed to flash: webqa@b2g-8:~$ adb shell df Filesystem Size Used Free Blksize /dev 90M 48K 90M 4096 /mnt/asec 90M 0K 90M 4096 /mnt/obb 90M 0K 90M 4096 /system 200M 173M 26M 4096 /data 161M 4M 156M 4096 /persist 4M 908K 3M 4096 /cache 40M 1M 38M 4096 We only have 26M free for /system -- is that typical?

So, in other, passing master/mozilla-central jobs, the free space, before we flash Gecko, looks like this: 14:20:19 Showing free space 14:20:19 Filesystem Size Used Free Blksize 14:20:19 /dev 90M 48K 90M 4096 14:20:19 /mnt/asec 90M 0K 90M 4096 14:20:19 /mnt/obb 90M 0K 90M 4096 14:20:19 /system 200M 140M 59M 4096 14:20:19 /data 161M 1M 159M 4096 14:20:19 /persist 4M 908K 3M 4096 14:20:19 /cache 40M 1M 38M 4096

Yeah something happened in the last week which caused the system memory usage to jump. I was able to flash my eng build a few days ago, but couldn't yesterday.

The best regression range I have right now from our Jenkins instance is: Failed > Console Output #479 Apr 23, 2014 3:07:34 PM Success > Console Output #478 Apr 23, 2014 2:18:56 PM

Gaia push log: https://github.com/mozilla-b2g/gaia/compare/0a4d2dea25a7162ee43db3a0db817798b70e7521...33b5bcb1a092172c983ef178766869aba738ad4d Gecko push log: http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=35737ab92f9e&tochange=3cd4615c60ba

I have a strong feeling that bug 968661 caused this. George - Can you check? If that ends up being the case, then can you backout?

I have executed |make PRODUCTION=1 MOZILLA_OFFICIAL=1| under below two commits and found out profile/webapps 's size are all 56MB . I think it's properly not my regression... 4317d7ab7fc4193f55655490fc343ba7bc718bb3 Bug 968661 - Extract new build module webapp-shared.js from webapp-zip.js 75951b8227b54b3dd39ec8878c0be96eda47efd9 (commit before 4317d7ab7fc4193f55655490fc343ba7bc718bb3)

Can someone download the ok build and the failed build and compare its |du -ch|?

https://docs.google.com/spreadsheets/d/1O29h36iASO3LjINgDpftq5h7J8E77HiNAAzDlicr5AQ/edit?usp=sharing Please reference above attachment.

Thanks Al, It looks like every Gaia app is getting larger .... need to bisect this range and find the offending commit.

Alright - marking window wanted to bisect this further

I am working on the range now.

b2g-inbound regression range: http://hg.mozilla.org/integration/b2g-inbound/pushloghtml?fromchange=1c5860c1ca51&tochange=fcf7b2b1ef09 double checked the last working and first failings too.

(In reply to George Duan [:gduan] [:喬智] from comment #9) > I have executed |make PRODUCTION=1 MOZILLA_OFFICIAL=1| under below two > commits and found out profile/webapps 's size are all 56MB . I think it's > properly not my regression... > > 4317d7ab7fc4193f55655490fc343ba7bc718bb3 Bug 968661 - Extract new build > module webapp-shared.js from webapp-zip.js > > 75951b8227b54b3dd39ec8878c0be96eda47efd9 (commit before > 4317d7ab7fc4193f55655490fc343ba7bc718bb3) Looks like the bisection seems to disagree with this - bug 968661 is the only patch in the push log. Let's get this backed out & find out why this isn't happening during local testing & only in pvtbuilds.

jsmith, while working on this regression range njpark found that this build was OK when using a Hamachi with a different base build. We reasoned that the different base build was slightly smaller. I found the range using the V1.2-device.cfg base build.

(In reply to Zac C (:zac) from comment #19) > jsmith, while working on this regression range njpark found that this build > was OK when using a Hamachi with a different base build. We reasoned that > the different base build was slightly smaller. > > I found the range using the V1.2-device.cfg base build. I have two buris with different base builds- one says ro.build.version.incremental=eng.cltbld.20140306.073741 ro.build.date=Thu Mar 6 08:02:39 EST 2014 and the other says ro.build.version.incremental=eng.tclxa.20131223.163538 │ ro.build.date=Mon Dec 23 16:36:04 CST 2013 The Dec 23 2013 one matches what zac sees, but with Mar 6 2014 base build, the phone is booting up normally. But it has a issue that it cannot recognize the sim card. I am working to flash a base build to see whether i can identify it is hw issue or base image issue. BTW, I think i got the Mar 6 base image from teleweb. Not sure whether that is the proper way to get the base image though.

Turns out Mar 6 2014 base image is not recommended for nhirata_. so zac's result should be correct.

Backed out in https://bugzilla.mozilla.org/show_bug.cgi?id=968661#c51.

I've tweaked the flash script to show free space after pushing gecko and gaia, which may help to identify or track how large these are getting. See https://gist.github.com/davehunt/7646076/revisions

Verified FIXED with the latest tinderbox build; we're back in action (still awaiting the twice-daily mozilla-central build to pick up the fix, but it'll be there). Thanks!

Flashing the nightly mozilla-central-hamachi-eng build still doesn't work on both of my hamachi devices - "no space left on device" error. I am using the latest flashing script as mentioned in comment 23. I even blew away the device's /system folder and reloaded the base image (V1.2-device.cfg) but still the same result. From the flashing script, before the clean: 13:04:03 [1mShowing free space[0m 13:04:03 Filesystem Size Used Free Blksize 13:04:03 /dev 90M 48K 90M 4096 13:04:03 /mnt/asec 90M 0K 90M 4096 13:04:03 /mnt/obb 90M 0K 90M 4096 13:04:03 /system 200M 194M 5M 4096 13:04:03 /data 161M 6M 154M 4096 13:04:03 /persist 4M 908K 3M 4096 13:04:03 /cache 40M 7M 32M 4096 After the clean: 13:04:05 [1mShowing free space[0m 13:04:05 Filesystem Size Used Free Blksize 13:04:05 /dev 90M 48K 90M 4096 13:04:05 /mnt/asec 90M 0K 90M 4096 13:04:05 /mnt/obb 90M 0K 90M 4096 13:04:05 /system 200M 140M 59M 4096 13:04:05 /data 161M 4M 156M 4096 13:04:05 /persist 4M 908K 3M 4096 13:04:05 /cache 40M 1M 38M 4096 Fails loading the webapps: 13:04:24 failed to copy 'shallowflashgaia_temp/gaia/profile/webapps/settings.gaiamobile.org/application.zip' to '/system/b2g/webapps/settings.gaiamobile.org/application.zip': No space left on device On the device after the flashing attempt: root@android:/ # df Filesystem Size Used Free Blksize /dev 90M 48K 90M 4096 /mnt/asec 90M 0K 90M 4096 /mnt/obb 90M 0K 90M 4096 /system 200M 197M 2M 4096 /data 161M 4M 156M 4096 /persist 4M 908K 3M 4096 /cache 40M 1M 38M 4096 Note: I tried flashing the TBPL hamachi central eng build, and it works. The gaia-ui endurance tests use the nightly build, so they are currently offline.

Rob - Can you open a new bug for this? The original root cause was backed out when this happened, so there is probably a new root cause that is causing this.

Ok, filed Bug 1004589, thanks