Chris AtLee [:catlee] Reporter
	Description • 15 years ago

At least the following builds have run out of disk space:

Electrolysis linux depend build (asked for 5, got 5.06 GB):
http://tinderbox.mozilla.org/showlog.cgi?log=Electrolysis/1255679363.1255683377.32120.gz

du -ms on the slave shows 5199 MB used in that directory.

Mac OSX debug build (asked for 3, got 3.72 GB):
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1254853004.1254853805.26011.gz

	Chris AtLee [:catlee] Reporter
	Comment 1 • 15 years ago

For bonus points, it would be nice to:
- Detect when builds fail because they're out of disk, and flag that somehow.  It could be that a new change has a bug that causes too much disk space to be used, but more likely it's just regular growth in the code base.

- Monitor the peak disk usage of a build, and warn when we're getting close to the specified limit.  Note that 'du' is slow, but 'df' is fast...we could run 'df' at the beginning of the build, and periodically throughout to give an idea of how much space is being used...assuming nothing else is writing to the disk.

	Nick Thomas [:nthomas] (UTC+12)
	Comment 2 • 15 years ago

Electrolysis unit on linux - asked for 5G and got 5.02G, ran out of space doing make package.
http://tinderbox.mozilla.org/showlog.cgi?log=Electrolysis/1255837061.1255840559.4578.gz

	John Ford [:jhford] CET/CEST Berlin Time
	Comment 3 • 15 years ago

I am looking at this right now.

	John Ford [:jhford] CET/CEST Berlin Time
	Comment 4 • 15 years ago

braindump:  At the end of a build, we could have the client post to a server the amount of disk space required for the type of build and have that used instead of a hard coded value.  This would likely add a couple minutes to get the directory size for.

As an immediate mitigation, i will bump the build space requested.

	Chris AtLee [:catlee] Reporter
	Comment 5 • 15 years ago

(In reply to comment #4)
> braindump:  At the end of a build, we could have the client post to a server
> the amount of disk space required for the type of build and have that used
> instead of a hard coded value.  This would likely add a couple minutes to get
> the directory size for.

I'd imagine that our peak usage at some point in the build is sometimes greater than the total space in the directory tree at the end of the build, so we'd run into problems with this method.  Indeed, the initial build space settings were derived from examining the total disk space in completed build directories.

	John Ford [:jhford] CET/CEST Berlin Time
	Comment 6 • 15 years ago

I am going to make this value set per platform+buildtype instead of branch+platform+buildtype in a patch coming soon

	John Ford [:jhford] CET/CEST Berlin Time
	Comment 8 • 15 years ago

This patch adds a dictionary that defines the build_space parameter on a platform-buildtype basis instead of the current branch-platform-buildtype basis.

I am currently testing this in staging

	Nick Thomas [:nthomas] (UTC+12)
	Comment 9 • 15 years ago

A mozilla-central debug everythingelse build got exactly the 0.5G it asked for, then died unpacking the symbols. IIRC that size is hard coded on the factory instantiation in misc.py. Could we move it into the dict too ?

	John Ford [:jhford] CET/CEST Berlin Time
	Comment 10 • 15 years ago

yep, that sounds like a good idea.  Can I do that as a follow up patch?

	John Ford [:jhford] CET/CEST Berlin Time
	Comment 11 • 15 years ago

Comment on attachment 407166 [details] [diff] [review]
build_space dictionary

I have tested this and it works in staging

	Nick Thomas [:nthomas] (UTC+12)
	Comment 12 • 15 years ago

Comment on attachment 407166 [details] [diff] [review]
build_space dictionary

>diff -r e19a365c45df mozilla2-staging/config.py
>+# BUILDSPACE
>+# This dictionary is used for figuring out required disk space
>+# per platform and buildtype
>+build_space = {}
>+build_space['win32'] = {'plain': 8, 'debug': 6, 'unit': 5}
>+build_space['macosx'] = {'plain': 6, 'debug': 4, 'unit': 5}
>+build_space['linux'] = {'plain': 6, 'debug': 4, 'unit': 5}
>+build_space['wince'] = {'plain': 5}
>+build_space['unittest'] = 5

r- for these two
* We don't end up using the |'unit': 5| part anywhere so it can get removed from each platform
* To address the Electrolysis unit running out of space you should bump unittest from 5 to 6 
Also
* I'd leave the wince size at 4 in the absence of reported failures. Was anything else incremented like that ?
* I'd suggest setting |build_space['default'] = 5| and using that here ...
>-    'default_build_space': 5,
>+    'default_build_space': build_space['unittest'],

>-BRANCHES['mozilla-1.9.1']['platforms']['linux-debug']['build_space'] = 3
>-BRANCHES['mozilla-1.9.1']['platforms']['win32-debug']['build_space'] = 4
>-BRANCHES['mozilla-1.9.1']['platforms']['macosx-debug']['build_space'] = 3

I'm wondering if switching to the global defaults is sub-optimal here. m-1.9.1 will be growing/changing very slowly. win32-debug will jump from 4 to 6G and cause more builds to be free-space clobbers, making compile times longer across the board.

>-BRANCHES['tracemonkey']['platforms']['linux']['build_space'] = 5
>-BRANCHES['tracemonkey']['platforms']['linux64']['build_space'] = 5
>-BRANCHES['tracemonkey']['platforms']['win32']['build_space'] = 5

I could make a similar argument here, where we don't do PGO for Win32 TraceMonkey, Places or Electrolysis. This patch would ask for another 3G of space that won't be used.

bhearsum and catlee, would you chose easy configuration w/ some wasted space, or more complicated config + less free space clobbers ?

	Chris AtLee [:catlee] Reporter
	Comment 13 • 15 years ago

(In reply to comment #12)
> bhearsum and catlee, would you chose easy configuration w/ some wasted space,
> or more complicated config + less free space clobbers ?

I'd choose complicated config + fewer free space clobbers.  It's more overhead on our part, but means more efficient slaves.

	Chris AtLee [:catlee] Reporter
	Comment 14 • 15 years ago

A nightly OSX build just failed:
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1256205720.1256213706.23497.gz&fulltext=1

Asked for 5.2 GB, got 5.41GB.

	John Ford [:jhford] CET/CEST Berlin Time
	Comment 15 • 15 years ago

we could have the 1.9.1 and 1.9.2 based builds have their own space requirements (i.e. build_space['win32']['debug-191']).  If we would prefer the status quo, that is fine by me and I will whip up the patch to bump up the required.

	(not currently active) Ted Mielczarek
	Comment 16 • 15 years ago

Another mac nightly failure:
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1256292120.1256296647.1121.gz

	Chris AtLee [:catlee] Reporter
	Comment 17 • 15 years ago

I wonder what would happen if we really over-specify the amount of space required for builds?  E.g. set it to 10 GB.

I _think_ you'd slightly increase the number of clobber builds, but not by much.  In effect you'd ensure that the build has the 5 or 6 GB free that it needs, plus an extra 4 GB of working space.

	John Ford [:jhford] CET/CEST Berlin Time
	Comment 18 • 15 years ago

I think that over-specifying is good, but are we going to do this on old branches as well?

	John Ford [:jhford] CET/CEST Berlin Time
	Comment 19 • 15 years ago

http://tinderbox.mozilla.org/showlog.cgi?tree=Firefox&errorparser=unittest&logfile=1256311202.1256314247.16241.gz&buildtime=1256311202&buildname=Linux%20mozilla-central%20leak%20test%20build&fulltext=

I will bump up buildspaces right now.

	(not currently active) Ted Mielczarek
	Comment 20 • 15 years ago

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1256311202.1256314247.16241.gz
Linux mozilla-central leak test build on 2009/10/23 08:20:02

	John Ford [:jhford] CET/CEST Berlin Time
	Comment 21 • 15 years ago

Unbitrot, fix issues.

I am going to create a patch to bump the numbers without a dictionary as we need something that can get checked in now if we don't do the dictionary.

	John Ford [:jhford] CET/CEST Berlin Time
	Comment 22 • 15 years ago

Here is something to stop the burning.

	Chris AtLee [:catlee] Reporter
	Comment 23 • 15 years ago

Comment on attachment 408037 [details] [diff] [review]
bump-buildspace non-dictionary

good enough for now.

	(not currently active) Ted Mielczarek
	Comment 24 • 15 years ago

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1256310827.1256315010.25915.gz
OS X 10.5.2 mozilla-central leak test build on 2009/10/23 08:13:47

	John Ford [:jhford] CET/CEST Berlin Time
	Comment 25 • 15 years ago

Comment on attachment 408037 [details] [diff] [review]
bump-buildspace non-dictionary

I can do the reconfig if somebody else can check it.

	Nick Thomas [:nthomas] (UTC+12)
	Comment 26 • 15 years ago

Comment on attachment 408037 [details] [diff] [review]
bump-buildspace non-dictionary

Here you go,
http://hg.mozilla.org/build/buildbot-configs/rev/e5e7a1c08f9f

	Daniel Holbert [:dholbert]
	Comment 27 • 15 years ago

this happened again, two hours after after the failure in comment 24:

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1256317656.1256320410.24885.gz
Linux mozilla-central leak test build on 2009/10/23 10:07:36

	Daniel Holbert [:dholbert]
	Comment 28 • 15 years ago

...and again, around 3 hours after the that:
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1256329760.1256332439.678.gz
Linux mozilla-central leak test build on 2009/10/23 13:29:20

	John Ford [:jhford] CET/CEST Berlin Time
	Comment 29 • 15 years ago

Linux debug build failed when it got 4.4GB and asked for 3gb.

	Lukas Blakk [:lsblakk] use ?needinfo
	Comment 30 • 15 years ago

Comment on attachment 408136 [details] [diff] [review]
bump up debug builds as well

http://hg.mozilla.org/build/buildbot-configs/rev/cf48442b58fb

	Nick Thomas [:nthomas] (UTC+12)
	Comment 31 • 15 years ago

http://hg.mozilla.org/build/buildbot-configs/rev/83a8d1840448

We hit another out-of-disk for a m-c linux debug. I checked a successful build and it was using about 5400M (after the fact), so I bumped the disk space requirement from 5G to 6G. This is probably fallout from switching to gstabs+ symbols in bug 519196.

Masters reconfig'd.

	Nick Thomas [:nthomas] (UTC+12)
	Comment 32 • 15 years ago

We should also sync these changes back to staging.

	Nick Thomas [:nthomas] (UTC+12)
	Comment 33 • 15 years ago

Builds like "WINNT 5.2 mozilla-central test debug mochitests-3/5" need more than 0.5G on slaves using FAT disks. I found one using 638M. NTFS uses less space.

	Chris AtLee [:catlee] Reporter
	Comment 34 • 15 years ago

Another debug mochitest failed on windows after getting 0.53GB free.

	Aki Sasaki (not active)
	Comment 35 • 15 years ago

I just theorized that this could be a possible cause of bug 509960 ... though it could also be a parallel make issue.

I'm kinda leaning towards doubling these numbers -- I think reliability takes precedence over speed.

	Aki Sasaki (not active)
	Comment 36 • 15 years ago

Out of disk space:
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1256675793.1256679213.8509.gz

I never was a fan of trying to squeeze every ounce of disk space. Maybe I'm old fashioned, but running builds on a disk where we expect to edge up against a gig or less of free space seems like we're asking for trouble.

If we double these numbers, we will clobber more often, slowing things down. We will also hit that hard limit much less often, and we can re-enable stricter nagios checks to tell us when those types of things happen.

We also expect to see significant disk usage reduction when bug 498522 lands.

Patch to double disk space requirements incoming.

	Aki Sasaki (not active)
	Comment 37 • 15 years ago

(In reply to comment #36)
> If we double these numbers, we will clobber more often, slowing things down. We
> will also hit that hard limit much less often, 

(^ ... which will result in fewer build/test failures, ...)

	Aki Sasaki (not active)
	Comment 38 • 15 years ago

While bumping these I realized that the lack of /scratchbox purging can cause a lot of linux burning; marked the bug dependent on bug 524820 .

We'll see if jhford gets the bind mount fix before I can update purge_builds.py; that is possibly the ideal solution.

	Nick Thomas [:nthomas] (UTC+12)
	Comment 39 • 15 years ago

Comment on attachment 408035 [details] [diff] [review]
build_space dictionary v2

Apologies for not getting to this patch earlier. Please reset r? if you still want review on it given the subsequent comments. I'd note that the staging and production changes aren't the same, eg there are still 'unit' entries in the staging dict.

	Nick Thomas [:nthomas] (UTC+12)
	Comment 40 • 15 years ago

To fix:

(In reply to comment #33)
> Builds like "WINNT 5.2 mozilla-central test debug mochitests-3/5" need more
> than 0.5G on slaves using FAT disks. I found one using 638M. NTFS uses less
> space than FAT.

	Nick Thomas [:nthomas] (UTC+12)
	Comment 41 • 15 years ago

Comment on attachment 410428 [details] [diff] [review]
Bump packaged unit test up to 1G.

http://hg.mozilla.org/build/buildbotcustom/rev/db5d3d010d67

	Nick Thomas [:nthomas] (UTC+12)
	Comment 42 • 15 years ago

A couple of linux unit tests for e10s have run out of space during make package today, asking for 5G and getting 5.01G. In production this syncs the m-c unittest_build_space of 6G to the project branches, and then syncs staging to prod.

	Nick Thomas [:nthomas] (UTC+12)
	Comment 43 • 15 years ago

Comment on attachment 411610 [details] [diff] [review]
Sync m-c unit to project branches

http://hg.mozilla.org/build/buildbot-configs/rev/9d766dbe92ea

	Aki Sasaki (not active)
	Comment 44 • 15 years ago

This is a somewhat vaguely worded bug that might never be closed.

If we want to keep it, it should probably be a [buildduty] bug; otherwise we should resolve + open new ones as needed.

	Chris AtLee [:catlee] Reporter
	Comment 46 • 15 years ago

Comment on attachment 414058 [details] [diff] [review]
increase mac builds to 7 GB

changeset:   1776:1fd91ffe4cb3

	Chris AtLee [:catlee] Reporter
	Comment 49 • 15 years ago

Comment on attachment 416170 [details] [diff] [review]
increase win32-debug builds on m-c and projects to 7 GB

changeset:   1832:7cc1b34623dd

	Armen [:armenzg]
	Comment 50 • 15 years ago

"Linux mozilla-central leak test build" failed at the "make package" step by lack of space on the device.
6.03 GB of space available at the beginning of the build.
http://production-master.build.mozilla.org:8010/builders/Linux%20mozilla-central%20leak%20test%20build/builds/5075

The patch requests increasing to 7GB of space.

	bhearsum@mozilla.com (:bhearsum)
	Comment 51 • 15 years ago

Comment on attachment 417697 [details] [diff] [review]
bump unittest space from 6GB to 7GB in all branches except m191

Actually, let's bump the linux-debug space, since that's the build that actually failed.

	Armen [:armenzg]
	Comment 52 • 15 years ago

Increasing linux-debug build space rather than unit test's build space.

	bhearsum@mozilla.com (:bhearsum)
	Comment 53 • 15 years ago

Comment on attachment 417703 [details] [diff] [review]
 bump leak test space from 6GB to 7GB in all branches except m191 and m192

Might have to bump 1.9.2 later, but this seems fine for now.

	Armen [:armenzg]
	Comment 54 • 15 years ago

Comment on attachment 417703 [details] [diff] [review]
 bump leak test space from 6GB to 7GB in all branches except m191 and m192

http://hg.mozilla.org/build/buildbot-configs/rev/d6872b792a77

	Chris AtLee [:catlee] Reporter
	Comment 56 • 15 years ago

Comment on attachment 418314 [details] [diff] [review]
Bump tracemonkey build space to 7 GB

changeset:   1871:866a524e5f8d

	Chris Cooper [:coop] (he/him)
	Comment 57 • 15 years ago

Are we still holding this open for bug 489410?

	Chris Cooper [:coop] (he/him)
	Comment 58 • 15 years ago

Let's file new bugs for this as it becomes an issue rather than holding one bug open indefinitely.