Right now we have multiple copies of each branch on our build slaves, one per builder. We have as many as 6 non-l10n non-mobile builders per branch currently. If we share a single srcdir and use a different objdir per build, we'll be able to save disk, clobber less, re-clone fewer times, die on hg clone errors fewer times. That isn't to say there isn't a lot of work to be done. There is, and even more testing. o mozconfigs should have their paths specified rather than copying into the srcdir. Keep the srcdir as pristine as possible, which should mean zero changes. o bhearsum requests that the objdir live inside the builder directory... e.g. /builds/slave/nameOfBuilder/objdir. I'm not sure if there's an easy way to get the absolute path of the builder dir without using SetProperty + pwd... go with whatever works. This step may not be the same on MSYS as true POSIX. o Leave l10n, mobile, and most likely release builders alone for now. We'll revisit in another bug. This will affect the helper scripts as well. One very obvious example is the clobberer, which will need to be updated to be srcdir aware. (If it cleans up the srcdir, it should do so once per machine, rather than once per builder. Also, we may just want to clean the objdir.) For clobberer specifically, you may want to set a bool that says this is a split objdir builder, and only clean up objdir for now. I'll file another srcdir-aware clobberer bug. Deal with any other tool changes similarly; get them working if possible, but also split out to other bugs if needed and the group can decide whether those changes can wait or not. You can decide how you want to split this up (per branch? per platform?), but I'd feel safest if you tested each builder affected in both depend and clobber mode (if applicable). Both of those would require a previous build, so that's up to 3 test builds per builder, assuming everything goes perfectly.

This is a proof of concept patch. It allows builders to share the same source tree. Diffed from hg.m.o/b/buildbotcustom 2c8a4ab0c551

This is a sample master.cfg to be used with the patch.

John: what do you think about adding the retry option in bug 503580 to your srcdir patch? It's bloat, but if we get it in before your testing we'll get good test coverage for it.

Aki: I think that would be easy enough to do. As per my comments in bug 503580 I will surround the hg call with a check of the return value.

This is a work in progress patch. I have only looked at MozillaBuildFactory and MercurialBuildFactory. I have moved a couple things (like how objdir is found) into the MozillaBuildFactory. If this is not ok, please let me know. I have changed self.objdir to be able to be used as a workdir param for any given step. I have also removed workdir='build....' hardcodes because build is no longer a valid assumption, as it might need to be '../mozilla-central' when using a shared source dir and invoking a command in the srcdir. I have also added an obdjirAbs member to MozillaBF which can be used to in the environment to set the MOZ_OBJDIR param or any command. I have changed the mozconfig from needing to copy the mozconfig file to srcdir/.mozconfig as that'd dirty up the srcdir. Instead, I have an environmental var, MOZCONFIG, which points to the mozconfig file inside the buildbot-configs checkout. This is only a WIP. It currently builds fine. I am adapting things like Codesighs and Leaktesting to our new assumptions

Because I don't have an hg.m.o user repo, i am working in http://hg.johnford.info/build/buildbotcustom/

Here's some high level comments, John: * MozillaBuildFactory is also the base for ReleaseFactory - which has children that don't care about scrdirs, objdirs, etc. I think that you need another class here. MozillaAppFactory, or something? * I'm not sure env, config* belong in a base - need to think about that one more though. * What's the hgHost manipulation for? I don't see you doing anything else with it. * Please find a better way to do the 'hg purge'. I'd rather not introduce more hacky bash-in-a-shellcommand steps. TinderboxShellCommand might help you out (it can ignore given exit codes).

(In reply to comment #7) > Here's some high level comments, John: > * MozillaBuildFactory is also the base for ReleaseFactory - which has children > that don't care about scrdirs, objdirs, etc. I think that you need another > class here. MozillaAppFactory, or something? Totally agree, I will make that class now. Is it ok to leave the builddir full path property there? > * I'm not sure env, config* belong in a base - need to think about that one > more though. They need to be in whatever class does objdir/srcdir stuff as the env is required to set the objdir. That also makes the config* stuff appropriate as I can set the mozconfig file in the dictionary > * What's the hgHost manipulation for? I don't see you doing anything else with > it. I will remove that > * Please find a better way to do the 'hg purge'. I'd rather not introduce more > hacky bash-in-a-shellcommand steps. TinderboxShellCommand might help you out > (it can ignore given exit codes). I will use this, i didn't know that it existed.

(In reply to comment #8) > (In reply to comment #7) > > Here's some high level comments, John: > > * MozillaBuildFactory is also the base for ReleaseFactory - which has children > > that don't care about scrdirs, objdirs, etc. I think that you need another > > class here. MozillaAppFactory, or something? > Totally agree, I will make that class now. Is it ok to leave the builddir full > path property there? I think so, yeah. It might end up being useful to other consumers.

This patch creates a new base class called MozillaAppBuildFactory which knows the basics of building a single source mozilla application. It provides many hooks for derived classes. This patch also modifies the existing MercurialBuildFactory to hook into MozAppBF. This patch is not fully working and is a WIP. There are a couple issues with the Steps orders.

I am working on this again. I am going to be keeping my work at http://hg.johnford.info/build-sharedsource/ and running my own buildbot master for development at http://johnford.info:8010

While this would be great, we should decide if we would prefer to go with build scripts before we implement this. If we still want to use buildbot factories to control build logic we can reopen this bug.