Gregory Szorc [:gps] Reporter
	Description • 11 years ago

https://hg.mozilla.org/build/tools/file/default/scripts/spidermonkey_builds/spidermonkey.sh#l84 and below are mostly build system implementation details and should live in the tree, not in some external repository because we want the logic for producing standalone SpiderMonkey builds to be versioned with the tree.

I propose we extract this code to an in-tree script (or even make file if it's appropriate) and then modify the RelEng repo to call into the in-tree script.

	Gregory Szorc [:gps] Reporter
	Comment 1 • 11 years ago

Pretty much the entire directory https://hg.mozilla.org/build/tools/file/default/scripts/spidermonkey_builds/ should be in mozilla-central.

I plan to code this up because it is blocking me from landing build system improvements. If someone says I can break standalone SpiderMonkey builds in the form of "all header removals will require a clobber," then I will happily stop working on this.

	Gregory Szorc [:gps] Reporter
	Comment 2 • 11 years ago

Actually, the change in bug 896797 didn't regress the JS build system behavior w.r.t. removed headers not being deleted from $(DIST)/include, so we went ahead and landed that patch in a way that won't break the TBPL SM builders.

I'm returning this bug to the pool. While I would love to see this bug fixed so we don't blow up the tree due to things beyond the tree's control, I don't think it's near the top of build system priorities ATM. The current configs - being out of tree - could blow up at any time. This is a ticking time bomb waiting to go off.

	Steve Fink [:sfink] [:s:]
	Comment 3 • 11 years ago

These builds will change in bug 898970, but not necessarily in a way that's relevant to this bug. Specifically, I plan to stop using spidermonkey.sh and related files, and switch to mozharness-based builds. But if I put the actual logic into the mozharness repo, then we'd just be moving from the build/tools repo to the build/mozharness repo.

But here's another reason to deprioritize this bug -- currently, the spidermonkey builds are *always* clobbers:

  # Always do clobber builds. They're fast.
  [ -d objdir ] && rm -rf objdir
  mkdir objdir
  cd objdir

So to keep these working, you just need to keep the standard 'autoconf-2.13; configure; make' dance working for the js/src/ tree. (Admittedly, these builds are not quite so fast now that they're building icu, but we're comparing to the full gecko build here.)

At the same time, if the build system grew a way to make these builds with something like |./mach build spidermonkey --please|, I'd be fine with switching them over. (Well, ok, the critical part is controlling the configure options, since that's what distinguishes the various spidermonkey builds from each other. But I'm not hung up on the exact mechanism for doing that. If I need to write out a mozconfig-like file, that's fine.)

	Gregory Szorc [:gps] Reporter
	Comment 4 • 11 years ago

The now-unblocked issue for us was that we made a change to js/src/Makefile that we *thought* would be executed at the top of *any* standalone JS build. The change assumed this. So, to have an out-of-tree process invoke js/src/Makefile in such a way that it wasn't the first thing that executed invalidated our assumptions and caused bustage. Not having the ability to merely check in a patch to mozilla-central is extremely annoying in these scenarios. You have to worry about supporting multiple trees, multiple channels, riding the trains, etc. It's very painful and prevents us from moving fast.

Low-level build system logic like this needs to live in the tree. I steadfastly believe that RelEng/automation should only be interacting with the tree via a set of well-defined APIs (e.g. mach commands or script files). Any attempts to bypass "sanctioned" APIs (invoking configure directly is one such bypass) should be forbidden. We should actively seek out and destroy all buildbot/mozharness configs that don't interact with the build system in this way. Step 1 of that is probably for the build peers to establish a set of "sanctioned" APIs. But that's for another bug.

	Steve Fink [:sfink] [:s:]
	Comment 5 • 11 years ago

(In reply to Steve Fink [:sfink] from comment #3)
> At the same time, if the build system grew a way to make these builds with
> something like |./mach build spidermonkey --please|, I'd be fine with
> switching them over. (Well, ok, the critical part is controlling the
> configure options, since that's what distinguishes the various spidermonkey
> builds from each other. But I'm not hung up on the exact mechanism for doing
> that. If I need to write out a mozconfig-like file, that's fine.)

Well... maybe I should back off on this a bit. This would serve the the purposes of the variety of spidermonkey builds we have. But those builds are currently *also* testing whether autoconf/configure/make still works, which spidermonkey developers and embedders expect. So if I moved the existing spidermonkey builds over to a toplevel mach invocation, it's likely that someone would want to add a new build just to keep autoconf/configure/make working. (Actually, embedders/distributions want 'make install' too, which is currently not tested.) Nobody would insist on it living outside of the tree, though.

	:Ms2ger (he/him; ⌚ UTC+1/+2)
	Comment 6 • 11 years ago

FWIW (and I'll try not to go on this tangent for too long), I find the configure/make way of building the js shell extremely painful, in particular if I need to reconfigure and then have to figure out what magic incantation I used last time...

	Steve Fink [:sfink] [:s:]
	Comment 7 • 11 years ago

(In reply to :Ms2ger from comment #6)
> FWIW (and I'll try not to go on this tangent for too long), I find the
> configure/make way of building the js shell extremely painful, in particular
> if I need to reconfigure and then have to figure out what magic incantation
> I used last time...

I always grab the configure options out of config.status (they're near the end). Though ./config.status --help says that you should just be able to do |./config.status --recheck|.

For the record, I'm not the one to convince about preserving configure/make. If direct use of configure is going to be removed, that should be a separate (contentious) bug.

	Steve Fink [:sfink] [:s:]
	Comment 8 • 10 years ago

I ended up doing this in bug 1079579 (though note that all the original logic remains in spidermonkey.sh for now to support old branches.)