GYP is Chromium's own build system generator: http://code.google.com/p/gyp/ We use source code from a few Google projects that use GYP as their build-system: at least ANGLE and (afaik) Breakpad. We are currently manually porting these to our own Makefiles, which is tedious and potentially has to be partly redone whenever we sync with upstream. Also, for part of ANGLE, we are currently using GYP-generated VS project files on Windows, which is causing some trouble: bug 641630, bug 633348. Having a GYP generator generating Mozilla Makefiles should be very useful for these reasons.

Now WebRTC as well (with hundreds of .gyp files...)

Breakpad's gyp files only cover the Windows client, so it wouldn't be totally useful there. For WebRTC, I'm first trying to see if we can just make the gyp-generated Makefiles work on Windows. If that turns out to not be feasible then this is probably the tack I'll take.

Relevant to gps's work on making our build system suck less.

I don't think it's particularly relevant, TBH. The likelihood of us using GYP for Mozilla at large is virtually nil.

This still isn't finished, but it's getting closer. It's an awful awful mess, so I apologize if anyone actually reads this patch. The basic concept here is to: a) Generate one Makefile per gyp target (because our build system only supports one target per-Makefile.in), putting them fairly close to the gyp file they were generated from. I generate a subdir per-target, named "gypfile_targetname". b) Generate a common.mk that gets included by every Makefile.in, because GYP files set things like DEFINES and INCLUDES per-target-type (Release vs. Debug), so that has to be chosen at build-time, so this just encapsulates that logic. c) Generate a top-level Makefile.in that simply uses DIRS to build all the other targets. This doesn't produce a working WebRTC build yet, but it does compile some things. I have a few more things to fix, the known ones include: * Adding extra libs to the link line * Checking that build order of targets is correct wrt dependencies * Proper paths to source files (I think object files are winding up in the srcdir right now) * Handling some other GYP bits, like actions/rules/copies * Mac support (ObjC, might need to pull settings from the XCodeSettings) * Test building on Windows

Sample output (for webrtc.gyp) can be seen in attachment 583771 [details] [diff] [review] on bug 688187.

Looking good! Would it make sense to put this code under build/ instead of /media/webrtc/? IMO the files constituting the build system are fragmented enough as it is. I'd like to see consolidation of all the code logic (especially the Python bits) in a common directory so we can treat them as a Python package and not have so much PYTHONPATH hackiness. ensure_directory_exists(path) is not thread safe. You probably want the same construct used in bug 629668: try: os.makedirs(dir) except OSError, ex: if ex.errno != errno.EEXIST: raise ex # else directory already exists. silently ignore and continue Unless of course there is no way this can be called from multiple threads. Also, you use underscore for that method but CamelCase elsewhere. Underscore is the standard Python naming convention. But it looks like GYP uses CamelCase. Hmmm. Same goes for getdepth(). Be consistent. Can you convert the Makefile-producing methods into generators? This loosely couples the act of generating the Makefiles from the GYP-specific side-effect of writing out a file. Personally, I would like this because my alternate build system could consume the Makefile as a "stream" and pipe it directly into my parsing logic without involving any filesystem overhead. It also makes it easier to test the API since, again, you don't involve the filesystem. If the GYP->Makefile generation happens at check-in time, not after checkout, then I don't care as much. On a higher-level, GYP has access to the global state of everything, no? If so, it would be awesome if you could produce a single Makefile from all the GYP files in one go. Of course, this would require adding adding support to rules.mk or circumventing the rules.mk logic. Those are probably beyond scope, but I'm throwing it out there just in case. Either way, my magical build system could consume your generated Makefiles since they conform to the existing convention, so the end result would be the same (if I can deliver).

I fixed a couple of the things off that list: * Extra libs now make it to the link line (although it's still failing to link the first test program it gets to, need to figure that out) * Paths to source files have been sorted out * I punted on actions/rules/copies, they don't seem to be used in webrtc except for building the protobuf compiler, which I hear is optional

If you'd like to try this out, the patch in bug 688187 was generated with the following command after applying this patch: luser@cuatro:~/build/alder/media/webrtc/trunk$ ./build/gyp_chromium --debug=all --format=mozmake --depth=. -D build_with_mozilla=1 -G relative_srcdir=media/webrtc/trunk webrtc.gyp

Very excited to see this! If you want a big and useful GYP testcase, see http://hg.mozilla.org/mozilla-central/file/c5b90ea7e475/gfx/angle/src/build_angle.gyp

(In reply to Gregory Szorc [:gps] from comment #7) > Would it make sense to put this code under build/ instead of /media/webrtc/? It very much would, since we already have at least ANGLE that also uses GYP and would benefit from this.

Unfortunately, this code has to live inside of the gyp folder, since gyp loads its generators by module name. Also, apparently the trunk version of gyp can't build the WebRTC project, so I had to use the copy of gyp that was bundled with the WebRTC source.

This fixes most of the previous issues, I'm able to generate a set of Makefiles and build WebRTC on Linux. I need to test this on Mac and Windows next.

Some slight revisions here. I changed things to generate Makefiles directly, instead of Makefile.ins in the srcdir. I'll attach a patch in bug 688187 showing how I'm doing this for WebRTC.

I landed an updated version of this on alder: http://hg.mozilla.org/projects/alder/rev/1d1d10574a84 I'll get post-facto review from someone once I get things working properly on Windows. (It's close there, but not quite yet.)

Ted landed a version that works on windows as well. Yeah ted! However, it has a few issues with the Win32 build servers 1) the use of the 'with' statement, which is not in Python 2.5 by default (see attachment , which has been landed on alder to fix this) 2) with 1) fixed, you find that the json module for python isn't on the WIn32 build servers: Updating projects from gyp files... *** Fix above errors and then restart with "make -f client.mk build" make[2]: Leaving directory `/e/builds/moz2_slave/a-w32/build' make[1]: Leaving directory `/e/builds/moz2_slave/a-w32/build' Traceback (most recent call last): File "e:/builds/moz2_slave/a-w32/build/media/webrtc/trunk/build/gyp_chromium", line 171, in <module> sys.exit(gyp.main(args)) File "e:\builds\moz2_slave\a-w32\build\media\webrtc\trunk\tools\gyp\pylib\gyp\__init__.py", line 471, in main options.circular_check) File "e:\builds\moz2_slave\a-w32\build\media\webrtc\trunk\tools\gyp\pylib\gyp\__init__.py", line 72, in Load generator = __import__(generator_name, globals(), locals(), generator_name) File "e:\builds\moz2_slave\a-w32\build\media\webrtc\trunk\tools\gyp\pylib\gyp\generator\mozmake.py", line 7, in <module> import json ImportError: No module named json configure: error: failed to generate WebRTC Makefiles

For json, we can either update python to 2.6 or later on Linux and Windows build servers, or we can remove the dependency on the json module from mozmake.py, or we could install simplejson on the build servers and alternatively import that: try: import json except ImportError: import simplejson as json

(In reply to Randell Jesup [:jesup] from comment #16) > 1) the use of the 'with' statement, which is not in Python 2.5 by default > (see attachment , which has been landed on alder to fix this) You should be able to use 'from __future__ import with_statement' to fix this. > 2) with 1) fixed, you find that the json module for python isn't on the > WIn32 build servers: JP just pinged me about this. This is something we can fix, he's going to file a bug and I'll have a look on Monday.

(In reply to Ben Hearsum [:bhearsum] from comment #18) > (In reply to Randell Jesup [:jesup] from comment #16) > > 1) the use of the 'with' statement, which is not in Python 2.5 by default > > (see attachment , which has been landed on alder to fix this) > > You should be able to use 'from __future__ import with_statement' to fix > this. That's exactly what the patch I've applied to alder does. :-) > > 2) with 1) fixed, you find that the json module for python isn't on the > > WIn32 build servers: > > JP just pinged me about this. This is something we can fix, he's going to > file a bug and I'll have a look on Monday The Linux build servers are also Python 2.5 (without json). As indicated in comment 17, I'm good with either updating python (yeah!) or installing simplejson (if it isn't there; I haven't checked yet) and using the snippet above. Mac seems ok.

just to note, turns out json was included but not needed; so i removed the json include from alder

I was looking at alder, and it appears when I touch certain headers, the correct stuff isn't rebuild (in this case sink_filter_windows.h). When I touch the source file (sink_filter_windows.cc) stuff rebuilds correctly.

Note known bug - it doesn't rebuild Makefiles if a gyp file changes

I landed a couple of follow-ups here: http://hg.mozilla.org/projects/alder/rev/3c906d1866b7 - Fix GNU make incompatible paths on Windows http://hg.mozilla.org/projects/alder/rev/3bfb70703e95 - Add rules to regenerate Makefiles when GYP files change I'm going to call this project FIXED now. These changes should get review before we land alder into mozilla-central, but this isn't blocking any WebRTC work anymore.