When building Thunderbird on comm-central, we've been seeing this error for the last few days: /builds/slave/tb-c-cen-osx64/build/objdir-tb/i386/mozilla/_virtualenv/bin/python /builds/slave/tb-c-cen-osx64/build/mozilla/config/expandlibs_exec.py --depend .deps/XUL.pp --target XUL --uselist -- /usr/local/bin/ccache /builds/slave/tb-c-cen-osx64/build/clang/bin/clang++ -arch i386 -Qunused-arguments -Qunused-arguments -Wall -Wpointer-arith -Woverloaded-virtual -Werror=return-type -Wtype-limits -Wempty-body -Wno-invalid-offsetof -Wno-c++0x-extensions -Wno-extended-offsetof -Wno-unknown-warning-option -Wno-return-type-c-linkage -Wno-mismatched-tags -isysroot /Developer/SDKs/MacOSX10.6.sdk -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -std=gnu++0x -pthread -DNO_X11 -pipe -DNDEBUG -DTRIMMED -g -O3 -fno-omit-frame-pointer -fPIC -o XUL nsStaticXULComponents.o nsUnicharUtils.o nsBidiUtils.o nsSpecialCasingData.o nsUnicodeProperties.o nsRDFResource.o -framework Cocoa -lobjc -framework ExceptionHandling -Wl,-executable_path,/builds/slave/tb-c-cen-osx64/build/objdir-tb/i386/mozilla/dist/bin -Wl,-dead_strip ../../toolkit/components/osfile/libosfile_s.a ../../toolkit/xre/libxulapp_s.a ../../staticlib/components/libnecko.a ../../staticlib/components/libuconv.a ../../staticlib/components/libi18n.a ../../staticlib/components/libchardet.a ../../staticlib/components/libjar50.a ../../staticlib/components/libstartupcache.a ../../staticlib/components/libpref.a ../../staticlib/components/libhtmlpars.a ../../staticlib/components/libidentity.a ../../staticlib/components/libimglib2.a ../../staticlib/components/libmediasniffer.a ../../staticlib/components/libgkgfx.a ../../staticlib/components/libgklayout.a ../../staticlib/components/libdocshell.a ../../staticlib/components/libembedcomponents.a ../../staticlib/components/libwebbrwsr.a ../../staticlib/components/libnsappshell.a ../../staticlib/components/libtxmgr.a ../../staticlib/components/libcommandlines.a ../../staticlib/components/libtoolkitcomps.a ../../staticlib/components/libpipboot.a ../../staticlib/components/libpipnss.a ../../staticlib/components/libappcomps.a ../../staticlib/components/libjsreflect.a ../../staticlib/components/libcomposer.a ../../staticlib/components/libtelemetry.a ../../staticlib/components/libjsinspector.a ../../staticlib/components/libjsdebugger.a ../../staticlib/components/libstoragecomps.a ../../staticlib/components/librdf.a ../../staticlib/components/libwindowds.a ../../staticlib/components/libjsctypes.a ../../staticlib/components/libjsperf.a ../../staticlib/components/libgkplugin.a ../../staticlib/components/libosxproxy.a ../../staticlib/components/libjsd.a ../../staticlib/components/libautoconfig.a ../../staticlib/components/libauth.a ../../staticlib/components/libcookie.a ../../staticlib/components/libpermissions.a ../../staticlib/components/libuniversalchardet.a ../../staticlib/components/libplaces.a ../../staticlib/components/libtkautocomplete.a ../../staticlib/components/libsatchel.a ../../staticlib/components/libpippki.a ../../staticlib/components/libimgicon.a ../../staticlib/components/libprofiler.a ../../staticlib/components/libwidget_mac.a ../../staticlib/components/libaccessibility.a ../../staticlib/components/libspellchecker.a ../../staticlib/components/libzipwriter.a ../../staticlib/components/libservices-crypto.a ../../staticlib/components/libxpautocomplete.a ../../staticlib/components/libmailcomps.a ../../staticlib/components/libmail.a ../../staticlib/components/libmsgsmime.a ../../staticlib/components/libimport.a ../../staticlib/components/libmozldap.a ../../staticlib/components/libmork.a ../../staticlib/components/libpeerconnection.a ../../staticlib/libjsipc_s.a ../../staticlib/libdomipc_s.a ../../staticlib/libdomplugins_s.a ../../staticlib/libmozipc_s.a ../../staticlib/libmozipdlgen_s.a ../../staticlib/libipcshell_s.a ../../staticlib/libgfxipc_s.a ../../staticlib/libhal_s.a ../../staticlib/libdombindings_s.a ../../staticlib/libxpcom_core.a ../../staticlib/libucvutil_s.a ../../staticlib/libchromium_s.a ../../staticlib/libsnappy_s.a ../../staticlib/libthebes.a ../../staticlib/libgl.a ../../staticlib/libycbcr.a -L../../dist/bin -L../../dist/lib /builds/slave/tb-c-cen-osx64/build/objdir-tb/i386/mozilla/dist/lib/libjs_static.a -L../../dist/bin -L../../dist/lib -lcrmf -lsmime3 -lssl3 -lnss3 -lnssutil3 -L../../dist/bin -L../../dist/lib -lldap60 -lprldap60 -lldif60 ../../dist/lib/libmozsqlite3.a /builds/slave/tb-c-cen-osx64/build/objdir-tb/i386/mozilla/modules/zlib/src/libmozz.a ../../dist/lib/libgkmedias.a ../../media/mtransport/build/libmtransport.a ../../media/webrtc/signaling/signaling_ecc/libecc.a ../../media/webrtc/signaling/signaling_sipcc/libsipcc.a -L../../dist/bin -L../../dist/lib -L/builds/slave/tb-c-cen-osx64/build/objdir-tb/i386/mozilla/dist/lib -lnspr4 -lplc4 -lplds4 ../../dist/lib/libmozalloc.a -dynamiclib -install_name @executable_path/XUL -compatibility_version 1 -current_version 1 -single_module -L/builds/slave/tb-c-cen-osx64/build/objdir-tb/i386/mozilla/dist/lib -lmozglue -framework OpenGL -lcups -framework SystemConfiguration -framework QTKit -framework IOKit -F/System/Library/PrivateFrameworks -framework CoreUI -framework QuartzCore -framework Carbon -framework CoreAudio -framework AudioToolbox -framework AudioUnit -framework AddressBook -framework OpenGL -framework Carbon -framework CoreAudio -framework AudioToolbox -framework AudioUnit -framework IOKit -framework Foundation -framework AppKit -framework Security clang: error: unable to execute command: posix_spawn failed: Argument list too long clang: error: linker command failed due to signal (use -v to see invocation)

Bug 644081 seems to have been a previous issue in this area that has been fixed. We do add a few extra libs and parameters of our own, but it wouldn't surprise me if FF is going to head towards hitting this soon.

We do use a response file under the hood, so it would seem to indicate a clang bug with long response files. Under the hood, it may just be giving a full list to some subprocess. GCC used to have the same problem, but fixed it by passing the response file to subprocesses, iirc.

I confirmed the before & after bustage command lines were the same, so this does look like the long response files. Apart from removing lots of source files, is there any way we could get around this? It is blocking Thunderbird on Mac building at the moment. I'm moving this to core, as I think resolving this is going to be in the Core build system rather than the Thunderbird specific parts.

Possibly a DUP of Bug 753223 (Intermittent permission denied from apps like ccache and nsinstall and python and posix_spawn on bld-lion-r5-* in mid-build, sometimes with clang thinking it's a clang bug)

(In reply to Philip Chee from comment #4) > Possibly a DUP of Bug 753223 (Intermittent permission denied from apps like > ccache and nsinstall and python and posix_spawn on bld-lion-r5-* in > mid-build, sometimes with clang thinking it's a clang bug) I doubt it is.

(In reply to Mike Hommey [:glandium] from comment #2) > We do use a response file under the hood, so it would seem to indicate a > clang bug with long response files. Under the hood, it may just be giving a > full list to some subprocess. GCC used to have the same problem, but fixed > it by passing the response file to subprocesses, iirc. Close. The heuristic that got implemented in gcc is that if it gets an @file, it should created one when running ld, so ld is run with just "ld @/tmp/foobar" and /tmp/foobar has all the command line options and files. In fact, Nathan was the one that implemented it :-) I have reported http://llvm.org/pr15171 to get the same heuristic in clang.

I did a try build with the patch from bug 837665: https://tbpl.mozilla.org/php/getParsedLog.php?id=19523215&tree=Thunderbird-Try&full=1 From the log, I don't think there's much we can do to the list file to reduce it down, so we'd need the clang fix. However, from the Thunderbird perspective, I have an idea of what we can temporarily disable that might get us going again - and I'm trying that out on try as well. I think its clear that Firefox & others will probably hit this soon so we'd should still push on the clang fix anyway.

I don't much motion in this, but it is stopping my Gecko 21 Mac builds. I don't see failures in tbpl at the moment, so is there some workaround for this? Or is this only seen on Universal Mac builds?

Patch submitted upstream: http://lists.cs.uiuc.edu/pipermail/cfe-commits/Week-of-Mon-20130311/075977.html I don't know what procedures (if any) we have for patching the compilers we use for builds. Willing to learn, though!

(In reply to Nathan Froyd (:froydnj) from comment #9) > Patch submitted upstream: > > http://lists.cs.uiuc.edu/pipermail/cfe-commits/Week-of-Mon-20130311/075977. > html > > I don't know what procedures (if any) we have for patching the compilers we > use for builds. Willing to learn, though! Rail is your man!

The Graphics branch would like to see some solution, since the combination of something on their branch and something in the merge in https://tbpl.mozilla.org/?tree=Graphics&rev=ab1eea09544b lost them the ability to build opt OS X.

So, the situation is that we have working patches. I assumed that we'd want to wait for upstream review, but that doesn't seem to be coming quickly. Rail, what are the steps for getting a new clang with the appropriate patches deployed? I see we have build/unix/build-clang/; is running the build-clang.py script in that directory on my machine and handing over the result sufficient? Or do I need to run those bits on some internal machines?

You can take a look at bug 823906 (or any of bugs mentioned in "hg log browser/config/tooltool-manifests/macosx64/releng.manifest") as a reference. Basically the work flow was like this: 1) create a new clang.tar.bz2 tarballs to be uploaded. Someone from releng can upload them. 2) test the manifest changes on try (using uploaded binaries) 3) if you are satisfied with the results push to mozilla-inbound Rafael used to be responsible for compiling and uploading new clang tarballs. Since we don't have anyone on point for this, these steps need to be done by someone from releng.

Nathan, do you need any help here? I'd really like to get this fixed up before the gecko 24 cycle if possible, just so we can sort it before the next Thunderbird major releases. I believe there may have been another bug/info page around where Rafael had described the steps that were used to actually create the clang tarballs.

(In reply to Mark Banner (:standard8) from comment #14) > Nathan, do you need any help here? I'd really like to get this fixed up > before the gecko 24 cycle if possible, just so we can sort it before the > next Thunderbird major releases. I believe there may have been another > bug/info page around where Rafael had described the steps that were used to > actually create the clang tarballs. I don't need any help here; I've figured out how to create clang releases etc. on the releng infrastructure. But it looks like the clang patches won't help; the OS X linker doesn't understand @files. So modifying clang to pass @files to the linker won't help any. Something in the Thunderbird build system will need to be changed.

(In reply to Nathan Froyd (:froydnj) from comment #15) > (In reply to Mark Banner (:standard8) from comment #14) > > Nathan, do you need any help here? I'd really like to get this fixed up > > before the gecko 24 cycle if possible, just so we can sort it before the > > next Thunderbird major releases. I believe there may have been another > > bug/info page around where Rafael had described the steps that were used to > > actually create the clang tarballs. > > I don't need any help here; I've figured out how to create clang releases > etc. on the releng infrastructure. > > But it looks like the clang patches won't help; the OS X linker doesn't > understand @files. So modifying clang to pass @files to the linker won't > help any. Something in the Thunderbird build system will need to be changed. The linker man page says it supports -filelist: https://developer.apple.com/library/mac/#documentation/Darwin/Reference/ManPages/man1/ld.1.html which is roughly equivalent to @files, though the -filelist file can't contain options, only input files.

Here's a patch I'm going to push to try.

(In reply to Nathan Froyd (:froydnj) from comment #17) > Created attachment 738010 [details] [diff] [review] > teach expandlibs_exec.py about OS X's -filelist linker option > > Here's a patch I'm going to push to try. We're not invoking ld directly. Does clang give -filelist to ld directly?

Also note that clang happily likes to ignore arguments it doesn't know, instead of throwing errors.

Hm, good point. Trying this instead, using -Wl.

(In reply to Nathan Froyd (:froydnj) from comment #15) > (In reply to Mark Banner (:standard8) from comment #14) > Something in the Thunderbird build system will need to be changed. Just FYI, we can't. There's basically too many files in the system. We've taken out WebRTC for now which is a lot of files, and I've not done an assessment but I suspect Firefox is going to hit this limit as well before too long.

(In reply to Mark Banner (:standard8) from comment #21) > (In reply to Nathan Froyd (:froydnj) from comment #15) > > (In reply to Mark Banner (:standard8) from comment #14) > > Something in the Thunderbird build system will need to be changed. > > Just FYI, we can't. There's basically too many files in the system. We've > taken out WebRTC for now which is a lot of files, and I've not done an > assessment but I suspect Firefox is going to hit this limit as well before > too long. Let me re-phrase that, we can change the build system to be closer to Firefox, but I'd bet Firefox is going to hit this soon as well with extra stuff that it is importing (xref comment 11).

This patch works much better on OS X; I can build Thunderbird with --enable-webrtc locally. The ordering of the tests in expandlibs.m4 matters, as does the quoting. Try run on Linux, OS X, Windows, and Android: https://tbpl.mozilla.org/?tree=Try&rev=28ed16f600bd

Comment on attachment 738181 [details] [diff] [review] teach expandlibs_exec.py about OS X's -filelist linker option Review of attachment 738181 [details] [diff] [review]: ----------------------------------------------------------------- ::: build/autoconf/expandlibs.m4 @@ +23,5 @@ > + dnl -filelist is for the OS X linker. We need to try -filelist > + dnl first because clang understands @file, but may pass an > + dnl oversized argument list to the linker depending on the > + dnl contents of @file. > + if AC_TRY_COMMAND(${CC-cc} -o conftest${ac_exeext} $LDFLAGS [-Wl,-filelist] [-Wl,conftest.list] $LIBS 1>&5) && test -s conftest${ac_exeext}; then -Wl,-filelist,conftest.list @@ +30,2 @@ > EXPAND_LIBS_LIST_STYLE=list > + else Trailing whitespace ::: config/expandlibs_exec.py @@ +99,5 @@ > content = ['INPUT("%s")\n' % obj for obj in objs] > + replacement = [tmp] > + elif conf.EXPAND_LIBS_LIST_STYLE == "filelist": > + content = ["%s\n" % obj for obj in objs] > + replacement = ["-Wl,-filelist", "-Wl," + tmp] If you use "-Wl,-filelist,"+tmp, you don't need to use lists here.

Applied with requested changes in: https://hg.mozilla.org/integration/mozilla-inbound/rev/4909e586d460