e.g. https://tbpl.mozilla.org/php/getParsedLog.php?id=8267665&full=1&branch=mozilla-inbound#error0 # we don't have branding yet, but we need it. Call it explicitly make[8]: Entering directory `/builds/slave/m-in-andrd-xul/build/obj-firefox/mobile/xul/branding/nightly/locales' /tools/python/bin/python2.5 /builds/slave/m-in-andrd-xul/build/config/JarMaker.py \ -j ../../../../../dist/bin/chrome \ -t /builds/slave/m-in-andrd-xul/build -f flat -c /builds/slave/m-in-andrd-xul/build/mobile/branding/nightly/locales/en-US -DNDEBUG -DTRIMMED -DAB_CD=en-US -DOSTYPE=\"Linux\" -DOSARCH=Linux -DANDROID=1 -DANDROID_VERSION=5 -DCROSS_COMPILE=1 -DMOZ_THUMB2=1 -DHAVE_ARM_SIMD=1 -DHAVE_ARM_NEON=1 -DMOZILLA_VERSION=\"12.0a1\" -DMOZILLA_VERSION_U=12.0a1 -DNO_PW_GECOS=1 -DD_INO=d_ino -DSTDC_HEADERS=1 -DHAVE_SSIZE_T=1 -DHAVE_ST_BLKSIZE=1 -DHAVE_SIGINFO_T=1 -DHAVE_UINT=1 -DHAVE_UINT_T=1 -DHAVE_UNAME_DOMAINNAME_FIELD=1 -DHAVE_VISIBILITY_HIDDEN_ATTRIBUTE=1 -DHAVE_VISIBILITY_ATTRIBUTE=1 -DHAVE_DIRENT_H=1 -DHAVE_GETOPT_H=1 -DHAVE_MEMORY_H=1 -DHAVE_UNISTD_H=1 -DHAVE_MALLOC_H=1 -DHAVE_SYS_STATFS_H=1 -DHAVE_SYS_SYSMACROS_H=1 -DHAVE_LINUX_QUOTA_H=1 -DHAVE_MMINTRIN_H=1 -DHAVE_SYS_CDEFS_H=1 -DHAVE_DLOPEN=1 -DHAVE_MEMMEM=1 -DNO_X11=1 -DHAVE_STRERROR=1 -DHAVE_LCHOWN=1 -DHAVE_FCHMOD=1 -DHAVE_SNPRINTF=1 -DHAVE_MEMMOVE=1 -DHAVE_RINT=1 -DHAVE_SETBUF=1 -DHAVE_ISATTY=1 -DHAVE_FLOCKFILE=1 -DHAVE_LOCALTIME_R=1 -DHAVE_STRTOK_R=1 -DMALLOC_H=\<malloc.h\> -DHAVE_STRNDUP=1 -DHAVE_MEMALIGN=1 -DHAVE_VALLOC=1 -DHAVE_I18N_LC_MESSAGES=1 -DNS_ALWAYS_INLINE=__attribute__\(\(always_inline\)\) -DNS_ATTR_MALLOC=__attribute__\(\(malloc\)\) -DNS_WARN_UNUSED_RESULT=__attribute__\(\(warn_unused_result\)\) -DMOZ_BUILD_APP=mobile/xul -DMOZ_WIDGET_ANDROID=1 -DMOZ_PDF_PRINTING=1 -DMOZ_INSTRUMENT_EVENT_LOOP=1 -DUSE_ARM_KUSER=1 -DMOZ_DISTRIBUTION_ID=\"org.mozilla\" -DIBMBIDI=1 -DACCESSIBILITY=1 -DNS_PRINTING=1 -DNS_PRINT_PREVIEW=1 -DMOZ_RAW=1 -DMOZ_OGG=1 -DATTRIBUTE_ALIGNED_MAX=64 -DMOZ_WEBM=1 -DVPX_ARM_ASM=1 -DMOZ_WAVE=1 -DMOZ_SYDNEYAUDIO=1 -DMOZ_MEDIA=1 -DMOZ_TREMOR=1 -DMOZ_XTF=1 -DENABLE_SYSTEM_EXTENSION_DIRS=1 -DMOZ_CRASHREPORTER=1 -DMOZ_CRASHREPORTER_ENABLE_PERCENT=100 -DMOZ_USE_NATIVE_POPUP_WINDOWS=1 -DMOZ_TREE_FREETYPE=1 -DHAVE_FT_BITMAP_SIZE_Y_PPEM=1 -DHAVE_FT_GLYPHSLOT_EMBOLDEN=1 -DHAVE_FT_LOAD_SFNT_TABLE=1 -DMOZ_UPDATER=1 -DMOZ_UPDATE_CHANNEL=default -DMOZ_DISABLE_DOMCRYPTO=1 -DMOZ_FEEDS=1 -DMOZ_GFX_OPTIMIZE_MOBILE=1 -DMOZ_DEBUG_SYMBOLS=1 -DMOZ_LOGGING=1 -DSIZEOF_INT_P=4 -DMOZ_MEMORY_SIZEOF_PTR_2POW=2 -DMOZ_MEMORY=1 -DMOZ_MEMORY_LINUX=1 -DMOZ_MEMORY_ANDROID=1 -DHAVE_JEMALLOC_VALLOC=1 -DHAVE_JEMALLOC_POSIX_MEMALIGN=1 -DHAVE_JEMALLOC_MEMALIGN=1 -DJSGC_INCREMENTAL=1 -DJS_CRASH_DIAGNOSTICS=1 -DHAVE__UNWIND_BACKTRACE=1 -DJS_DEFAULT_JITREPORT_GRANULARITY=3 -DMOZ_OMNIJAR=1 -DMOZ_USER_DIR=\".mozilla\" -DMOZ_STATIC_JS=1 -DHAVE_STDINT_H=1 -DHAVE_INTTYPES_H=1 -DMOZ_TREE_CAIRO=1 -DHAVE_UINT64_T=1 -DMOZ_TREE_PIXMAN=1 -DMOZ_GRAPHITE=1 -DMOZ_ENABLE_SKIA=1 -DMOZ_XUL=1 -DMOZ_PROFILELOCKING=1 -DBUILD_CTYPES=1 -DMOZ_PLACES=1 -DMOZ_SERVICES_SYNC=1 -DMOZ_APP_COMPONENT_INCLUDE=\"nsBrowserComponents.h\" -DMOZ_APP_UA_NAME=\"\" -DMOZ_APP_UA_VERSION=\"12.0a1\" -DMOZ_UA_FIREFOX_VERSION=\"12.0a1\" -DFIREFOX_VERSION=12.0a1 -DMOZ_UA_BUILDID=\"\" -DMOZ_DLL_SUFFIX=\".so\" -DXP_UNIX=1 -DUNIX_ASYNC_DNS=1 \ /builds/slave/m-in-andrd-xul/build/mobile/xul/branding/nightly/locales/jar.mn /tools/android-sdk-r13/platforms/android-13/../../platform-tools/aapt package -f -M AndroidManifest.xml -I /tools/android-sdk-r13/platforms/android-13/android.jar -S res -J . --custom-package org.mozilla.gecko /tools/android-sdk-r13/platforms/android-13/../../platform-tools/aapt: /usr/lib/libz.so.1: no version information available (required by /tools/android-sdk-r13/platforms/android-13/../../platform-tools/aapt) processing /builds/slave/m-in-andrd-xul/build/mobile/xul/branding/nightly/locales/jar.mn make[8]: Leaving directory `/builds/slave/m-in-andrd-xul/build/obj-firefox/mobile/xul/branding/nightly/locales' /tools/python/bin/python2.5 /builds/slave/m-in-andrd-xul/build/config/Preprocessor.py -DAB_CD=en-US -DOSTYPE=\"Linux\" -DOSARCH=Linux \ -DBRANDPATH="/builds/slave/m-in-andrd-xul/build/obj-firefox/embedding/android/locales/../../../dist/bin/chrome/en-US/locale/branding/brand.dtd" \ -DSTRINGSPATH="/builds/slave/m-in-andrd-xul/build/embedding/android/locales/en-US/android_strings.dtd" \ /builds/slave/m-in-andrd-xul/build/embedding/android/locales/../strings.xml.in \ > ../res/values/strings.xml res/values/strings.xml:1: error: Error parsing XML: no element found res/layout/crash_reporter.xml:7: error: Error: No resource found that matches the given name (at 'text' with value '@string/crash_message'). res/layout/crash_reporter.xml:13: error: Error: No resource found that matches the given name (at 'text' with value '@string/crash_help_message'). res/layout/crash_reporter.xml:18: error: Error: No resource found that matches the given name (at 'text' with value '@string/crash_send_report_message'). res/layout/crash_reporter.xml:23: error: Error: No resource found that matches the given name (at 'text' with value '@string/crash_include_url'). res/layout/crash_reporter.xml:33: error: Error: No resource found that matches the given name (at 'text' with value '@string/crash_close_label'). res/layout/crash_reporter.xml:40: error: Error: No resource found that matches the given name (at 'text' with value '@string/crash_restart_label'). make[6]: *** [R.java] Error 1 make[6]: *** Waiting for unfinished jobs.... make[7]: Leaving directory `/builds/slave/m-in-andrd-xul/build/obj-firefox/embedding/android/locales' make[6]: Leaving directory `/builds/slave/m-in-andrd-xul/build/obj-firefox/embedding/android' make[5]: *** [libs] Error 2 make[5]: Leaving directory `/builds/slave/m-in-andrd-xul/build/obj-firefox/embedding' make[4]: *** [libs_tier_platform] Error 2 make[4]: Leaving directory `/builds/slave/m-in-andrd-xul/build/obj-firefox' make[3]: *** [tier_platform] Error 2 make[3]: Leaving directory `/builds/slave/m-in-andrd-xul/build/obj-firefox' make[2]: *** [default] Error 2 make[2]: Leaving directory `/builds/slave/m-in-andrd-xul/build/obj-firefox' make[1]: Leaving directory `/builds/slave/m-in-andrd-xul/build' make[1]: *** [realbuild] Error 2 make: *** [build] Error 2 program finished with exit code 2 Because it's Android, we just retrigger, sometimes pretending that a clobber will do something, but it would be nicer if we didn't have a build which constantly fails. https://tbpl.mozilla.org/php/getParsedLog.php?id=8265160&tree=Firefox https://tbpl.mozilla.org/php/getParsedLog.php?id=8265779&tree=Firefox https://tbpl.mozilla.org/php/getParsedLog.php?id=8265860&tree=Firefox https://tbpl.mozilla.org/php/getParsedLog.php?id=8264177&tree=Firefox https://tbpl.mozilla.org/php/getParsedLog.php?id=8234905&tree=Firefox https://tbpl.mozilla.org/php/getParsedLog.php?id=8267755&tree=Mozilla-Inbound

This is kinda the same problem as bug 708437. Which we don't understand, really. It calls the preprocessor to create strings.xml, and it seems like we're so parallel that it's reading the created file before it was finished writing. At least that seems to be the only thing left in my mind. Kyle and I stare at Ted in bug 708437, too. Next thing we could try is to create the preprocessor output in a temporary file, and then mv it to the location of the target.

this is a similar patch to what I did in bug 712380. I haven't seen bug 708437 since that landed, so hopefully this will do similar black magic voodoo here.

Phil, mind filing a blocked bug to start trees? This bug has a patch and possibly useful comments, star comments are just making the actual bug harder to fix.

do we know when this started happening?

https://tbpl.mozilla.org/php/getParsedLog.php?id=8276641&tree=Firefox Yes, I do mind moving to another bug, this is the bug I filed. My assumption was it started with http://hg.mozilla.org/mozilla-central/rev/b230ea62de17#l1.54 from bug 708015 - it landed last weekend, smelling like the sort of thing that would require a clobber, and it's entirely possible that a clobber slows things down enough that we don't race, and actually works. That would be plenty to let us ignore it for a week that didn't have many pushes.

https://tbpl.mozilla.org/php/getParsedLog.php?id=8278665&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=8278635&tree=Mozilla-Inbound Set inbound to clobber yet again (last one was on the 28th, probably from this).

https://hg.mozilla.org/integration/mozilla-inbound/rev/ea1181d6a816 Doesn't seem to have made any difference, first time failure on the push and the next two: https://tbpl.mozilla.org/php/getParsedLog.php?id=8290286&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=8290524&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=8292019&tree=Mozilla-Inbound I set yet another clobber, to get some breathing room, so we'll probably see some green now (and who knows, the build system's mostly made of magic, maybe it'll just start working perfectly after that).

https://tbpl.mozilla.org/php/getParsedLog.php?id=8300275&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=8300529&tree=Mozilla-Inbound Got a pretty nice bit of peace out of that morning clobber, but I'm starting to think about -j1 until someone gets a handle on it, assuming -j1 will stop us racing.

https://tbpl.mozilla.org/php/getParsedLog.php?id=8313735&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=8314307&tree=Mozilla-Inbound

https://tbpl.mozilla.org/php/getParsedLog.php?id=8308419&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=8307467&tree=Mozilla-Inbound

https://tbpl.mozilla.org/php/getParsedLog.php?id=8320197&tree=Mozilla-Aurora - I wonder how long it would take Aurora to start failing.

https://tbpl.mozilla.org/php/getParsedLog.php?id=8319001&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=8320258&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=8321856&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=8322967&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=8328943&tree=Mozilla-Inbound

After talking to ted I think this is the best approach. What happens is that embedding/android has a custom libs rule that it wants to run after all of the standard libs rules complete. There's no way to set up that dependency in the current build system though. Instead, we use the directory ordering to enforce this dependency. The canonical way to do this is with a build/ subdir, but I think it's better not to move everything we have sitting around. Blassey, can you test this patch and confirm that fennec still builds with it?

Comment on attachment 586086 [details] [diff] [review] Patch no dice flyingfox:objdir-droid-xul blassey$ make -C embedding/android/ /usr/bin/make export make[1]: Nothing to be done for `export'. /usr/bin/make libs make[1]: *** No rule to make target `res/values/strings.xml', needed by `R.java'. Stop. make: *** [default] Error 2

Er, you have to make -C objdir/embedding ...

I'm testing by doing: $> make -C ../objdir-droid-xul/embedding/android/ clean $> make -C ../objdir-droid-xul/embedding/android/ and I get: make[1]: *** No rule to make target `res/values/strings.xml', needed by `R.java'. Stop. which I now realize just plain won't work with this patch, but is a fairly common work flow when working on the java code

attachment 579864 [details] [diff] [review] in bug 708437 (which is the corresponding bug for mobile/android/base) allows to do that still, by creating a rule for strings.xml that explicitly calls into locales, and doesn't rely on DIRS to be done. Which still seems to be the underlying weirdness, do we know if that's on purpose?

Resuming posting the logs here, since whilst the comment 18 landing has reduced the frequency, this issue still occurs: https://tbpl.mozilla.org/php/getParsedLog.php?id=8408504&tree=Firefox

The underlying weirdness here is that this makefile wants to run custom libs targets after all the builtin targets are completed (since looping over the subdirectories is the last thing the builtin targets do). The mechanism for that synchronization already exists, it's the subdirectory system. If you don't want to build android/locales from embedding/ then you need to put the relevant things in an android/build directory or something similar.

Summary: This Makefile is broken. You can take khuey's patch, or you can rework the Makefile to put the relevant rules in another subdir. We're not going to take a hacky band-aid patch to work around the brokenness.

OK, thanks for the clarification. Also makes me think that attachment 579864 [details] [diff] [review] is the right way to fix the Makefiles, and should be ported to embedding/android, too. review nag :-)

No, that's the band-aid. The ordering of DIRS should take care of those things if Makefiles are written correctly.

If one could explicitly order the current dir, I'd buy that.

Our infrastructure makes silly assumptions like "we won't leave a platform failing to build for more than a week," so this is now doing fun things like causing us to not get nightly builds. I'm pretty sure I know what caused this, and the right thing to do would be to back that out. Shall I?

Backing out bug 708015 is not an option.

https://tbpl.mozilla.org/php/getParsedLog.php?id=8461660&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=8461381&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=8460862&tree=Mozilla-Inbound

Comment on attachment 587514 [details] [diff] [review] patch Review of attachment 587514 [details] [diff] [review]: ----------------------------------------------------------------- Ted gave the equivalent patch r- in Bug 708437.

(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #70) > Comment on attachment 587514 [details] [diff] [review] > patch > > Review of attachment 587514 [details] [diff] [review]: > ----------------------------------------------------------------- > > Ted gave the equivalent patch r- in Bug 708437. any reason

See comments 31-33 in this bug.

(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #72) > See comments 31-33 in this bug. read them. I guess I disagree with this being a band-aid now that I understand the cause.

https://tbpl.mozilla.org/php/getParsedLog.php?id=8471773&tree=Mozilla-Inbound (the 5th attempt to build for that push, but I accidentally triggered the 6th one at the same time, so maybe it'll manage to build)

This ensures proper dependency: chrome is the rule that goes through subdirectories and ends up creating res/values/strings.xml. This is what needs to be depended upon instead of res/values/strings.xml.

https://tbpl.mozilla.org/php/getParsedLog.php?id=8474662&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=8474516&tree=Mozilla-Inbound

https://tbpl.mozilla.org/php/getParsedLog.php?id=8479056&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=8476568&tree=Mozilla-Inbound

https://tbpl.mozilla.org/php/getParsedLog.php?id=8480821&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=8480980&tree=Mozilla-Inbound

https://tbpl.mozilla.org/php/getParsedLog.php?id=8487834&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=8487014&tree=Mozilla-Inbound

https://tbpl.mozilla.org/php/getParsedLog.php?id=8492576&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=8493051&tree=Mozilla-Inbound

https://tbpl.mozilla.org/php/getParsedLog.php?id=8496184&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=8496338&tree=Mozilla-Inbound

https://tbpl.mozilla.org/php/getParsedLog.php?id=8502465&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=8502857&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=8503502&tree=Mozilla-Inbound

https://tbpl.mozilla.org/php/getParsedLog.php?id=8515380&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=8516292&tree=Mozilla-Inbound

https://tbpl.mozilla.org/php/getParsedLog.php?id=8517301&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=8517779&tree=Mozilla-Inbound

https://tbpl.mozilla.org/php/getParsedLog.php?id=8578097&tree=Firefox https://tbpl.mozilla.org/php/getParsedLog.php?id=8579002&tree=Firefox

Landed attachment 587627 [details] [diff] [review] in https://hg.mozilla.org/integration/mozilla-inbound/rev/ec8ded10b50c because you'll like me doing that better than you'll like the alternatives.

Comment on attachment 587627 [details] [diff] [review] Avoid race condition for objdir/embedding/android/R.java dependencies Review of attachment 587627 [details] [diff] [review]: ----------------------------------------------------------------- Post-facto review. Whatever. This is still broken but this is the least-shitty solution I guess since nobody actually wants to rewrite the Makefiles to make them correct.

Given how things are now, i.e., no more tbpl noise on central, should we mark this bug fixed, and request approval for the hack to be backported to aurora?

That's pretty funny, "tbpl noise," when you very obviously never ever look at the tree itself.

(In reply to Phil Ringnalda (:philor) from comment #152) > That's pretty funny, "tbpl noise," when you very obviously never ever look > at the tree itself. what does that mean?