https://tbpl.mozilla.org/php/getParsedLog.php?id=17081223&tree=Mozilla-Inbound b2g_mozilla-inbound_panda_dep on 2012-11-15 14:58:39 PST for push dc61a415f3e9 slave: bld-linux64-ec2-044 15:01:22 INFO - HOST_BUILD_TYPE=release 15:01:22 INFO - BUILD_ID=OPENMASTER 15:01:22 INFO - OUT_DIR=out 15:01:22 INFO - ============================================ 15:02:28 WARNING - external/busybox/Android.mk:33: warning: overriding commands for target `out/target/product/panda/obj/EXECUTABLES/busybox_intermediates/busybox' 15:02:28 WARNING - build/core/prebuilt.mk:117: warning: ignoring old commands for target `out/target/product/panda/obj/EXECUTABLES/busybox_intermediates/busybox' 15:02:37 WARNING - external/screencap-gonk/Android.mk:35: warning: overriding commands for target `out/target/product/panda/system/bin/screencap-gonk' 15:02:37 WARNING - build/core/base_rules.mk:532: warning: ignoring old commands for target `out/target/product/panda/system/bin/screencap-gonk' 15:02:41 INFO - find: `../../external/nist-sip/java': No such file or directory 15:02:51 WARNING - gaia/Android.mk:32: warning: overriding commands for target `out/target/product/panda/data/local/gaia' 15:02:51 WARNING - build/core/base_rules.mk:532: warning: ignoring old commands for target `out/target/product/panda/data/local/gaia' 15:02:51 WARNING - gonk-misc/Android.mk:26: warning: overriding commands for target `out/target/product/panda/obj/ETC/init.rc_intermediates/init.rc' 15:02:51 WARNING - build/core/prebuilt.mk:117: warning: ignoring old commands for target `out/target/product/panda/obj/ETC/init.rc_intermediates/init.rc' 15:02:51 WARNING - gonk-misc/Android.mk:155: warning: overriding commands for target `out/target/product/panda/system/gecko' 15:02:51 WARNING - build/core/base_rules.mk:532: warning: ignoring old commands for target `out/target/product/panda/system/gecko' 15:02:51 WARNING - gonk-misc/Android.mk:214: warning: overriding commands for target `out/target/product/panda/obj/DATA/gecko_intermediates/gecko' 15:02:51 WARNING - build/core/prebuilt.mk:117: warning: ignoring old commands for target `out/target/product/panda/obj/DATA/gecko_intermediates/gecko' 15:02:58 INFO - No private recovery resources for TARGET_DEVICE panda 15:03:01 INFO - target Strip: libdl (out/target/product/panda/obj/lib/libdl.so) 15:03:01 INFO - target Strip: libz (out/target/product/panda/obj/lib/libz.so) 15:03:01 INFO - prebuilt/linux-x86/toolchain/arm-linux-androideabi-4.4.x/bin/arm-linux-androideabi-objcopy:out/target/product/panda/obj/lib/stjsaA4o: cannot create debug link section `out/target/product/panda/symbols/system/lib/libdl.so': Invalid operation 15:03:01 INFO - make: *** [out/target/product/panda/obj/lib/libdl.so] Error 1 15:03:01 INFO - make: *** Deleting file `out/target/product/panda/obj/lib/libdl.so' 15:03:01 INFO - make: *** Waiting for unfinished jobs.... 15:03:01 INFO - target Strip: libstlport (out/target/product/panda/obj/lib/libstlport.so) 15:03:01 INFO - prebuilt/linux-x86/toolchain/arm-linux-androideabi-4.4.x/bin/arm-linux-androideabi-objcopy:out/target/product/panda/obj/lib/stDNKEFx: cannot create debug link section `out/target/product/panda/symbols/system/lib/libz.so': Invalid operation 15:03:01 INFO - make: *** [out/target/product/panda/obj/lib/libz.so] Error 1 15:03:01 INFO - make: *** Deleting file `out/target/product/panda/obj/lib/libz.so' 15:03:01 INFO - prebuilt/linux-x86/toolchain/arm-linux-androideabi-4.4.x/bin/arm-linux-androideabi-objcopy:out/target/product/panda/obj/lib/stugRsDy: cannot create debug link section `out/target/product/panda/symbols/system/lib/libstlport.so': Invalid operation 15:03:01 INFO - make: *** [out/target/product/panda/obj/lib/libstlport.so] Error 1 15:03:01 INFO - make: *** Deleting file `out/target/product/panda/obj/lib/libstlport.so' 15:03:01 INFO - real 1m40.497s 15:03:01 INFO - user 0m10.095s 15:03:01 INFO - sys 0m4.577s 15:03:01 INFO - 15:03:01 INFO - > Build failed! < 15:03:01 INFO - Build with |./build.sh -j1| for better messages 15:03:01 INFO - If all else fails, use |rm -rf objdir-gecko| to clobber gecko and |rm -rf out| to clobber everything else. 15:03:01 ERROR - Return code: 2 15:03:01 INFO - Running command: ['mock_mozilla', '-r', u'mozilla-centos6-i386', '-q', '--cwd', '/builds/slave/b2g-m-in-panda-dep/build', '--unpriv', '--shell', '/usr/bin/env "LESSOPEN=|/usr/bin/lesspipe.sh %s" LOGNAME=cltbld USER=cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin SYMBOL_SERVER_USER=b2gbld CCACHE_UMASK=002 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash SHLVL=1 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_b2g/ SYMBOL_SERVER_HOST=symbols1.dmz.phx1.mozilla.com MAIL=/var/spool/mail/cltbld CCACHE_COMPRESS=1 _=/tools/buildbot/bin/python CCACHE_BASEDIR=/builds/slave/b2g-m-in-panda-dep/build HOSTNAME=bld-linux64-ec2-044.build.aws-us-west-1.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/b2gbld_dsa HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/b2g-m-in-panda-dep PROPERTIES_FILE=/builds/slave/b2g-m-in-panda-dep/buildprops.json CCACHE_DIR=/builds/ccache ccache -s'] in /builds/slave/b2g-m-in-panda-dep/build 15:03:01 INFO - Copy/paste: mock_mozilla -r mozilla-centos6-i386 -q --cwd /builds/slave/b2g-m-in-panda-dep/build --unpriv --shell "/usr/bin/env \"LESSOPEN=|/usr/bin/lesspipe.sh %s\" LOGNAME=cltbld USER=cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin SYMBOL_SERVER_USER=b2gbld CCACHE_UMASK=002 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash SHLVL=1 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_b2g/ SYMBOL_SERVER_HOST=symbols1.dmz.phx1.mozilla.com MAIL=/var/spool/mail/cltbld CCACHE_COMPRESS=1 _=/tools/buildbot/bin/python CCACHE_BASEDIR=/builds/slave/b2g-m-in-panda-dep/build HOSTNAME=bld-linux64-ec2-044.build.aws-us-west-1.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/b2gbld_dsa HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/b2g-m-in-panda-dep PROPERTIES_FILE=/builds/slave/b2g-m-in-panda-dep/buildprops.json CCACHE_DIR=/builds/ccache ccache -s" 15:03:02 INFO - cache directory /builds/ccache 15:03:02 INFO - cache hit (direct) 0 15:03:02 INFO - cache hit (preprocessed) 0 15:03:02 INFO - cache miss 0 15:03:02 INFO - files in cache 228951 15:03:02 INFO - cache size 9.3 Gbytes 15:03:02 INFO - max cache size 10.0 Gbytes 15:03:02 INFO - Return code: 0 15:03:02 FATAL - failed to build 15:03:02 FATAL - Exiting 2

https://tbpl.mozilla.org/php/getParsedLog.php?id=17081732&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=17084318&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=17085101&tree=Mozilla-Inbound

https://tbpl.mozilla.org/php/getParsedLog.php?id=17106912&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=17105786&tree=Mozilla-Inbound

https://tbpl.mozilla.org/php/getParsedLog.php?id=17123660&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=17123914&tree=Mozilla-Inbound

Not sure who is on the hook for B2G build stuff; Ted/Mike, can you help or else suggest who might be able to? :-)

(In reply to Ed Morley [:edmorley UTC+0] from comment #17) > Not sure who is on the hook for B2G build stuff; Ted/Mike, can you help or > else suggest who might be able to? :-) That would be mwu.

I've never seen this in a build. Maybe some sort of build env issue? tzimmermann, have you ever seen this error?

It's possible this is related to my symbol dumping changes. We're probably running strip on these binaries now. Would that break things?

You can test this theory by running "build.sh buildsymbols" locally.

mwu thinks the symbol stripping might be breaking things, so I filed a gonk PR to stop doing that: https://github.com/mozilla-b2g/gonk-misc/pull/56

Sorry, I've never seen this problem during builds. It looks like you first strip the libraries and later try to access the (now removed) debugging symbols.

We're using GNU gold 2.20 and related tools. Without deeper research, I'd say you trigger this bug: http://sourceware.org/bugzilla/show_bug.cgi?id=11072

https://tbpl.mozilla.org/php/getParsedLog.php?id=17181830&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=17181314&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=17182525&tree=Mozilla-Inbound

https://tbpl.mozilla.org/php/getParsedLog.php?id=17187469&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=17188106&tree=Mozilla-Inbound

https://tbpl.mozilla.org/php/getParsedLog.php?id=17194648&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=17194509&tree=Mozilla-Inbound

https://tbpl.mozilla.org/php/getParsedLog.php?id=17215063&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=17215607&tree=Mozilla-Inbound

https://tbpl.mozilla.org/php/getParsedLog.php?id=17219627&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=17219626&tree=Mozilla-Inbound

https://tbpl.mozilla.org/php/getParsedLog.php?id=17220030&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=17220012&tree=Mozilla-Inbound

https://tbpl.mozilla.org/php/getParsedLog.php?id=17222437&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=17222161&tree=Mozilla-Inbound

https://tbpl.mozilla.org/php/getParsedLog.php?id=17223161&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=17222965&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=17223142&tree=Mozilla-Inbound

https://tbpl.mozilla.org/php/getParsedLog.php?id=17223521&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=17223531&tree=Mozilla-Inbound

The pull request mentioned in comment 28 was closed 4 days ago. I lose track of how the emulator builds work - someone has to manually update it, right? https://tbpl.mozilla.org/php/getParsedLog.php?id=17280075&tree=Mozilla-Inbound

(In reply to Ed Morley [:edmorley, UTC+0, email: emorley at moco] from comment #84) > The pull request mentioned in comment 28 was closed 4 days ago. I lose track > of how the emulator builds work - someone has to manually update it, right? Oh bug 810994 comment 13.

https://tbpl.mozilla.org/php/getParsedLog.php?id=17287127&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=17287650&tree=Mozilla-Inbound

Ted, this is another fail even after updating gonk-misc to aafa9ca4d2e9e755fe9964018a9797eac4ecc7de: https://tbpl.mozilla.org/php/getParsedLog.php?id=17292460&tree=Mozilla-Inbound In the few builds after I landed the snapshot update we had green on both clobber and dep builds, but only had 6 builds in total until this fail.

Is this fixed by bug 810994 or not? The most recent comment there seems to indicate it is, but we still have failure logs here on inbound. Does inbound just need a clobber?

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #96) > Is this fixed by bug 810994 or not? The most recent comment there seems to > indicate it is, but we still have failure logs here on inbound. Does inbound > just need a clobber? I don't believe this is fixed. However, I can't re-clobber just to be sure; due to bug 815556.

I clobbered inbound for this at 2012-11-24 16:44:55, so would love to know if there were any fails after that which are not recorded here. I'll have a look in the buildbot db too.

There has been one failure on inbound since the clobber, which was an unrelated code bustage. There has been one failure on mozilla-beta since I landed+clobbered there, also unrelated code error. No failures on central since the 20th, and can't see any other branches with issues. I've just landed on Aurora to complete the set. ---> This bug is fixed.