https://tbpl.mozilla.org/php/getParsedLog.php?id=33324075&tree=B2g-Inbound b2g_b2g-inbound_leo_eng_dep on 2014-01-21 02:43:08 PST for push cb77095b283f slave: bld-linux64-ec2-504 03:17:12 INFO - checking build system type... i686-pc-linux-gnu 03:17:12 INFO - checking host system type... arm-linux-android 03:17:12 INFO - checking target system type... arm-linux-android 03:17:12 INFO - checking for a BSD-compatible install... /usr/bin/install -c 03:17:13 INFO - checking whether build environment is sane... configure: error: newly created file is older than distributed files! 03:17:13 INFO - Check your system clock 03:17:13 INFO - configure: error: /builds/slave/b2g_b2g-in_leo_eng_dep-0000000/build/gecko/js/src/../../js/src/ctypes/libffi/configure failed for js/src/ctypes/libffi 03:17:13 INFO - configure: error: /builds/slave/b2g_b2g-in_leo_eng_dep-0000000/build/gecko/js/src/configure failed for js/src 03:17:13 INFO - *** Fix above errors and then restart with\ 03:17:13 INFO - "make -f client.mk build" 03:17:13 ERROR - make[3]: *** [configure] Error 1 03:17:13 ERROR - make[2]: *** [/builds/slave/b2g_b2g-in_leo_eng_dep-0000000/build/objdir-gecko/Makefile] Error 2 03:17:13 ERROR - make[1]: *** [build] Error 2 03:17:13 INFO - make: *** [out/target/product/leo/obj/DATA/gecko_intermediates/gecko] Error 2 03:17:13 INFO - real 19m46.069s 03:17:13 INFO - user 15m34.699s 03:17:13 INFO - sys 1m30.806s 03:17:13 INFO - 03:17:13 INFO - > Build failed! < 03:17:13 INFO - Build with |./build.sh -j1| for better messages 03:17:13 INFO - If all else fails, use |rm -rf objdir-gecko| to clobber gecko and |rm -rf out| to clobber everything else. 03:17:13 ERROR - Return code: 2 03:17:13 INFO - Running command: ['mock_mozilla', '-r', u'mozilla-centos6-i386', '-q', '--cwd', '/builds/slave/b2g_b2g-in_leo_eng_dep-0000000/build', '--unpriv', '--shell', u'/usr/bin/env VARIANT=eng GAIA_OPTIMIZE=1 "LESSOPEN=|/usr/bin/lesspipe.sh %s" TMOUT=86400 CVS_RSH=ssh LOGNAME=cltbld USER=cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin:/builds/slave/b2g_b2g-in_leo_eng_dep-0000000/build/compare-locales/scripts SYMBOL_SERVER_USER=b2gbld "WGET_OPTS=-c -q" CCACHE_UMASK=002 LANG=en_US.UTF-8 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash MOZ_BUILD_DATE=20140121022701 SHLVL=1 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_b2g/ LOCALES_FILE=/builds/slave/b2g_b2g-in_leo_eng_dep-0000000/build/gaia/locales/languages_dev.json SYMBOL_SERVER_HOST=symbolpush.mozilla.org PYTHONPATH=:/builds/slave/b2g_b2g-in_leo_eng_dep-0000000/build/compare-locales/lib L10NBASEDIR=/builds/slave/b2g_b2g-in_leo_eng_dep-0000000/build/gecko-l10n MOZILLA_OFFICIAL=1 "MOZ_CHROME_MULTILOCALE=es-ES pl pt-BR" CCACHE_DIR=/builds/ccache CCACHE_COMPRESS=1 B2G_SYSTEM_APPS=1 _=/tools/buildbot/bin/python CCACHE_BASEDIR=/builds/slave/b2g_b2g-in_leo_eng_dep-0000000/build HOSTNAME=bld-linux64-ec2-504.build.releng.usw2.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/b2gbld_dsa LOCALE_BASEDIR=/builds/slave/b2g_b2g-in_leo_eng_dep-0000000/build/gaia-l10n HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/b2g_b2g-in_leo_eng_dep-0000000 PROPERTIES_FILE=/builds/slave/b2g_b2g-in_leo_eng_dep-0000000/buildprops.json MAIL=/var/spool/mail/cltbld B2G_UPDATER=1 B2G_UPDATE_CHANNEL=leo/default/default ccache -s'] in /builds/slave/b2g_b2g-in_leo_eng_dep-0000000/build 03:17:13 INFO - Copy/paste: mock_mozilla -r mozilla-centos6-i386 -q --cwd /builds/slave/b2g_b2g-in_leo_eng_dep-0000000/build --unpriv --shell "/usr/bin/env VARIANT=eng GAIA_OPTIMIZE=1 \"LESSOPEN=|/usr/bin/lesspipe.sh %s\" TMOUT=86400 CVS_RSH=ssh LOGNAME=cltbld USER=cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin:/builds/slave/b2g_b2g-in_leo_eng_dep-0000000/build/compare-locales/scripts SYMBOL_SERVER_USER=b2gbld \"WGET_OPTS=-c -q\" CCACHE_UMASK=002 LANG=en_US.UTF-8 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash MOZ_BUILD_DATE=20140121022701 SHLVL=1 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_b2g/ LOCALES_FILE=/builds/slave/b2g_b2g-in_leo_eng_dep-0000000/build/gaia/locales/languages_dev.json SYMBOL_SERVER_HOST=symbolpush.mozilla.org PYTHONPATH=:/builds/slave/b2g_b2g-in_leo_eng_dep-0000000/build/compare-locales/lib L10NBASEDIR=/builds/slave/b2g_b2g-in_leo_eng_dep-0000000/build/gecko-l10n MOZILLA_OFFICIAL=1 \"MOZ_CHROME_MULTILOCALE=es-ES pl pt-BR\" CCACHE_DIR=/builds/ccache CCACHE_COMPRESS=1 B2G_SYSTEM_APPS=1 _=/tools/buildbot/bin/python CCACHE_BASEDIR=/builds/slave/b2g_b2g-in_leo_eng_dep-0000000/build HOSTNAME=bld-linux64-ec2-504.build.releng.usw2.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/b2gbld_dsa LOCALE_BASEDIR=/builds/slave/b2g_b2g-in_leo_eng_dep-0000000/build/gaia-l10n HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/b2g_b2g-in_leo_eng_dep-0000000 PROPERTIES_FILE=/builds/slave/b2g_b2g-in_leo_eng_dep-0000000/buildprops.json MAIL=/var/spool/mail/cltbld B2G_UPDATER=1 B2G_UPDATE_CHANNEL=leo/default/default ccache -s" 03:17:14 INFO - cache directory /builds/ccache 03:17:14 INFO - cache hit (direct) 0 03:17:14 INFO - cache hit (preprocessed) 0 03:17:14 INFO - cache miss 5047 03:17:14 INFO - called for link 204 03:17:14 INFO - preprocessor error 1 03:17:14 INFO - unsupported source language 334 03:17:14 INFO - autoconf compile/link 12 03:17:14 INFO - unsupported compiler option 9 03:17:14 INFO - no input file 2 03:17:14 INFO - files in cache 79887 03:17:14 INFO - cache size 2.3 Gbytes 03:17:14 INFO - max cache size 10.0 Gbytes 03:17:14 INFO - Return code: 0 03:17:14 FATAL - failed to build 03:17:14 FATAL - Running post_fatal callback... 03:17:14 FATAL - Exiting 2

If you look higher up, you can see time going backwards: 10:46:51 INFO - Unpacking objects: 50% (1/2) 10:46:51 INFO - Unpacking objects: 100% (2/2) 10:46:51 INFO - Unpacking objects: 100% (2/2), done. 02:46:52 INFO - From https://git.mozilla.org/external/caf/platform/external/webp 02:46:52 INFO - * [new tag] AU_LINUX_ANDROID_JB_3.2.1_RB1.04.03.00.101.064 -> AU_LINUX_ANDROID_JB_3.2.1_RB1.04.03.00.101.064 02:46:52 INFO - * [new tag] LNX.LA.3.2.1-06410-8x10.0 -> LNX.LA.3.2.1-06410-8x10.0 02:46:52 INFO - 02:46:52 INFO - Fetching projects: 36% (33/90) Fetching project platform_build

Jan 21 10:42:54 bld-linux64-ec2-504 collectd[1760]: Initialization complete, entering read-loop. Jan 21 10:42:54 bld-linux64-ec2-504 puppet-agent[1183]: (/Stage[main]/Collectd/Service[collectd]/ensure) ensure changed 'stopped' to 'running' Jan 21 10:42:54 bld-linux64-ec2-504 puppet-agent[1183]: (/Stage[main]/Collectd/Service[collectd]) Unscheduling refresh on Service[collectd] Jan 21 10:42:54 bld-linux64-ec2-504 puppet-agent[1183]: Finished catalog run in 11.53 seconds Jan 21 02:46:51 bld-linux64-ec2-504 ntpd[1117]: synchronized to 10.12.48.10, stratum 3 Jan 21 02:46:51 bld-linux64-ec2-504 ntpd[1117]: time reset -28799.666698 s Jan 21 02:46:51 bld-linux64-ec2-504 ntpd[1117]: kernel time sync status change 2001 Jan 21 02:56:59 bld-linux64-ec2-504 ntpd[1117]: synchronized to 10.12.48.10, stratum 3

To make sure that ntpd works properly we need to run ntpdate on start not only on test slaves.

Comment on attachment 8363006 [details] [diff] [review] always use ntpdate on boot remote: https://hg.mozilla.org/build/puppet/rev/4405ed2ac404 remote: https://hg.mozilla.org/build/puppet/rev/b20d4ef94118

Comment on attachment 8363006 [details] [diff] [review] always use ntpdate on boot remote: https://hg.mozilla.org/build/puppet/rev/74ad96a85992 remote: https://hg.mozilla.org/build/puppet/rev/0efd49e9d34a Tue Jan 21 08:07:37 -0800 2014 Puppet (err): Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate declaration: Service[ntpdate] is already declared in file /etc/puppet/production/modules/ntp/manifests/atboot.pp at line 12; cannot redeclare on node bld-lion-r5-079.build.releng.scl3.mozilla.com

I tested it on bld-lion (no changes generated), bld-linux64 (see below) and talos-linux64 (no changes) slaves. bld-linux64 log snippet (--noop): Notice: /Stage[main]/Ntp::Atboot/Service[ntpdate]/enable: current_value false, should be true (noop) Notice: Class[Ntp::Atboot]: Would have triggered 'refresh' from 1 events

Comment on attachment 8363023 [details] [diff] [review] always use ntpdate on boot So this basically makes ntp::daemon = ntp::atboot + a daemon

Comment on attachment 8363023 [details] [diff] [review] always use ntpdate on boot remote: https://hg.mozilla.org/build/puppet/rev/1643c7045bf5 remote: https://hg.mozilla.org/build/puppet/rev/c6ce8cf3cf2b

I believe this is fixed now. I haven't seen any errors like this on the new (HVM) instances I added last night. https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=bld-linux64-ec2-20 https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=bld-linux64-ec2-50