it should probably be enough to copy the list of i686 packages we install for desktop builds into b2g-config.py and change mock_target.

This converts all required-by-build package dependencies to .i686 packages, and adds the list of extra packages needed-because-yum-sucks, picked from the "linux" platform in mozilla/config.py. This should work but I can't guarantee that 100%. Please run on staging?

Comment on attachment 8513303 [details] [diff] [review] Use mozilla-centos6-x86_64 chroots for linux32 b2g builds This fails mock install with: Error: Protected multilib versions: freetype-2.3.11-6.el6_1.8.i686 != freetype-2.4.12-6.el6.1.x86_64 The repos used are http://mockbuild-repos.pub.build.mozilla.org/releng/public/CentOS/6/x86_64 http://mockbuild-repos.pub.build.mozilla.org/mirrors/centos/6/latest/updates/x86_64 http://mockbuild-repos.pub.build.mozilla.org/mirrors/centos/6/latest/os/x86_64 which have no 2.3.11 patch level 2.9 for x86_64, but we do have a 2.4.12 in the releng repo instead. The firefox desktop builds on x64_64 seem to use 2.4.12 even though they specify 2.3.11 p2.9. Downgrading the freetype requirement to freetype-2.3.11-6.el6_1.8 gets through mock install, but you also need to add the x64_64 package too. So r- for you to figure out the best solution. Builds are running (or will do soon) at http://dev-master1.build.mozilla.org:8710/builders/b2g_mozilla-central_linux32_gecko%20nightly/builds/7 http://dev-master1.build.mozilla.org:8710/builders/b2g_mozilla-central_linux32_gecko-debug%20build/builds/0 http://dev-master1.build.mozilla.org:8710/builders/b2g_mozilla-central_linux32_gecko_localizer%20nightly/builds/1

Let's go with 1.8 with x64_64 added.

Comment on attachment 8514112 [details] [diff] [review] Use mozilla-centos6-x86_64 chroots for linux32 b2g builds Downloading B2G SDK... wget -q -c "http://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/2014/08/2014-08-12-04-02-01-mozilla-central/b2g-34.0a1.multi.linux-x86_64.tar.bz2" tar xjf "b2g-34.0a1.multi.linux-x86_64.tar.bz2" -C "/builds/slave/m-cen-linux32_g-ntly-000000000/build/gaia/b2g_sdk/34.0a1-2014-08-12-04-02-01" test -f /builds/slave/m-cen-linux32_g-ntly-000000000/build/gaia/b2g_sdk/34.0a1-2014-08-12-04-02-01/b2g/xpcshell mkdir -p /builds/slave/m-cen-linux32_g-ntly-000000000/build/gaia/build_stage /builds/slave/m-cen-linux32_g-ntly-000000000/build/gaia/b2g_sdk/34.0a1-2014-08-12-04-02-01/b2g/xpcshell: error while loading shared libraries: libXt.so.6: cannot open shared object file: No such file or directory make[6]: *** [app] Error 127

ah, b2g builds use xpcshell...

I took the mock image 58d76c6acca148a1aedcbec7fc1b8212e12807b4, got the b2g tarball with xpcshell, and installed packages until xpcshell would start... which turned out to be libXt.x86_64 only.

Comment on attachment 8514729 [details] [diff] [review] Use mozilla-centos6-x86_64 chroots for linux32 b2g builds Seems fine to me, holding off on review until I've chucked at staging overnight.

Comment on attachment 8514729 [details] [diff] [review] Use mozilla-centos6-x86_64 chroots for linux32 b2g builds Green in staging for these builders: b2g_mozilla-central_linux32_gecko nightly b2g_mozilla-central_linux32_gecko-debug build b2g_mozilla-central_linux32_gecko_localizer nightly Will we need a clobber when this takes gets merged to production ?

This shouldn't need a clobber.

From: https://treeherder.mozilla.org/ui/logviewer.html#?job_id=3572615&repo=mozilla-inbound ======================================================= Fetching... retry: Calling <function run_with_timeout at 0x7f80dead7578> with args: (['/tools/tooltool.py', '--url', 'http://runtime-binaries.pvt.build.mozilla.org/tooltool', '--overwrite', '-m', 'b2g/config/tooltool-manifests/linux32/releng.manifest', 'fetch', '-c', '/builds/tooltool_cache'], 300, None, None, False, True), kwargs: {}, attempt #1 Executing: ['/tools/tooltool.py', '--url', 'http://runtime-binaries.pvt.build.mozilla.org/tooltool', '--overwrite', '-m', 'b2g/config/tooltool-manifests/linux32/releng.manifest', 'fetch', '-c', '/builds/tooltool_cache'] INFO - rm tree: moztt INFO - untarring "moztt.tar.bz2" Bootstraping... + rm -rf sccache + tar -xjf sccache.tar.bz2 program finished with exit code 0 elapsedTime=3.343323 ========= Finished 'sh /builds/slave/m-in-linux32_g-000000000000000/tools/scripts/tooltool/tooltool_wrapper.sh ...' (results: 0, elapsed: 3 secs) (at 2014-11-04 05:08:31.348376) ========= ========= Started compile failed (results: 2, elapsed: 4 secs) (at 2014-11-04 05:08:31.348765) ========= mock_mozilla -r mozilla-centos6-x86_64 --cwd /builds/slave/m-in-linux32_g-000000000000000/build --unpriv --shell '/usr/bin/env HG_SHARE_BASE_DIR="/builds/hg-shared" LOCALES_FILE="locales/languages_dev.json" TOOLTOOL_HOME="/builds" SYMBOL_SERVER_HOST="symbolpush.mozilla.org" CCACHE_DIR="/builds/ccache" POST_SYMBOL_UPLOAD_CMD="/usr/local/bin/post-symbol-upload.py" MOZ_AUTOMATION="1" MOZ_OBJDIR="obj-firefox" SYMBOL_SERVER_SSH_KEY="/home/cltbld/.ssh/ffxbld_rsa" LOCALE_BASEDIR="/builds/slave/m-in-linux32_g-000000000000000/build-gaia-l10n" TINDERBOX_OUTPUT="1" CCACHE_COMPRESS="1" TOOLTOOL_CACHE="/builds/tooltool_cache" SYMBOL_SERVER_PATH="/mnt/netapp/breakpad/symbols_ffx/" PATH="/tools/python27-mercurial/bin:/tools/python27/bin:${PATH}:/tools/buildbot/bin" MOZ_CRASHREPORTER_NO_REPORT="1" SYMBOL_SERVER_USER="ffxbld" WGET_OPTS="-q -c" LC_ALL="C" CCACHE_UMASK="002" make -f client.mk build '"'"'MOZ_BUILD_DATE=20141104050144'"'"'' in dir /builds/slave/m-in-linux32_g-000000000000000/build (timeout 10800 secs) (maxTime 19800 secs) watching logfiles {} argv: ['mock_mozilla', '-r', 'mozilla-centos6-x86_64', '--cwd', '/builds/slave/m-in-linux32_g-000000000000000/build', '--unpriv', '--shell', u'/usr/bin/env HG_SHARE_BASE_DIR="/builds/hg-shared" LOCALES_FILE="locales/languages_dev.json" TOOLTOOL_HOME="/builds" SYMBOL_SERVER_HOST="symbolpush.mozilla.org" CCACHE_DIR="/builds/ccache" POST_SYMBOL_UPLOAD_CMD="/usr/local/bin/post-symbol-upload.py" MOZ_AUTOMATION="1" MOZ_OBJDIR="obj-firefox" SYMBOL_SERVER_SSH_KEY="/home/cltbld/.ssh/ffxbld_rsa" LOCALE_BASEDIR="/builds/slave/m-in-linux32_g-000000000000000/build-gaia-l10n" TINDERBOX_OUTPUT="1" CCACHE_COMPRESS="1" TOOLTOOL_CACHE="/builds/tooltool_cache" SYMBOL_SERVER_PATH="/mnt/netapp/breakpad/symbols_ffx/" PATH="/tools/python27-mercurial/bin:/tools/python27/bin:${PATH}:/tools/buildbot/bin" MOZ_CRASHREPORTER_NO_REPORT="1" SYMBOL_SERVER_USER="ffxbld" WGET_OPTS="-q -c" LC_ALL="C" CCACHE_UMASK="002" make -f client.mk build \'MOZ_BUILD_DATE=20141104050144\''] environment: CCACHE_COMPRESS=1 CCACHE_DIR=/builds/ccache CCACHE_HASHDIR= CCACHE_UMASK=002 CVS_RSH=ssh G_BROKEN_FILENAMES=1 HG_SHARE_BASE_DIR=/builds/hg-shared HISTCONTROL=ignoredups HISTSIZE=1000 HOME=/home/cltbld HOSTNAME=bld-linux64-spot-008.build.releng.use1.mozilla.com LANG=en_US.UTF-8 LC_ALL=C LESSOPEN=|/usr/bin/lesspipe.sh %s LOCALES_FILE=locales/languages_dev.json LOCALE_BASEDIR=/builds/slave/m-in-linux32_g-000000000000000/build-gaia-l10n LOGNAME=cltbld MAIL=/var/spool/mail/cltbld MOZ_AUTOMATION=1 MOZ_CRASHREPORTER_NO_REPORT=1 MOZ_OBJDIR=obj-firefox PATH=/tools/python27-mercurial/bin:/tools/python27/bin:/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin:/tools/buildbot/bin POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/m-in-linux32_g-000000000000000/build SHELL=/bin/bash SHLVL=1 SYMBOL_SERVER_HOST=symbolpush.mozilla.org SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_ffx/ SYMBOL_SERVER_SSH_KEY=/home/cltbld/.ssh/ffxbld_rsa SYMBOL_SERVER_USER=ffxbld TERM=linux TINDERBOX_OUTPUT=1 TMOUT=86400 TOOLTOOL_CACHE=/builds/tooltool_cache TOOLTOOL_HOME=/builds USER=cltbld WGET_OPTS=-q -c _=/tools/buildbot/bin/python using PTY: False INFO: mock_mozilla.py version 1.0.3 starting... State Changed: init plugins INFO: selinux disabled State Changed: start State Changed: lock buildroot State Changed: shell ( echo 'export MOZ_AUTOMATION_BUILD_SYMBOLS=1'; echo 'export MOZ_AUTOMATION_L10N_CHECK=1'; echo 'export MOZ_AUTOMATION_PACKAGE=1'; echo 'export MOZ_AUTOMATION_PACKAGE_TESTS=1'; echo 'export MOZ_AUTOMATION_INSTALLER=0'; echo 'export MOZ_AUTOMATION_UPDATE_PACKAGING=0'; echo 'export MOZ_AUTOMATION_UPLOAD=1'; echo 'export MOZ_AUTOMATION_UPLOAD_SYMBOLS=0'; echo 'export MOZ_AUTOMATION_BUILD_SYMBOLS=1'; echo 'export MOZ_AUTOMATION_L10N_CHECK=1'; echo 'export MOZ_AUTOMATION_PACKAGE=1'; echo 'export MOZ_AUTOMATION_PACKAGE_TESTS=1'; echo 'export MOZ_AUTOMATION_INSTALLER=0'; echo 'export MOZ_AUTOMATION_UPDATE_PACKAGING=0'; echo 'export MOZ_AUTOMATION_UPLOAD=1'; echo 'export MOZ_AUTOMATION_UPLOAD_SYMBOLS=0'; echo 'export SCCACHE_BUCKET=mozilla-releng-s3-cache-us-east-1-prod'; echo 'export SCCACHE_NAMESERVER=169.254.169.253'; echo 'export UPLOAD_EXTRA_FILES+=sccache.log.gz'; ) > /builds/slave/m-in-linux32_g-000000000000000/build/obj-firefox/.mozconfig.mk make -f /builds/slave/m-in-linux32_g-000000000000000/build/client.mk realbuild CREATE_MOZCONFIG_JSON= Adding client.mk options from /builds/slave/m-in-linux32_g-000000000000000/build/.mozconfig: AUTOCLOBBER=1 export MOZ_AUTOMATION_BUILD_SYMBOLS=1 export MOZ_AUTOMATION_L10N_CHECK=1 export MOZ_AUTOMATION_PACKAGE=1 export MOZ_AUTOMATION_PACKAGE_TESTS=1 export MOZ_AUTOMATION_INSTALLER=0 export MOZ_AUTOMATION_UPDATE_PACKAGING=0 export MOZ_AUTOMATION_UPLOAD=1 export MOZ_AUTOMATION_UPLOAD_SYMBOLS=0 AUTOCLOBBER=1 export MOZ_AUTOMATION_BUILD_SYMBOLS=1 export MOZ_AUTOMATION_L10N_CHECK=1 export MOZ_AUTOMATION_PACKAGE=1 export MOZ_AUTOMATION_PACKAGE_TESTS=1 export MOZ_AUTOMATION_INSTALLER=0 export MOZ_AUTOMATION_UPDATE_PACKAGING=0 export MOZ_AUTOMATION_UPLOAD=1 export MOZ_AUTOMATION_UPLOAD_SYMBOLS=0 CONFIG_GUESS=i686-pc-linux export SCCACHE_BUCKET=mozilla-releng-s3-cache-us-east-1-prod export SCCACHE_NAMESERVER=169.254.169.253 MOZ_PREFLIGHT_ALL+=build/sccache.mk MOZ_POSTFLIGHT_ALL+=build/sccache.mk export UPLOAD_EXTRA_FILES+=sccache.log.gz MOZ_OBJDIR=/builds/slave/m-in-linux32_g-000000000000000/build/obj-firefox OBJDIR=/builds/slave/m-in-linux32_g-000000000000000/build/obj-firefox FOUND_MOZCONFIG=/builds/slave/m-in-linux32_g-000000000000000/build/.mozconfig make[1]: Entering directory `/builds/slave/m-in-linux32_g-000000000000000/build' set -e; \ for mkfile in build/sccache.mk; do \ make -f /builds/slave/m-in-linux32_g-000000000000000/build/$mkfile preflight_all TOPSRCDIR=/builds/slave/m-in-linux32_g-000000000000000/build OBJDIR=/builds/slave/m-in-linux32_g-000000000000000/build/obj-firefox MOZ_OBJDIR=/builds/slave/m-in-linux32_g-000000000000000/build/obj-firefox; \ done make[2]: Entering directory `/builds/slave/m-in-linux32_g-000000000000000/build' # Terminate any sccache server that might still be around python2.7 /builds/slave/m-in-linux32_g-000000000000000/build/sccache/sccache.py > /dev/null 2>&1 make[2]: Leaving directory `/builds/slave/m-in-linux32_g-000000000000000/build' Generating /builds/slave/m-in-linux32_g-000000000000000/build/configure using autoconf cd /builds/slave/m-in-linux32_g-000000000000000/build; /usr/bin/autoconf-2.13 Generating /builds/slave/m-in-linux32_g-000000000000000/build/js/src/configure using autoconf cd /builds/slave/m-in-linux32_g-000000000000000/build/js/src; /usr/bin/autoconf-2.13 make[2]: Entering directory `/builds/slave/m-in-linux32_g-000000000000000/build' cp /builds/slave/m-in-linux32_g-000000000000000/build/.mozconfig /builds/slave/m-in-linux32_g-000000000000000/build/obj-firefox/.mozconfig cd /builds/slave/m-in-linux32_g-000000000000000/build/obj-firefox /builds/slave/m-in-linux32_g-000000000000000/build/configure Adding configure options from /builds/slave/m-in-linux32_g-000000000000000/build/.mozconfig --enable-crashreporter --enable-release --enable-crashreporter --enable-release --enable-elf-hack --enable-stdcxx-compat --target=i686-pc-linux --host=i686-pc-linux --x-libraries=/usr/lib --enable-update-channel= --enable-update-packaging --enable-signmar --with-compiler-wrapper=python2.7 /builds/slave/m-in-linux32_g-000000000000000/build/sccache/sccache.py --enable-application=b2g --disable-elf-hack loading cache ./config.cache checking host system type... i686-pc-linux-gnu checking target system type... i686-pc-linux-gnu checking build system type... i686-pc-linux-gnu checking for gawk... (cached) gawk checking for python2.7... (cached) /tools/python27/bin/python2.7 Creating Python environment checking Python environment is Mozilla virtualenv... /builds/slave/m-in-linux32_g-000000000000000/build/obj-firefox/_virtualenv/bin/python: error while loading shared libraries: libpython2.7.so.1.0: wrong ELF class: ELFCLASS64 configure: error: Python environment does not appear to be sane. ------ config.log ------ This file contains any messages produced by compilers while running configure, to aid debugging if configure makes a mistake. configure:1185: checking host system type configure:1206: checking target system type configure:1224: checking build system type configure:1300: checking for gawk configure:1385: checking for python2.7 configure:1495: checking Python environment is Mozilla virtualenv configure: error: Python environment does not appear to be sane. *** Fix above errors and then restart with\ "make -f client.mk build" make[2]: *** [configure] Error 1 make[2]: Leaving directory `/builds/slave/m-in-linux32_g-000000000000000/build' make[1]: *** [/builds/slave/m-in-linux32_g-000000000000000/build/obj-firefox/config.status] Error 2 make[1]: Leaving directory `/builds/slave/m-in-linux32_g-000000000000000/build' make: *** [build] Error 2 State Changed: unlock buildroot program finished with exit code 2 elapsedTime=4.701965 ========= Finished compile failed (results: 2, elapsed: 4 secs) (at 2014-11-04 05:08:36.072368) ========= ========= Skipped (results: not started, elapsed: not started) =========

This error appears to be about a python 32 bit interpreter trying to use a python 64 bit library - I think this is likely to be fall out from this bug, so will roll back, and reconfig. In addition, this bug got picked up in a reconfig, but bugzilla API failed to update this bug to say so - have raised bug 1093600 about this.

Next one: https://treeherder.mozilla.org/ui/#/jobs?repo=mozilla-inbound&revision=1dd40ce6fc5a Backing out now...

also that this hit in https://treeherder.mozilla.org/ui/#/jobs?repo=mozilla-inbound&revision=1dd40ce6fc5a only the B2g Linux Builds opt and debug , Builds like Linux opt are ok

Reconfig in progress

pmoore was wondering if this just needed a clobber instead. That looks likely given that it's the objdir/_virtualenv/python that's failing, since it copies the system python into the objdir when setting up the virtualenv. I doubt it knows to copy the new binary when the system python changes.

(In reply to Michael Shal [:mshal] from comment #16) > pmoore was wondering if this just needed a clobber instead. That looks > likely given that it's the objdir/_virtualenv/python that's failing, since > it copies the system python into the objdir when setting up the virtualenv. > I doubt it knows to copy the new binary when the system python changes. In fairness, I was just stealing nthomas's idea in comment 8 =)

So to summarize - I backed this out, reconfig'd, and then Tomcat clobbered, and all is ok again. The fact Tomcat needed to clobber after the backout also suggests that a clobber was probably necessary. Therefore, it might be worth landing this change again, and then coordinating a clobber with a reconfig, to see if that fixes everything.

Indeed, this looks very much like it needs a clobber of the 32-bits b2g builders. Relanded. https://hg.mozilla.org/build/buildbot-configs/rev/d029e3d04215

Reconfiging now... We will clobber after reconfig completes.

In production: https://hg.mozilla.org/build/buildbot-configs/rev/d029e3d04215

Tomcat has clobbered