See https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=16ce8562a975 buildid: Traceback (most recent call last): File "build/config/printconfigsetting.py", line 5, in <module> import configobj ImportError: No module named configobj or ImportError: No module named configobj --revision 16ce8562a975 --release-to-tinderbox-dated-builds --release-to-latest-tinderbox-builds Traceback (most recent call last): File "/builds/slave/m-beta-m64-d-00000000000000000/build/build/upload.py", line 328, in <module> package=options.package, verbose=True) File "/builds/slave/m-beta-m64-d-00000000000000000/build/build/upload.py", line 232, in UploadFiles output = DoSSHCommand('%s "%s" %s' % (post_upload_command, path, file_list), user, host, port=port, ssh_key=ssh_key) File "/builds/slave/m-beta-m64-d-00000000000000000/build/build/upload.py", line 96, in DoSSHCommand output = f(cmdline, stderr=STDOUT).strip() File "/builds/slave/m-beta-m64-d-00000000000000000/build/python/redo/redo/__init__.py", line 185, in _retriable_wrapper **retry_kwargs) File "/builds/slave/m-beta-m64-d-00000000000000000/build/python/redo/redo/__init__.py", line 140, in retry return action(*args, **kwargs) File "/tools/python/lib/python2.7/subprocess.py", line 544, in check_output raise CalledProcessError(retcode, cmd, output=output) subprocess.CalledProcessError: Command '['ssh', '-o', 'IdentityFile=~/.ssh/ffxbld_rsa', 'ffxbld@upload.ffxbld.productdelivery.prod.mozaws.net', 'post_upload.py --tinderbox-builds-dir mozilla-beta-macosx64-debug -p firefox -i Traceback (most recent call last):\n File "build/config/printconfigsetting.py", line 5, in <module>\n import configobj\nImportError: No module named configobj --revision 16ce8562a975 --release-to-tinderbox-dated-builds --release-to-latest-tinderbox-builds "/media/ephemeral0/tmp/tmp.ssLqaKKHrV/" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//firefox-44.0.en-US.mac64.dmg" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV/mac64/xpi/firefox-44.0.en-US.langpack.xpi" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//mozharness.zip" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//firefox-44.0.en-US.mac64.common.tests.zip" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//firefox-44.0.en-US.mac64.cppunittest.tests.zip" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//firefox-44.0.en-US.mac64.xpcshell.tests.zip" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//firefox-44.0.en-US.mac64.mochitest.tests.zip" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//firefox-44.0.en-US.mac64.talos.tests.zip" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//firefox-44.0.en-US.mac64.reftest.tests.zip" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//firefox-44.0.en-US.mac64.web-platform.tests.zip" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//firefox-44.0.en-US.mac64.crashreporter-symbols.zip" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//firefox-44.0.en-US.mac64.txt" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//firefox-44.0.en-US.mac64.json" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//firefox-44.0.en-US.mac64.mozinfo.json" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//test_packages.json" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//jsshell-mac64.zip" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV/host/bin/mar" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV/host/bin/mbsdiff" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//firefox-44.0.en-US.mac64.checksums" "/media/ephemeral0/tmp/tmp.ssLqaKKHrV//firefox-44.0.en-US.mac64.checksums.asc"']' returned non-zero exit status 1 make[1]: *** [upload] Error 1 make: *** [upload] Error 2 retry: Giving up on <function run_with_timeout at 0x1004edc08> Unable to successfully run ['make', 'upload'] after 5 attempts program finished with exit code 1 elapsedTime=907.006275 mshal pointed me to bug 957911, bug 1213839, bug 957951 FTR, m-a uses mozharness, beta uses old-style buildbot factories

The signature for re.sub changed between 2.6 and 2.7: see https://docs.python.org/2.6/library/re.html and https://docs.python.org/2/library/re.html. The very beginning of the log at http://archive.mozilla.org/pub/mobile/tinderbox-builds/mozilla-beta-android-api-9/1450134357/mozilla-beta-android-api-9-bm94-build1-build3.txt.gz is clear: builder: mozilla-beta-android-api-9 slave: bld-linux64-spot-256 starttime: 1450128869.6 results: failure (2) buildid: Traceback (most recent call last): File "build/config/printconfigsetting.py", line 17, in <module> content = re.sub('^\s*;', '#', fh.read(), flags=re.M) TypeError: sub() got an unexpected keyword argument 'flags' builduid: 82d1647865324b138fa85a47f4ba97a5 revision: 3f02f1340ad2 We'll need to bump the Python version, special case the version, or write equivalent code that doesn't use the new signature.

this is going to be fun.. okay so I'm making 3 wild assumptions: 1) we can call 'python2.7' on windows aws builders and it will be found in path <- markco? 2) we can use the new style of printconfigsettings.py on m-r/esr <- glandium? 3) the only reason TB used comms printconfigsettings.py was a workaround for not having this patch dump master diff: http://people.mozilla.org/~jlund/printconfigsettings_fix_bbotcustom-dump-master.diff

I pushed a hack to see if we can just avoid this issue: https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=d1a4b331d80b Hopefully we're green by tomorrow AM!

That's giving us green on Linux and Android, but not on Mac or Windows.

Comment on attachment 8698310 [details] [diff] [review] 151214_bug_1232466_printconfigsettings-buildbotcustom.patch Review of attachment 8698310 [details] [diff] [review]: ----------------------------------------------------------------- In overall the patch looks good to me. Given the urgency, let's land it and see what else breaks. :)

Comment on attachment 8698310 [details] [diff] [review] 151214_bug_1232466_printconfigsettings-buildbotcustom.patch https://hg.mozilla.org/build/buildbotcustom/rev/694085d310b7

> 1) we can call 'python2.7' on windows aws builders and it will be found in > path <- markco? Yes, "C:\mozilla-build\python27" is within the default path.

I'm going to back it out in a bit. It didn't work. :( I was watching http://buildbot-master85.bb.releng.scl3.mozilla.com:8001/builders/OS%20X%2010.7%2064-bit%20mozilla-beta%20leak%20test%20build/builds/106 and it failed. python2.7 /builds/slave/m-beta-m64-d-00000000000000000/build/mach python build/config/printconfigsetting.py build/obj-firefox/dist/bin/application.ini App BuildID in dir /builds/slave/m-beta-m64-d-00000000000000000/build/obj-firefox (timeout 1200 secs) ... /builds/slave/m-beta-m64-d-00000000000000000/build/obj-firefox/_virtualenv/bin/python: can't open file 'build/config/printconfigsetting.py': [Errno 2] No such file or directory I tried to run it from /builds/slave/m-beta-m64-d-00000000000000000 on the same host and got this first time: New python executable in /builds/slave/m-beta-m64-d-00000000000000000/build/obj-x86_64-apple-darwin11.2.0/_virtualenv/bin/python Installing setuptools, pip...done. Python 2.7.3 or greater (but not Python 3) is required to build. You are running Python 2.7.1. Run |mach bootstrap| to ensure your system is up to date. If you still receive this error, your shell environment is likely detecting another Python version. Ensure a modern Python can be found in the paths defined by the $PATH environment variable and try again. Error running mach: ['python', 'build/config/printconfigsetting.py', 'build/obj-firefox/dist/bin/application.ini', 'App', 'BuildID'] The error occurred in code that was called by the mach command. This is either a bug in the called code itself or in the way that mach is calling it. You should consider filing a bug for this issue. If filing a bug, please include the full output of mach, including this error message. The details of the failure are as follows: Exception: Error populating virtualenv. File "/builds/slave/m-beta-m64-d-00000000000000000/build/python/mach_commands.py", line 76, in python self._activate_virtualenv() File "/builds/slave/m-beta-m64-d-00000000000000000/build/python/mozbuild/mozbuild/base.py", line 665, in _activate_virtualenv self.virtualenv_manager.ensure() File "/builds/slave/m-beta-m64-d-00000000000000000/build/python/mozbuild/mozbuild/virtualenv.py", line 128, in ensure return self.build() File "/builds/slave/m-beta-m64-d-00000000000000000/build/python/mozbuild/mozbuild/virtualenv.py", line 393, in build raise Exception('Error populating virtualenv.') second run was less chatty: [cltbld@bld-lion-r5-068.build.releng.scl3.mozilla.com m-beta-m64-d-00000000000000000]$ python2.7 build/config/printconfigsetting.py build/obj-firefox/dist/bin/ application.ini App BuildID Traceback (most recent call last): File "build/config/printconfigsetting.py", line 5, in <module> import configobj ImportError: No module named configobj [cltbld@bld-lion-r5-068.build.releng.scl3.mozilla.com m-beta-m64-d-00000000000000000]$ python2.7 /builds/slave/m-beta-m64-d-00000000000000000/build/mach python build/config/printconfigsetting.py build/obj-firefox/dist/bin/application.ini App BuildID Traceback (most recent call last): File "build/config/printconfigsetting.py", line 5, in <module> import configobj ImportError: No module named configobj

> I tried to run it from /builds/slave/m-beta-m64-d-00000000000000000 on the > same host and got this first time: > New python executable in > /builds/slave/m-beta-m64-d-00000000000000000/build/obj-x86_64-apple-darwin11. > 2.0/_virtualenv/bin/python > Installing setuptools, pip...done. > Python 2.7.3 or greater (but not Python 3) is required to build. You are > running Python 2.7.1. > Run |mach bootstrap| to ensure your system is up to date. the osx one is using the wrong python2.7. I guess mach needs 2.7.3 or greater but we are using 2.7.1 :( [cltbld@bld-lion-r5-084.build.releng.scl3.mozilla.com ~]$ python --version Python 2.7.1 [cltbld@bld-lion-r5-084.build.releng.scl3.mozilla.com ~]$ which python /usr/bin/python for mozharness builds, we use python 2.7.3 interpreter [cltbld@bld-lion-r5-084.build.releng.scl3.mozilla.com ~]$ /tools/buildbot/bin/python --version Python 2.7.3 this is defined from 'mozharness_python': https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla/config.py#930

> second run was less chatty: > > [cltbld@bld-lion-r5-068.build.releng.scl3.mozilla.com > m-beta-m64-d-00000000000000000]$ python2.7 > build/config/printconfigsetting.py build/obj-firefox/dist/bin/ > application.ini App BuildID > > Traceback (most recent call last): > File "build/config/printconfigsetting.py", line 5, in <module> > import configobj > ImportError: No module named configobj > [cltbld@bld-lion-r5-068.build.releng.scl3.mozilla.com > m-beta-m64-d-00000000000000000]$ python2.7 > /builds/slave/m-beta-m64-d-00000000000000000/build/mach python > build/config/printconfigsetting.py > build/obj-firefox/dist/bin/application.ini App BuildID > > Traceback (most recent call last): > File "build/config/printconfigsetting.py", line 5, in <module> > import configobj > ImportError: No module named configobj this osx one I suspect I screwed up the objdir workdir to run this because ... osx this is what mozharness runs.. ['/tools/buildbot/bin/python', '/builds/slave/m-aurora-m64-00000000000000000/build/src/mach', 'python', '/builds/slave/m-aurora-m64-00000000000000000/build/src/config/printconfigsetting.py', '/builds/slave/m-aurora-m64-00000000000000000/build/src/obj-firefox/x86_64/dist/bin/application.ini', 'App', 'SourceStamp'] in /builds/slave/m-aurora-m64-00000000000000000/build/src/obj-firefox/x86_64 this is what my buildbot patch ran: python2.7 /builds/slave/m-beta-m64-d-00000000000000000/build/mach python build/config/printconfigsetting.py build/obj-firefox/dist/bin/application.ini App BuildID in dir /builds/slave/m-beta-m64-d-00000000000000000/build/obj-firefox good cwd: /builds/slave/m-aurora-m64-00000000000000000/build/src/obj-firefox/x86_64 bad cwd: /builds/slave/m-beta-m64-d-00000000000000000/build/obj-firefox

(In reply to Jordan Lund (:jlund) from comment #11) > > second run was less chatty: > > > > [cltbld@bld-lion-r5-068.build.releng.scl3.mozilla.com > > m-beta-m64-d-00000000000000000]$ python2.7 > > build/config/printconfigsetting.py build/obj-firefox/dist/bin/ > > application.ini App BuildID > > > > Traceback (most recent call last): > > File "build/config/printconfigsetting.py", line 5, in <module> > > import configobj > > ImportError: No module named configobj > > [cltbld@bld-lion-r5-068.build.releng.scl3.mozilla.com > > m-beta-m64-d-00000000000000000]$ python2.7 > > /builds/slave/m-beta-m64-d-00000000000000000/build/mach python > > build/config/printconfigsetting.py > > build/obj-firefox/dist/bin/application.ini App BuildID > > > > Traceback (most recent call last): > > File "build/config/printconfigsetting.py", line 5, in <module> > > import configobj > > ImportError: No module named configobj > > this osx one I suspect I screwed up the objdir workdir to run this because > ... osx maybe not. the buildbot patched job in question failed doing a leak (debug) run. in the mozharness world, they both use the same objdir: 06:39:11 INFO - Copy/paste: /tools/buildbot/bin/python /builds/slave/m-aurora-m64-d-000000000000000/build/src/mach python /builds/slave/m-aurora-m64-d-000000000000000/build/src/config/printconfigsetting.py /builds/slave/m-aurora-m64-d-000000000000000/build/src/obj-firefox/dist/bin/application.ini App BuildID

interdiff: http://people.mozilla.org/~jlund/151215_bug_1232466_printconfigsettings-buildbotcustom-v2-interdiff.patch full dump_master diff: http://people.mozilla.org/~jlund/printconfigsettings_fix_bbotcustom-dump-master-v2.diff so, I think we had two issues based on logs 1) we were using a lesser version of python to invoke mach 2) we were using broken relative paths for app.ini and printconfigsetting (because we are now calling this from the objdir) I did not test this on a master but I ran a test cmd locally that I think was mirrors what the patch will do based on dump_master and filling in the build prop vars: [cltbld@bld-lion-r5-068.build.releng.scl3.mozilla.com obj-firefox]$ cd /builds/slave/m-beta-m64-d-00000000000000000/build/obj-firefox[cltbld@bld-lion-r5-068.build.releng.scl3.mozilla.com obj-firefox]$ Apple_PubSub_Socket_Render=/tmp/launch-FL7jGa/Render CCACHE_COMPRESS=1 CCACHE_DIR=/builds/ccache CCACHE_UMASK=002 DISPLAY=/tmp/launch-7n2SR1/org.x:0 GIT_SHARE_BASE_DIR=/builds/git-shared HG_SHARE_BASE_DIR=/builds/hg-shared HOME=/Users/cltbld IDLEIZER_DISABLE_SHUTDOWN=true LC_ALL=C LOGNAME=cltbld MOZ_AUTOMATION=1 MOZ_CRASHREPORTER_NO_REPORT=1 PATH=/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin PWD=/builds/slave/m-beta-m64-d-00000000000000000/build/obj-firefox SHELL=/bin/bash SSH_AUTH_SOCK=/tmp/launch-92ZnER/Listeners TINDERBOX_OUTPUT=1 TMPDIR=/var/folders/1t/62sfn2j55sx4w7c3jzgg8_kw00000w/T/ TOOLTOOL_CACHE=/builds/tooltool_cache TOOLTOOL_HOME=/builds TWISTD_LOG_PATH=/builds/slave/twistd.log USER=cltbld VERSIONER_PYTHON_PREFER_32_BIT=no VERSIONER_PYTHON_VERSION=2.7 XPCOM_DEBUG_BREAK=stack-and-abort __CF_USER_TEXT_ENCODING=0x1C:0:0 /tools/buildbot/bin/python /builds/slave/m-beta-m64-d-00000000000000000/build/mach python /builds/slave/m-beta-m64-d-00000000000000000/build/config/printconfigsetting.py /builds/slave/m-beta-m64-d-00000000000000000/build/obj-firefox/dist/bin/application.ini App BuildID 20151214205110

Comment on attachment 8698562 [details] [diff] [review] 151215_bug_1232466_printconfigsettings-buildbotcustom-v2.patch landed and reconfig'd. reconfig finished at about 230pm PT. after that I re-triggered some windows and osx builders. you can see those still running (grey) here: https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=d1a4b331d80b you need vpn to see the logs running live but I am watching them and they have been able to make it farther than before. so far one has finished and it is looking good: https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=d1a4b331d80b&filter-searchStr=os%20x%20debug%20build

I think this patch is working so far. we have one failure (orange) on osx opt build[1]. But I think that has just shown itself now that we have made it past the printconfigsetting.py issue. tracking the known failure [1] https://treeherder.mozilla.org/logviewer.html#?job_id=684965&repo=mozilla-beta

1) failing to run `make package MOZ_PKG_PRETTYNAMES=1` # Package mozharness /builds/slave/m-beta-m64-0000000000000000000/build/obj-firefox/x86_64/_virtualenv/bin/python -m mozbuild.action.test_archive mozharness /builds/slave/m-beta-m64-0000000000000000000/build/obj-firefox/x86_64/dist/mac/en-US/mozharness.zip Traceback (most recent call last): File "/tools/python/lib/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/tools/python/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/builds/slave/m-beta-m64-0000000000000000000/build/python/mozbuild/mozbuild/action/test_archive.py", line 332, in <module> sys.exit(main(sys.argv[1:])) File "/builds/slave/m-beta-m64-0000000000000000000/build/python/mozbuild/mozbuild/action/test_archive.py", line 313, in main with open(args.outputfile, 'wb') as fh: IOError: [Errno 2] No such file or directory: '/builds/slave/m-beta-m64-0000000000000000000/build/obj-firefox/x86_64/dist/mac/en-US/mozharness.zip' make[3]: *** [stage-package] Error 255 make[2]: *** [make-package] Error 2 make[1]: *** [default] Error 2 make: *** [package] Error 2 iirc - taskcluster started packaging mozharness.zip. not sure if it's related. 2) failing to run `make -C obj-firefox/x86_64/tools/update-packaging MOZ_PKG_PRETTYNAMES=1` log: mkdir -p '/builds/slave/m-beta-m64-0000000000000000000/build/obj-firefox/x86_64/dist/update/mac/en-US/' MAR=/builds/slave/m-beta-m64-0000000000000000000/build/obj-firefox/x86_64/dist/host/bin/mar \ /builds/slave/m-beta-m64-0000000000000000000/build/tools/update-packaging/make_full_update.sh \ '../../dist/update/mac/en-US/nightly-44.0.complete.mar' \ '/builds/slave/m-beta-m64-0000000000000000000/build/obj-firefox/x86_64/dist/universal/firefox/Nightly.app' /builds/slave/m-beta-m64-0000000000000000000/build/obj-firefox/x86_64/dist/universal/firefox/Nightly.app /builds/slave/m-beta-m64-0000000000000000000/build/obj-firefox/x86_64/tools/update-packaging precomplete file is missing! make: *** [complete-patch] Error 1

I guess linux has been hitting this already.. https://treeherder.mozilla.org/logviewer.html#?job_id=684419&repo=mozilla-beta windows just hit this too. anything that runs pretty names (opt builds)

Rail filed bug 1232887 for the mozharness packaging error. Hopefully update packaging will work once normal packaging can create the precomplete file.

Comment on attachment 8698310 [details] [diff] [review] 151214_bug_1232466_printconfigsettings-buildbotcustom.patch Looks like you figured it out.

This is still busted, actually: ========= Started set props: buildid (results: 2, elapsed: 0 secs) (at 2015-12-15 22:22:50.417902) ========= /tools/buildbot/bin/python /builds/slave/m-beta-lx-d-000000000000000000/build/mach python /builds/slave/m-beta-lx-d-000000000000000000/build/config/printconfigsetting.py /builds/slave/m-beta-lx-d-000000000000000000/build/obj-firefox/dist/bin/application.ini App BuildID in dir /builds/slave/m-beta-lx-d-000000000000000000/build/obj-firefox (timeout 1200 secs) watching logfiles {} argv: ['/tools/buildbot/bin/python', '/builds/slave/m-beta-lx-d-000000000000000000/build/mach', 'python', '/builds/slave/m-beta-lx-d-000000000000000000/build/config/printconfigsetting.py', '/builds/slave/m-beta-lx-d-000000000000000000/build/obj-firefox/dist/bin/application.ini', 'App', 'BuildID'] environment: CCACHE_COMPRESS=1 CCACHE_DIR=/builds/ccache CCACHE_HASHDIR= CCACHE_UMASK=002 DISPLAY=:2 G_BROKEN_FILENAMES=1 HG_SHARE_BASE_DIR=/builds/hg-shared HISTCONTROL=ignoredups HISTSIZE=1000 HOME=/home/cltbld HOSTNAME=bld-linux64-spot-036.build.releng.use1.mozilla.com LANG=en_US.UTF-8 LC_ALL=C LD_LIBRARY_PATH=obj-firefox/dist/bin LESSOPEN=|/usr/bin/lesspipe.sh %s LOGNAME=cltbld MAIL=/var/spool/mail/cltbld MOZ_AUTOMATION=1 MOZ_CRASHREPORTER_NO_REPORT=1 PATH=/tools/buildbot/bin:/usr/local/bin:/usr/lib/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin PWD=/builds/slave/m-beta-lx-d-000000000000000000/build/obj-firefox SHELL=/bin/bash SHLVL=1 TERM=linux TINDERBOX_OUTPUT=1 TMOUT=86400 TOOLTOOL_CACHE=/builds/tooltool_cache TOOLTOOL_HOME=/builds USER=cltbld XPCOM_DEBUG_BREAK=stack-and-abort _=/tools/buildbot/bin/python using PTY: False Error loading mozconfig: /builds/slave/m-beta-lx-d-000000000000000000/build/.mozconfig Evaluation of your mozconfig exited with an error. This could be triggered by a command inside your mozconfig failing. Please change your mozconfig to not error and/or to catch errors in executed commands. mozconfig output: ------BEGIN_AC_OPTION --enable-debug ------END_AC_OPTION ------BEGIN_AC_OPTION --enable-dmd ------END_AC_OPTION ------BEGIN_AC_OPTION --enable-verify-mar ------END_AC_OPTION ------BEGIN_AC_OPTION --with-google-oauth-api-keyfile=/builds/google-oauth-api.key ------END_AC_OPTION ------BEGIN_MK_OPTION AUTOCLOBBER=1 ------END_MK_OPTION ------BEGIN_AC_OPTION --enable-crashreporter ------END_AC_OPTION ------BEGIN_AC_OPTION --enable-release ------END_AC_OPTION ------BEGIN_MK_OPTION export MOZ_AUTOMATION_BUILD_SYMBOLS=1 ------END_MK_OPTION ------BEGIN_MK_OPTION export MOZ_AUTOMATION_L10N_CHECK=0 ------END_MK_OPTION ------BEGIN_MK_OPTION export MOZ_AUTOMATION_PACKAGE=1 ------END_MK_OPTION ------BEGIN_MK_OPTION export MOZ_AUTOMATION_PACKAGE_TESTS=1 ------END_MK_OPTION ------BEGIN_MK_OPTION export MOZ_AUTOMATION_INSTALLER=0 ------END_MK_OPTION ------BEGIN_MK_OPTION export MOZ_AUTOMATION_UPDATE_PACKAGING=0 ------END_MK_OPTION ------BEGIN_MK_OPTION export MOZ_AUTOMATION_UPLOAD=1 ------END_MK_OPTION ------BEGIN_MK_OPTION export MOZ_AUTOMATION_UPLOAD_SYMBOLS=0 ------END_MK_OPTION ------BEGIN_MK_OPTION export MOZ_AUTOMATION_SDK=0 ------END_MK_OPTION ------BEGIN_MK_OPTION PATH=/builds/slave/m-beta-lx-d-000000000000000000/build/gcc/bin:/tools/buildbot/bin:/usr/local/bin:/usr/lib/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin ------END_MK_OPTION ------BEGIN_AC_OPTION --enable-elf-hack ------END_AC_OPTION ------BEGIN_AC_OPTION --enable-stdcxx-compat ------END_AC_OPTION ------BEGIN_AC_OPTION --enable-default-toolkit=cairo-gtk3 ------END_AC_OPTION ------BEGIN_MK_OPTION export FONTCONFIG_PATH=/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/usr/local/etc/fonts ------END_MK_OPTION ------BEGIN_MK_OPTION export PANGO_SYSCONFDIR=/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/usr/local/etc ------END_MK_OPTION ------BEGIN_MK_OPTION export PANGO_LIBDIR=/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/usr/local/lib ------END_MK_OPTION ------BEGIN_MK_OPTION export GDK_PIXBUF_MODULE_FILE=/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/usr/local/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache ------END_MK_OPTION ------BEGIN_MK_OPTION export GDK_PIXBUF_MODULEDIR=/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/usr/local/lib/gdk-pixbuf-2.0/2.10.0/loaders ------END_MK_OPTION ------BEGIN_MK_OPTION export LD_LIBRARY_PATH=/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/usr/local/lib ------END_MK_OPTION /builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/setup.sh: ./usr/local/bin/pango-querymodules: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory /builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/setup.sh: ./usr/local/bin/gdk-pixbuf-query-loaders: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory /builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/setup.sh: ./usr/local/bin/fc-cache: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory program finished with exit code 1 elapsedTime=0.316526 buildid: 'Error loading mozconfig: /builds/slave/m-beta-lx-d-000000000000000000/build/.mozconfig\n\nEvaluation of your mozconfig exited with an error. This could be triggered\nby a command inside your mozconfig failing. Please change your mozconfig\nto not error and/or to catch errors in executed commands.\n\nmozconfig output:\n\n------BEGIN_AC_OPTION\n--enable-debug\n------END_AC_OPTION\n------BEGIN_AC_OPTION\n--enable-dmd\n------END_AC_OPTION\n------BEGIN_AC_OPTION\n--enable-verify-mar\n------END_AC_OPTION\n------BEGIN_AC_OPTION\n--with-google-oauth-api-keyfile=/builds/google-oauth-api.key\n------END_AC_OPTION\n------BEGIN_MK_OPTION\nAUTOCLOBBER=1\n------END_MK_OPTION\n------BEGIN_AC_OPTION\n--enable-crashreporter\n------END_AC_OPTION\n------BEGIN_AC_OPTION\n--enable-release\n------END_AC_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_BUILD_SYMBOLS=1\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_L10N_CHECK=0\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_PACKAGE=1\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_PACKAGE_TESTS=1\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_INSTALLER=0\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_UPDATE_PACKAGING=0\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_UPLOAD=1\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_UPLOAD_SYMBOLS=0\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport MOZ_AUTOMATION_SDK=0\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nPATH=/builds/slave/m-beta-lx-d-000000000000000000/build/gcc/bin:/tools/buildbot/bin:/usr/local/bin:/usr/lib/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin\n------END_MK_OPTION\n------BEGIN_AC_OPTION\n--enable-elf-hack\n------END_AC_OPTION\n------BEGIN_AC_OPTION\n--enable-stdcxx-compat\n------END_AC_OPTION\n------BEGIN_AC_OPTION\n--enable-default-toolkit=cairo-gtk3\n------END_AC_OPTION\n------BEGIN_MK_OPTION\nexport FONTCONFIG_PATH=/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/usr/local/etc/fonts\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport PANGO_SYSCONFDIR=/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/usr/local/etc\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport PANGO_LIBDIR=/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/usr/local/lib\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport GDK_PIXBUF_MODULE_FILE=/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/usr/local/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport GDK_PIXBUF_MODULEDIR=/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/usr/local/lib/gdk-pixbuf-2.0/2.10.0/loaders\n------END_MK_OPTION\n------BEGIN_MK_OPTION\nexport LD_LIBRARY_PATH=/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/usr/local/lib\n------END_MK_OPTION\n/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/setup.sh: ./usr/local/bin/pango-querymodules: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory\n/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/setup.sh: ./usr/local/bin/gdk-pixbuf-query-loaders: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory\n/builds/slave/m-beta-lx-d-000000000000000000/build/gtk3/setup.sh: ./usr/local/bin/fc-cache: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory' ========= master_lag: 0.01 ========= ========= Finished set props: buildid (results: 2, elapsed: 0 secs) (at 2015-12-15 22:22:50.748150) ========= This subsequently fails in post-upload because POST_UPLOAD_CMD has a busted value.

Or https://treeherder.mozilla.org/logviewer.html#?job_id=685601&repo=mozilla-beta if you like it opt, where instead of trying to upload with an error message in the commandline, we try to post to graphserver with an error message in the commandline.

sounds like we need to run those commands inside mock environments or install 32-bit libraries on builders.

still needs to be tested

Comment on attachment 8698957 [details] [diff] [review] use_mock.diff https://hg.mozilla.org/build/buildbotcustom/rev/73e830d00659

In production: https://hg.mozilla.org/build/buildbotcustom/rev/73e830d00659

Comment on attachment 8698957 [details] [diff] [review] use_mock.diff https://hg.mozilla.org/build/buildbotcustom/rev/2293d99fd1fc

Whee, no longer blocks 44beta thanks to disabling GTK+3, but those last little bits, making Linux32 work for it, block 45beta since GTK+3 is planning on riding the next train, so six weeks to get it fixed and somehow tested. [Tracking Requested - why for this release]: Because if it isn't fixed, you can't even ship the first beta of 45.

In production: https://hg.mozilla.org/build/buildbotcustom/rev/73e830d00659

Tracking to make sure it is fixed before the aurora => beta merge.

I think this bug outlived its usefulness. Closing this, feel free to reopen if I'm wrong.