comm-central builds are broken since bug 957911 landed, with errors like File "build/mozilla/config/printconfigsetting.py", line 17, in <module> content = re.sub('^\s*;', '#', fh.read(), flags=re.M) TypeError: sub() got an unexpected keyword argument 'flags' 'flags' requires python 2.7, so the suspicion is that printconfigsetting.py is being run by python 2.6.

nthomas suggested in bug 957911 comment 18: I suggest you compare the setup in PLATFORM_VARS in http://hg.mozilla.org/build/buildbot-configs/file/default/mozilla/thunderbird_config.py with the corresponding ones in http://hg.mozilla.org/build/buildbot-configs/file/default/mozilla/config.py Ask in #releng if that doesn't help.

(In reply to aleth [:aleth] from comment #1) > nthomas suggested in bug 957911 comment 18: > > I suggest you compare the setup in PLATFORM_VARS in > > http://hg.mozilla.org/build/buildbot-configs/file/default/mozilla/ > thunderbird_config.py > with the corresponding ones in > http://hg.mozilla.org/build/buildbot-configs/file/default/mozilla/config.py As a guess, the Firefox PLATFORM_VARS have 'mozharness_python': '/tools/buildbot/bin/python' But that likely wouldn't help for Thunderbird as mozharness isn't enabled there yet?

mozharness_python shouldn't be related. I think, PATH is different, python 2.7 should be first in the list.

(In reply to Rail Aliiev [:rail] from comment #3) > mozharness_python shouldn't be related. I think, PATH is different, python > 2.7 should be first in the list. In the config.py files linked above, the PATH in 'env' matches however. Could it be that 'TOOLTOOL_CACHE': '/builds/tooltool_cache', 'TOOLTOOL_HOME': '/builds', is missing in the TB env?

I've also diffed the configs and the TOOLTOOL variables are the only major differences in environment I see. Could it be that the master is running with a different python version?

I don't think it's related to tooltool, we don't deploy python using it. Can you give a link to the failed build?

Failed Linux x64 Nightly from today: https://treeherder.mozilla.org/#/jobs?repo=comm-central&filter-searchStr=Linux%20x64%20opt%20Nightly%20(N)

Ah, it should just be running inside mock. Using MockProperty or RetryingMockProperty instead of SetProperty should help.

So, in case this is not fixed by bug 1213715, would this be the right thing to do? I've replaced all occurrences of SetProperty with MockProperty where python is being called.

Comment on attachment 8673833 [details] [diff] [review] Fix - v1 Almost! :) You also need to pass: target=self.mock_target, mock=self.use_mock, and maybe mock_workdir_prefix=None,

What do we need to do to move this forward?

I need to make above changes to the patch, and if requested also test this on staging. Alternatively we could try running printconfigsettings.py using mach python, but this patch seems easier. Leaving needinfo to remind me to update the patch.

Ok, this version adds the missing properties as you've suggested. In contrast to v1, I've reverted the SetProperty steps that happen before mock init. One thing that came to mind today, isn't mock only used on Linux? This error happens for all platforms. I haven't run this through staging yet, do you think it is necessary or is this a fairly safe patch? Another thing we could try is hardcoding python2.7 instead of python just for printconfigsetting.py. What do you think?

Comment on attachment 8675925 [details] [diff] [review] Fix - v2 Review of attachment 8675925 [details] [diff] [review]: ----------------------------------------------------------------- It should work (TM) :) Let's land it tomorrow - today we have some FTP migration related work.

(In reply to Rail Aliiev [:rail] from comment #14) > It should work (TM) :) > > Let's land it tomorrow - today we have some FTP migration related work. Does this need the checkin-needed flag now?

No, Fallen can push whenever he feels it's best.

https://hg.mozilla.org/build/buildbotcustom/rev/2104c6cc8081971e55b6555cacb0cf3c86b4cb57 Bug 1213839 - printconfigsetting.py fails on TB builders as it requires python 2.7. r=rail

I had to back it out :/ https://hg.mozilla.org/build/buildbotcustom/rev/b9780f76d2b4 It broke biuldbot-configs for some reason.

In production: https://hg.mozilla.org/build/buildbotcustom/rev/2104c6cc8081

(In reply to Rail Aliiev [:rail] from comment #18) > I had to back it out :/ > > https://hg.mozilla.org/build/buildbotcustom/rev/b9780f76d2b4 > > It broke biuldbot-configs for some reason. Do you have any further details that might help diagnose this?

From the travis log https://travis-ci.org/mozilla/build-buildbot-configs/jobs/86912426 2015-10-22 21:00:58,136 - loading test-output/bm69-tests1-windows/master.cfg 2015-10-22 21:01:14,174 - Couldn't load test-output/bm75-try1/master.cfg Traceback (most recent call last): File "/home/travis/build/mozilla/build-buildbot-configs/.tox/braindump/buildbot-related/dump_master_json.py", line 111, in dump_master c = loadMaster(path) File "/home/travis/build/mozilla/build-buildbot-configs/.tox/braindump/buildbot-related/dump_master_json.py", line 26, in loadMaster execfile(path, g, g) File "/home/travis/build/mozilla/build-buildbot-configs/test-output/bm75-try1/master.cfg", line 83, in <module> getattr(passwords, 'secrets', None)) File "/home/travis/build/mozilla/build-buildbot-configs/.tox/buildbotcustom/misc.py", line 1744, in generateBranchObjects mozilla2_dep_factory = factory_class(**factory_kwargs) File "/home/travis/build/mozilla/build-buildbot-configs/.tox/buildbotcustom/process/factory.py", line 1856, in __init__ MercurialBuildFactory.__init__(self, **kwargs) File "/home/travis/build/mozilla/build-buildbot-configs/.tox/buildbotcustom/process/factory.py", line 1063, in __init__ self.addBuildSteps() File "/home/travis/build/mozilla/build-buildbot-configs/.tox/buildbotcustom/process/factory.py", line 1196, in addBuildSteps self.addBuildAnalysisSteps() File "/home/travis/build/mozilla/build-buildbot-configs/.tox/buildbotcustom/process/factory.py", line 1418, in addBuildAnalysisSteps mock_workdir_prefix=None, File "/home/travis/build/mozilla/build-buildbot-configs/.tox/buildbotcustom/steps/mock.py", line 125, in __init__ self.super_class.__init__(self, **kwargs) File "/home/travis/build/mozilla/build-buildbot-configs/.tox/buildbotcustom/steps/mock.py", line 54, in __init__ assert 'workdir' in kwargs.keys(), "You *must* specify workdir" AssertionError: You *must* specify workdir In addBuildAnalysisSteps, MockProperty hasn't a workdir and the parent class is MockCommand which throws this error because workdir is missing in the kwargs: https://hg.mozilla.org/build/buildbotcustom/file/3d4e1078bcd9/steps/mock.py#l125 https://hg.mozilla.org/build/buildbotcustom/file/3d4e1078bcd9/steps/mock.py#l31

Add the workdirs (2x) which were missing as described in comment 21.

Comment on attachment 8680924 [details] [diff] [review] fix - v3 Review of attachment 8680924 [details] [diff] [review]: ----------------------------------------------------------------- Just a tiny fix required: ::: process/factory.py @@ +1682,2 @@ > )) > + self.MockProperty(MockProperty( this should be self.addStep(...

Comment on attachment 8681187 [details] [diff] [review] fix - v4 Thank you!

Pushed as http://hg.mozilla.org/build/buildbotcustom/rev/7c7cc793219d

Was this responsible for the Linux opt failure https://treeherder.mozilla.org/#/jobs?repo=comm-central&filter-searchStr=34f285eafca829de3954e2be19cf5ba6bcbc5d3b ?

It caused some upload issues :/ https://treeherder.mozilla.org/logviewer.html#?job_id=616893&repo=mozilla-beta

backout http://hg.mozilla.org/build/buildbotcustom/rev/c807f6272371

(In reply to Rail Aliiev [:rail] from comment #29) > backout http://hg.mozilla.org/build/buildbotcustom/rev/c807f6272371 Thanks and sorry for the inconvenience. (In reply to aleth [:aleth] from comment #27) > Was this responsible for the Linux opt failure > https://treeherder.mozilla.org/#/jobs?repo=comm-central&filter- > searchStr=34f285eafca829de3954e2be19cf5ba6bcbc5d3b ? Yes. From the build which was only orange: python build/config/printconfigsetting.py build/objdir-tb/dist/bin/application.ini App BuildID in dir /builds/slave/tb-c-cen-lx-000000000000000000/. (timeout 1200 secs) From the burning one: mock_mozilla -r mozilla-centos6-x86_64 --cwd . --unpriv --shell '/usr/bin/env python build/config/printconfigsetting.py build/objdir-tb/dist/bin/application.ini App BuildID' in dir /builds/slave/tb-c-cen-lx-000000000000000000/. (timeout 1200 secs) Any idea why it doesn't find the file now? Uploading /builds/slave/tb-c-cen-lx-000000000000000000/build/objdir-tb/dist/thunderbird-45.0a1.en-US.linux-i686.checksums Uploading /builds/slave/tb-c-cen-lx-000000000000000000/build/objdir-tb/dist/thunderbird-45.0a1.en-US.linux-i686.checksums.asc Running post-upload command: post_upload.py --tinderbox-builds-dir comm-central-linux -p thunderbird -i INFO: mock_mozilla.py version 1.0.3 starting... State Changed: init plugins INFO: selinux disabled State Changed: start State Changed: lock buildroot State Changed: shell python: can't open file 'build/config/printconfigsetting.py': [Errno 2] No such file or directory State Changed: unlock buildroot --revision 2ef88c6fc965 --release-to-tinderbox-dated-builds --release-to-latest-tinderbox-builds Traceback (most recent call last): File "/builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/build/upload.py", line 328, in <module> package=options.package, verbose=True) File "/builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/build/upload.py", line 232, in UploadFiles output = DoSSHCommand('%s "%s" %s' % (post_upload_command, path, file_list), user, host, port=port, ssh_key=ssh_key) File "/builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/build/upload.py", line 96, in DoSSHCommand output = f(cmdline, stderr=STDOUT).strip() File "/builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/python/redo/redo/__init__.py", line 185, in _retriable_wrapper **retry_kwargs) File "/builds/slave/tb-c-cen-lx-000000000000000000/build/mozilla/python/redo/redo/__init__.py", line 140, in retry return action(*args, **kwargs) File "/tools/python27/lib/python2.7/subprocess.py", line 573, in check_output raise CalledProcessError(retcode, cmd, output=output) subprocess.CalledProcessError: Command '['ssh', '-o', 'IdentityFile=~/.ssh/tbirdbld_dsa', 'tbirdbld@upload.tbirdbld.productdelivery.prod.mozaws.net', 'post_upload.py --tinderbox-builds-dir comm-central-linux -p thunderbird -i INFO: mock_mozilla.py version 1.0.3 starting...\nState Changed: init plugins\nINFO: selinux disabled\nState Changed: start\nState Changed: lock buildroot\nState Changed: shell\npython: can\'t open file \'build/config/printconfigsetting.py\': [Errno 2] No such file or directory\nState Changed: unlock buildroot --revision 2ef88c6fc965 --release-to-tinderbox-dated-builds --release-to-latest-tinderbox-builds "/tmp/tmp.g3J0ywUUmg/" "/tmp/tmp.g3J0ywUUmg//thunderbird-45.0a1.en-US.linux-i686.tar.bz2" "/tmp/tmp.g3J0ywUUmg/linux-i686/xpi/thunderbird-45.0a1.en-US.langpack.xpi" "/tmp/tmp.g3J0ywUUmg//mozharness.zip" "/tmp/tmp.g3J0ywUUmg//thunderbird-45.0a1.en-US.linux-i686.common.tests.zip" "/tmp/tmp.g3J0ywUUmg//thunderbird-45.0a1.en-US.linux-i686.cppunittest.tests.zip" "/tmp/tmp.g3J0ywUUmg//thunderbird-45.0a1.en-US.linux-i686.xpcshell.tests.zip" "/tmp/tmp.g3J0ywUUmg//thunderbird-45.0a1.en-US.linux-i686.mochitest.tests.zip" "/tmp/tmp.g3J0ywUUmg//thunderbird-45.0a1.en-US.linux-i686.talos.tests.zip" "/tmp/tmp.g3J0ywUUmg//thunderbird-45.0a1.en-US.linux-i686.reftest.tests.zip" "/tmp/tmp.g3J0ywUUmg//thunderbird-45.0a1.en-US.linux-i686.web-platform.tests.zip" "/tmp/tmp.g3J0ywUUmg//thunderbird-45.0a1.en-US.linux-i686.crashreporter-symbols.zip" "/tmp/tmp.g3J0ywUUmg//thunderbird-45.0a1.en-US.linux-i686.txt" "/tmp/tmp.g3J0ywUUmg//thunderbird-45.0a1.en-US.linux-i686.json" "/tmp/tmp.g3J0ywUUmg//thunderbird-45.0a1.en-US.linux-i686.mozinfo.json" "/tmp/tmp.g3J0ywUUmg//test_packages.json" "/tmp/tmp.g3J0ywUUmg//jsshell-linux-i686.zip" "/tmp/tmp.g3J0ywUUmg/host/bin/mar" "/tmp/tmp.g3J0ywUUmg/host/bin/mbsdiff" "/tmp/tmp.g3J0ywUUmg//thunderbird-45.0a1.en-US.linux-i686.checksums" "/tmp/tmp.g3J0ywUUmg//thunderbird-45.0a1.en-US.linux-i686.checksums.asc"']' returned non-zero exit status 1 make[1]: *** [upload] Error 1 make[1]: Leaving directory `/builds/slave/tb-c-cen-lx-000000000000000000/build/objdir-tb/mail/installer' make: *** [upload] Error 2 retry: Calling <function run_with_timeout at 0x7f82a89156e0> with args: (['make', 'upload'], 2460, None, None, False, True), kwargs: {}, attempt #2

(In reply to Sebastian H. [:aryx][:archaeopteryx] from comment #30) > (In reply to Rail Aliiev [:rail] from comment #29) > > backout http://hg.mozilla.org/build/buildbotcustom/rev/c807f6272371 > Thanks and sorry for the inconvenience. > > (In reply to aleth [:aleth] from comment #27) > > Was this responsible for the Linux opt failure > > https://treeherder.mozilla.org/#/jobs?repo=comm-central&filter- > > searchStr=34f285eafca829de3954e2be19cf5ba6bcbc5d3b ? > Yes. > > From the build which was only orange: > python build/config/printconfigsetting.py > build/objdir-tb/dist/bin/application.ini App BuildID > in dir /builds/slave/tb-c-cen-lx-000000000000000000/. (timeout 1200 secs) > > From the burning one: > mock_mozilla -r mozilla-centos6-x86_64 --cwd . --unpriv --shell > '/usr/bin/env python build/config/printconfigsetting.py > build/objdir-tb/dist/bin/application.ini App BuildID' > in dir /builds/slave/tb-c-cen-lx-000000000000000000/. (timeout 1200 secs) > > Any idea why it doesn't find the file now? Might be worth checking if this is related to Bug 1220501 (ie. premature printconfigsetting calls)

(In reply to aleth [:aleth] from comment #31) > (In reply to Sebastian H. [:aryx][:archaeopteryx] from comment #30) > > (In reply to Rail Aliiev [:rail] from comment #29) > > > backout http://hg.mozilla.org/build/buildbotcustom/rev/c807f6272371 > > Thanks and sorry for the inconvenience. > > > > ... > > Any idea why it doesn't find the file now? > > Might be worth checking if this is related to Bug 1220501 (ie. premature > printconfigsetting calls) were you able to determine if it is related? (Bug 1220501 seems to have stalled)

(In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment #32) > were you able to determine if it is related? (Bug 1220501 seems to have > stalled) Bug 1220501 is effectively fixed (just waiting on a double-checking needinfo before resolving)

Bug 1213715 was landed instead and fixed this issue, so while this approach might be better in the long run, this bug should be resolved.