We're not doing well https://archive.mozilla.org/pub/thunderbird/candidates/52.0-candidates/build1/logs/release-comm-esr52-win32_repack_1-bm72-build1-build0.txt.gz command: output: a15a29e4a07c Updated to revision a15a29e4a07c2d5182985adf47b54d177a33cf34. Warning: Preferring MOZILLA_REV (THUNDERBIRD_52_0_RELEASE) over passed ... File "c:\builds\moz2_slave\tb-rel-c-esr52-w32_rpk_1-00000\comm-esr52\mozilla\python\mozbuild\mozbuild\util.py", line 943, in __get__ setattr(instance, name, self.func(instance)) File "c:\builds\moz2_slave\tb-rel-c-esr52-w32_rpk_1-00000\comm-esr52\mozilla\python\mozbuild\mozbuild\configure\__init__.py", line 90, in result return self.func(*resolved_args) File "c:\builds\moz2_slave\tb-rel-c-esr52-w32_rpk_1-00000\comm-esr52\mozilla\python\mozbuild\mozbuild\configure\__init__.py", line 932, in wrapped return new_func(*args, **kwargs) File "c:/builds/moz2_slave/tb-rel-c-esr52-w32_rpk_1-00000/comm-esr52/mozilla/build/moz.configure/init.configure", line 119, in mozconfig mozconfig = loader.read_mozconfig(mozconfig, moz_build_app=current_project) File "c:\builds\moz2_slave\tb-rel-c-esr52-w32_rpk_1-00000\comm-esr52\mozilla\python\mozbuild\mozbuild\mozconfig.py", line 257, in read_mozconfig raise MozconfigLoadException(path, MOZCONFIG_BAD_EXIT_CODE, lines) mozbuild.mozconfig.MozconfigLoadException: Evaluation of your mozconfig exited with an error. This could be triggered by a command inside your mozconfig failing. Please change your mozconfig to not error and/or to catch errors in executed commands. *** Fix above errors and then restart with\ "c:/builds/moz2_slave/tb-rel-c-esr52-w32_rpk_1-00000/comm-esr52/mozmake.exe -f client.mk build" c:/builds/moz2_slave/tb-rel-c-esr52-w32_rpk_1-00000/comm-esr52/client.mk:360: recipe for target 'configure' failed mozmake.exe[1]: Leaving directory 'c:/builds/moz2_slave/tb-rel-c-esr52-w32_rpk_1-00000/comm-esr52' mozmake.exe[1]: *** [configure] Error 1 client.mk:217: recipe for target 'configure' failed mozmake.exe: *** [configure] Error 2 command: ERROR Traceback (most recent call last): File "c:\builds\moz2_slave\tb-rel-c-esr52-w32_rpk_1-00000\scripts\lib\python\util\commands.py", line 52, in run_cmd return subprocess.check_call(cmd, **kwargs) File "c:\mozilla-build\python27\lib\subprocess.py", line 542, in check_call raise CalledProcessError(retcode, cmd) CalledProcessError: Command '['c:\\mozilla-build\\python27\\python.exe', 'c:\\builds\\moz2_slave\\tb-rel-c-esr52-w32_rpk_1-00000\\comm-esr52/build/pymake/make.py', '-f', 'client.mk', 'configure']' returned non-zero exit status 2 command: END (194.68s elapsed) Traceback (most recent call last): File "c:/builds/moz2_slave/tb-rel-c-esr52-w32_rpk_1-00000/scripts/scripts/l10n/create-release-repacks.py", line 384, in <module> bucket_prefix=branchConfig['bucket_prefix'], File "c:/builds/moz2_slave/tb-rel-c-esr52-w32_rpk_1-00000/scripts/scripts/l10n/create-release-repacks.py", line 108, in createRepacks makeDirs, env, tooltoolManifest, tooltool_script, tooltool_urls) File "c:\builds\moz2_slave\tb-rel-c-esr52-w32_rpk_1-00000\scripts\lib\python\build\l10n.py", line 85, in l10nRepackPrep env=env) File "c:\builds\moz2_slave\tb-rel-c-esr52-w32_rpk_1-00000\scripts\lib\python\util\commands.py", line 52, in run_cmd return subprocess.check_call(cmd, **kwargs) File "c:\mozilla-build\python27\lib\subprocess.py", line 542, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['c:\\mozilla-build\\python27\\python.exe', 'c:\\builds\\moz2_slave\\tb-rel-c-esr52-w32_rpk_1-00000\\comm-esr52/build/pymake/make.py', '-f', 'client.mk', 'configure']' returned non-zero exit status 2 program finished with exit code 1 On 3/9/2017 4:57 PM, release--- via release-automation-notifications-thunderbird wrote: > repack_1/10 failed for Thunderbird 52.0 build1 on win32 > > Status: failure > > Full details are available at: > http://buildbot-master72.bb.releng.usw2.mozilla.com:8001/builders/release-comm-esr52-win32_repack_1%2F10/builds/0 > > -buildbot

Mac repacks did not fail. FWIW, windows nightly repacks also failing bug 1345422

further up, is this normal? mkdir -p 'c:/builds/moz2_slave/tb-rel-c-esr52-w32_rpk_1-00000/comm-esr52/obj-l10n/' client.mk:203: c:/builds/moz2_slave/tb-rel-c-esr52-w32_rpk_1-00000/comm-esr52/obj-l10n/.mozconfig.mk: No such file or directory

(blocking release)

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #2) > further up, is this normal? > > mkdir -p > 'c:/builds/moz2_slave/tb-rel-c-esr52-w32_rpk_1-00000/comm-esr52/obj-l10n/' > client.mk:203: > c:/builds/moz2_slave/tb-rel-c-esr52-w32_rpk_1-00000/comm-esr52/obj-l10n/. > mozconfig.mk: No such file or directory That's also in the log for the repack of 52.0b4: https://archive.mozilla.org/pub/thunderbird/candidates/52.0b4-candidates/build1/logs/release-comm-beta-win32_repack_1-bm91-build1-build3.txt.gz The first line missing in the release log compare to the one for 52.0b4 is > TOOLTOOL_DIR=/c/builds/moz2_slave/tb-rel-c-beta-w32_rpk_1-000000/comm-beta Next significant different seems to be: beta: >python2.7 c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_1-000000/comm-beta/client.py checkout --hg-options='--time' --hgtool=../tools/buildfarm/utils/hgtool.py --hgtool1=../scripts/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --tinderbox-print release: >python2.7 c:/builds/moz2_slave/tb-rel-c-esr52-w32_rpk_1-00000/comm-esr52/client.py checkout --hg-options='--time' --hgtool=../tools/buildfarm/utils/hgtool.py --hgtool1=../scripts/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --tinderbox-print --mozilla-repo=https://hg.mozilla.org/releases/mozilla-esr52 --mozilla-rev=THUNDERBIRD_52_VERBRANCH See https://dxr.mozilla.org/comm-central/rev/d03f6d1f69374c85d4939d0c7cb92021614600f4/client.py#445 Down the line: >Warning: Preferring MOZILLA_REV (THUNDERBIRD_52_0_RELEASE) over passed revision (THUNDERBIRD_52_VERBRANCH)

This looks like the ESR version of bug 1286440. Reading bug 1286440 comment 100 it is still not clear to me why we are not consistent with --disable-compile-environment. I propose we do the same buildbot-configs fix as in that bug, plus add --disable-compile-environment to all win/linux mozconfigs. aleth, can you confirm this is ok?

Here is a patch for the buildbot-configs part. I've also taken the liberty to copy the missing staging config from the esr45 files, if this is not needed anymore or the simple s/45/52/ doesn't quite do it please let me know.

Here are the mozconfig changes. I created the patch with a comm-esr52 repo, but we should probably push this to all repos once we have confirmed it works for 52.

Comment on attachment 8848267 [details] [diff] [review] mozconfig changes - v1 Review of attachment 8848267 [details] [diff] [review]: ----------------------------------------------------------------- Jörg, if you have insight into this issue and aleth is not available I am happy to take your review.

Comment on attachment 8848267 [details] [diff] [review] mozconfig changes - v1 (Greetings from Catalonia where my phone/internet still work ;-)) I have dug around in bug 1322402 disabling universal builds, and I was surprised about the inconsistencies with respect to this flag. Reading through bug 1286440 comment #77 downwards and also reading bug 301197 (which first introduced the flag) I think the proposed change here is right, but we should make sure we fix all branches once TB 52 builds fine (I can do that). Besides, it's not such a drastic change. So r=jorgk when you get the buildbot part approved. Thanks for the patches, BTW.

Comment on attachment 8848266 [details] [diff] [review] buildbot-configs changes - v1 Review of attachment 8848266 [details] [diff] [review]: ----------------------------------------------------------------- I'm not a buildbot peer, sorry.

:Fallen, might it work if you include the following line to l10n-mozconfig: . "$topsrcdir/build/mozconfig.win-common"

ewong, the patches I created were based on reading the linked bug, I wasn't actually able to verify it locally. I don't have a non-mac machine set up for testing at the moment, so I also cannot test if including mozconfig.win-common helps. Can you clarify why you believe that would fix it? Would this be instead of what I proposed, or in addition?

(In reply to Philipp Kewisch [:Fallen] from comment #12) > ewong, the patches I created were based on reading the linked bug, I wasn't > actually able to verify it locally. I don't have a non-mac machine set up > for testing at the moment, so I also cannot test if including > mozconfig.win-common helps. Can you clarify why you believe that would fix > it? Would this be instead of what I proposed, or in addition? Hi Fallen, I apologize for adding my $0.02 when in fact, the issue is the basic topsrcdir != expected topsrcdir. I've tried replicating the repack on a loaner; but the way the repacks are done, it isn't the easiest thing to do namely because of all the processes being lumped under the create-release-repacks.py script. That said, while I'm still having some problems getting past a certain point, the issue I think with the current MOZCONFIG bustage is that you'll need your buildbot-config patch. It's another one of those topsrcdir isn't the right topsrcdir that is expected in the mozconfig loading. Particularly this line: +releaseConfig['extra_signing_env'] = {'TOOLTOOL_DIR': '%(basedir)s/comm-esr52'} As for the mozconfig patch, my suggestion was based on ignorance. Mind you, with our buildbot config patch, I don't think the --disable-compile-environment is necessary; but now that I've looked at how TB does its repacks, it could be a different thing. So my humblest apologies. particularly it was (at least in my loaner env) trying to access the VS2015u3 path. 'c:/builds/slave/tbtmp/comm-esr52/mozilla/build/win32/mozconfig.vs2015-win64: line 6: cd: /c/builds/slave/tbtmp/comm-esr52/mozilla/vs2015u3: No such file or directory' I believe that it's sufficient with the buildbot-configs patch.

ewong, no worries at all! I appreciate your input very much, I just had trouble understanding why it was needed and asked for clarification. I'm sorry if this came across blunt. If the disable-compile-environment patch doesn't hurt I think it would be good for consistency. I hope the buildbot-configs changes will work as expected. If you are successful testing with both patches applied that would be very helpful!

https://hg.mozilla.org/build/buildbot-configs/rev/5fc070519675cee83236ed49e0cf8013ce712e40 (default branch)

In production: https://hg.mozilla.org/build/buildbot-configs/rev/5fc070519675

In production: https://hg.mozilla.org/build/buildbot-configs/rev/5fc070519675

Landing this on all branches, here comes C-C: https://hg.mozilla.org/comm-central/rev/0d9f3ed365b7912b6e57fcd21ad11de536f3384c

C-A (TB 54): https://hg.mozilla.org/releases/comm-aurora/rev/48de8f8524b74f4867bcc5b647e4c72e0e10291a C-B (TB 53): https://hg.mozilla.org/releases/comm-beta/rev/32087f2e8d4e72e1765d0d9a5dd9c1a6adc466bc

ESR 52: https://hg.mozilla.org/releases/comm-esr52/rev/a730476809a84b349be34db4b59013288ebc2803

In production: https://hg.mozilla.org/build/buildbot-configs/rev/5fc070519675

Fallen, I assume you no longer aleth feedback. If that is incorrect please flag someone else.