tracking wiki: https://wiki.mozilla.org/Releases/Firefox_41.0b1/BuildNotes

release-runner failed on first attempt of beta1 build1 post ship-it: 2015-08-11 14:30:20,302 - INFO - command: output: 2015-08-11 14:30:20,303 - INFO - e928b851d8d9 2015-08-11 14:30:20,304 - INFO - mark as failed Fennec-41.0b1-build1 2015-08-11 14:30:20,304 - DEBUG - Request to https://ship-it.mozilla.org/releases/Fennec-41.0b1-build1 2015-08-11 14:30:20,305 - DEBUG - Data sent: {'ready': False, 'status': "Failed: KeyError('nextVersion',)", 'csrf_token': '20150811220015##a6621c41e288d1dfd091141212e7a62fb439a0e6'} 2015-08-11 14:30:20,305 - DEBUG - attempt 1/5 2015-08-11 14:30:20,306 - DEBUG - retry: Calling <function _req at 0x1205578> with args: (), kwargs: {}, attempt #1 2015-08-11 14:30:20,356 - DEBUG - "POST /releases/Fennec-41.0b1-build1 HTTP/1.1" 200 0 2015-08-11 14:30:20,357 - INFO - mark as failed Firefox-41.0b1-build1 2015-08-11 14:30:20,357 - DEBUG - Request to https://ship-it.mozilla.org/releases/Firefox-41.0b1-build1 2015-08-11 14:30:20,357 - DEBUG - Data sent: {'ready': False, 'status': "Failed: KeyError('nextVersion',)", 'csrf_token': '20150811220015##a6621c41e288d1dfd091141212e7a62fb439a0e6'} 2015-08-11 14:30:20,357 - DEBUG - attempt 1/5 2015-08-11 14:30:20,358 - DEBUG - retry: Calling <function _req at 0x138d488> with args: (), kwargs: {}, attempt #1 2015-08-11 14:30:20,402 - DEBUG - "POST /releases/Firefox-41.0b1-build1 HTTP/1.1" 200 0 2015-08-11 14:30:20,403 - DEBUG - Releasing lock: /builds/releaserunner/tools/buildfarm/release/.release-runner.lock Traceback (most recent call last): File "release-runner.py", line 482, in <module> main(options) File "release-runner.py", line 404, in main ssh_username=hg_username, ssh_key=hg_ssh_key) File "/builds/releaserunner/tools/lib/python/util/hg.py", line 634, in apply_and_push changer(localrepo, 1) File "release-runner.py", line 385, in process_configs sendMailRD(smtp_server, notify_from, "%s/mozilla/%s" % (configs_workdir, cfgFile), release) File "release-runner.py", line 215, in sendMailRD release_config = readReleaseConfig(cfgFile) File "/builds/releaserunner/tools/lib/python/release/info.py", line 69, in readReleaseConfig return readConfig(configfile, keys=['releaseConfig'], required=required) File "/builds/releaserunner/tools/lib/python/release/info.py", line 87, in readConfig execfile(configfile, c) File "buildbot-configs/mozilla/release-fennec-mozilla-beta.py", line 46, in <module> 'nextVersion': releaseConfig['nextVersion'] KeyError: 'nextVersion' The full log is available on buildbot-master81 in /builds/releaserunner/release-runner.log I'll sleep for 60 seconds before retry nthomas is landing: https://bugzilla.mozilla.org/attachment.cgi?id=8646602 and we are going to reconfig and start again

how do I test this?

I tried the patch and it was still, which I think is due to the comparitor tracking which mozconfig each line is in. So here's a counter-offer which passes for me.

Comment on attachment 8646700 [details] [diff] [review] [tools] Fix whitelist for 41.0b1 http://hg.mozilla.org/build/tools/rev/87d69be3cd53

bouncer submitter errors for fennec: 18:53:12 INFO - Checking if Fennec-41.0b1 already exists 18:53:12 INFO - retry: Calling _api_call with args: ('product_show?product=Fennec-41.0b1', None), kwargs: {}, attempt #1 18:53:12 INFO - Submitting to https://bounceradmin.mozilla.com/api/product_show?product=Fennec-41.0b1 18:53:13 INFO - Server response 18:53:13 INFO - <?xml version="1.0" encoding="utf-8"?><products/> 18:53:13 INFO - Products found: 0 18:53:13 INFO - Adding installer... 18:53:13 INFO - retry: Calling _api_call with args: ('product_add/', {'languages': ['ach', 'af', 'ak', 'ar', 'an', 'as', 'az', 'be', 'bn-IN', 'br', 'ca', 'cak', 'cs', 'cy', 'da', 'de', 'dsb', 'el', 'en-GB', 'en-ZA', 'eo', 'es-AR', 'es-ES', 'es-MX', 'et', 'eu', 'fa', 'ff', 'fi', 'fr', 'fy-NL', 'ga-IE', 'gd', 'gl', 'gu-IN', 'he', 'hi-IN', 'hr', 'hsb', 'hu', 'hy-AM', 'id', 'is', 'it', 'ja-JP-mac', 'ja', 'kk', 'km', 'kn', 'ko', 'lg', 'lij', 'lt', 'lv', 'mai', 'ml', 'mr', 'ms', 'my', 'nb-NO', 'nl', 'nn-NO', 'nso', 'or', 'pa-IN', 'pl', 'pt-BR', 'pt-PT', 'ro', 'ru', 'sk', 'sl', 'son', 'sq', 'sr', 'sv-SE', 'sw', 'ta', 'te', 'th', 'tr', 'tsz', 'uk', 'uz', 'vi', 'wo', 'xh', 'zh-CN', 'zh-TW', 'zu'], 'product': 'Fennec-41.0b1'}), kwargs: {}, attempt #1 18:53:13 INFO - POST data: languages=ach&languages=af&languages=ak&languages=ar&languages=an&languages=as&languages=az&languages=be&languages=bn-IN&languages=br&languages=ca&languages=cak&languages=cs&languages=cy&languages=da&languages=de&languages=dsb&languages=el&languages=en-GB&languages=en-ZA&languages=eo&languages=es-AR&languages=es-ES&languages=es-MX&languages=et&languages=eu&languages=fa&languages=ff&languages=fi&languages=fr&languages=fy-NL&languages=ga-IE&languages=gd&languages=gl&languages=gu-IN&languages=he&languages=hi-IN&languages=hr&languages=hsb&languages=hu&languages=hy-AM&languages=id&languages=is&languages=it&languages=ja-JP-mac&languages=ja&languages=kk&languages=km&languages=kn&languages=ko&languages=lg&languages=lij&languages=lt&languages=lv&languages=mai&languages=ml&languages=mr&languages=ms&languages=my&languages=nb-NO&languages=nl&languages=nn-NO&languages=nso&languages=or&languages=pa-IN&languages=pl&languages=pt-BR&languages=pt-PT&languages=ro&languages=ru&languages=sk&languages=sl&languages=son&languages=sq&languages=sr&languages=sv-SE&languages=sw&languages=ta&languages=te&languages=th&languages=tr&languages=tsz&languages=uk&languages=uz&languages=vi&languages=wo&languages=xh&languages=zh-CN&languages=zh-TW&languages=zu&product=Fennec-41.0b1 18:53:13 INFO - Submitting to https://bounceradmin.mozilla.com/api/product_add/ 18:53:13 WARNING - Cannot access https://bounceradmin.mozilla.com/api/product_add/ 18:53:13 WARNING - Returned page source: 18:53:13 WARNING - <?xml version="1.0" encoding="utf-8"?><error number="103">invalid language code(s)</error> this seems to be because we have new locales that are not in bouncer: https://hg.mozilla.org/releases/mozilla-beta/file/FENNEC_41_0b1_RELEASE/mobile/android/locales/all-locales ^ most recent additions tsz and cak: https://hg.mozilla.org/releases/mozilla-beta/rev/432e6a592cf9 I've filed https://bugzil.la/1193613 to get these locales put into bouncer

looks like all desktop firefox repacks are failing e.g. make[4]: Leaving directory `/builds/slave/rel-m-beta-l64_rpk_9-000000000/mozilla-beta/obj-l10n/config/external/sqlite' make[4]: *** No rule to make target `../../../db/sqlite3/src/libdb_sqlite3_src.a', needed by `libmozsqlite3.so'. Stop. make[3]: *** [config/external/sqlite/target] Error 2 make[3]: Leaving directory `/builds/slave/rel-m-beta-l64_rpk_9-000000000/mozilla-beta/obj-l10n' make[2]: *** [compile] Error 2 make[2]: Leaving directory `/builds/slave/rel-m-beta-l64_rpk_9-000000000/mozilla-beta/obj-l10n' make[1]: *** [compile] Error 2 make[1]: Leaving directory `/builds/slave/rel-m-beta-l64_rpk_9-000000000/mozilla-beta/obj-l10n/config' make: *** [default] Error 2 command: ERROR Traceback (most recent call last): File "/builds/slave/rel-m-beta-l64_rpk_9-000000000/scripts/lib/python/util/commands.py", line 51, in run_cmd return subprocess.check_call(cmd, **kwargs) File "/usr/lib64/python2.6/subprocess.py", line 502, in check_call raise CalledProcessError(retcode, cmd) CalledProcessError: Command '['make']' returned non-zero exit status 2 command: END (1.23s elapsed) Traceback (most recent call last): File "/builds/slave/rel-m-beta-l64_rpk_9-000000000/scripts/scripts/l10n/create-release-repacks.py", line 377, in <module> mozillaSrcDir=mozillaSrcDir File "/builds/slave/rel-m-beta-l64_rpk_9-000000000/scripts/scripts/l10n/create-release-repacks.py", line 104, in createRepacks makeDirs, env, tooltoolManifest, tooltool_script, tooltool_urls) File "/builds/slave/rel-m-beta-l64_rpk_9-000000000/scripts/lib/python/build/l10n.py", line 96, in l10nRepackPrep env=env) File "/builds/slave/rel-m-beta-l64_rpk_9-000000000/scripts/lib/python/util/commands.py", line 51, in run_cmd return subprocess.check_call(cmd, **kwargs) File "/usr/lib64/python2.6/subprocess.py", line 502, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['make']' returned non-zero exit status 2 investigating

also of note, xulrunner win64 failed: http://buildbot-master82.bb.releng.scl3.mozilla.com:8001/builders/release-mozilla-beta-xulrunner_win64_build/builds/10

repack 'make config' error in comment 7 was patched via 1193620 force builds of the repacks yielded a new error: macosx64 example fail: + rsync -a /builds/slave/rel-m-beta-m64_rpk_2-000000000/mozilla-beta/obj-l10n/dist/unpack.tmp/Nightly.app firefox rsync: link_stat "/builds/slave/rel-m-beta-m64_rpk_2-000000000/mozilla-beta/obj-l10n/dist/unpack.tmp/Nightly.app" failed: No such file or directory (2) rsync error: some files could not be transferred (code 23) at /SourceCache/rsync/rsync-42/rsync/main.c(992) [sender=2.6.9] make: *** [/builds/slave/rel-m-beta-m64_rpk_2-000000000/mozilla-beta/obj-l10n/dist/l10n-stage/firefox/Nightly.app/Contents/Resources] Error 23 command: ERROR Traceback (most recent call last): File "/builds/slave/rel-m-beta-m64_rpk_2-000000000/scripts/lib/python/util/commands.py", line 51, in run_cmd return subprocess.check_call(cmd, **kwargs) File "/tools/python27/lib/python2.7/subprocess.py", line 511, in check_call raise CalledProcessError(retcode, cmd) CalledProcessError: Command '['make', 'installers-bn-BD']' returned non-zero exit status 2 command: END (32.21s elapsed) Traceback (most recent call last): File "/builds/slave/rel-m-beta-m64_rpk_2-000000000/scripts/scripts/l10n/create-release-repacks.py", line 131, in createRepacks mozillaDir=mozillaDir, mozillaSrcDir=mozillaSrcDir) File "/builds/slave/rel-m-beta-m64_rpk_2-000000000/scripts/lib/python/build/l10n.py", line 191, in repackLocale run_cmd(make + ["installers-%s" % locale], cwd=localeSrcDir, env=env) File "/builds/slave/rel-m-beta-m64_rpk_2-000000000/scripts/lib/python/util/commands.py", line 51, in run_cmd return subprocess.check_call(cmd, **kwargs) File "/tools/python27/lib/python2.7/subprocess.py", line 511, in check_call raise CalledProcessError(retcode, cmd) CalledProcessError: Command '['make', 'installers-bn-BD']' returned non-zero exit status 2 linux64 example fail: command: START command: perl ../../../tools/update-packaging/unwrap_full_update.pl /builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/dist/update/linux-x86_64/bg/firefox-41.0b1.complete.mar command: cwd: /builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/dist/current command: env: {'MOZ_MAKE_COMPLETE_MAR': '1', 'AB_CD': 'bg', 'MOZ_SIGN_CMD': 'python /builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/release/signing/signtool.py --cachedir /builds/slave/rel-m-beta-l64_rpk_1-000000000/signing_cache -t /builds/slave/rel-m-beta-l64_rpk_1-000000000/token -n /builds/slave/rel-m-beta-l64_rpk_1-000000000/nonce -c /builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/release/signing/host.cert -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing5.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:b2gmar:emevoucher:signing6.srv.releng.scl3.mozilla.com:9120 -H dmg:mac-signing2.srv.releng.scl3.mozilla.com:9120 -H dmg:mac-signing3.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9120', 'UPLOAD_SSH_KEY': '~/.ssh/ffxbld_rsa', 'LOCALE_MERGEDIR': '/builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/browser/locales/merged', 'COMM_REV': 'FIREFOX_41_0b1_RELEASE', 'MOZ_OBJDIR': 'obj-l10n', 'MAR': '/builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/dist/host/bin/mar', 'UPLOAD_TO_TEMP': '1', 'LD_LIBRARY_PATH': '/tools/gcc-4.3.3/installed/lib64', 'ZIP_IN': '/builds/slave/rel-m-beta-l64_rpk_1-000000000/firefox.tar.bz2', 'UPLOAD_USER': 'ffxbld', 'MBSDIFF': '/builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/dist/host/bin/mbsdiff', 'UPLOAD_HOST': 'stage.mozilla.org', 'POST_UPLOAD_CMD': 'post_upload.py -p firefox -n 1 -v 41.0b1 --release-to-candidates-dir --signed', 'MOZILLA_REV': 'FIREFOX_41_0b1_RELEASE', 'MOZ_PKG_PRETTYNAMES': '1', 'MOZ_PKG_VERSION': '41.0b1'} command: output: ERROR: could not open file in mar_open() /builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/dist/update/linux-x86_64/bg/firefox-41.0b1.complete.mar: No such file or directory Couldn't run "/builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/dist/host/bin/mar" -t at ../../../tools/update-packaging/unwrap_full_update.pl line 51. command: ERROR Traceback (most recent call last): File "/builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/lib/python/util/commands.py", line 51, in run_cmd return subprocess.check_call(cmd, **kwargs) File "/usr/lib64/python2.6/subprocess.py", line 502, in check_call raise CalledProcessError(retcode, cmd) CalledProcessError: Command '['perl', '../../../tools/update-packaging/unwrap_full_update.pl', '/builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/dist/update/linux-x86_64/bg/firefox-41.0b1.complete.mar']' returned non-zero exit status 255 command: END (0.01s elapsed) Traceback (most recent call last): File "/builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/scripts/l10n/create-release-repacks.py", line 131, in createRepacks mozillaDir=mozillaDir, mozillaSrcDir=mozillaSrcDir) File "/builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/lib/python/build/l10n.py", line 197, in repackLocale cwd=path.join(nativeDistDir, 'current'), env=env) File "/builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/lib/python/util/commands.py", line 51, in run_cmd return subprocess.check_call(cmd, **kwargs) File "/usr/lib64/python2.6/subprocess.py", line 502, in check_call raise CalledProcessError(retcode, cmd) CalledProcessError: Command '['perl', '../../../tools/update-packaging/unwrap_full_update.pl', '/builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/dist/update/linux-x86_64/bg/firefox-41.0b1.complete.mar']' returned non-zero exit status 255 The following tracebacks were detected during repacks: ach: Traceback (most recent call last): File "/builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/scripts/l10n/create-release-repacks.py", line 131, in createRepacks mozillaDir=mozillaDir, mozillaSrcDir=mozillaSrcDir) File "/builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/lib/python/build/l10n.py", line 197, in repackLocale cwd=path.join(nativeDistDir, 'current'), env=env) File "/builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/lib/python/util/commands.py", line 51, in run_cmd return subprocess.check_call(cmd, **kwargs) File "/usr/lib64/python2.6/subprocess.py", line 502, in check_call raise CalledProcessError(retcode, cmd) CalledProcessError: Command '['perl', '../../../tools/update-packaging/unwrap_full_update.pl', '/builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/dist/update/linux-x86_64/ach/firefox-41.0b1.complete.mar']' returned non-zero exit status 255 I am going to have to pick these up tomorrow.

Fix for l10n branding: https://hg.mozilla.org/releases/mozilla-beta/rev/41eefdf2aed4 (relbranch) https://hg.mozilla.org/releases/mozilla-beta/rev/aabfbc74aa01 (relbranch move tag) https://hg.mozilla.org/releases/mozilla-beta/rev/3e5094441819 (default) This gets us back to the state we had in 40.0 betas - no branding config in l10n-mozconfig, and the repack script adds --enable-official-branding at runtime (!!).

(In reply to Nick Thomas [:nthomas] from comment #11) > Fix for l10n branding: > https://hg.mozilla.org/releases/mozilla-beta/rev/41eefdf2aed4 (relbranch) > https://hg.mozilla.org/releases/mozilla-beta/rev/aabfbc74aa01 (relbranch > move tag) > https://hg.mozilla.org/releases/mozilla-beta/rev/3e5094441819 (default) > > This gets us back to the state we had in 40.0 betas - no branding config in > l10n-mozconfig, and the repack script adds --enable-official-branding at > runtime (!!). all repacks completed successfully. thank you for your help here!

firefox desktop has been pushed to beta channel, signedoff the beta channel and post release steps complete. however, xulrunner post release steps failed: ERROR: push to mirrors directory, /pub/mozilla.org/xulrunner/releases/41.0b1/, does not exist on stage.mozilla.org ERROR: Did you push to mirrors before running post release? Traceback (most recent call last): File "/builds/slave/rel-m-beta-xr_pr-0000000000000/scripts/scripts/release/stage-tasks.py", line 390, in <module> target=ftpSymlinkName) File "/builds/slave/rel-m-beta-xr_pr-0000000000000/scripts/scripts/release/stage-tasks.py", line 192, in updateSymlink sshKey=stageSshKey) File "/builds/slave/rel-m-beta-xr_pr-0000000000000/scripts/lib/python/util/commands.py", line 75, in run_remote_cmd return run_cmd(cmd_prefix + cmd, **kwargs) File "/builds/slave/rel-m-beta-xr_pr-0000000000000/scripts/lib/python/util/commands.py", line 51, in run_cmd return subprocess.check_call(cmd, **kwargs) File "/tools/python27/lib/python2.7/subprocess.py", line 511, in check_call raise CalledProcessError(retcode, cmd) this is likely due to https://bugzilla.mozilla.org/show_bug.cgi?id=1188493#c9

fennec mobile has also been pushed to the play store, waiting for QE signoff before running post release steps

still planning bouncer solution for single locale fennec fix. tracking that effort in: https://bugzilla.mozilla.org/show_bug.cgi?id=1193613

(In reply to Jordan Lund (:jlund) from comment #14) > fennec mobile has also been pushed to the play store, waiting for QE signoff > before running post release steps ritu helped track this down. the sign off fell through the cracks due to other stuff going on. I triggered the post-release steps for fennec and it failed on the last step because of https://bugzilla.mozilla.org/show_bug.cgi?id=1193613 and comment 5 Like the factory in comment 5, I am not going to fix it. the real fix is fixing 1193613 for future betas

that's a wrap for beta1 starting beta2 tracking: https://wiki.mozilla.org/Releases/Firefox_41.0b2/BuildNotes ship-it for beta2 started and failed due to l10n changesets missing. A new one is about to start

[release] Firefox 41.0b2 build1: failed at repack_9/10 on win64 was a slow slave and timed out in configure step. nick re-ran this builder

r+ by bhearsum over ff hello Firefox 41.0b2 build1: failed at firefox_beta_updates this may be something we need to do on every beta2 as patcher configs mangled beta and release status-updates. this verify step or at least this assert will probably go away (https://bugzil.la/1191074) I am going to land, bump the two beta2 tags and rebuild this builder

Comment on attachment 8649409 [details] [diff] [review] 150818_release_41_update_patcher_configs_beta2.patch landed and bumped release tags: remote: https://hg.mozilla.org/build/tools/rev/e0a6768fcf2b remote: https://hg.mozilla.org/build/tools/rev/c047c7f595be

beta2 complete. beta3 submitted to release-kickoff(ship-it): * win32 build failed because of slow win slave trying to remove old buildirs (retriggered) * win{32,64}_xul failed because of https://bugzil.la/1170212 (win32_xul didn't even make it that far because it too timed out trying to remove old builddirs) I am going to ignore the xulrunner failures and file a dep bug to fix for beta4

Pre-emptive strike so that the updates builder doesn't fail for 41.0b3 build1 - 2nd hunk of attachment 8649409 [details] [diff] [review] again: https://hg.mozilla.org/build/tools/rev/ab2eb5392456

[release] Firefox 41.0b3 build1: failed at update_verify_beta_4/6 on win32 ^ hung for 8 hours. cancelled and re-triggered because it was due to windows getting in a hung state of a tree removeal. this blocked "[release] Firefox 41.0b3 build1: ready to ship to the beta channel" email. also of note, dozens of the ui_update_verify jobs were hung for over 8 hours too. I've canceled them as they are not blocking.

14:01 <•nthomas> coop: did you want to turn off auotmated partner repacks for 41.0 build2 ? 14:01 <•coop|mtg> nthomas: yes, i think we should 14:04 <jlund> do we want to turn this off for both Fennec and Firefox? 14:04 <•nthomas> we were talking about it a meeting, and c.oop wants to switch over to the github based system 14:04 <•nthomas> it's on for Fennec ? 14:04 <•nthomas> we'll generate 41.0 bulid2 partner repacks manually 14:14 <•nthomas> but anyway, we are just looking at firefox release right now 14:14 <jlund> okay. I'll revert my fennec parts in the patch

Comment on attachment 8662049 [details] [diff] [review] 150916_ff_41_disable_doPartnerRepacks-bbot-cfgs.patch http://hg.mozilla.org/build/buildbot-configs/rev/7cbf830b20f1 and live in prod

started RC build3 (https://wiki.mozilla.org/index.php?title=Releases/Firefox_41.0) currently failing: beta_final_verification failed for Firefox 41.0 build3 logs: first fail: http://buildbot-master73.bb.releng.usw2.mozilla.com:8001/builders/release-mozilla-release-beta_final_verification/builds/1 reason: It is getting 404s from download links from obtained balrog xml. however, if I take one of those failed verify urls, e.g. the complete mar in here: https://aus4.mozilla.org/update/3/Firefox/30.0/20140529161749/WINNT_x86-msvc/sv-SE/beta-cdntest/default/default/default/update.xml?force=1 it downloads fine. Because of that, I re-triggered a second build.. second fail(after rebuild): http://buildbot-master73.bb.releng.usw2.mozilla.com:8001/builders/release-mozilla-release-beta_final_verification/builds/2 this failed much faster. the logs suggest missing xml but again I can download the xml: log snippet: Thu Sep 17 23:49:55 PDT 2015: FAILURE 1: Update xml file not available Thu Sep 17 23:49:55 PDT 2015: Thu Sep 17 23:49:55 PDT 2015: Download url: https://aus4.mozilla.org/update/3/Firefox/39.0/20150609130336/Linux_x86-gcc3/en-ZA/beta-cdntest/default/default/default/update.xml?force=1 Thu Sep 17 23:49:55 PDT 2015: Curl returned exit code: 52 ben: would you be able to have a peek at this when you come online or 302 it to someone?

(In reply to Jordan Lund (:jlund) from comment #26) > started RC build3 > (https://wiki.mozilla.org/index.php?title=Releases/Firefox_41.0) > > currently failing: beta_final_verification failed for Firefox 41.0 build3 > > logs: > > first fail: > http://buildbot-master73.bb.releng.usw2.mozilla.com:8001/builders/release- > mozilla-release-beta_final_verification/builds/1 > > reason: > It is getting 404s from download links from obtained balrog xml. however, > if I take one of those failed verify urls, e.g. the complete mar in here: > https://aus4.mozilla.org/update/3/Firefox/30.0/20140529161749/WINNT_x86-msvc/ > sv-SE/beta-cdntest/default/default/default/update.xml?force=1 > > it downloads fine. Because of that, I re-triggered a second build.. > > > second fail(after rebuild): > http://buildbot-master73.bb.releng.usw2.mozilla.com:8001/builders/release- > mozilla-release-beta_final_verification/builds/2 > > this failed much faster. the logs suggest missing xml but again I can > download the xml: > log snippet: > Thu Sep 17 23:49:55 PDT 2015: FAILURE 1: Update xml file not available > Thu Sep 17 23:49:55 PDT 2015: > Thu Sep 17 23:49:55 PDT 2015: Download url: > https://aus4.mozilla.org/update/3/Firefox/39.0/20150609130336/Linux_x86-gcc3/ > en-ZA/beta-cdntest/default/default/default/update.xml?force=1 > Thu Sep 17 23:49:55 PDT 2015: Curl returned exit code: 52 > > ben: would you be able to have a peek at this when you come online or 302 it > to someone? I think these were both intermittent issues. The first one might be related to the move to Cloudfront, as Nick suggested in e-mail, and the second one is probably related to https://bugzilla.mozilla.org/show_bug.cgi?id=1199721. A third run fully passed.

bc we disabled partner repacks for 41 but still tried to do post release partner steps: cp: cannot stat `/pub/mozilla.org/firefox/candidates/41.0-candidates/build3/partner-repacks/bing/mac/en-US/Firefox 41.0.dmg': No such file or directory command: ERROR