For "Unable to download config file(s) or config files are missing from repo". More here[1]. [1]: https://taskcluster-artifacts.net/Qqs8HC4VTbySKXjBbcwJPQ/0/public/logs/live_backing.log

I took a look into this. The update verify config generation failed upstream of final verify, which means we also didn't run update verify from 60.0 to 60.0.1. The config gen failed because it didn't find 60.0esr, from [1]: INFO - retry: Calling _urlopen with args: (), kwargs: {'url': 'https://product-details.mozilla.org/1.0/firefox.json'}, attempt #1 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-9.0b6 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-9.0b5 .... INFO - Skipping release that's behind the last watershed: firefox-7.0 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-61.0b5 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-61.0b4 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-61.0b3 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-60.0b9 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-60.0b8 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-60.0b7 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-60.0b6 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-60.0b5 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-60.0b4 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-60.0b3 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-60.0b16 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-60.0b15 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-60.0b14 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-60.0b13 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-60.0b12 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-60.0b11 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-60.0b10 INFO - Skipping release that's behind the last watershed: firefox-60.0 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-6.0b5 INFO - Skipping release whose version doesn't match any include_version pattern: firefox-6.0b4 ... FATAL - Didn't find any update paths, cannot continue FATAL - Running post_fatal callback... FATAL - Exiting -1 INFO - [mozharness: 2018-05-16 06:07:15.612524Z] Finished gather-info step (failed) So this comes back to ship-it's product-details code. [1] https://taskcluster-artifacts.net/cozae-D_QRizRojtryvL0A/0/public/logs/live_backing.log)

Merged and deployed to ship-it: https://github.com/mozilla-releng/ship-it/commit/90794c58730ec20d00a184399f12b04cadc45484 Munged up a copy of the 60.0.1esr graph for verify config and u.v. tasks at https://tools.taskcluster.net/groups/TiiSjWAFRluTqpVupwoZUQ but it's still failing. Last watershed is set to 60.0esr so we get [task 2018-05-24T00:12:26.283Z] 00:12:26 INFO - Skipping release that's behind the last watershed: firefox-60.0esr Thinks 60.0.1 should be included but fails to look up shipped-locales: [task 2018-05-24T00:12:26.285Z] 00:12:26 INFO - retry: Calling _urlopen with args: (), kwargs: {'url': 'https://archive.mozilla.org/pub/firefox/candidates/60.0.1-candidates/build1/win64_info.txt'}, attempt #1 [task 2018-05-24T00:12:26.866Z] 00:12:26 INFO - retry: Calling _urlopen with args: (), kwargs: {'url': 'https://hg.mozilla.org/releases/mozilla-esr60/raw-file/FIREFOX_60_0_1_RELEASE/browser/locales/shipped-locales'}, attempt #1 [task 2018-05-24T00:12:27.106Z] 00:12:27 INFO - retry: attempt #1 caught HTTPError exception: HTTP Error 404: Not Found .... [run out of retries] ... [task 2018-05-24T00:24:28.935Z] 00:24:28 INFO - [mozharness: 2018-05-24 00:24:28.935426Z] Finished gather-info step (failed) Here's the call to the config creation script: [task 2018-05-24T00:12:25.755Z] 00:12:25 INFO - Run as testing/mozharness/scripts/release/update-verify-config-creator.py --config internal_pypi.py --product firefox --stage-product firefox --app-name browser --platform win64 --to-version 60.0.1esr --to-app-version 60.0.1 --to-build-number 1 --to-buildid 20180516032417 --to-revision efbcc205a0d3562691bf62bb0cb55bb7e891e8ff --output-file update-verify.cfg --repo-path releases/mozilla-esr60 --partial-version 60.0esr --updater-platform linux-x86_64 --channel default --archive-prefix https://archive.mozilla.org/pub --aus-server https://aus5.mozilla.org --include-version ^\d+\.\d+(\.\d+)?$ --last-watershed 60.0esr The default channel differs from what we do on other branches.

The Firefox 61.0b8 release tasks tried to include 60.0esr as well, so I've backed out the change in ship-it until I can up with a solution for the config generator and input args that works across all branches: https://github.com/mozilla-releng/ship-it/commit/0097b55e7faae10ee2356a406e22ed557dfc5e71

This patch solves a few issues with ESR60 * we should include firefox-60.0esr in firefox.json in product-details, so that we can generate update verify configs including it * if we do that we can't tell the difference between a 60.0 on release and esr, due to trimming the esr off the release name in the config gen script (eg, comment #4). This cross-talk will also bite us if we do a 60.0.2 on release or esr, because 60.0.1esr is already in firefox.json * so leave the esr in the release name when looking for releases to consider, and the existing regexes will ignore esr releases on beta & release. Add a new regex to handle the esr case * we also needed to set the channel to esr-localtest for update verify on esr60, otherwise it ends up as default Testing: * redid the 60.0.1esr update verify config and update verify jobs https://tools.taskcluster.net/groups/VrKJf36lQZegfM3ea84UUw (munged up graph/tasks from release, and a copy of firefox.json with 60.0esr added) * local testing to make sure update verify configs from Fx & Deved 61.0b8, 60.0.1, 60.0 on beta & release are not modified Deployment: * land this patch everywhere. It'll need a merge on mozilla-esr60 because include-version is set to nonbeta there * new PR to reland https://github.com/mozilla-releng/ship-it/pull/222 and push that out

Comment on attachment 8981379 [details] [diff] [review] [gecko] Fix up ESR60 handling Review of attachment 8981379 [details] [diff] [review]: ----------------------------------------------------------------- Thanks for fixing this up.

Pushed by nthomas@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/7998b3ccc005 Update verify fixes for ESR60 branch, r=bhearsum DONTBUILD

Comment on attachment 8981379 [details] [diff] [review] [gecko] Fix up ESR60 handling https://hg.mozilla.org/releases/mozilla-beta/rev/0c6274aa49006fbc6170b8d0f355b8eb84dec09c https://hg.mozilla.org/releases/mozilla-release/rev/13fdf7dee9147f3d70b6f6da3d48eb19a790a100 https://hg.mozilla.org/releases/mozilla-esr60/rev/ba558623bafd83c5b113d2072f7ef23653c3bb4c

https://github.com/mozilla-releng/ship-it/pull/224 to re-add 60.0esr to firefox.json.

Merged #224 of shipit and deployed to stage+prod. We should be all done here.