eg

[task 2020-03-02T23:24:30.082Z] TEST-UNEXPECTED-FAIL: expected buildID 20200302184608 does not match actual 20200117190643
[task 2020-03-02T23:24:30.082Z] TEST-UNEXPECTED-FAIL: [72.0.1 ach complete] download_mars returned non-zero exit code: 1
...
[task 2020-03-02T23:24:30.082Z] TEST-UNEXPECTED-FAIL: expected buildID 20200302184608 does not match actual 20200117190643
[task 2020-03-02T23:24:30.082Z] TEST-UNEXPECTED-FAIL: [61.0.1 rm complete] download_mars returned non-zero exit code: 1
[task 2020-03-02T23:24:30.082Z] TEST-UNEXPECTED-FAIL: expected buildID 20200302184608 does not match actual 20200117190643
[task 2020-03-02T23:24:30.082Z] TEST-UNEXPECTED-FAIL: [61.0.1 ro complete] download_mars returned non-zero exit code: 1
...
[task 2020-03-02T23:24:30.084Z] TEST-UNEXPECTED-FAIL: expected buildID 20200302184608 does not match actual 20200117190643
[task 2020-03-02T23:24:30.084Z] TEST-UNEXPECTED-FAIL: [61.0 xh complete] download_mars returned non-zero exit code: 1

ie versions older than the 72.0.2 watershed are being given 72.0.2 (20200117190643), but update verify still expects it should be 74.0 because of the config.

Comment on attachment 9130297 [details]
Bug 1619469 - Update release watershed to 72.0.2, r=tomprince

Beta/Release Uplift Approval Request

• User impact if declined: None
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This is limited to automation configurationn for tests.
• String changes made/needed:

https://hg.mozilla.org/releases/mozilla-release/rev/940a2b0c8932

I ran a customized promotion action to rerun update verify with the fix. The first attempt (FgwZDyquSp-EGkia0yqVuA) picked up the buildID from fix builds, which was not intended and caused every job to fail. The second attempt worked as we wanted - see graph JaJhBm30TTaS6dNYFUGHrQ.

To get it running I extracted the action input from the task definition of the original promotion task (NYzQGXMNRKuA2hY_QZXRRg), then modified it with

+   "release_enable_partners": false,
+   "release_enable_emefree": false,
+   "rebuild_kinds": ["release-update-verify-config"],
    "previous_graph_ids": [
-        "ZXQF9KhiQv6E3ZSlOGI3Mg"
+        "ZXQF9KhiQv6E3ZSlOGI3Mg", "NYzQGXMNRKuA2hY_QZXRRg"
     ]

This was then submitted via Treeherder, using the Custom Push Action for the fix push, with the release promotion action.

TBH, disabling the partners and emefree was a carryover from testing locally with
$ ./mach taskgraph test-action-callback --input input.json release-promotion > /dev/null

Graph was green after a few reruns for transient download errors.

https://hg.mozilla.org/mozilla-central/rev/0a8b4f760303