TC and buildbot jobs are publishing to the same index on beta. For rev 91faf7ec36cd18a8ebdc0e2edac966e5bbb15da2 the linux64 buildbot build [1] finished first, and TC build second [2], so TC was what the index pointed to [3]. The other platforms were OK (linux build finished in the other order, mac failed on TC, and no windows TC builds). Since we don't sign TC builds we don't have an artifact matching checksums.asc, and can't continue. bug 1304259 to handle that more gracefully, but the real issue for release automation is we should only use buildbot builds. Perhaps we've just been lucky so far, or perhaps something changed in the recent merge to beta. We shouldn't be vulnerable to timing though. I'm attempting to work around this by re-running the buildbot job. [1] https://tools.taskcluster.net/task-inspector/#HD5JE_paQ-SwH1_fit1ugQ/0 [2] https://tools.taskcluster.net/task-inspector/#DWkAFJPNQb6I1QtJwmB2ug/0 [3] https://tools.taskcluster.net/index/#gecko.v2.mozilla-beta.revision.91faf7ec36cd18a8ebdc0e2edac966e5bbb15da2.firefox/gecko.v2.mozilla-beta.revision.91faf7ec36cd18a8ebdc0e2edac966e5bbb15da2.firefox.linux64-opt returned DWkAFJPNQb6I1QtJwmB2ug

(In reply to Nick Thomas [:nthomas] from comment #0) > TC and buildbot jobs are publishing to the same index on beta. For rev > 91faf7ec36cd18a8ebdc0e2edac966e5bbb15da2 the linux64 buildbot build [1] > finished first, and TC build second [2], so TC was what the index pointed to > [3]. Specifically, the upload-files action in buildbot finished before the whole TC job.

(In reply to Nick Thomas [:nthomas] from comment #0) > I'm attempting to work around this by re-running the buildbot job. That worked, the index points to https://tools.taskcluster.net/task-inspector/#PcbuNN7uQpyXtA7dC4fFpQ/ now. The release was then restarted. Leaving this open for now. Could use this bug or a more general one for the tc vs buildbot race.

hmm, are TC builds tier1 now? If not, we may need to reopen bug 1274311.

https://queue.taskcluster.net/v1/task/DWkAFJPNQb6I1QtJwmB2ug is missing "tier" in the definition. According to https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/task/legacy.py#493-498 this won't set the rank to 0, so TC wins in this case. Maybe we should reverse the logic and make it set the rank to 0 if no "tier" is set?

Looks like the issue is http://hg.mozilla.org/releases/mozilla-beta/file/tip/taskcluster/ci/legacy/tasks/builds/opt_linux64.yml#l18 vs http://hg.mozilla.org/releases/mozilla-beta/file/tip/taskcluster/ci/legacy/tasks/builds/opt_linux32.yml#l24

The patch is for beta only.

(In reply to Rail Aliiev [:rail] from comment #4) > Maybe we should reverse the logic and make it set the rank to 0 if no "tier" > is set? Yeah, when I added the rank-0 code for tier-2+, for some reason I made it default to tier-1 if no tier was present. However, dustin made this more sensible in bug 1286075 by defaulting to tier-3 (rank 0). That isn't on beta yet, though. I think your approach of just setting the builds you don't want in the index as tier-2 should be sufficient.

Thanks!

Comment on attachment 8793275 [details] Bug 1304260 - opt-linux64 should be tier 2 https://reviewboard.mozilla.org/r/80066/#review78848

Comment on attachment 8793275 [details] Bug 1304260 - opt-linux64 should be tier 2 This is required to prevent Tier 2 Taskcluster builds overwrite Buildbot "uploads". The patch if for beta only. The patch is trivial enough, so I didn't even tested it. :) What can possibly go wrong - linux32 config contains the line I added in this patch. [Risks and why]: Very low, this affects only TC builds and this is reimplemented in 51. [String/UUID change made/needed]: None.

Comment on attachment 8793275 [details] Bug 1304260 - opt-linux64 should be tier 2 NPOTB, Beta50+

Comment on attachment 8793275 [details] Bug 1304260 - opt-linux64 should be tier 2 https://hg.mozilla.org/releases/mozilla-beta/rev/178bd0d028dd