Initial motivation for this was reflected in bug 1501167 and bug 1537710. A temporary solution is going to be implemented in bug 1579415 to prepare for upcoming more-frequent betas. But our long-term goal is to ensure we have idempotency for our release graph, for each of the individual tasks. So this is to track the long-term solution for the above.

I was thinking about this a bit more, and if we want true idempotency for Balrog submissions, I think it necessitates chunking in a way where we only write to every platform+locale combination once. In any scenario where we have multiple tasks that write the same platform+locale, we have to deal with list merging or overwriting, which wrecks any idempotency guarantees.

We could still group multiple locales for the same platform together, but we can't separate submissions for different MARs for the same platform+locale. When we do the submission, for these, we can simply overwrite the entire locale block, rather than trying to submit just a single MAR (and letting the server do merging).

We've implemented an index check that helps mitigate any duplicate nightlies/betas triggered after the decision task finishes, so we've reduced the race condition window to the duration of the first decision task.