action/cron - don't trigger nightlies/betas off same rev
Categories
(Release Engineering :: General, defect)
Tracking
(Not tracked)
People
(Reporter: mtabara, Unassigned)
References
Details
If two nightlies are based off the same revision, it'll end up submitting duplicate hash entries in the balrog blobs. While end goal is to make jobs idempotent and solve this in balrogscript jobs, we still want a(n) additional/temporary solution graph-wise.
This means we add a sanity checking at the beginning of the graph to ensure we don't trigger nightlies/betas off the same revision.
More context from other bugs' comments:
That might be tricky while nightlies are a hook creating a new graph, but might be doable.
I think it'll become easier when we're triggering promotion on shippable builds.
Could we have the hook logic check something in the index to determine if we've already done nightlies for a given revision?
This was discussed briefly at channel meeting - We can have something in index check if nightlies were already run, but it would be cludgy and harder to validate (e.g. do we want to prevent triggering windows again on a given rev if all linux finished?)
That said, there will be work in Q2 that should solve that aspect as well, which is Nightly Promotion. I say we wait for that unless this becomes more of a prevalent problem.
Comment 2•5 years ago
|
||
The main question seems to be 'How do we determine if a nightly release has happened, given a revision?' Balrog contains a buildID, which we determine in the decision task, so we could also get information there about what has already been released - and this covers us in case we built something which ended up in the index, but wasn't released.
Do we want there to be different behaviour between cron and a human triggering a nightly through the hook? That is, do we want cron to bail early if it thinks there's already a released nightly for this revision, but do we want to trust a human knows what they're doing? Or should a human need to create a new (potentially no-op) revision in this situation?
Comment 3•5 years ago
|
||
We can create/update an index when a given graph has passed a certain point of green (this may just be a green decision task; or we may choose to only update the index if we start submitting to balrog, or some other criteria). The next hook trigger can check the index, and trace it back to the revision. If the revision matches the revision we're being triggered off of, don't schedule any tasks.
And yes, there should be a force
boolean in the hook, to allow us to intentionally trigger a hook off the same revision.
Comment 5•4 years ago
|
||
Nightly failed to update today, both partial and total. Checked with sheriffs, and it looks like we hit this again.
Comment 6•4 years ago
|
||
Should a fix or bandaid for this be prioritized?
Comment 7•4 years ago
|
||
Callek tells me that he thinks this won't be as much of a problem after the v2 balrog changes land (bug 1634757). We'll still potentially have two nightlies off the same rev, but they shouldn't result in broken updates.
Comment 8•3 years ago
|
||
Solved for nightlies in bug 1380123.
Betas I think Relman keeps a closer eye on; we could add similar logic if wanted.
Description
•