Balrog version 2.41 is ready to be pushed to prod. Please deploy the new Docker images (master-36cb816f5d3270472f6f4aeb77bbfc9e5cfa68b7) for admin, public, and the agent. We'd like the production push for this to happen sometime between 11am and 1pm pacific on Wednesday, October 11th. If anything goes wrong with the new version in production, we can safely rollback to the previous versions of admin, public, and agent. It's also worth noting that this push includes both the patch that caused last week's performance issues when nightlies ran (that caused us to rollback admin), as well as an in-app fix that should make those queries massively more efficient. I wasn't able to verify the patch in staging under equivalent load, but I was able to reproduce the slow queries by hand, and verify the improved performance of the new ones. I will be on hand when the first set of nightlies submit after this push (~7:30pm eastern) in case anything goes wrong.

(In reply to Ben Hearsum (:bhearsum) from comment #0) > If anything goes wrong with the new version in production, we can safely > rollback to the previous versions of admin, public, and agent And for clarity, we can rollback just one of these if just one is broken, or we can choose to rollback all of them. For this specific deploy, things are still out of sync since last time, so rolling back admin would be to v2.39, and rolling back public/agent would be to v2.40.