The idea here is to avoid long-running processes which suffer from things like memory leaks. It'd be great to fix those, but we don't always have time to. I'm thinking something like this...On Sunday night, for each pool of masters, gracefully restart each master in a pool in series. Pools can be processed in parallel. On Monday morning, buildduty (with help from nagios!) checks to see if the restarts completed successfully, or if they're hung somewhere.

This sounds pretty difficult. Let's get periodic *manual* restarts going first. I'd suggest doing this with slavealloc - bring a new instance up, disable the old in slavealloc, and graceful it. Then we don't lose capacity at any point. Anyway, futuristic for now..

Found in triage.

Do we still care about this now that we're moving most things to Taskcluster?

We have nagios alerts for buildbot process age, and we restart them manually periodically.