This can take a millisecond or two, and should be easy to incrementalize, so maybe we should just do that. You just want to make sure nobody does any graph operations while it is in some weird torn down state, but we shouldn't be doing them anyways. Note that RemoveObjectFromGraph() actually does currently do a remove when we're in CleanupPhase, so that would have to be fixed.