Currently, we are unable to do compositor animations/transitions for frames bigger than the viewport. This leaves the smoothness of the animation at the mercy of mainthread scheduling - See bug 1825391 comment 6.
Work was done to partially prerender frames to address this (bug 1321412), and is hidden behind layout.animation.prerender.partial pref, but enabling it can cause worse behaviour so it's disabled by default (bug 1760918).
Would be nice to be able to improve and eventually enable this, since it seems that real websites (bug 1825391 example being Amazon) would benefit from it.

Currently, bug 1656473 contains the collection of known issues with partial prerender. Should we move those dependents to this bug?

Hm - probably a good idea, and having bug 1656473 be the metabug, though it could be argued that this is a dupe of that bug.