The containerless scrolling design implemented in bug 1158392 and dependencies relies on the invariant that if there is an async zoom container layer, then any layers that carry RCD-RSF scroll metadata are descendants of it.

The reason this invariant is important is that the RCD-RSF APZC's async transform needs to apply to all layers carrying RCD-RSF scroll metadata, but the zoom component of this transform is lifted to the async zoom container layer at composition time. If there is an RCD-RSF layer outside the zoom container, the zoom component will not apply to it, resulting in incorrect rendering.

In this bug I will add some code to assert this invariant whenever the compositor receives a layers transaction.

I implemented the assertion. A Try push is showing it failing on several Android mochitests:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1&revision=db0c70b697cba956e2c80a7682b5d2a1bb0aa5e0

Apparently, we can get nested async zoom containers in the compositor layer tree!

The invariant that layers containing RCD-RSF metadata are inside the respective async zoom container is being respected, but the invariant-checking code doesn't expect there to be nested containers. Other code may not expect that, either...

(In reply to Botond Ballo [:botond] from comment #2)

Apparently, we can get nested async zoom containers in the compositor layer tree!

This can happen due to OOP iframes. It can't actually occur in production code at the moment, and based on a conversation with Nika, the facilities we'd need to guarding against it have not landed yet (but will do so before OOP iframes ship).

I checked and all the test failures are caused by this. For now, I'm going to bail from the check if we detect a nested async zoom container. I'll file a follow-up for not building an async zoom container for the RCD of an OOP iframe.

(We can also get a nested async zoom container with RDM once we enable desktop zooming. There, we'll need a different fix, as we want the inner async zoom container and not the outer one.)

(In reply to Botond Ballo [:botond] from comment #3)

I'll file a follow-up for not building an async zoom container for the RCD of an OOP iframe.

Filed bug 1534459.

https://hg.mozilla.org/mozilla-central/rev/2d47c27e5526