Reftests need to read back the entire window contents into a main memory buffer. When we draw into OS compositor layers, there is no such single buffer to read back from. We'll need to find a solution so that we can enable gfx.webrender.compositor by default without breaking reftests.

I think there are a number of potential solutions:

1. Read back the update rect contents from each surface individually, and do a simple "software compositing" step to assemble the window buffer on the CPU.
2. Make the reftest drawing step trigger an extra WR render which goes down the traditional path and doesn't use OS compositor surfaces.
3. Make it Gecko's responsibility to ask for the OS's compositing of our OS surfaces.
4. Others?

Solution 1 only works as long as the OS compositor is simple enough, i.e. as long as it doesn't support opacity / transforms or other effects. Solutions 1 and 2 would miss bugs that are introduced at the OS compositor level.
Solution 3 might not be possible on all platforms.

It would be good to find out what EdgeHTML and Safari do for their reftests.

On windows, reftest seemed to work with os compositor by Bug 1585619.
https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=279479858&revision=3a5cea30145a9b720fa2b20352d999e97e610651

I'm still investigating options for Mac. It looks like Safari does not test their CoreAnimation path in their reftests. At least web platform tests seem to be going through Safari's main thread rendering path, which does not support 3d transforms. As a result, tests which contain 3d transforms that can't be flattened fail in Safari, such as this one: https://wpt.fyi/results/css/css-transforms/css-transform-3d-transform-style.html?label=experimental&label=master&aligned

https://github.com/WebKit/webkit/blob/a71e69ec66df96246fad2b984ad5fcdef62677b0/Source/WebCore/rendering/RenderLayer.cpp#L1029-L1033

I'm morphing this bug into the macOS version, since bug 1585619 took care of Windows.

I explored two options for macOS. I wanted to have option 3, i.e. rather than doing a "fake" rendering, read back exactly what the native layers put into the window, including any compositing effects from those native layers.

1. First I tried using CGWindowListCreateImage. When called as CGWindowListCreateImage(CGRectNull, kCGWindowListOptionIncludingWindow, windowID, kCGWindowImageBoundsIgnoreFraming | kCGWindowImageBestResolution), it successfully rendered the window contents, even if the window was covered by other windows on the screen. However, there was a big problem: I frequently got old frames. And I couldn't find a way to wait for the window server to render our CATransaction. I also couldn't find any other way of establishing a fence between committing a CATransaction and the CGWindowListCreateImage render. There may still be a way to do this, but I stopped looking at some point. (For example, I didn't investigate what the CAContext fence port is.)
2. Then I explored using CARenderer. CARenderer can synchronously render our CALayer tree into an OpenGL context. This worked! I'm going to attach patches that use it. I ran into a few problems along the way:
   • CARenderer only picks up CALayer changes that have truly been "committed" by the root CATransaction. On the main thread, the root CATransaction is always an implicit transaction that gets committed by the event loop. So we can't render use CARenderer on the main thread, at least not in a synchronous fashion. But that's fine, we want to do this on the Renderer thread anyway.
   • Whether IOSurfaces render upside down or not is affected by the geometryFlipped property on ancestor CALayers. The WebRender CALayers in the Firefox window are rooted at the backing layer of an NSView which has isFlipped == YES. In order to get the same results when rendering layers with CARenderer, the layers need to be rooted at a CALayer which has geometryFlipped set to YES.
   • CALayers can be either on screen (i.e. somehow attached to an NSView's layer) or they can be attached to a CARenderer. Not both at the same time! That's because the CARenderer has its own CAContext, and a CALayer can only be associated with one CAContext at any given time. This requirement means that our NativeLayerRoot now needs to maintain two CALayer trees: an on-screen tree and an off-screen tree. The on-screen tree is attached to the NSView, and the off-screen tree is attached to the CARenderer.

This makes it more similar to how SwapBuffers was used.
This patch also makes us call glFlush directly when using native layers, rather than going through the misleadingly-named GLContext::SwapBuffers method.

Depends on D56668

Depends on D56669

Depends on D56670

Depends on D56671

(This is the approach that returned old frames. It's no good. I'm just attaching it for future reference.)

Depends on D54863

Depends on D57060

This makes it more similar to how SwapBuffers was used.
This patch also makes us call glFlush directly when using native layers, rather than going through the misleadingly-named GLContext::SwapBuffers method.

Depends on D57061

Depends on D57062

Depends on D57063

Depends on D57064

This will allow us to have two representations per NativeLayerCA in the next patch.

Depends on D57065

The onscreen representation is attached to the NSView.
The offscreen representation is free-floating but will be used in a CARenderer in an upcoming patch.

Each representation is only updated on demand.

Depends on D57066

Suggestions for a better name than "snapshotter" are welcome.
This is a separate object so that the lifetime of its GLContext isn't governed by the lifetime of the NativeLayerRootCA.
The NativeLayerRootCA gets destroyed on the main thread, but GLContext uses non-threadsafe weak pointer support, so it wants to be destroyed on the same thread that it was created on.
So now the GLContext lives on the snapshotter, which is created and destroyed on the renderer thread.

Depends on D57067

Depends on D57068

https://hg.mozilla.org/mozilla-central/rev/5433f2ea1f4b
https://hg.mozilla.org/mozilla-central/rev/f478a2e999e4
https://hg.mozilla.org/mozilla-central/rev/b5cf7c3c4771
https://hg.mozilla.org/mozilla-central/rev/90d4d9ab6493
https://hg.mozilla.org/mozilla-central/rev/2effeae528d3
https://hg.mozilla.org/mozilla-central/rev/b041f2395c05
https://hg.mozilla.org/mozilla-central/rev/8d7fd1220a4d
https://hg.mozilla.org/mozilla-central/rev/c15a3eb03899
https://hg.mozilla.org/mozilla-central/rev/2bb6e9c84513
https://hg.mozilla.org/mozilla-central/rev/781f53bf9c78