At the moment, OS compositor tiles for scrolled content have a width of 2048. This means that we potentially waste a lot of memory, and spend a lot of time clearing pixels unnecessarily. We should switch to 1024x1024 tiles for scrolled content, and maybe something even smaller on lower resolution displays.
Gecko has gfxVars::TileSize() which is used in non-WebRender and automatically adjusts to the resolution, see gfxPlatform::ComputeTileSize(). We could forward that value into WebRender.

For reference, at the moment, if I have my window set to 1024 logical pixels on my HiDPI display, i.e. 2048 physical pixels, the amount of OS compositor surface memory almost doubles when I increase the window width by one pixel. (It jumps from 374% to 729%. This percentage is relative to the window size. The baseline is currently 3x as much as it needs needs to be due to memory wasted on hidden layers, see bug 1591526.)

In addition to this change, I also want to:

• Change small tile size from 128 x 128 to something more like 512 x 128. This should be a better balance of number of tiles, and reduce the number of OS surfaces we need to create.
• Add special size bins for scroll bars (we have this information in the prim flags now) that are 512 x 64 and 64 x 512. This will reduce the tile counts for scroll bar slices significantly and be more efficient.

(1) Reduce size of content tiles from 2048 x 512 to 1024 x 512.

This reduces the large extra overhead of tiles on screens that
are not 4k resolution.

(2) Remove small 128 x 128 tiles.

These typically create a lot of OS compositor surfaces, which
adds significant overhead on some platforms. Having them be
normal content tiles also means no resizing of the picture
cache for the UI when the awesome bar and/or status bar is on.

(3) Add support for very narrow cache surfaces.

This is a special use case for scrollbar caches. It's worth
supporting this special case for the significant memory and
performance savings for these surfaces.

This does seem to move around the talos performance results a bit [1]. Three tests improve by 3-6%, and 3 tests regress by 2-3%. I think that's reasonable, and we could land this, for the memory savings + reduced OS compositor overhead.

[1] https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-central&newProject=try&newRevision=87ac7aefb4e3ad14957832ea9289730b3c14ba19&framework=1&selectedTimeRange=172800

https://hg.mozilla.org/mozilla-central/rev/5647ec4ba6f2

== Change summary for alert #23674 (as of Mon, 04 Nov 2019 16:38:14 GMT) ==

Improvements:

3% tp5o_scroll linux64-shippable-qr opt e10s stylo 2.17 -> 2.10
2% tscrollx linux64-shippable-qr opt e10s stylo 1.15 -> 1.13

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=23674