1. Enable classic scrollbars.
2. Remove all but test_layerization.html and a couple other tests from gfx/layers/apz/test/mochitest.ini so the following steps complete in a shorter amount of time.
3. Edit test_layerization.html to remove the ".then(SimpleTest.finish, SimpleTest.finishWithFailure);" line and add a semicolon to the line before so the test stays open after finished.
4. ./mach mochitest gfx/layers/apz/test/mochitest/ -f plain --enable-webrender
5. Right click, inspect on the test iframe, add style="height: 1000px;" to the iframe. Close inspector.
6. Try to scroll the outermost scrollable element in the test by panning on the touchpad while the mouse cursor is over the classic scrollbar of it.

Presumably you don't need the step 2, you can just running test_layerization.html with --run-until-failure to run the test in the iframe in question.

(In reply to Hiroyuki Ikezoe (:hiro) from comment #1)

Presumably you don't need the step 2, you can just running test_layerization.html with --run-until-failure to run the test in the iframe in question.

Ah, good tip. That seems like it would work.

Adding to See Also some other bugs where this assertion failure was seen.