Found while fuzzing m-c 20220904-c731914e8096 (--enable-debug --enable-fuzzing)

To reproduce via Grizzly Replay:

$ pip install fuzzfetch grizzly-framework
$ python -m fuzzfetch -d --fuzzing -n firefox
$ python -m grizzly.replay ./firefox/firefox testcase.html

Assertion failure: inverted (Attempted to get the inverse of a non-invertible matrix), at /builds/worker/workspace/obj-build/dist/include/mozilla/gfx/Matrix.h:1329

#0 0x7fac2c176365 in mozilla::layers::APZCTreeManager::GetScreenToApzcTransform(mozilla::layers::AsyncPanZoomController const*) const /builds/worker/checkouts/gecko/gfx/layers/apz/src/APZCTreeManager.cpp
#1 0x7fac2c17fe8d in mozilla::layers::TransformDisplacement(mozilla::layers::APZCTreeManager*, mozilla::layers::AsyncPanZoomController*, mozilla::layers::AsyncPanZoomController*, mozilla::gfx::PointTyped<mozilla::ParentLayerPixel, float>&, mozilla::gfx::PointTyped<mozilla::ParentLayerPixel, float>&) /builds/worker/checkouts/gecko/gfx/layers/apz/src/APZCTreeManager.cpp:2610:21
#2 0x7fac2c18115e in mozilla::layers::APZCTreeManager::DispatchFling(mozilla::layers::AsyncPanZoomController*, mozilla::layers::FlingHandoffState const&) /builds/worker/checkouts/gecko/gfx/layers/apz/src/APZCTreeManager.cpp:2731:12
#3 0x7fac2c1ab530 in mozilla::layers::AsyncPanZoomController::HandleFlingOverscroll(mozilla::gfx::PointTyped<mozilla::ParentLayerPixel, float> const&, mozilla::SideBits, RefPtr<mozilla::layers::OverscrollHandoffChain const> const&, RefPtr<mozilla::layers::AsyncPanZoomController const> const&) /builds/worker/checkouts/gecko/gfx/layers/apz/src/AsyncPanZoomController.cpp:3815:27
#4 0x7fac2c1abab8 in mozilla::layers::AsyncPanZoomController::HandleSmoothScrollOverscroll(mozilla::gfx::PointTyped<mozilla::ParentLayerPixel, float> const&, mozilla::SideBits) /builds/worker/checkouts/gecko/gfx/layers/apz/src/AsyncPanZoomController.cpp:3841:3
#5 0x7fac2c1ffd61 in applyImpl<mozilla::layers::AsyncPanZoomController, void (mozilla::layers::AsyncPanZoomController::*)(const mozilla::gfx::PointTyped<mozilla::ParentLayerPixel, float> &, mozilla::SideBits), StoreCopyPassByConstLRef<mozilla::gfx::PointTyped<mozilla::ParentLayerPixel, float> >, StoreCopyPassByConstLRef<mozilla::SideBits>, 0UL, 1UL> /builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:1147:12
#6 0x7fac2c1ffd61 in apply<mozilla::layers::AsyncPanZoomController, void (mozilla::layers::AsyncPanZoomController::*)(const mozilla::gfx::PointTyped<mozilla::ParentLayerPixel, float> &, mozilla::SideBits)> /builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:1153:12
#7 0x7fac2c1ffd61 in mozilla::detail::RunnableMethodImpl<mozilla::layers::AsyncPanZoomController*, void (mozilla::layers::AsyncPanZoomController::*)(mozilla::gfx::PointTyped<mozilla::ParentLayerPixel, float> const&, mozilla::SideBits), true, (mozilla::RunnableKind)0, mozilla::gfx::PointTyped<mozilla::ParentLayerPixel, float>, mozilla::SideBits>::Run() /builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:1200:13
#8 0x7fac2af3a68e in mozilla::RunnableTask::Run() /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:538:16
#9 0x7fac2af12d39 in mozilla::TaskController::DoExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:851:26
#10 0x7fac2af118c3 in mozilla::TaskController::ExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:683:15
#11 0x7fac2af11b33 in mozilla::TaskController::ProcessPendingMTTask(bool) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:461:36
#12 0x7fac2af3dee6 in operator() /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:187:37
#13 0x7fac2af3dee6 in mozilla::detail::RunnableFunction<mozilla::TaskController::InitializeInternal()::$_0>::Run() /builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:531:5
#14 0x7fac2af277ff in nsThread::ProcessNextEvent(bool, bool*) /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:1205:16
#15 0x7fac2af2de0d in NS_ProcessNextEvent(nsIThread*, bool) /builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.cpp:465:10
#16 0x7fac2bb08fd6 in mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) /builds/worker/checkouts/gecko/ipc/glue/MessagePump.cpp:85:21
#17 0x7fac2ba2e9b7 in MessageLoop::RunInternal() /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:381:10
#18 0x7fac2ba2e8c2 in RunHandler /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:374:3
#19 0x7fac2ba2e8c2 in MessageLoop::Run() /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:356:3
#20 0x7fac2fdf7308 in nsBaseAppShell::Run() /builds/worker/checkouts/gecko/widget/nsBaseAppShell.cpp:150:27
#21 0x7fac31e31484 in nsAppStartup::Run() /builds/worker/checkouts/gecko/toolkit/components/startup/nsAppStartup.cpp:295:30
#22 0x7fac31f570d3 in XREMain::XRE_mainRun() /builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp:5725:22
#23 0x7fac31f584a0 in XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) /builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp:5919:8
#24 0x7fac31f58c79 in XRE_main(int, char**, mozilla::BootstrapConfig const&) /builds/worker/checkouts/gecko/toolkit/xre/nsAppRunner.cpp:5975:21
#25 0x5590d461a32f in do_main /builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp:229:22
#26 0x5590d461a32f in main /builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp:433:16
#27 0x7fac418d6082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16
#28 0x5590d45f01cc in _start (/home/worker/builds/m-c-20220904093956-fuzzing-debug/firefox-bin+0x161cc) (BuildId: 9eec3addc971eda4ac84f38f59ceaae0af4e4356)

A Pernosco session is available here: https://pernos.co/debug/cM1lzQwZySvMV4NhY_GFjg/index.html

Bugmon Analysis
Verified bug as reproducible on mozilla-central 20220907093209-663615ef7a19.
The bug appears to have been introduced in the following build range:

Start: bdb42cfe62138374343d5be83ac208826812cd2d (20220810161147)
End: a5ef26cc165936d1c01c42c0e5d2c597ebcc5a8f (20220810181917)
Pushlog: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=bdb42cfe62138374343d5be83ac208826812cd2d&tochange=a5ef26cc165936d1c01c42c0e5d2c597ebcc5a8f

The severity field is not set for this bug.
:botond, could you have a look please?

For more information, please visit auto_nag documentation.

Marking as S3 but I plan to look at this when I'm back from PTO.

Does the regression range look correct to you?

Bug 793686 in the regression range could have "caused this" in that it could have moved around column layout and the testcase uses columns, but if that is the case there isn't much to be learned from that I don't think.

As Timothy says, the regression range here is likely to be incidental. This seems like a long-standing issue that can probably be triggered even before that change on a slightly different testcase.

Looking at the testcase, it uses a transform with a z-scale of 0, which makes the transform matrix non-invertible. That causes us to not render the transformed element.

The reason for this seems to be the following provisions of the CSS Transforms spec:

If a transform function causes the current transformation matrix of an object to be non-invertible, the object and its content do not get displayed.

The assertion is during a codepath that tries to invert the transformation matrix for purposes related to hit-testing. Since the element does not get rendered, it does not need to be hit-tested either, so we could add an invertibility check there and bail without affecting correctness.

However, I've noticed that if I instead modify the transform to have an x-scale or y-scale of 0 rather than a z-scale of 0, the element still does not render but the assertion does not fire either. This suggests that there is a check earlier in the pipeline (e.g. in layout or display list building code) that catches those cases and causes the transform to not even be sent to APZ. So, before adding a check to the APZ code, we should first find that earlier check and evaluate whether it would be more appropriate to have it handle the z-scale=0 case as well.

Bugmon was unable reproduce this issue.
Removing bugmon keyword as no further action possible. Please review the bug and re-add the keyword for further analysis.

A change to the Taskcluster build definitions over the weekend caused Bugmon to fail when reproducing issues. This issue has been corrected. Re-enabling bugmon.