Found while fuzzing m-c 20220803-ae4cb105d717 (--enable-address-sanitizer --enable-fuzzing)

To reproduce via Grizzly Replay:

$ pip install fuzzfetch grizzly-framework
$ python -m fuzzfetch -a --fuzzing -n firefox
$ python -m grizzly.replay ./firefox/firefox testcase.html

To help catch this issue ASAN_OPTIONS=max_allocation_size_mb=1024 was used.

Hit MOZ_CRASH(out of memory: 0x00000000936B2A18 bytes requested) at src/memory/mozalloc/mozalloc_abort.cpp:35

==543018==WARNING: AddressSanitizer failed to allocate 0x936b2a18 bytes
=================================================================
==543018==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000001 (pc 0x557bc9e33733 bp 0x7f4547099f30 sp 0x7f4547099f10 T33)
==543018==The signal is caused by a WRITE memory access.
==543018==Hint: address points to the zero page.
    #0 0x557bc9e33733 in MOZ_Crash /builds/worker/workspace/obj-build/dist/include/mozilla/Assertions.h:261:3
    #1 0x557bc9e33733 in mozalloc_abort src/memory/mozalloc/mozalloc_abort.cpp:35:3
    #2 0x557bc9e32e85 in mozalloc_handle_oom(unsigned long) src/memory/mozalloc/mozalloc_oom.cpp:51:3
    #3 0x7f4583736236 in mozglue_static::oom_hook::hook::h50cf97f565b59932 src/mozglue/static/rust/lib.rs:115:13
    #4 0x7f4586e2b017 in rust_oom (/home/user/workspace/browsers/m-c-20220803212830-fuzzing-asan-opt/libxul.so+0x210fa017) (BuildId: 6bb0ae42c3bfc363a5916a543c21ba4ce188c965)
    #5 0x7f4586e72466 in __rg_oom crtstuff.c
    #6 0x7f4586e720f6 in alloc::alloc::handle_alloc_error::rt_error::h0f0cef0b1f7fae80 (/home/user/workspace/browsers/m-c-20220803212830-fuzzing-asan-opt/libxul.so+0x211410f6) (BuildId: 6bb0ae42c3bfc363a5916a543c21ba4ce188c965)
    #7 0x7f4586e71f36 in core::ops::function::FnOnce::call_once::h0c14d51cbb5833de (.llvm.16445399339973007591) crtstuff.c
    #8 0x7f4586e71ec5 in core::intrinsics::const_eval_select::hf0f7f377a4ede30b (.llvm.16445399339973007591) crtstuff.c
    #9 0x7f456f6685a5 in alloc::alloc::handle_alloc_error::h9af2e230ffc0d1dd (/home/user/workspace/browsers/m-c-20220803212830-fuzzing-asan-opt/libxul.so+0x99375a5) (BuildId: 6bb0ae42c3bfc363a5916a543c21ba4ce188c965)
    #10 0x7f458230143b in alloc::raw_vec::RawVec$LT$T$C$A$GT$::allocate_in::h4c556232da3cc6b1 /builds/worker/fetches/rust/library/alloc/src/raw_vec.rs:190:27
    #11 0x7f458230143b in alloc::raw_vec::RawVec$LT$T$C$A$GT$::with_capacity_zeroed_in::h3be7a91333e817c1 /builds/worker/fetches/rust/library/alloc/src/raw_vec.rs:139:9
    #12 0x7f458230143b in _$LT$u8$u20$as$u20$alloc..vec..spec_from_elem..SpecFromElem$GT$::from_elem::hba50227ab86c26cb /builds/worker/fetches/rust/library/alloc/src/vec/spec_from_elem.rs:39:31
    #13 0x7f458230143b in alloc::vec::from_elem::h21eedb641a2aee5e /builds/worker/fetches/rust/library/alloc/src/vec/mod.rs:2423:5
    #14 0x7f458230143b in webrender::platform::unix::font::FontContext::rasterize_glyph::hcd2aa7e18d9d61d4 src/gfx/wr/webrender/src/platform/unix/font.rs:907:32
    #15 0x7f458230143b in webrender::glyph_rasterizer::GlyphRasterizer::flush_glyph_requests::_$u7b$$u7b$closure$u7d$$u7d$::h9e24d5076795a025 src/gfx/wr/webrender/src/glyph_rasterizer/mod.rs:157:25
    #16 0x7f4581d1abf8 in webrender::glyph_rasterizer::GlyphRasterizer::flush_glyph_requests::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h5d8b6b20172a12a5 src/gfx/wr/webrender/src/glyph_rasterizer/mod.rs:211:31
    #17 0x7f4581d1abf8 in core::ops::function::impls::_$LT$impl$u20$core..ops..function..FnMut$LT$A$GT$$u20$for$u20$$RF$F$GT$::call_mut::h17162e7f5d4c6b53 /builds/worker/fetches/rust/library/core/src/ops/function.rs:268:13
    #18 0x7f4581d1abf8 in _$LT$core..slice..iter..Iter$LT$T$GT$$u20$as$u20$core..iter..traits..iterator..Iterator$GT$::for_each::h6dd10a453f4fd379 /builds/worker/fetches/rust/library/core/src/slice/iter/macros.rs:211:21
    #19 0x7f4581d1abf8 in _$LT$rayon..iter..for_each..ForEachConsumer$LT$F$GT$$u20$as$u20$rayon..iter..plumbing..Folder$LT$T$GT$$GT$::consume_iter::h08418ed6642b7090 src/third_party/rust/rayon/src/iter/for_each.rs:55:9
    #20 0x7f4581d1abf8 in rayon::iter::plumbing::Producer::fold_with::h792d1a825779829c src/third_party/rust/rayon/src/iter/plumbing/mod.rs:110:9
    #21 0x7f4581d1abf8 in rayon::iter::plumbing::bridge_producer_consumer::helper::h8226c144bd727b99 src/third_party/rust/rayon/src/iter/plumbing/mod.rs:438:13
    #22 0x7f4581aaba9d in rayon::iter::plumbing::bridge_producer_consumer::helper::_$u7b$$u7b$closure$u7d$$u7d$::ha9ca4a65983657ef src/third_party/rust/rayon/src/iter/plumbing/mod.rs:418:21
    #23 0x7f4581aaba9d in rayon_core::join::join_context::call_a::_$u7b$$u7b$closure$u7d$$u7d$::hae959dea1a3262c1 src/third_party/rust/rayon-core/src/join/mod.rs:124:17
    #24 0x7f4581aaba9d in _$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h2ca16d93bd59ad02 /builds/worker/fetches/rust/library/core/src/panic/unwind_safe.rs:271:9
    #25 0x7f4581aaba9d in std::panicking::try::do_call::he121dc77655f9ff2 /builds/worker/fetches/rust/library/std/src/panicking.rs:492:40
    #26 0x7f4581aaba9d in std::panicking::try::hca2026e013c18bea /builds/worker/fetches/rust/library/std/src/panicking.rs:456:19
    #27 0x7f4581aaba9d in std::panic::catch_unwind::h9ef07c876388e442 /builds/worker/fetches/rust/library/std/src/panic.rs:137:14
    #28 0x7f4581aaba9d in rayon_core::unwind::halt_unwinding::hcf2a2044ba3d17a4 src/third_party/rust/rayon-core/src/unwind.rs:17:5
    #29 0x7f4581aaba9d in rayon_core::join::join_context::_$u7b$$u7b$closure$u7d$$u7d$::hb2fd77cc9f9962fe src/third_party/rust/rayon-core/src/join/mod.rs:141:24
    #30 0x7f4581d1b124 in rayon_core::registry::in_worker::h8fb85bafd53050ac src/third_party/rust/rayon-core/src/registry.rs:877:13
    #31 0x7f4581d1b124 in rayon_core::join::join_context::h6fd0bc06a162088a src/third_party/rust/rayon-core/src/join/mod.rs:132:5
    #32 0x7f4581d1b124 in rayon::iter::plumbing::bridge_producer_consumer::helper::h8226c144bd727b99 src/third_party/rust/rayon/src/iter/plumbing/mod.rs:416:47
    #33 0x7f45820bbbfc in rayon::iter::plumbing::bridge_producer_consumer::helper::_$u7b$$u7b$closure$u7d$$u7d$::hdea74fe81dd61a3e src/third_party/rust/rayon/src/iter/plumbing/mod.rs:427:21
    #34 0x7f45820bbbfc in rayon_core::join::join_context::call_b::_$u7b$$u7b$closure$u7d$$u7d$::h88680835ce5369fa src/third_party/rust/rayon-core/src/join/mod.rs:129:25
    #35 0x7f45820bbbfc in _$LT$rayon_core..job..StackJob$LT$L$C$F$C$R$GT$$u20$as$u20$rayon_core..job..Job$GT$::execute::call::_$u7b$$u7b$closure$u7d$$u7d$::h4bc78cee0142540a src/third_party/rust/rayon-core/src/job.rs:113:21
    #36 0x7f45820bbbfc in _$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h799e006bc0160bdc /builds/worker/fetches/rust/library/core/src/panic/unwind_safe.rs:271:9
    #37 0x7f45820bbbfc in std::panicking::try::do_call::hfde46896857c83fd /builds/worker/fetches/rust/library/std/src/panicking.rs:492:40
    #38 0x7f45820bbbfc in std::panicking::try::h6f270a3feb2acda7 /builds/worker/fetches/rust/library/std/src/panicking.rs:456:19
    #39 0x7f45820bbbfc in std::panic::catch_unwind::hea0f989b8a18eca6 /builds/worker/fetches/rust/library/std/src/panic.rs:137:14
    #40 0x7f45820bbbfc in rayon_core::unwind::halt_unwinding::h3badd8664940cf01 src/third_party/rust/rayon-core/src/unwind.rs:17:5
    #41 0x7f45820bbbfc in _$LT$rayon_core..job..StackJob$LT$L$C$F$C$R$GT$$u20$as$u20$rayon_core..job..Job$GT$::execute::h7933b7d112b27553 src/third_party/rust/rayon-core/src/job.rs:119:38
    #42 0x7f456f64320c in rayon_core::job::JobRef::execute::hd57a822047bd1d9a src/third_party/rust/rayon-core/src/job.rs:59:9
    #43 0x7f456f64320c in rayon_core::registry::WorkerThread::execute::h285ffd813e1315df src/third_party/rust/rayon-core/src/registry.rs:752:9
    #44 0x7f456f64320c in rayon_core::registry::WorkerThread::wait_until_cold::he37c2bd0719c95e0 src/third_party/rust/rayon-core/src/registry.rs:729:17
    #45 0x7f4586baf57c in rayon_core::registry::WorkerThread::wait_until::h36144a6348e0cc2d src/third_party/rust/rayon-core/src/registry.rs:703:13
    #46 0x7f4586baf57c in rayon_core::registry::main_loop::h73d60f560002a0f8 src/third_party/rust/rayon-core/src/registry.rs:836:5
    #47 0x7f4586baf57c in rayon_core::registry::ThreadBuilder::run::hd8b18a91c97dbd15 src/third_party/rust/rayon-core/src/registry.rs:55:18
    #48 0x7f4586ba0446 in _$LT$rayon_core..registry..DefaultSpawn$u20$as$u20$rayon_core..registry..ThreadSpawn$GT$::spawn::_$u7b$$u7b$closure$u7d$$u7d$::h053d858faa76a9c8 src/third_party/rust/rayon-core/src/registry.rs:100:20
    #49 0x7f4586ba0446 in std::sys_common::backtrace::__rust_begin_short_backtrace::h9fc32286fab2436d /builds/worker/fetches/rust/library/std/src/sys_common/backtrace.rs:122:18
    #50 0x7f4586ba44c8 in std::thread::Builder::spawn_unchecked_::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h80a1fa31f6869abe /builds/worker/fetches/rust/library/std/src/thread/mod.rs:501:17
    #51 0x7f4586ba44c8 in _$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h8e11d16d91e8b5c6 /builds/worker/fetches/rust/library/core/src/panic/unwind_safe.rs:271:9
    #52 0x7f4586ba44c8 in std::panicking::try::do_call::hc4998255eb6f3c63 /builds/worker/fetches/rust/library/std/src/panicking.rs:492:40
    #53 0x7f4586ba44c8 in std::panicking::try::h3d27a1f561f0a178 /builds/worker/fetches/rust/library/std/src/panicking.rs:456:19
    #54 0x7f4586ba44c8 in std::panic::catch_unwind::h8dee73848e97447c /builds/worker/fetches/rust/library/std/src/panic.rs:137:14
    #55 0x7f4586ba44c8 in std::thread::Builder::spawn_unchecked_::_$u7b$$u7b$closure$u7d$$u7d$::h142c7fdb1aef8b33 /builds/worker/fetches/rust/library/std/src/thread/mod.rs:500:30
    #56 0x7f4586ba44c8 in core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf82fb59964141bcd /builds/worker/fetches/rust/library/core/src/ops/function.rs:248:5
    #57 0x7f4586e43432 in std::sys::unix::thread::Thread::new::thread_start::h296aa11e8c800360 std.f6da9894-cgu.4
    #58 0x7f459816c608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
    #59 0x7f4597d33132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Bugmon Analysis
Unable to reproduce bug 1783090 using build mozilla-central 20220803212830-ae4cb105d717. Without a baseline, bugmon is unable to analyze this bug.
Removing bugmon keyword as no further action possible. Please review the bug and re-add the keyword for further analysis.

Glenn, any ideas? It looks like a relatively small test case and loads very sluggishly for me.

Although the testcase is small, the use of

<style>
* {
  scale: 7 0.169882921075 1;
  rotate: 38deg;
}
</style>

means that once an element (like the <pre> text in the testcase) is nested inside several ancestors, some very extreme scaling (and rotation) is being applied, because each level of nesting multiplies another level of transformation onto it. Here, the text is inside about 7 or 8 levels of this transformation.

So I think this is just a case of the font being scaled to a very extreme size, and that's stressing the rasterization pretty hard.

Lee, this seems like a bug, I guess - since we have a maximum glyph size we rasterize? Perhaps that limit doesn't apply if we are rasterizing the glyph in post-transform space?

I can't seem to reproduce this one even with the recommended fuzzing build?

Verified no longer reproducible with m-c 20220818-6e2c5eb109ce.