Found while fuzzing m-c 20211217-ba22a155be2e (--enable-address-sanitizer --enable-fuzzing)

This was found by enabling the float-cast-overflow check in UBSan. This type of issue can create inconsistencies across platforms, architectures and optimization levels.

To enable this check add the following to your mozconfig:

ac_add_options --enable-undefined-sanitizer="float-cast-overflow"

To reproduce with the attached test case use the following commands:

$ pip install grizzly-framework
$ python -m grizzly.replay <ubsan-build>/firefox ./testcase.html --xvfb

src/gl.cc:202:17: runtime error: 2.51151e+09 is outside the range of representable values of type 'int'
    #0 0x7fb2ecbb9114 in FloatRange::__glsl_round() const src/gfx/wr/swgl/src/gl.cc:202:17
    #1 0x7fb2ecb1f461 in IntRange clip_distance_range<void draw_quad_spans<unsigned int>(int, glsl::vec2_scalar*, unsigned int, glsl::vec3*, Texture&, Texture&, ClipRect const&)::Edge>(void draw_quad_spans<unsigned int>(int, glsl::vec2_scalar*, unsigned int, glsl::vec3*, Texture&, Texture&, ClipRect const&)::Edge const&, void draw_quad_spans<unsigned int>(int, glsl::vec2_scalar*, unsigned int, glsl::vec3*, Texture&, Texture&, ClipRect const&)::Edge const&) src/gfx/wr/swgl/src/rasterize.h:591:63
    #2 0x7fb2ecb1f461 in void draw_quad_spans<unsigned int>(int, glsl::vec2_scalar*, unsigned int, glsl::vec3*, Texture&, Texture&, ClipRect const&) src/gfx/wr/swgl/src/rasterize.h:951:31
    #3 0x7fb2ec5f529d in draw_quad(int, Texture&, Texture&) src/gfx/wr/swgl/src/rasterize.h:1615:5
    #4 0x7fb2ec5f0638 in void draw_elements<unsigned short>(int, int, unsigned long, VertexArray&, Texture&, Texture&) src/gfx/wr/swgl/src/rasterize.h:1648:7
    #5 0x7fb2ec5f01a6 in DrawElementsInstanced src/gfx/wr/swgl/src/gl.cc:2738:7
    #6 0x7fb2eb441f01 in webrender::device::gl::Device::draw_indexed_triangles_instanced_u16::h037961f88c92ec4f src/gfx/wr/webrender/src/device/gl.rs:3639:9
    #7 0x7fb2eb8d9560 in webrender::renderer::Renderer::draw_instanced_batch::h1adce3b75edd9c4a src/gfx/wr/webrender/src/renderer/mod.rs:2498:17
    #8 0x7fb2eb3e7680 in webrender::renderer::Renderer::draw_alpha_batch_container::ha869cb4deeb18150 src/gfx/wr/webrender/src/renderer/mod.rs:2988:17
    #9 0x7fb2eb3f1a29 in webrender::renderer::Renderer::draw_picture_cache_target::h3ce95858f8e72ff5 src/gfx/wr/webrender/src/renderer/mod.rs:2808:9
    #10 0x7fb2eb3f1a29 in webrender::renderer::Renderer::draw_frame::ha0dabf5c0503f358 src/gfx/wr/webrender/src/renderer/mod.rs:4701:21
    #11 0x7fb2eb3d592a in webrender::renderer::Renderer::render_impl::hc522075a3854dfce src/gfx/wr/webrender/src/renderer/mod.rs:2002:17
    #12 0x7fb2eb3d2787 in webrender::renderer::Renderer::render::h4e11dd3761dd7f7a src/gfx/wr/webrender/src/renderer/mod.rs:1724:30
    #13 0x7fb2eb19b62d in wr_renderer_render src/gfx/webrender_bindings/src/bindings.rs:622:11
    #14 0x7fb2dd0eac41 in mozilla::wr::RendererOGL::UpdateAndRender(mozilla::Maybe<mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> > const&, mozilla::Maybe<mozilla::wr::ImageFormat> const&, mozilla::Maybe<mozilla::Range<unsigned char> > const&, bool*, mozilla::wr::RendererStats*) src/gfx/webrender_bindings/RendererOGL.cpp:185:8
    #15 0x7fb2dd0e9147 in mozilla::wr::RenderThread::UpdateAndRender(mozilla::wr::WrWindowId, mozilla::layers::BaseTransactionId<mozilla::VsyncIdType> const&, mozilla::TimeStamp const&, bool, mozilla::Maybe<mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> > const&, mozilla::Maybe<mozilla::wr::ImageFormat> const&, mozilla::Maybe<mozilla::Range<unsigned char> > const&, bool*) src/gfx/webrender_bindings/RenderThread.cpp:516:31
    #16 0x7fb2dd0e8225 in mozilla::wr::RenderThread::HandleFrameOneDoc(mozilla::wr::WrWindowId, bool) src/gfx/webrender_bindings/RenderThread.cpp:368:3
    #17 0x7fb2dd10a13b in decltype(*(fp).*fp0(Get<0ul>(fp1).PassAsParameter(), Get<1ul>(fp1).PassAsParameter())) mozilla::detail::RunnableMethodArguments<mozilla::wr::WrWindowId, bool>::applyImpl<mozilla::wr::RenderThread, void (mozilla::wr::RenderThread::*)(mozilla::wr::WrWindowId, bool), StoreCopyPassByConstLRef<mozilla::wr::WrWindowId>, StoreCopyPassByConstLRef<bool>, 0ul, 1ul>(mozilla::wr::RenderThread*, void (mozilla::wr::RenderThread::*)(mozilla::wr::WrWindowId, bool), mozilla::Tuple<StoreCopyPassByConstLRef<mozilla::wr::WrWindowId>, StoreCopyPassByConstLRef<bool> >&, std::integer_sequence<unsigned long, 0ul, 1ul>) src/objdir-ff-ubsan/dist/include/nsThreadUtils.h:1147:12
    #18 0x7fb2dd109efb in decltype(applyImpl(fp, fp0, *(this).mArguments, std::integer_sequence<unsigned long, 0ul, 1ul>{})) mozilla::detail::RunnableMethodArguments<mozilla::wr::WrWindowId, bool>::apply<mozilla::wr::RenderThread, void (mozilla::wr::RenderThread::*)(mozilla::wr::WrWindowId, bool)>(mozilla::wr::RenderThread*, void (mozilla::wr::RenderThread::*)(mozilla::wr::WrWindowId, bool)) src/objdir-ff-ubsan/dist/include/nsThreadUtils.h:1153:12
    #19 0x7fb2dd109efb in mozilla::detail::RunnableMethodImpl<mozilla::wr::RenderThread*, void (mozilla::wr::RenderThread::*)(mozilla::wr::WrWindowId, bool), true, (mozilla::RunnableKind)0, mozilla::wr::WrWindowId, bool>::Run() src/objdir-ff-ubsan/dist/include/nsThreadUtils.h:1200:13
    #20 0x7fb2d9ca8c37 in nsThread::ProcessNextEvent(bool, bool*) src/xpcom/threads/nsThread.cpp:1177:16
    #21 0x7fb2d9cb21d6 in NS_ProcessNextEvent(nsIThread*, bool) src/xpcom/threads/nsThreadUtils.cpp:467:10
    #22 0x7fb2db253744 in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) src/ipc/glue/MessagePump.cpp:330:5
    #23 0x7fb2db0b1235 in MessageLoop::RunInternal() src/ipc/chromium/src/base/message_loop.cc:331:10
    #24 0x7fb2db0b1235 in MessageLoop::RunHandler() src/ipc/chromium/src/base/message_loop.cc:324:3
    #25 0x7fb2db0b1235 in MessageLoop::Run() src/ipc/chromium/src/base/message_loop.cc:306:3
    #26 0x7fb2d9ca1c09 in nsThread::ThreadFunc(void*) src/xpcom/threads/nsThread.cpp:391:10
    #27 0x7fb3045bb499 in _pt_root src/nsprpub/pr/src/pthreads/ptthread.c:201:5
    #28 0x7fb3041ef6da in start_thread /build/glibc-S9d2JN/glibc-2.27/nptl/pthread_create.c:463
    #29 0x7fb3031cd71e in __clone /build/glibc-S9d2JN/glibc-2.27/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95

A Pernosco session is available here: https://pernos.co/debug/FExVhd-f6nPHpMqsqvXoyg/index.html

The severity field is not set for this bug.
:jimm, could you have a look please?

For more information, please visit auto_nag documentation.

This issue is currently triggered while fuzzing with the 'float-cast-overflow' UBSan check enabled. This issue will need to be addressed before the check can be enabled by default.

If it requires too much effort to fix immediately please ni? me and let me know. If necessary it will be added to a suppression list. Thank you :)

Lee, could you comment on https://bugzilla.mozilla.org/show_bug.cgi?id=1746913#c4 ?

https://hg.mozilla.org/mozilla-central/rev/e50be4953b80