This is triggered when running gtests with an UBSan build.

To enable this check add the following to your mozconfig:

ac_add_options --enable-address-sanitizer
ac_add_options --enable-undefined-sanitizer="shift"
ac_add_options --disable-jemalloc

Traceback:

[----------] 3 tests from Cairo
[ RUN      ] Cairo.Simple
TestCairo:TryArcs centerY 0.000000, radius 14.000000
TestCairo:TryArcs centerY 1.000000, radius 22.400000
gfx/cairo/cairo/src/cairo-image-surface.c:2537:34: runtime error: left shift of 255 by 24 places cannot be represented in type 'int'
    #0 0x7f993a53531c in color_to_uint32 gfx/cairo/cairo/src/cairo-image-surface.c:2537:34
    #1 0x7f993a535175 in color_to_pixel gfx/cairo/cairo/src/cairo-image-surface.c:2563:9
    #2 0x7f993a531c5a in _composite_boxes gfx/cairo/cairo/src/cairo-image-surface.c:2979:2
    #3 0x7f993a53167c in _clip_and_composite_boxes gfx/cairo/cairo/src/cairo-image-surface.c:3090:14
    #4 0x7f993a538125 in _clip_and_composite_trapezoids gfx/cairo/cairo/src/cairo-image-surface.c:3246:9
    #5 0x7f993a537ca6 in _clip_and_composite_polygon gfx/cairo/cairo/src/cairo-image-surface.c:3625:15
    #6 0x7f993a5229ec in _cairo_image_surface_fill gfx/cairo/cairo/src/cairo-image-surface.c:3816:15
    #7 0x7f993a5855b8 in _cairo_surface_fill gfx/cairo/cairo/src/cairo-surface.c:2352:11
    #8 0x7f993a515b6c in _cairo_gstate_fill gfx/cairo/cairo/src/cairo-gstate.c:1290:15
    #9 0x7f993a5ade16 in _moz_cairo_fill_preserve gfx/cairo/cairo/src/cairo.c:2473:14
    #10 0x7f9931acbc22 in mozilla::layers::TryCircle(double, double, double) gfx/2d/unittest/TestCairo.cpp:24:3
    #11 0x7f9931acb9a8 in mozilla::layers::Cairo_Simple_Test::TestBody() gfx/2d/unittest/TestCairo.cpp:33:3
    #12 0x7f993192f55b in testing::Test::Run() testing/gtest/gtest/src/gtest.cc:2519:5
    #13 0x7f9931930405 in testing::TestInfo::Run() testing/gtest/gtest/src/gtest.cc:2695:11
    #14 0x7f9931930bf1 in testing::TestCase::Run() testing/gtest/gtest/src/gtest.cc:2813:28
    #15 0x7f993193e642 in testing::internal::UnitTestImpl::RunAllTests() testing/gtest/gtest/src/gtest.cc:5179:43
    #16 0x7f993193e106 in testing::UnitTest::Run() testing/gtest/gtest/src/gtest.cc:4788:10
    #17 0x7f99319774f7 in mozilla::RunGTestFunc(int*, char**) testing/gtest/mozilla/GTestRunner.cpp:158:10
    #18 0x7f993c152ee0 in XREMain::XRE_mainStartup(bool*) toolkit/xre/nsAppRunner.cpp:3788:16
    #19 0x7f993c15baf9 in XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) toolkit/xre/nsAppRunner.cpp:4722:12
    #20 0x7f993c15c561 in XRE_main(int, char**, mozilla::BootstrapConfig const&) toolkit/xre/nsAppRunner.cpp:4816:21
    #21 0x560fad060cae in do_main(int, char**, char**) browser/app/nsBrowserApp.cpp:218:22
    #22 0x560fad060344 in main browser/app/nsBrowserApp.cpp:300:16

There's a r+ patch which didn't land and no activity in this bug for 2 weeks.
:jnicol, could you have a look please?
For more information, please visit auto_nag documentation.

This was fixed by updating pixman in bug 1689998