This was found by enabling the function check in UBSan and running existing tests. This type of issue can create inconsistencies across platforms, architectures and optimization levels.

Found with m-c 20220124-9b23d1bb84b.

To enable this check add the following to your mozconfig:

ac_add_options --enable-undefined-sanitizer="function"

This issue is found by the existing test: dom/media/webrtc/tests/mochitests/test_peerConnection_basicH264Video.html

INFO - GECKO(6052) | /builds/worker/checkouts/gecko/dom/media/gmp/GMPLoader.cpp:49:12: runtime error: call to function GMPInit through pointer to incorrect function type 'GMPErr (*)(const GMPPlatformAPI *)'
INFO - GECKO(6052) | /builds/worker/checkouts/gecko/dom/media/gmp-plugin-openh264/gmp-fake-openh264.cpp:392: note: GMPInit defined here
INFO - GECKO(6052) |     #0 0x7fb36acbac05 in mozilla::gmp::PassThroughGMPAdapter::GMPInit(GMPPlatformAPI const*) /builds/worker/checkouts/gecko/dom/media/gmp/GMPLoader.cpp:49:12
INFO - GECKO(6052) |     #1 0x7fb36aca08f2 in mozilla::gmp::GMPLoader::Load(char const*, unsigned int, GMPPlatformAPI const*, mozilla::gmp::GMPAdapter*) /builds/worker/checkouts/gecko/dom/media/gmp/GMPLoader.cpp:116:17
INFO - GECKO(6052) |     #2 0x7fb36aca0060 in mozilla::gmp::GMPChild::RecvStartPlugin(nsTString<char16_t> const&) /builds/worker/checkouts/gecko/dom/media/gmp/GMPChild.cpp:551:20
INFO - GECKO(6052) |     #3 0x7fb3666af5c7 in mozilla::gmp::PGMPChild::OnMessageReceived(IPC::Message const&, IPC::Message*&) /builds/worker/workspace/obj-build/ipc/ipdl/PGMPChild.cpp:754:52
INFO - GECKO(6052) |     #4 0x7fb366330ab2 in mozilla::ipc::MessageChannel::DispatchSyncMessage(mozilla::ipc::ActorLifecycleProxy*, IPC::Message const&, IPC::Message*&) /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:2006:25
INFO - GECKO(6052) |     #5 0x7fb36632e8ed in mozilla::ipc::MessageChannel::DispatchMessage(IPC::Message&&) /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:1960:9
INFO - GECKO(6052) |     #6 0x7fb36632fcad in mozilla::ipc::MessageChannel::RunMessage(mozilla::ipc::MessageChannel::MessageTask&) /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:1823:3
INFO - GECKO(6052) |     #7 0x7fb366330378 in mozilla::ipc::MessageChannel::MessageTask::Run() /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:1851:14
INFO - GECKO(6052) |     #8 0x7fb3662438f8 in MessageLoop::RunTask(already_AddRefed<nsIRunnable>) /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:454:11
INFO - GECKO(6052) |     #9 0x7fb36624463e in MessageLoop::DeferOrRunPendingTask(MessageLoop::PendingTask&&) /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:463:5
INFO - GECKO(6052) |     #10 0x7fb366244ecb in MessageLoop::DoWork() /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:538:13
INFO - GECKO(6052) |     #11 0x7fb366246237 in base::MessagePumpDefault::Run(base::MessagePump::Delegate*) /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_pump_default.cc:35:31
INFO - GECKO(6052) |     #12 0x7fb3662434a1 in RunInternal /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:331:10
INFO - GECKO(6052) |     #13 0x7fb3662434a1 in RunHandler /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:324:3
INFO - GECKO(6052) |     #14 0x7fb3662434a1 in MessageLoop::Run() /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:306:3
INFO - GECKO(6052) |     #15 0x7fb370a194a8 in XRE_InitChildProcess(int, char**, XREChildData const*) /builds/worker/checkouts/gecko/toolkit/xre/nsEmbedFunctions.cpp:707:34
INFO - GECKO(6052) |     #16 0x555911d835f5 in content_process_main(mozilla::Bootstrap*, int, char**) /builds/worker/checkouts/gecko/ipc/app/../contentproc/plugin-container.cpp:57:28
INFO - GECKO(6052) |     #17 0x555911d838a5 in main /builds/worker/checkouts/gecko/ipc/app/MozillaRuntimeMain.cpp:72:11
INFO - GECKO(6052) |     #18 0x7fb35f0edb96 in __libc_start_main /tmp/glibc/csu/../csu/libc-start.c:310
INFO - GECKO(6052) |     #19 0x555911cd26dc in _start (/builds/worker/workspace/build/application/firefox/plugin-container+0x6d6dc)

Set it to P3/S3 for now and CCed Bryce in case he thinks this deserves a higher priority.

This issue is currently triggered in CI when the 'function' UBSan check is enabled. This issue will need to be addressed before the check can be enabled by default.

If it requires too much effort to fix immediately please ni? me and let me know. If necessary it will be added to a suppression list. Thank you :)

Stealing NI. I'll try to get this sorted in the near future. If I haven't in the next week, please ping me.

Think this should be straight forward. Our fake openh264 used in testing is missing a const, let's get that added in there. Patch incoming.

The signature for GMPInit should have a const arg. This is missing for the fake
plugin here, and that is potentially problematic (if for no other reason than it
breaks UBSan checking of functions). Given that the function doesn't do anything
with its arg for the fake plugin, we can trivially make it const.

https://hg.mozilla.org/mozilla-central/rev/9b719aca4e38