Found with m-c:
BuildID=20190331215222
SourceStamp=70a50fe09a18b2516e9ffdb2083debfab7de720d

Assertion failure: mSize > 0 (invalid size), at src/ipc/glue/Shmem.cpp:257

#0 mozilla::ipc::Shmem::AssertInvariants() const src/ipc/glue/Shmem.cpp:256:3
#1 mozilla::ipc::Shmem::ShareTo(mozilla::ipc::Shmem::PrivateIPDLCaller, int, int) src/ipc/glue/Shmem.cpp:421:3
#2 mozilla::ipc::IToplevelProtocol::ToplevelState::CreateSharedMemory(unsigned long, mozilla::ipc::SharedMemory::SharedMemoryType, bool, int*) src/ipc/glue/ProtocolUtils.cpp:736:13
#3 mozilla::ipc::IProtocol::AllocShmem(unsigned long, mozilla::ipc::SharedMemory::SharedMemoryType, mozilla::ipc::Shmem*) src/ipc/glue/ProtocolUtils.cpp:447:31
#4 mozilla::RemoteDecoderChild::Decode(mozilla::MediaRawData*) src/dom/media/ipc/RemoteDecoderChild.cpp:107:8
#5 non-virtual thunk to mozilla::RemoteDecoderChild::Decode(mozilla::MediaRawData*) src/dom/media/ipc/RemoteDecoderChild.cpp
#6 mozilla::RemoteMediaDataDecoder::Decode(mozilla::MediaRawData*)::$_4::operator()() const src/dom/media/ipc/RemoteMediaDataDecoder.cpp:66:62
#7 mozilla::detail::ProxyFunctionRunnable<mozilla::RemoteMediaDataDecoder::Decode(mozilla::MediaRawData*)::$_4, mozilla::MozPromise<nsTArray<RefPtr<mozilla::MediaData> >, mozilla::MediaResult, true> >::Run() src/obj-firefox/dist/include/mozilla/MozPromise.h:1419:29
#8 mozilla::EventTargetWrapper::Runner::Run() src/xpcom/threads/AbstractThread.cpp:113:25
#9 nsThread::ProcessNextEvent(bool, bool*) src/xpcom/threads/nsThread.cpp:1180:14
#10 NS_ProcessNextEvent(nsIThread*, bool) src/xpcom/threads/nsThreadUtils.cpp:482:10
#11 mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) src/ipc/glue/MessagePump.cpp:333:5
#12 MessageLoop::RunInternal() src/ipc/chromium/src/base/message_loop.cc:315:10
#13 MessageLoop::Run() src/ipc/chromium/src/base/message_loop.cc:290:3
#14 nsThread::ThreadFunc(void*) src/xpcom/threads/nsThread.cpp:454:11
#15 _pt_root src/nsprpub/pr/src/pthreads/ptthread.c:201:5

Jean-Yves,
The test case above results in aSample->Size() returning 0 here[1]. Should we check for aSample->Size() > 0 and if not, reject with NS_ERROR_DOM_MEDIA_DECODE_ERR like here[2]?

[1] https://searchfox.org/mozilla-central/source/dom/media/ipc/RemoteDecoderChild.cpp#107
[2] https://searchfox.org/mozilla-central/source/dom/media/ipc/RemoteDecoderChild.cpp#99

Or would it make more sense to zero size here[1] and return false?

[1] https://searchfox.org/mozilla-central/source/ipc/glue/ProtocolUtils.cpp#443

What would make the most sense is to never return a sample with zero content. We already do that for mp4.
https://searchfox.org/mozilla-central/source/dom/media/mp4/MP4Demuxer.cpp#500

should do the same with webm

:bryce, is this something you can look at? thanks

To make sure I understand what's happening here: the underlying demuxers should not return 0 sized samples. Mp4 already does this, webm doesn't, so we need a change the webm demuxer and that should sort this (without modifying any of the other code mentioned here).

Will start work based on the above understanding, so let me know if it's incorrect.

(In reply to Bryce Seager van Dyk (:bryce) from comment #4)

To make sure I understand what's happening here: the underlying demuxers should not return 0 sized samples. Mp4 already does this, webm doesn't, so we need a change the webm demuxer and that should sort this (without modifying any of the other code mentioned here).

Will start work based on the above understanding, so let me know if it's incorrect.

That sounds correct to me.

The mp4 demuxer already does this, so have the WebM case follow suit. This
avoids asserts with creating 0 size shmems in the case where the demuxer is
being used with the remote data decoder.

Depends on D25846

Backed out for crashtest assertion failures on RemoteDecoderManagerParent.cpp

backout: https://hg.mozilla.org/integration/autoland/rev/9b412ab507f36e2efb015a8be29170671c504b95

push: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=a9048220ad8bddb66327733185cecca91af6db9f&selectedJob=238427219

failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=238427219&repo=autoland&lineNumber=12843

16:13:45 INFO - REFTEST TEST-START | file:///Z:/task_1554479951/build/tests/reftest/tests/dom/media/test/crashtests/empty-samples.webm
16:13:45 INFO - REFTEST TEST-LOAD | file:///Z:/task_1554479951/build/tests/reftest/tests/dom/media/test/crashtests/empty-samples.webm | 641 / 3708 (17%)
16:13:45 INFO - ++DOMWINDOW == 310 (12BE6400) [pid = 5988] [serial = 2083] [outer = 01072820]
16:13:46 INFO - ### XPCOM_MEM_BLOAT_LOG defined -- logging bloat/leaks to c:\users\task_1554479951\appdata\local\temp\tmput8cu4.mozrunner\runreftest_leaks_rdd_pid5200.log
16:13:46 INFO - [RDD 5200, Main Thread] WARNING: NS_ENSURE_TRUE(InitStaticMembers()) failed: file z:/build/build/src/modules/libpref/Preferences.cpp, line 4848
16:13:46 INFO - [RDD 5200, Main Thread] WARNING: NS_ENSURE_TRUE(InitStaticMembers()) failed: file z:/build/build/src/obj-firefox/dist/include/mozilla/Preferences.h, line 192
16:13:46 INFO - [RDD 5200, Main Thread] WARNING: NS_ENSURE_TRUE(InitStaticMembers()) failed: file z:/build/build/src/modules/libpref/Preferences.cpp, line 4584
16:13:46 INFO - [RDD 5200, Main Thread] WARNING: NS_ENSURE_TRUE(InitStaticMembers()) failed: file z:/build/build/src/modules/libpref/Preferences.cpp, line 4584
16:13:46 INFO - [RDD 5200, Main Thread] WARNING: NS_ENSURE_TRUE(InitStaticMembers()) failed: file z:/build/build/src/modules/libpref/Preferences.cpp, line 4619
16:13:46 INFO - [RDD 5200, Main Thread] WARNING: NS_ENSURE_TRUE(InitStaticMembers()) failed: file z:/build/build/src/modules/libpref/Preferences.cpp, line 4584
16:13:46 INFO - [Child 5988, MediaPDecoder #1] WARNING: Unimplemented function NotifyDataArrived: file z:/build/build/src/dom/media/mp3/MP3Demuxer.cpp, line 72
16:13:46 INFO - Assertion failure: (((HRESULT)(hr)) >= 0), at z:/build/build/src/dom/media/ipc/RemoteDecoderManagerParent.cpp:87
16:14:08 INFO - #01: nsThread::ProcessNextEvent(bool,bool *) [xpcom/threads/nsThread.cpp:1180]
16:14:08 INFO -
16:14:08 INFO - #02: NS_ProcessNextEvent(nsIThread *,bool) [xpcom/threads/nsThreadUtils.cpp:486]
16:14:08 INFO -
16:14:08 INFO - #03: mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate *) [ipc/glue/MessagePump.cpp:333]
16:14:08 INFO -
16:14:08 INFO - #04: MessageLoop::RunInternal() [ipc/chromium/src/base/message_loop.cc:315]
16:14:08 INFO -
16:14:08 INFO - #05: MessageLoop::RunHandler() [ipc/chromium/src/base/message_loop.cc:309]
16:14:08 INFO -
16:14:08 INFO - #06: MessageLoop::Run() [ipc/chromium/src/base/message_loop.cc:291]
16:14:08 INFO -
16:14:08 INFO - #07: nsThread::ThreadFunc(void *) [xpcom/threads/nsThread.cpp:456]
16:14:08 INFO -
16:14:09 INFO - #08: _PR_NativeRunThread [nsprpub/pr/src/threads/combined/pruthr.c:397]
16:14:09 INFO -
16:14:09 INFO - #09: static unsigned int pr_root(void *) [nsprpub/pr/src/md/windows/w95thred.c:137]
16:14:09 INFO -
16:14:09 INFO - #10: ucrtbase.DLL + 0x3e16f
16:14:09 INFO -
16:14:09 INFO - #11: kernel32.dll + 0x53c45
16:14:09 INFO -
16:14:09 INFO - #12: static void patched_BaseThreadInitThunk(int, void *, void *) [mozglue/build/WindowsDllBlocklist.cpp:705]
16:14:09 INFO -
16:14:09 INFO - #13: ntdll.dll + 0x637f5
16:14:09 INFO -
16:14:09 INFO - #14: ntdll.dll + 0x637c8
16:14:09 INFO -
16:14:09 INFO - ###!!! [Child][MessageChannel::SendAndWait] Error: Channel error: cannot send/recv
16:14:09 INFO - IPDL protocol error: constructor for actor failed

Hoisted by my own crash test. Unclear to me at this stage exactly what the issue is. Based on the line it appears something is going wrong with us initializing the COM lib.

:mjf, have you seen this before?

Looks like Bug 1535704 to me.

(In reply to Michael Froman [:mjf] from comment #11)

Looks like Bug 1535704 to me.

Thanks. I'm going to mark that as a blocker and take another look once it's resolved.

https://hg.mozilla.org/mozilla-central/rev/1a102b35a05d
https://hg.mozilla.org/mozilla-central/rev/d8571fb523a0

Crash test landed with fix.