I am working on getting our tests running on windows11, this has been pretty smooth with a few hiccups. One larger remaining area to look into is the mda failures on asan/debug for wmfme variant.

You can see a try push here:
https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&searchStr=wmfme&revision=e60dd525bd3709a9a1bf548e60618cdfcc113492

I tried skipping a lot of the tests which failed frequently as timedout but more and more kept failing.

in debug mode, I see an assertion which looks to be the root cause:

[task 2022-12-29T02:38:28.144Z] 02:38:28     INFO - TEST-START | dom/media/test/test_VideoPlaybackQuality.html
[task 2022-12-29T02:38:28.145Z] 02:38:28     INFO - GECKO(8880) | ### XPCOM_MEM_BLOAT_LOG defined -- logging bloat/leaks to C:\Users\task_167227772687855\AppData\Local\Temp\tmpwsggwx18.mozrunner\runtests_leaks_tab_pid6820.log
[task 2022-12-29T02:38:28.146Z] 02:38:28     INFO - GECKO(8880) | [6820, Main Thread] WARNING: XPCOM_MEM_BLOAT_LOG is set, disabling native allocations.: file /builds/worker/checkouts/gecko/tools/profiler/core/platform.cpp:339
[task 2022-12-29T02:38:28.147Z] 02:38:28     INFO - GECKO(8880) | ### XPCOM_MEM_BLOAT_LOG defined -- logging bloat/leaks to C:\Users\task_167227772687855\AppData\Local\Temp\tmpwsggwx18.mozrunner\runtests_leaks_utility_pid9220.log
[task 2022-12-29T02:38:28.147Z] 02:38:28     INFO - GECKO(8880) | [9220, Main Thread] WARNING: XPCOM_MEM_BLOAT_LOG is set, disabling native allocations.: file /builds/worker/checkouts/gecko/tools/profiler/core/platform.cpp:339
[task 2022-12-29T02:38:28.148Z] 02:38:28     INFO - GECKO(8880) | [Parent 1680, Main Thread] WARNING: IPC message 'PContent::Msg_UpdateMediaCodecsSupported' discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:504
[task 2022-12-29T02:38:28.149Z] 02:38:28     INFO - GECKO(8880) | [Child 1392, Main Thread] WARNING: Extra shutdown CC: 'i < NORMAL_SHUTDOWN_COLLECTIONS', file /builds/worker/checkouts/gecko/xpcom/base/nsCycleCollector.cpp:3384
[task 2022-12-29T02:38:28.150Z] 02:38:28     INFO - GECKO(8880) | [Child 1392, Main Thread] WARNING: NS_ENSURE_TRUE(InitStaticMembers()) failed: file /builds/worker/workspace/obj-build/dist/include/mozilla/Preferences.h:129
[task 2022-12-29T02:38:28.151Z] 02:38:28     INFO - GECKO(8880) | [Child 1392, Main Thread] WARNING: NS_ENSURE_TRUE(Preferences::InitStaticMembers()) failed: file /builds/worker/checkouts/gecko/modules/libpref/Preferences.cpp:4663
[task 2022-12-29T02:38:28.151Z] 02:38:28     INFO - GECKO(8880) | [2022-12-29T02:38:14Z WARN  rkv::backend::impl_safe::environment] `load_ratio()` is irrelevant for this storage backend.
[task 2022-12-29T02:38:28.151Z] 02:38:28     INFO - GECKO(8880) | MEMORY STAT | vsize 2103811MB | vsizeMaxContiguous 104087157MB | residentFast 78MB | heapAllocated 9MB
[task 2022-12-29T02:38:28.152Z] 02:38:28     INFO - GECKO(8880) | JavaScript error: , line 0: uncaught exception: Object
[task 2022-12-29T02:38:28.153Z] 02:38:28     INFO - GECKO(8880) | Assertion failure: !mManagerThread->IsOnCurrentThread(), at /builds/worker/checkouts/gecko/dom/media/platforms/wmf/MFMediaSource.cpp:537
[task 2022-12-29T02:38:28.153Z] 02:38:28     INFO - GECKO(8880) | #01: mozilla::MFMediaSource::GetService(_GUID const&, _GUID const&, void**) [dom/media/platforms/wmf/MFMediaSource.cpp:406]
[task 2022-12-29T02:38:28.153Z] 02:38:28     INFO - GECKO(8880) | #02: MFGetService [C:\Windows\System32\mfcore.dll + 0x29d17]
[task 2022-12-29T02:38:28.154Z] 02:38:28     INFO - GECKO(8880) | #03: MFCreateTopologyNode [C:\Windows\System32\mfcore.dll + 0x275b4]
[task 2022-12-29T02:38:28.154Z] 02:38:28     INFO - GECKO(8880) | #04: MFCreateTopologyNode [C:\Windows\System32\mfcore.dll + 0x291d4]
[task 2022-12-29T02:38:28.154Z] 02:38:28     INFO - GECKO(8880) | #05: MFGetService [C:\Windows\System32\mfcore.dll + 0x29d17]
[task 2022-12-29T02:38:28.155Z] 02:38:28     INFO - GECKO(8880) | #06: MFCreateSequencerSource [C:\Windows\System32\mfcore.dll + 0x934e3]
[task 2022-12-29T02:38:28.156Z] 02:38:28     INFO - GECKO(8880) | #07: MFCreateSequencerSource [C:\Windows\System32\mfcore.dll + 0x93331]
[task 2022-12-29T02:38:28.156Z] 02:38:28     INFO - GECKO(8880) | #08: DllGetClassObject [C:\Windows\System32\MFMediaEngine.dll + 0x5688f]
[task 2022-12-29T02:38:28.157Z] 02:38:28     INFO - GECKO(8880) | #09: DllCanUnloadNow [C:\Windows\System32\MFMediaEngine.dll + 0x11036e]
[task 2022-12-29T02:38:28.157Z] 02:38:28     INFO - GECKO(8880) | #10: ??? [C:\Windows\System32\MFMediaEngine.dll + 0x2c787]
[task 2022-12-29T02:38:28.157Z] 02:38:28     INFO - GECKO(8880) | #11: DllCanUnloadNow [C:\Windows\System32\MFMediaEngine.dll + 0x19e2c7]
[task 2022-12-29T02:38:28.158Z] 02:38:28     INFO - GECKO(8880) | #12: mozilla::MFMediaEngineParent::DestroyEngineIfExists(mozilla::Maybe<mozilla::MediaResult> const&) [dom/media/ipc/MFMediaEngineParent.cpp:99]

Are we missing some VC runtime on the system, or some media toolkit?

jmaher: please ni :alwu when he gets back next week

Could you point me how to run those tests on win 11? Then I can verify if my patch works or not. Also, how frequently those test would fail?

Thanks!

these are perma fail- not always the same tests failing, but the same symptoms in the log file. debug and asan are not too happy. Here is how to run them:
./mach try fuzzy --no-artifact --worker-override="win10-64-2004=gecko-t/win11-64-2009-beta" -q 'test-windows10-64 mochitest-media wmfme'

Based on the callstack, this function can be called on the manager
thread as well.

https://hg.mozilla.org/mozilla-central/rev/9cee25fca2ca

debug is all green, but asan still fails:

[task 2023-01-10T17:29:27.173Z] 17:29:27     INFO - GECKO(6992) | =================================================================
[task 2023-01-10T17:29:27.174Z] 17:29:27    ERROR - GECKO(6992) | ==4196==ERROR: AddressSanitizer: heap-use-after-free on address 0x11552e405268 at pc 0x7ff981d3b6f2 bp 0x003124c7e020 sp 0x003124c7e068
[task 2023-01-10T17:29:27.174Z] 17:29:27     INFO - GECKO(6992) | READ of size 2 at 0x11552e405268 thread T94
[task 2023-01-10T17:29:27.330Z] 17:29:27     INFO - GECKO(6992) | ==4196==WARNING: Can't read from symbolizer at fd 3976
[task 2023-01-10T17:29:27.331Z] 17:29:27     INFO - GECKO(6992) | ==4196==WARNING: Can't write to symbolizer at fd 3972
[task 2023-01-10T17:29:27.358Z] 17:29:27     INFO - GECKO(6992) | ==4196==WARNING: Can't read from symbolizer at fd 3968
[task 2023-01-10T17:29:27.358Z] 17:29:27     INFO - GECKO(6992) | ==4196==WARNING: Can't write to symbolizer at fd 3976
[task 2023-01-10T17:29:27.383Z] 17:29:27     INFO - GECKO(6992) | ==4196==WARNING: Can't read from symbolizer at fd 3972
[task 2023-01-10T17:29:27.384Z] 17:29:27     INFO - GECKO(6992) | ==4196==WARNING: Can't write to symbolizer at fd 3968
[task 2023-01-10T17:29:27.410Z] 17:29:27     INFO - GECKO(6992) | ==4196==WARNING: Can't read from symbolizer at fd 3976
[task 2023-01-10T17:29:27.410Z] 17:29:27     INFO - GECKO(6992) | ==4196==WARNING: Can't write to symbolizer at fd 3972
[task 2023-01-10T17:29:27.412Z] 17:29:27     INFO - GECKO(6992) | ==4196==WARNING: Failed to use and restart external symbolizer!
[task 2023-01-10T17:29:28.500Z] 17:29:28     INFO - GECKO(6992) |     #0 0x7ff981d3b6f1  (Z:\task_167337071820987\build\application\firefox\xul.dll+0x18845b6f1)
[task 2023-01-10T17:29:28.502Z] 17:29:28     INFO - GECKO(6992) |     #1 0x7ff97a521f7c  (Z:\task_167337071820987\build\application\firefox\xul.dll+0x180c41f7c)
[task 2023-01-10T17:29:28.503Z] 17:29:28     INFO - GECKO(6992) |     #2 0x7ff97a55b821  (Z:\task_167337071820987\build\application\firefox\xul.dll+0x180c7b821)
[task 2023-01-10T17:29:28.503Z] 17:29:28     INFO - GECKO(6992) |     #3 0x7ff97a548574  (Z:\task_167337071820987\build\application\firefox\xul.dll+0x180c68574)
[task 2023-01-10T17:29:28.504Z] 17:29:28     INFO - GECKO(6992) |     #4 0x7ff97a55739d in NS_ProcessNextEvent /builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.cpp:476
[task 2023-01-10T17:29:28.504Z] 17:29:28     INFO - GECKO(6992) |     #5 0x7ff97bb5c84e in mozilla::ipc::MessagePumpForNonMainThreads::Run /builds/worker/checkouts/gecko/ipc/glue/MessagePump.cpp:300
[task 2023-01-10T17:29:28.505Z] 17:29:28     INFO - GECKO(6992) |     #6 0x7ff97ba73a82 in MessageLoop::RunHandler /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:374
[task 2023-01-10T17:29:28.506Z] 17:29:28     INFO - GECKO(6992) |     #7 0x7ff97ba73857 in MessageLoop::Run /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:356
[task 2023-01-10T17:29:28.506Z] 17:29:28     INFO - GECKO(6992) |     #8 0x7ff97a53e0d7 in nsThread::ThreadFunc /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:383
[task 2023-01-10T17:29:28.507Z] 17:29:28     INFO - GECKO(6992) |     #9 0x7ff998d31a75 in _PR_NativeRunThread /builds/worker/checkouts/gecko/nsprpub/pr/src/threads/combined/pruthr.c:399
[task 2023-01-10T17:29:28.508Z] 17:29:28     INFO - GECKO(6992) |     #10 0x7ff998d0b55b in pr_root /builds/worker/checkouts/gecko/nsprpub/pr/src/md/windows/w95thred.c:139
[task 2023-01-10T17:29:28.508Z] 17:29:28     INFO - GECKO(6992) |     #11 0x7ff9c6ab9362 in recalloc+0xa2 (C:\Windows\System32\ucrtbase.dll+0x180029362)
[task 2023-01-10T17:29:28.509Z] 17:29:28     INFO - GECKO(6992) |     #12 0x7ff997f1abb3 in __asan::AsanThread::ThreadStart /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/asan_thread.cpp:277
[task 2023-01-10T17:29:28.510Z] 17:29:28     INFO - GECKO(6992) |     #13 0x7ff9c74826bc in BaseThreadInitThunk+0x1c (C:\Windows\System32\KERNEL32.DLL+0x1800126bc)
[task 2023-01-10T17:29:28.511Z] 17:29:28     INFO - GECKO(6992) |     #14 0x7ff99c5b927e in patched_BaseThreadInitThunk /builds/worker/checkouts/gecko/toolkit/xre/dllservices/mozglue/WindowsDllBlocklist.cpp:587
[task 2023-01-10T17:29:28.511Z] 17:29:28     INFO - GECKO(6992) |     #15 0x7ff9c962dfb7 in RtlUserThreadStart+0x27 (C:\Windows\SYSTEM32\ntdll.dll+0x18005dfb7)
[task 2023-01-10T17:29:28.512Z] 17:29:28     INFO - GECKO(6992) | 0x11552e405268 is located 40 bytes inside of 280-byte region [0x11552e405240,0x11552e405358)
...

If there is something at the system level I need to do, let me know. I have heard reports that win11/ASAN is not running as expected ( Bug 1809415), our other tests seem to work (after fixing bug 1806091) - this failure only seems to happen on wmfme.

:alwu, if you could take another look at this when you get some time, I would appreciate that!

That error on asan doesn't look like the same error, I've filed another bug 1809502 to track it.