User Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0

Steps to reproduce:

Log in to https://open.spotify.com/

Actual results:

The bar at the top of the page says "WidevineCDM plugin has crashed."

Expected results:

Spotify is able to play music.

The crash report: https://crash-stats.mozilla.org/report/index/92723eb0-d1ee-4725-9292-1f8b60210814

I installed Firefox as a Flatpak -- and this version does not crash in the same environment.
Firefox 91.0 (64-bit) mozilla-flatpak - 1.0
Widevine 4.10.2209.1

The crashing version is the same 91 64-bit and with the same Widevine 4.10.2209.1 just installed natively from Fedora package repository.

I also tried this in a fresh clean profile in non-flatpak Firefox -- still crashes.

Have the same issue on Fedora 35.
Running with MOZ_DISABLE_GMP_SANDBOX=1 solves the issue.
See below some sandbox related logs before crash:
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: attempt to open unexpected file /usr/lib64/firefox/libdl.so.2
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: attempt to open unexpected file /lib64/libdl.so.2
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: attempt to open unexpected file /lib64/glibc-hwcaps/x86-64-v3/libdl.so.2
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: seccomp sandbox violation: pid 7487, tid 7487, syscall 262, args 4294967196 140737200729248 140737200729440 0 4294967295 140737200729248.
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: attempt to open unexpected file /lib64/glibc-hwcaps/x86-64-v2/libdl.so.2
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: seccomp sandbox violation: pid 7487, tid 7487, syscall 262, args 4294967196 140737200729248 140737200729440 0 4294967295 140737200729248.
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: attempt to open unexpected file /lib64/tls/haswell/x86_64/libdl.so.2
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: seccomp sandbox violation: pid 7487, tid 7487, syscall 262, args 4294967196 140737200729248 140737200729440 0 4294967295 140737200729248.
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: attempt to open unexpected file /lib64/tls/haswell/libdl.so.2
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: seccomp sandbox violation: pid 7487, tid 7487, syscall 262, args 4294967196 140737200729248 140737200729440 0 4294967295 140737200729248.
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: attempt to open unexpected file /lib64/tls/x86_64/libdl.so.2
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: seccomp sandbox violation: pid 7487, tid 7487, syscall 262, args 4294967196 140737200729248 140737200729440 0 4294967295 140737200729248.
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: attempt to open unexpected file /lib64/tls/libdl.so.2
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: seccomp sandbox violation: pid 7487, tid 7487, syscall 262, args 4294967196 140737200729248 140737200729440 0 4294967295 140737200729248.
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: attempt to open unexpected file /lib64/haswell/x86_64/libdl.so.2
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: seccomp sandbox violation: pid 7487, tid 7487, syscall 262, args 4294967196 140737200729248 140737200729440 0 4294967295 140737200729248.
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: attempt to open unexpected file /lib64/haswell/libdl.so.2
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: seccomp sandbox violation: pid 7487, tid 7487, syscall 262, args 4294967196 140737200729248 140737200729440 0 4294967295 140737200729248.
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: attempt to open unexpected file /lib64/x86_64/libdl.so.2
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: seccomp sandbox violation: pid 7487, tid 7487, syscall 262, args 4294967196 140737200729248 140737200729440 0 4294967295 140737200729248.
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: attempt to open unexpected file /lib64/libdl.so.2
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: seccomp sandbox violation: pid 7487, tid 7487, syscall 262, args 4294967196 140737200729248 140737200729440 0 4294967295 140737200729248.
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: attempt to open unexpected file /usr/lib64/glibc-hwcaps/x86-64-v3/libdl.so.2
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: seccomp sandbox violation: pid 7487, tid 7487, syscall 262, args 4294967196 140737200729248 140737200729440 0 4294967295 140737200729248.
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: attempt to open unexpected file /usr/lib64/glibc-hwcaps/x86-64-v2/libdl.so.2
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: seccomp sandbox violation: pid 7487, tid 7487, syscall 262, args 4294967196 140737200729248 140737200729440 0 4294967295 140737200729248.
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: attempt to open unexpected file /usr/lib64/tls/haswell/x86_64/libdl.so.2
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: seccomp sandbox violation: pid 7487, tid 7487, syscall 262, args 4294967196 140737200729248 140737200729440 0 4294967295 140737200729248.
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: attempt to open unexpected file /usr/lib64/tls/haswell/libdl.so.2
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: seccomp sandbox violation: pid 7487, tid 7487, syscall 262, args 4294967196 140737200729248 140737200729440 0 4294967295 140737200729248.
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: attempt to open unexpected file /usr/lib64/tls/x86_64/libdl.so.2
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: seccomp sandbox violation: pid 7487, tid 7487, syscall 262, args 4294967196 140737200729248 140737200729440 0 4294967295 140737200729248.
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: attempt to open unexpected file /usr/lib64/tls/libdl.so.2
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: seccomp sandbox violation: pid 7487, tid 7487, syscall 262, args 4294967196 140737200729248 140737200729440 0 4294967295 140737200729248.
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: attempt to open unexpected file /usr/lib64/haswell/x86_64/libdl.so.2
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: seccomp sandbox violation: pid 7487, tid 7487, syscall 262, args 4294967196 140737200729248 140737200729440 0 4294967295 140737200729248.
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: attempt to open unexpected file /usr/lib64/haswell/libdl.so.2
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: seccomp sandbox violation: pid 7487, tid 7487, syscall 262, args 4294967196 140737200729248 140737200729440 0 4294967295 140737200729248.
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: attempt to open unexpected file /usr/lib64/x86_64/libdl.so.2
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: seccomp sandbox violation: pid 7487, tid 7487, syscall 262, args 4294967196 140737200729248 140737200729440 0 4294967295 140737200729248.
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: attempt to open unexpected file /usr/lib64/libdl.so.2
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: seccomp sandbox violation: pid 7487, tid 7487, syscall 262, args 4294967196 140737200729248 140737200729440 0 4294967295 140737200729248.

Thanks for the report and the follow up. Alex, could you check if disabling the GMP sandbox, as in comment 5, also helps your case?

Comment 5 points to syscall 262 causing issues. syscall 262 looks to be newfstatat. Looks like bug 1673770 is some prior art in this area.

:jld, do you have any thoughts as to what needs to be fixed here?

Yes, confirming that setting MOZ_DISABLE_GMP_SANDBOX=1 the environment solves it for me.

(In reply to Andrey from comment #5)

Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: attempt to open unexpected file /usr/lib64/firefox/libdl.so.2
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: attempt to open unexpected file /lib64/libdl.so.2
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: attempt to open unexpected file /lib64/glibc-hwcaps/x86-64-v3/libdl.so.2

This looks like the dynamic loader trying to resolve libdl.so.2; a copy of libwidevinecdm.so I have locally (version 4.10.2209.1 if it matters) depends on it. But, the libxul.so from my Firefox build also depends on it, and so does the one in Fedora's package, so it should already be loaded. There's something odd going on here.

I have opened bug for Fedora too (for tracking).
Please find backtrace of the crash here: https://bugzilla.redhat.com/show_bug.cgi?id=1993821#c1

(In reply to Jed Davis [:jld] ⟨⏰|UTC-6⟩ ⟦he/him⟧ from comment #9)

(In reply to Andrey from comment #5)

Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: attempt to open unexpected file /usr/lib64/firefox/libdl.so.2
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: attempt to open unexpected file /lib64/libdl.so.2
Aug 16 10:43:07 dragonfly firefox-wayland.desktop[7487]: Sandbox: attempt to open unexpected file /lib64/glibc-hwcaps/x86-64-v3/libdl.so.2

This looks like the dynamic loader trying to resolve libdl.so.2; a copy of libwidevinecdm.so I have locally (version 4.10.2209.1 if it matters) depends on it. But, the libxul.so from my Firefox build also depends on it, and so does the one in Fedora's package, so it should already be loaded. There's something odd going on here.

On Fedora 35 libxul.so has no libdl.so.2 dependency. And no any other binary obj in /usr/lib64/firefox :

$ ldd /home/()/.mozilla/firefox/()/gmp-widevinecdm/4.10.2209.1/libwidevinecdm.so |grep libdl
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f7c749a0000)
$ ldd /usr/lib64/firefox/libxul.so |grep libdl
$
$ ldd /usr/lib64/firefox/firefox-bin |grep libdl
$
$ find /usr/lib64/firefox/ -name *.so -exec ldd {} \; | grep libdl
$

(In reply to Andrey from comment #11)

On Fedora 35 libxul.so has no libdl.so.2 dependency. And no any other binary obj in /usr/lib64/firefox :

Thanks for the correction. I was looking at a package from the wrong version of Fedora. Now things make sense: glibc 2.34 moved dlopen et al. into libc and left libdl as a stub for compatibility, which explains why Firefox (and NSPR and so on) don't depend on it anymore when they're built for Fedora ≥35. This should be a one-line fix.

If anyone needs a workaround, Mozilla's builds of Firefox should still work.

https://hg.mozilla.org/mozilla-central/rev/49bf466c3f24

Comment on attachment 9237836 [details]
Bug 1725828 - Preload dependencies for the Widevine CDM when sandboxing it on Linux.

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: Widevine EME and H.264 WebRTC may be broken depending on how Firefox is built.
• User impact if declined: This doesn't affect Mozilla's builds and probably won't affect them for some time, but downstream builds (already on some Linux distributions, and probably more and more in the future) won't be able to use the Widevine plugin for EME or OpenH264 for WebRTC.
• Fix Landed on Version: 94
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): We just preload some libraries which are normally already loaded; the patch is small and should have no effect on builds that weren't affected by this bug (like Mozilla's).

One alternative is to require downstream distributions to apply the patch themselves if they intend to build ESR91 with a recent glibc, but as far as I know there's no good way to communicate that to everyone who needs to hear it, and failures are likely to result in more bug reports for us.

• String or UUID changes made by this patch: none

Comment on attachment 9237836 [details]
Bug 1725828 - Preload dependencies for the Widevine CDM when sandboxing it on Linux.

Approved for 91.3esr.

https://hg.mozilla.org/releases/mozilla-esr91/rev/f6e36e1fb791