Downstream bug: https://bugzilla.redhat.com/show_bug.cgi?id=2140424

glxinfo crashes regularly on NVIDIA, we may ignore NVIDIA for now.

https://bugzilla.redhat.com/show_bug.cgi?id=2144151 has a better backtrace.

Is this about XDisplayString? Then we could close bug 1804974 and bug 1787182 as duplicate.

(Darkspirit from bug 1804974 comment #1)
[..]

Deprecated libva-vdpau-driver doesn't support vaGetDisplayDRM and crashes.
Therefore the VAAPI test has been moved into the glxtest process (bug 1787182, bug 1758473/bug 1777927) to disable VAAPI in case vdpau_drv_video.so is present.

Could a sandbox rule just block vdpau_drv_video.so from being loaded in RDD and glxtest?

(Darkspirit from bug 1758473 comment #9)

It might be this one: https://salsa.debian.org/multimedia-team/attic/vdpau-video/-/blob/63450ffea86143d418c6e83cb8d2828d3a7beb25/src/vdpau_driver.c#L188

const char * const x11_dpy_name = XDisplayString(driver_data->x11_dpy);

https://bugs.archlinux.org/task/72241#comments

vaGetDisplayDRM() doesn't fill ->x11_dpy

VAAPI should be blocked for vdpau_drv_video.so.
vdpau_drv_video.so is deprecated and has been removed from Debian.
Debian Buster (oldstable) was the last release that had a package for it: https://packages.debian.org/oldstable/vdpau-va-driver
https://tracker.debian.org/pkg/vdpau-video
https://salsa.debian.org/multimedia-team/attic/vdpau-video

(Jed Davis [:jld] ⟨⏰|UTC-8⟩ ⟦he/him⟧ from bug 1804974 comment 2)

Unfortunately we don't have anything that could block a specific library at the sandbox level, but it'd be pretty simple for glxtest to use dl_iterate_phdr to see if a library with a given name was loaded.

https://bugzilla.redhat.com/show_bug.cgi?id=2140424#c7 mentions nvidia_drv_video.so, which in Fedora 37 is provided by https://github.com/elFarto/nvidia-vaapi-driver. That's wrong, right, as that one is the modern one?

Assuming it's caused by libva-vdpau-driver: IMO Fedora should remove that package. It's unmaintained and broken and causes issues for apps.

(Darkspirit from bug 1804974 comment #1)
Is this about XDisplayString? Then we could close bug 1804974 and bug 1787182 as duplicate.

Please leave them open for now, will investigate that later.

It seems nvidia_drv_video.so can be vdpau_drv_video.so:

https://packages.fedoraproject.org/pkgs/libva-vdpau-driver/libva-vdpau-driver/
creates

• nvidia_drv_video.so
  • I assume this is a symlink to vdpau_drv_video.so: https://github.com/elFarto/nvidia-vaapi-driver#removal, https://github.com/intel/libva/pull/554#issuecomment-1035978419
    • I have also found https://bbs.archlinux.org/viewtopic.php?id=282638 (XDisplayString + nvidia_drv_video.so) + https://github.com/intel/libva/issues/675.
• s3g_drv_video.so
• vdpau_drv_video.so

on Fedora 36 and 37.

rawhide package:

• nvidia_drv_video.so has been removed: https://packages.fedoraproject.org/pkgs/libva-vdpau-driver/libva-vdpau-driver/fedora-rawhide.html

  Swich upstream - Exclude nvidia_drv_video to prevent conflicts

• It seems to be based on https://github.com/xtknight/vdpau-va-driver-vp9 and the package might already contain a NULL-Check for XDisplayString: https://github.com/xtknight/vdpau-va-driver-vp9/pull/14/files

I think we need a short term and long term solution. Short them one may be to avoid va-api checks on NVIDIA as va-api is disabled there anyway and we should not produce cores for every run if we want to enable va-api by default. Long term may be to move remote code forward and run it before HW check.

Let's make this one about disabled VA-API test in gfxinfo. We're disabling VA-API on NVIDIA anyway so there's no point to probe it in gfxinfo. VA-API will still work if enabled by pref (media.ffmpeg.vaapi.enabled).

bug 1756459 should still work for nvidia-vaapi-driver.

Would something like the following logic at the beginning of vaapitest() work?
if dl_iterate_phdr (bug 1804974 comment 2) finds "[..]vdpau_drv_video.so" {
record_warning("VA-API force-disabled: Please uninstall deprecated libva-vdpau-driver "
"which doesn't support vaGetDisplayDRM and most likely crashes");
return;
}

I assume bug 1804974 occured due to force-enabling: Would force-enabling be reliably prevented by this code?

(In reply to Darkspirit from comment #8)

bug 1756459 should still work for nvidia-vaapi-driver.

Would something like the following logic at the beginning of vaapitest() work?
if dl_iterate_phdr (bug 1804974 comment 2) finds "[..]vdpau_drv_video.so" {
record_warning("VA-API force-disabled: Please uninstall deprecated libva-vdpau-driver "
"which doesn't support vaGetDisplayDRM and most likely crashes");
return;
}

I assume bug 1804974 occured due to force-enabling: Would force-enabling be reliably prevented by this code?

We may create it as a next iteration of the patch. Right now we can't print available codecs on about:support for NIVIDIA but given it's state I don't think it's so big deal.

I'm looking to enable VA-API by default which means we can't generate coredumps on NVIDIA systems. GfxInfo.cpp just reads data from remote process (which crashes on broken NVIDIA), that's not the detection itself.

Depends on D168003

Depends on D168004

Wouldn't comment 11 mean that vdpau_drv_video.so with force-enabled VAAPI wouldn't crash in glxtest anymore and instead in the RDD process?

(In reply to Martin Stránský [:stransky] (ni? me) from comment #9)

Right now we can't print available codecs on about:support for NIVIDIA

I'm looking to enable VA-API by default which means we can't generate coredumps on NVIDIA systems.

My understanding:

• nvidia-vaapi-driver does not crash and codec detection works because glxtest does not run in any sandbox. (Or is it broken for you?)
• Only deprecated libva-vdpau-driver crashes in XDisplayString.

If the user has an Nvidia GPU and if the presence of vdpau_drv_video.so is detected with dl_iterate_phdr (bug 1804974 comment 2)
vaapitest() should be early returned,
childvaapitest() could not crash,
glxtest would not print VAAPI_SUPPORTED true,
mIsVAAPISupported would be false in Gfxinfo.cpp:
My question was if this code would then realibly force-disable VAAPI and ignore media.hardware-video-decoding.force-enabled=true, so that's impossible to force-enable VAAPI as long as vdpau_drv_video.so is present on the Nvidia system.

(In reply to Darkspirit from comment #13)

Wouldn't comment 11 mean that vdpau_drv_video.so with force-enabled VAAPI wouldn't crash in glxtest anymore and instead in the RDD process?

Yes, exactly.

My understanding:

• nvidia-vaapi-driver does not crash and codec detection works because glxtest does not run in any sandbox. (Or is it broken for you?)
• Only deprecated libva-vdpau-driver crashes in XDisplayString.

Yes, that's correct.

If the user has an Nvidia GPU and if the presence of vdpau_drv_video.so is detected with dl_iterate_phdr (bug 1804974 comment 2)
vaapitest() should be early returned,
childvaapitest() could not crash,
glxtest would not print VAAPI_SUPPORTED true,
mIsVAAPISupported would be false in Gfxinfo.cpp:
My question was if this code would then realibly force-disable VAAPI and ignore media.hardware-video-decoding.force-enabled=true, so that's impossible to force-enable VAAPI as long as vdpau_drv_video.so is present on the Nvidia system.

That's perhaps possible. But I don't see worth to do such extensive diagnostics for NVIDIA how. It's still experimental and rather unstable solution which may crash/be broken for various reasons. Even working vaapitest() doesn't meat the decoding works so the vaapitest() doesn't give much value here (only advantage is correct states at about:support page).

https://hg.mozilla.org/mozilla-central/rev/b1339935f65d
https://hg.mozilla.org/mozilla-central/rev/e71e7781c14a
https://hg.mozilla.org/mozilla-central/rev/08a8e868d828