nvidia-vaapi-driver requires MOZ_DISABLE_RDD_SANDBOX=1 (in addition to usual LIBVA_DRIVER_NAME=nvidia) to work. Without it, the following errors are shown in the log:

[RDD 68813: MediaPDecoder #1]: D/PlatformDecoderModule FFVPX: Initialising VA-API FFmpeg decoder
[RDD 68813: MediaPDecoder #1]: D/PlatformDecoderModule FFVPX: codec vp9 : Google VP9
libva info: VA-API version 1.14.0
libva info: User environment variable requested driver 'nvidia'
libva info: Trying to open /usr/lib64/dri/nvidia_drv_video.so
libva info: Found init function __vaDriverInit_1_0
libva error: /usr/lib64/dri/nvidia_drv_video.so init failed
libva info: va_openDriver() returns 1
[RDD 68813: MediaPDecoder #1]: D/PlatformDecoderModule FFVPX: vaInitialize failed.
[RDD 68813: MediaPDecoder #1]: D/PlatformDecoderModule FFVPX: Failed to create VA-API device context

With the environmental variable set, VA-API is initialised successfully:

[RDD 70170: MediaPDecoder #1]: D/PlatformDecoderModule FFVPX: Initialising VA-API FFmpeg decoder
[RDD 70170: MediaPDecoder #1]: D/PlatformDecoderModule FFVPX: codec vp9 : Google VP9
libva info: VA-API version 1.14.0
libva info: User environment variable requested driver 'nvidia'
libva info: Trying to open /usr/lib64/dri/nvidia_drv_video.so
libva info: Found init function __vaDriverInit_1_0
[Child 69898: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder rejects requested type
[Child 69898: Main Thread]: D/PlatformDecoderModule Sandbox Utility decoder supports requested type
[Child 69898: Main Thread]: D/PlatformDecoderModule Sandbox RDD decoder supports requested type
libva info: va_openDriver() returns 0
[AVHWDeviceContext @ 0x7f3398876600] Format 0x3231564e -> nv12.
[AVHWDeviceContext @ 0x7f3398876600] VAAPI driver: VA-API NVDEC driver.
[AVHWDeviceContext @ 0x7f3398876600] Driver not found in known nonstandard list, using standard behaviour.

Thanks for the report! This is a duplicate of bug 1748460 and desired behavior.
https://github.com/elFarto/nvidia-vaapi-driver requires CUDA which doesn't save you power and is blocked by the media process sandbox. Allowing CUDA would require further media process sandbox exceptions/holes.

https://github.com/elFarto/nvidia-vaapi-driver/issues/74#issuecomment-1100918826

It's a driver issue. As soon as you initialise cuda, it forces the GPU into a higher power state (at least P2 IIRC), and the nvdec implementation forces you to use cuda to interact with it, so this always happens. It means that using nvdec will never save you power unless you were going to use cuda anyway.