User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:74.0) Gecko/20100101 Firefox/74.0

Steps to reproduce:

Load nightly build 76.0a1 (2020-03-23). Start with MOZ_ENABLE_WAYLAND=1 and set
widget.wayland-dmabuf-webgl.enabled, widget.wayland-dmabuf-vaapi.enabled to true
and
media.peerconnection.video.vp9_enabled, media.av1.enabled to false

Video doesn't start, and I got this errors on console(it similar for different videos)

[Child 1782361, MediaDecoderStateMachine #1] WARNING: Decoder=7f0e09377c00 Decode error: NS_ERROR_DOM_MEDIA_DECODE_ERR (0x806e0004) - virtual mozilla::MediaResult mozilla::FFmpegVideoDecoder<58>::DoDecode(mozilla::MediaRawData *, uint8_t *, int, bool *, MediaDataDecoder::DecodedData &): avcodec_send_packet error: -1094995529: file /builds/worker/checkouts/gecko/dom/media/MediaDecoderStateMachine.cpp, line 3367
[Child 1782361, MediaDecoderStateMachine #1] WARNING: Decoder=7f0df1e6f400 Decode error: NS_ERROR_DOM_MEDIA_DECODE_ERR (0x806e0004) - virtual mozilla::MediaResult mozilla::FFmpegVideoDecoder<58>::DoDecode(mozilla::MediaRawData *, uint8_t *, int, bool *, MediaDataDecoder::DecodedData &): avcodec_send_packet error: -1094995529: file /builds/worker/checkouts/gecko/dom/media/MediaDecoderStateMachine.cpp, line 3367

When I start without MOZ_ENABLE_WAYLAND=1 it works fine on same profile and machine.

$ lsb_release -a
LSB Version: n/a
Distributor ID: ManjaroLinux
Description: Manjaro Linux
Release: 19.0.2
Codename: Kyria
$ vainfo
vainfo: VA-API version: 1.6 (libva 2.6.0)
vainfo: Driver version: Mesa Gallium driver 19.3.5 for AMD Radeon R7 Graphics (CARRIZO, DRM 3.35.0, 5.4.23-1-MANJARO, LLVM 9.0.1)
vainfo: Supported profile and entrypoints
VAProfileMPEG2Simple : VAEntrypointVLD
VAProfileMPEG2Main : VAEntrypointVLD
VAProfileVC1Simple : VAEntrypointVLD
VAProfileVC1Main : VAEntrypointVLD
VAProfileVC1Advanced : VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
VAProfileH264Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointEncSlice
VAProfileH264High : VAEntrypointVLD
VAProfileH264High : VAEntrypointEncSlice
VAProfileHEVCMain : VAEntrypointVLD
VAProfileJPEGBaseline : VAEntrypointVLD
VAProfileNone : VAEntrypointVideoProc

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

If this happens only for vp9 videos you can try setting media.ffvpx.enabled to false (bug 1623584)

Oh, just saw your hardware does not support vp9 - so if you tried a vp9 video (youtube usually uses that) see bug 1624103

If this happens only for vp9 videos you can try setting media.ffvpx.enabled to false
As I understand I disable vp9 video media.peerconnection.video.vp9_enabled setting to false.

(In reply to alsoijw from comment #4)

As I understand I disable vp9 video media.peerconnection.video.vp9_enabled setting to false.

Mind the "peerconnection". AFAIK this is only about WebRTC. But setting media.webm.enabled to false should do.

You can reenable media.peerconnection.video.vp9_enabled (VAAPI has not effect on WebRTC) and media.ffvpx.enabled. You might need to disable the sandbox with security.sandbox.content.level=0 (that's dangerous) as long as bug 1619585 and its dependencies (bug 1624743, bug 1595994) are not fixed. If you have multiple GPUs, it's bug 1622132.

With https://addons.mozilla.org/firefox/addon/enhanced-h264ify/ you can disable other codecs than H264 on YouTube.

Yes, it is poblem with multiple gpu.

$ ls /dev/dri
by-path card0 card1 renderD128 renderD129
$ vainfo --display drm --device /dev/dri/renderD128
vainfo: VA-API version: 1.6 (libva 2.6.0)
vainfo: Driver version: Mesa Gallium driver 19.3.5 for AMD HAINAN (DRM 2.50.0, 5.4.23-1-MANJARO, LLVM 9.0.1)
vainfo: Supported profile and entrypoints
VAProfileMPEG2Simple : VAEntrypointVLD
VAProfileMPEG2Main : VAEntrypointVLD
VAProfileNone : VAEntrypointVideoProc
$ vainfo --display drm --device /dev/dri/renderD129
vainfo: VA-API version: 1.6 (libva 2.6.0)
vainfo: Driver version: Mesa Gallium driver 19.3.5 for AMD Radeon R7 Graphics (CARRIZO, DRM 3.35.0, 5.4.23-1-MANJARO, LLVM 9.0.1)
vainfo: Supported profile and entrypoints
VAProfileMPEG2Simple : VAEntrypointVLD
VAProfileMPEG2Main : VAEntrypointVLD
VAProfileVC1Simple : VAEntrypointVLD
VAProfileVC1Main : VAEntrypointVLD
VAProfileVC1Advanced : VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
VAProfileH264Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointEncSlice
VAProfileH264High : VAEntrypointVLD
VAProfileH264High : VAEntrypointEncSlice
VAProfileHEVCMain : VAEntrypointVLD
VAProfileJPEGBaseline : VAEntrypointVLD
VAProfileNone : VAEntrypointVideoProc
$

When I run using this command video played correctly
LIBVA_DRIVER_NAME=gallium MOZ_ENABLE_WAYLAND=1 GDK_BACKEND=wayland ./firefox
On this I have error
MOZ_ENABLE_WAYLAND=1 GDK_BACKEND=wayland ./firefox

by-path card0 card1 renderD128 renderD129

Then it's bug 1622132. Thank you!

vainfo: Driver version: Mesa Gallium driver 19.3.5 for AMD HAINAN (DRM 2.50.0, 5.4.23-1-MANJARO, LLVM 9.0.1)

When I run using this command video played correctly
LIBVA_DRIVER_NAME=gallium MOZ_ENABLE_WAYLAND=1 GDK_BACKEND=wayland ./firefox

One of MOZ_ENABLE_WAYLAND=1 or GDK_BACKEND=wayland is enough, they mean the same.

https://wiki.archlinux.org/index.php/Hardware_video_acceleration#Configuring_VA-API says:

Since version 12.0.1 libva-mesa-driver provides radeonsi instead of gallium.

Manjaro ships version 19.3.5-1: https://discover.manjaro.org/packages/libva-mesa-driver
In bug 1622132 comment 0 I'm using radeonsi_drv_video.so and it also says: Driver version: Mesa Gallium driver 19.3.3
I guess you would get an error from $ LIBVA_DRIVER_NAME=gallium vainfo. Therefore you might be decoding without VAAPI in this case.

On this I have error
MOZ_ENABLE_WAYLAND=1 GDK_BACKEND=wayland ./firefox

I assume this tries to use VAAPI, but your first graphics device does not support h264, so it fails. (h264 variant of bug 1624103)