Bug 1698778 mostly fixed the RDD sandbox for VA-API and allowed people to test it, which has turned up a few more issues; I'm making this bug to keep track of them.

Right now VA-API playback is blocked by Bug 1751363 and Bug 1751709 so MOZ_DISABLE_RDD_SANDBOX=1 is needed.

Guys, any update here?

The Bug 1751363 and Bug 1751709 are blocking whole VA-API stuff on Linux right now (and I don't think we want users to set MOZ_DISABLE_RDD_SANDBOX env variable).

I found some time to look into this, and I think I have it working at least for amdgpu on X11, but it's kind of messy, and in particular the RDD process didn't already have a connection to the X server (it doesn't start GTK — and arguably it shouldn't, because it doesn't need any of the widget rendering etc.) so it had to be set up to connect via the broker, which is currently only done for the content process.

The Wayland case may be interesting, because currently we have EGL borrow the display from GDK (bug 1460603) and there's a comment that some drivers need to be given a display (instead of being passed EGL_DEFAULT_DISPLAY and opening their own connection), but that wouldn't work in the RDD process if I understand correctly. I haven't tried it yet.

Thanks for looking at it!

(In reply to Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧ from comment #3)

The Wayland case may be interesting, because currently we have EGL borrow the display from GDK (bug 1460603) and there's a comment that some drivers need to be given a display (instead of being passed EGL_DEFAULT_DISPLAY and opening their own connection), but that wouldn't work in the RDD process if I understand correctly. I haven't tried it yet.

This whole block can be replaced by wl_display_connect(nullptr) call (if Gdk/Gdk is missing), we don't need to use Gdk here.

I wonder how MacOS handles that - does it use EGL on RDD process? Our primary need is Bug 1712588 for that (i.e. create a snapshot of dmabuf texture).

I wonder how MacOS handles that

FWIW macOS sandboxing is completely different, and I think the HW decoder acceleration is also completely different, so I don't entirely get the question :-/

(In reply to Gian-Carlo Pascutto [:gcp] from comment #6)

I wonder how MacOS handles that

FWIW macOS sandboxing is completely different, and I think the HW decoder acceleration is also completely different, so I don't entirely get the question :-/

I did some investigation here and it looks like MacOS doesn't seem to use GL to map MacIOSurface to from GPU to userspace. It uses direct mapping without GL. Linux also supports that but for Intel only, AMD has to use GL.

I hope this will be fixed quickly. Having to use MOZ_DISABLE_RDD_SANDBOX=1 is a major security risk.