We create X11/EGL headless GL context on Wayland but we're supposed to create a Wayland one.

It affects RDD process only and it's because we're missing display connection there.

The correct regression bug is Bug 1635451 where GdkIsX11Display() was replaced by GdkIsWaylandDisplay() at gfx/gl/GLContextProviderWayland.cpp.
As both GdkIsX11Display()/GdkIsWaylandDisplay() returns false on headless we switched from EGL to X11 for headless.

But the most important point of failure is here:
https://searchfox.org/mozilla-central/rev/24c1cdc33ccce692612276cd0d3e9a44f6c22fd3/gfx/gl/GLContextProviderEGL.cpp#1140

So when GdkIsWaylandDisplay() is false on headless we open a new X11 connection and create X11/PBuffer surface.

Backed out for causing bustages on GLContextProviderEGL.cpp. CLOSED TREE

Backout link : https://hg.mozilla.org/integration/autoland/rev/3666e4a93f79f15d1241d870b2cb626d36c72dda

Push with failures : https://treeherder.mozilla.org/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception%2Crunnable&revision=d13c0a1c7a19860435622b1786d6059a701ab5d1&selectedTaskRun=Gv-ffoUOS0ekLA3JO8w1mQ.0

Link to failure log : https://treeherder.mozilla.org/logviewer?job_id=378585761&repo=autoland&lineNumber=48858

Failure line: /builds/worker/checkouts/gecko/gfx/gl/GLContextProviderEGL.cpp:820:5: error: use of undeclared identifier 'nsGbmLib'

Updated, try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=15060de1613f1d45df5e628bd371bfe67af05925

https://hg.mozilla.org/mozilla-central/rev/c1ef18d0a27e

Jed, this patch unifies how headless GL context is created on Wayland/X11, i.e. where display connection is missing we don't create one but we use headless GL display based on dmabuf.

I tested that wit patches from Bug 1769182 and I'm getting this one (for both X11 / Wayland):

Sandbox: seccomp sandbox violation: pid 26657, tid 26706, syscall 27, args 140104273555456 4096 140104683351920 0 1 1073741824. Killing process.

Is that something which can be fixed?
Thanks.

(In reply to Martin Stránský [:stransky] (ni? me) from comment #10)

Jed, this patch unifies how headless GL context is created on Wayland/X11, i.e. where display connection is missing we don't create one but we use headless GL display based on dmabuf.

I tested that wit patches from Bug 1769182 and I'm getting this one (for both X11 / Wayland):

Sandbox: seccomp sandbox violation: pid 26657, tid 26706, syscall 27, args 140104273555456 4096 140104683351920 0 1 1073741824. Killing process.

I also ran into this while trying it out this morning. That's mincore, which we already allowed for content processes, because it's part of a hack that Mesa EGL uses to let the “native display” be one of several types of display pointer (or, previously, an fbdev fd cast from int to void*), determined at runtime.

I've confirmed that it works under X if mincore is allowed. And I've confirmed it doesn't need to connect to the X server now, so a lot of bug 1769182 can be turned off (which is good, even though the RDD sandbox is less critical than the content sandbox). I'll make some patches.

(Edited to correct the comment about EGL native display types.)

Great, Thanks. I need to address Bug 1770407 on Intel - AMD seems to be ok.

I have WIP implementation of surfaceless GL backend and seems to be working OK on AMD and with recent sandbox rules we don't need Bug 1770520 (mincore). Will try on Intel tomorrow and if that works I'll replace the GBM backend with that (or at lease use surfaceless backend for VA-API surface snapshot and leave GBM for WebGL - AFAIK it's not guaranteed that surfaceless GL backed is WH accelerated).