I found as part of the work in bug 1675768 that our talos instances for Linux don't have the libegl package installed. Today we rely upon GLX, but we would like to transition to using EGL; presently this just means preferring EGL in 85 nightly for WebRender users, but going forward it will eventually be deployed to beta/release.

See https://treeherder.mozilla.org/logviewer?job_id=321652834&repo=try&lineNumber=1449 for the failures showing no EGL.

The current linux64 hardware workers for talos/perf testing are on Ubuntu 16.04
They are being upgraded to Ubuntu 18.04 through bug 1572739

If the EGL libraries (or a modern enough version) are available only on 18.04, then this may depend on bug 1572739

The linux talos workers are installed with Ubuntu desktop (not server). So it *should just be installed on the 18.04 workers?

Hm, desktop by default should have what we need in 18.04. Similar for 16.04. The following commands should reveal whether the libraries are missing or have a different path on 16.04 (the loading code hasn't changed since 2012 so it should be fine).

dpkg -L libegl1-mesa
dpkg -L libegl1

Here's what I see for those listings:

[root@t-linux64-ms-001 ~]# dpkg -L libegl1-mesa
/.
/usr
/usr/lib
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/mesa-egl
/usr/lib/x86_64-linux-gnu/mesa-egl/libEGL.so.1.0.0
/usr/lib/x86_64-linux-gnu/mesa-egl/ld.so.conf
/usr/share
/usr/share/lintian
/usr/share/lintian/overrides
/usr/share/lintian/overrides/libegl1-mesa
/usr/share/bug
/usr/share/bug/libegl1-mesa
/usr/share/bug/libegl1-mesa/script
/usr/share/bug/libegl1-mesa/control
/usr/share/doc
/usr/share/doc/libegl1-mesa
/usr/share/doc/libegl1-mesa/copyright
/usr/lib/x86_64-linux-gnu/mesa-egl/libEGL.so.1
/usr/share/doc/libegl1-mesa/changelog.Debian.gz
[root@t-linux64-ms-001 ~]# dpkg -L libegl1     
dpkg-query: package 'libegl1' is not installed
Use dpkg --info (= dpkg-deb --info) to examine archive files,
and dpkg --contents (= dpkg-deb --contents) to list their contents.

Interesting.... and for?

dpkg -L libgles1-mesa

https://searchfox.org/mozilla-central/rev/44e6dfd7e02edd95e5fd4d4c25c8b946131f92cd/gfx/gl/GLLibraryEGL.cpp#390

should have loaded /usr/lib/x86_64-linux-gnu/mesa-egl/libEGL.so.1 from the package details in comment 5. I didn't think libgles would be missing, but that actually can cause the same error message (missing EGL).

https://wiki.ubuntu.com/X/EGLDriverPackagingHOWTO

It appears that 16.04 packages their EGL implementation differently than 18.04+. So there is some extra magic we need to do to select the EGL implementation as the default. The upgrade to 18.04 will make this go away. If is landed this quarter, then that just means delaying EGL by an extra release since 85 will take us into next year.... I'm willing to wait.

:aerickson, could :aosmond test this on the 1804 test pool? If using EGL works on those, then maybe this is a no-op and depends on that getting deployed to the production pool?

(In reply to Andrew Osmond [:aosmond] from comment #6)

Interesting.... and for?

dpkg -L libgles1-mesa

[root@t-linux64-ms-001 ~]# dpkg -L libgles1-mesa
dpkg-query: package 'libgles1-mesa' is not installed
Use dpkg --info (= dpkg-deb --info) to examine archive files,
and dpkg --contents (= dpkg-deb --contents) to list their contents.

(In reply to Andrew Osmond [:aosmond] from comment #7)

https://searchfox.org/mozilla-central/rev/44e6dfd7e02edd95e5fd4d4c25c8b946131f92cd/gfx/gl/GLLibraryEGL.cpp#390

should have loaded /usr/lib/x86_64-linux-gnu/mesa-egl/libEGL.so.1 from the package details in comment 5. I didn't think libgles would be missing, but that actually can cause the same error message (missing EGL).

I'm not surprised if we have non-standard gl libraries on here. When first setting these up, I had to force removing the graphics drivers and libglx that the Ubuntu 16.04 install automatically picked because of conflicts when installing the intel graphics driver package.

(In reply to Dave House [:dhouse] from comment #10)

(In reply to Andrew Osmond [:aosmond] from comment #7)

https://searchfox.org/mozilla-central/rev/44e6dfd7e02edd95e5fd4d4c25c8b946131f92cd/gfx/gl/GLLibraryEGL.cpp#390

should have loaded /usr/lib/x86_64-linux-gnu/mesa-egl/libEGL.so.1 from the package details in comment 5. I didn't think libgles would be missing, but that actually can cause the same error message (missing EGL).

I'm not surprised if we have non-standard gl libraries on here. When first setting these up, I had to force removing the graphics drivers and libglx that the Ubuntu 16.04 install automatically picked because of conflicts when installing the intel graphics driver package.

Hearing that makes me reluctant to fiddle with the configuration too much and causing headaches. beta/release and nightly would diverge with the proposed patch, and possibly requiring different configurations.

(In reply to Dave House [:dhouse] from comment #9)

:aerickson, could :aosmond test this on the 1804 test pool? If using EGL works on those, then maybe this is a no-op and depends on that getting deployed to the production pool?

I could run both the GLX and EGL variants on a 1804 pool I guess, mostly I wanted to see the difference to make sure prior to landing I didn't cause major regressions and/or see if there are any improvements given optimizations we have on the EGL path. However that is less useful than being able to land, which I can't with talos busted silently busted by my change :).

It sounds like if I am patient, and we get 18.04 in the next 2 months, then all will be well, because Ubuntu changed how they manage these packages (afaict). I think that's a compromise I can live with it and save us all some trouble.

Output from a 1804 host attached. We're working on getting a loaner setup also.

Sorry for the attachment spam. Bugzilla said it wasn't working, but it was.

:aosmond: if you're changing existing talos jobs, you can target the test pool of 1804 talos workers (https://firefox-ci-tc.services.mozilla.com/provisioners/releng-hardware/worker-types/gecko-t-linux-talos-dw) with try commands like:

./mach try fuzzy --full --query "'linux 'raptor \!shippable 'tp6 \!cold 'qr" --worker-suffix t-linux-talos=-dw --no-push

clearing the NI as it doesn't seem like a loaner is needed.

:aosmond, please re-NI if you do or have any issues with the test pool.

During glxtest, it now decides if EGL is available or not for use by gecko. If it is missing, it will fallback to GLX, so it won't block me landing future EGL work now. As well, the image update should resolve any remaining issues. I think we can close this.