Every emulator test job in buildbot is reporting the following error when the emulator is launched: emulator: Initializing hardware OpenGLES emulation support Failed to load libGL.so error libGL.so: cannot open shared object file: No such file or directory Failed to load libGL.so error libGL.so: cannot open shared object file: No such file or directory This happens with both successful and failing jobs. It's nice that this isn't causing us to fail, but it may be causing us to run more slowly than we would otherwise, and could be the source of other problems. The emulator expects this file to live at /usr/lib/x86_64-linux-gnu/libGL.so (for 64-bit linux) and should be symlinked from wherever it is to that location.

Good catch. For reference: > sudo ln -s /usr/lib/i386-linux-gnu/mesa/libGL.so.1 /usr/lib/i386-linux-gnu/libGL.so (s/i386/x86_64) From https://developer.mozilla.org/en-US/Firefox_OS/Firefox_OS_build_prerequisites#Emulator_build_issues

An alternative is to have libgl1-mesa-dev installed, but I am not sure how to do this?

Andrew, do you know how should I fix this for test machine, is this about mozharness?

The code seems in mozharness [1], I tried to import mozharness to my mecurial repository, but it doesn't show up. Tried also Bitbucket, but the build was failed with HTTP 404 [2], probably the build server is not allow to access external resource. [1] http://dxr.mozilla.org/build:mozharness/source/scripts/android_emulator_build.py#309 [2] https://treeherder.mozilla.org/#/jobs?repo=try&revision=d5a6d75f91d2

Try tries (sic!) to "probe" https://bitbucket.org/janus926/mozharness/rev/bug-1013634 per http://hg.mozilla.org/build/tools/file/0d8d4f943f6e/buildfarm/utils/repository_manifest.py#l148, fails to retrieve it and doesn't try to clone the repo after that. If you use your user repo on hg.m.o it should work as expected.

Thanks for the information! I've already filed bug 1115600 for the issue of cloning mozharness to my user repo.

The code I pointed out at comment 4 is not the right one, the same error message still shows up.

Try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=b2d3572748f6

This would work, but seems like it would be better if the machines just came pre-configured with this. Rail, how difficult would it be to get them set up with this symlink? > sudo ln -s /usr/lib/i386-linux-gnu/mesa/libGL.so.1 /usr/lib/i386-linux-gnu/libGL.so

(In reply to Andrew Halberstadt [:ahal] from comment #9) > This would work, but seems like it would be better if the machines just came > pre-configured with this. Rail, how difficult would it be to get them set up > with this symlink? > > > sudo ln -s /usr/lib/i386-linux-gnu/mesa/libGL.so.1 /usr/lib/i386-linux-gnu/libGL.so Definitely it would be better than adding hacks to the code. libgl1-mesa-dev has the file in the package, installing it should fix the problem. One of the downsides though will be some other extra packages installed (all of them are *-dev, so should not interfere with other libraries). I'll create a puppet patch for this.

Grabbing the bug to track the deployment.

I'm going to deploy this ... not today. :) In the worst case scenario on Mon/Tue.

Comment on attachment 8542970 [details] [diff] [review] [puppet] install libgl1-mesa-dev remote: https://hg.mozilla.org/build/puppet/rev/e9d1afd8c374 remote: https://hg.mozilla.org/build/puppet/rev/2cf318e35f3c This will show up in production tomorrow.

:ting, can you verify that the problem is fixed without applying your patch (attachment 8542490 [details] [diff] [review])?

Comment on attachment 8542490 [details] [diff] [review] patch v1 Thanks Rail!

I'm still seeing the libGL error in recent inbound qemu.logs: http://mozilla-releng-blobs.s3.amazonaws.com/blobs/mozilla-inbound/sha512/fd46662f417e689374c9724b5f47399cf60e44268109bf72eb43ee8e815b7094cb4f96c599f6b350f0c467a4f231e7671d891bfdf2dde7b3696fcaceae5debfe

Hmm, the patch symlinks 32-bit libraries, probably we need to install libgl1-mesa-dev:i386. I'll poke this next week. Meanwhile I'll backout the puppet patch.

Comment on attachment 8542970 [details] [diff] [review] [puppet] install libgl1-mesa-dev remote: https://hg.mozilla.org/build/puppet/rev/f6e23a227371 remote: https://hg.mozilla.org/build/puppet/rev/37806d3f195a

It'd be nice if we can fix this by the end of this week.

this is what's expected to be installed: $ apt-get install libgl1-mesa-dev:i386 Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: libdrm-dev:i386 libkms1:i386 libpthread-stubs0:i386 libpthread-stubs0-dev:i386 libx11-dev:i386 libx11-doc libxau-dev:i386 libxcb1-dev:i386 libxdmcp-dev:i386 libxext-dev:i386 mesa-common-dev:i386 x11proto-core-dev x11proto-input-dev x11proto-kb-dev x11proto-xext-dev xorg-sgml-doctools xtrans-dev Suggested packages: libxcb-doc:i386 Recommended packages: libx11-doc:i386 The following NEW packages will be installed: libdrm-dev:i386 libgl1-mesa-dev:i386 libkms1:i386 libpthread-stubs0:i386 libpthread-stubs0-dev:i386 libx11-dev:i386 libx11-doc libxau-dev:i386 libxcb1-dev:i386 libxdmcp-dev:i386 libxext-dev:i386 mesa-common-dev:i386 x11proto-core-dev x11proto-input-dev x11proto-kb-dev x11proto-xext-dev xorg-sgml-doctools xtrans-dev 0 upgraded, 18 newly installed, 0 to remove and 121 not upgraded. Need to get 4893 kB of archives. After this operation, 19.1 MB of additional disk space will be used.

Comment on attachment 8546392 [details] [diff] [review] install-mesa-32bit.diff remote: https://hg.mozilla.org/build/puppet/rev/d7eaf090016f remote: https://hg.mozilla.org/build/puppet/rev/561fab128519 We'll see the results tomorrow. Sorry for the delay, upcoming releases and merge day keep me busy.

This should be fixed now. \o/

Thank you Rail!