https://developer.android.com/studio/releases/emulator suggests there are a lot of improvements between r28 and r29. Let's see how our tests behave with r29.

29.0.11 is currently available and works well for me locally.

This version includes some GL improvements which may be of interest to kats, if we can use it.
This version includes some network improvements which may (long shot here) help bug 1534732.

In my initial trial, I updated adb as well, to 1.0.41 (28.0.3-5475833). I hit intermittent mkdir failures, especially in xpcshell, which caused adb-initiated task retries. I will fall back to the adb previously in use, 1.0.40 (28.0.2-5303910).

We've updated to adb 1.0.41 on bitbar's containers.

The mkdir failures continue with adb 1.0.40!

There are a few issues when the emulator is updated:

• intermittent xpcshell mkdir failures produce task retries
• a few mochitests perma-fail
• marionette tests frequently report network issues (port in use)
• wrench tests are unable to download the new archive from tooltool

I'm working through these...

It is hard to say definitively, but I believe the emulator update does resolve bug 1534732.

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=e407352ca0ceba397620926b41db29780a98ab66

With 29.0.11, it is also possible to use '-gpu on' on packet.net, instead of '-gpu swiftshader_indirect'.

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=f118882e1e3213e24fdd8d29084d099d9d4f2f03

That changes:

06-14 18:28:44.223  2795  2847 I GeckoConsole: Version: OpenGL ES 2.0 (OpenGL ES 3.0 SwiftShader 4.0.0.1)
06-14 18:28:44.223  2795  2847 I GeckoConsole: Vendor: Google (Google Inc.)
06-14 18:28:44.223  2795  2847 I GeckoConsole: Renderer: Android Emulator OpenGL ES Translator (Google SwiftShader)
06-14 18:28:44.223  2795  2847 I GeckoConsole: FBO Texture Target: TEXTURE_2D

to

06-14 20:47:41.210  2790  2841 I GeckoConsole: Version: OpenGL ES 2.0 (3.3 (Core Profile) Mesa 18.0.5)
06-14 20:47:41.210  2790  2841 I GeckoConsole: Vendor: Google (VMware, Inc.)
06-14 20:47:41.210  2790  2841 I GeckoConsole: Renderer: Android Emulator OpenGL ES Translator (llvmpipe (LLVM 6.0, 256 bits))
06-14 20:47:41.210  2790  2841 I GeckoConsole: FBO Texture Target: TEXTURE_2D

Note that many of the reftest failures are UNEXPECTED-PASS.

This seems to provide a 20-30% performance improvement for several tasks, compared to current mozilla-central.

Note that https://issuetracker.google.com/issues/133730690 should be available in 29.1.x.

Jamie, do you know if the change :gbrown describes above would make the emulator more or less representative of real world usage for WR?

It seems like it might be worth doing for non-WR runs at least.

Confirmation that non-WR reftest-plain can run with fewer annotations and significantly faster with the new emulator and '-gpu on':

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=f41be705da76ca842bd446753803ed51db13c945

I'd prefer if you landed the emulator update for non-WR reftest-plain sooner rather than later. I'm working on annotating the WR (on pixel2) reftest-plain failures and merging the two will be easier if you land first.

I think neither the swiftshader or GPU backend will give us a perfect picture of running on actual devices, and both will have their own set of bugs. But if it runs faster and passes more tests it sounds like a good idea.

Upgrade to version 29.0.11 of the emulator and use '-gpu on' rather than
swiftshader_indirect, for most Android 7.0 tests. The upgrade appears to
finally resolve bug 1534732, improves reftest performance dramatically, and
allows us to reduce reftest "fuzz" for many tests.
marionette tests are excluded because they intermittently fail with network
errors (address in use); these tests are near end-of-life, so I don't think
this issue is worth investigating, but I'll file a follow-up bug to record
the issue.
web-platform tests are excluded because they are not very stable on the
existing emulator, making it difficult to compare results. I will file a
follow-up and work with :maja_zf to see if they can be upgraded soon.

These changes are needed for consistently green runs with the new emulator with
"-gpu on".
Most changes are simple removal of fuzzy-if(geckoview) but I also needed to add
at least one new fuzzy-if.
In this configuration we can run reftests in just 2 chunks (20 minutes each on
opt/30 minutes on debug).

Updating to the new emulator avoids the network oddities observed in bug 1534732, but
there are a couple of tests that need to be skipped for a consistently green run.
Performance is improved, so we can run in fewer chunks. /opt is reliable, so moving
to tier 1; /debug is reliable except for existing perma-fail bug 1560736.

Leave open until wpt handled (and maybe wrench?).

https://hg.mozilla.org/mozilla-central/rev/989d4fa5731b
https://hg.mozilla.org/mozilla-central/rev/1c97fe096067
https://hg.mozilla.org/mozilla-central/rev/8776353e9c15

There are some intermittent failures in xpcshell tests (eg bug 1562344)
in the new emulator; let's revert to the older emulator until those can
be figured out.

https://hg.mozilla.org/mozilla-central/rev/49f22f8f91f9

Most tests use 29.0.11 now. Marionette still uses the older emulator; I don't think it is worth the effort to upgrade, given that there are new failures introduced with 29.0.11 and Mn is fennec-only, with a limited lifetime. xpcshell still uses the older emulator; given the trouble we have seen with the new emulator, I think this should be revisited the next time we update the emulator, but probably not before.