At some point we'd like to switch to using the android-x86 builds to generate the PGO profile for Android builds. Currently we use the android-api-16 builds and the arm emulator, which is quite slow. Using android-x86 should speed up the run task from 40+ minutes down to about 5 minutes. Additionally, we may want to stop using the arm emulator in automation for tests, so presumably we'd no longer want to depend on it for PGO.

The main blocker for this is bug 1545497.

Adding [geckoview:p1] whiteboard tag because we'd like Android PGO to move from ARM emulators to x86_64 soon. The Fennec tests running on the ARM emulators will be turned off soon during 71 Nightly (bug 1575973). Android PGO profile generation will then be the only task using the ARM emulators. Once we move PGO profile generation to x86-64, we can retire the ARM emulators for good.

(In reply to Chris Peterson [:cpeterson] from comment #1)

Once we move PGO profile generation to x86-64, we can retire the ARM emulators for good.

Just to be clear, are we using the x86-64 profiles for ARM, or are we using x86 profiles? Because the x86 profiles are likely to work much better due to bitnesses matching and therefore mangled names matching (more) closely. The same situation applies in reverse to a future AArch64 PGO build--we'd want to use x86-64 profiles there, not x86 ones.

(In reply to Nathan Froyd [:froydnj] from comment #2)

(In reply to Chris Peterson [:cpeterson] from comment #1)

Once we move PGO profile generation to x86-64, we can retire the ARM emulators for good.

Just to be clear, are we using the x86-64 profiles for ARM, or are we using x86 profiles? Because the x86 profiles are likely to work much better due to bitnesses matching and therefore mangled names matching (more) closely. The same situation applies in reverse to a future AArch64 PGO build--we'd want to use x86-64 profiles there, not x86 ones.

We currently use the android-api-16 profile on both android-api-16 and android-aarch64 builds. There were perf wins for aarch64 according to talos even though the bitness doesn't match.

My intention here was to simply replace the android-api-16 profile generation stage with android-x86 profile generation for both, but once we have that running it shouldn't be hard to experiment with android-x86_64 for aarch64 to see if there's any additional improvement there.

Bug 1582221 landed today, so Android PGO profiles will now be generated using the GeckoView Test Runner Activity (geckoview-androidTest.apk) instead of Fennec. That should simplify moving profile generation from android-api-16 (ARM) to android-x86_64.

AFAIK, we stopped running tests on android-x86 once the tests were all green on android-x86_64 (bug 1498298). So we should use android-x86_64, not x86, for profile generation.

I have some patches for switching from android-api-16 to android-x86, which could probably be pretty easily adapted to android-x86_64. I'll post them here for reference, though note that they may have bitrotted since originally written since the android-api-16 instrumented build & generate-profile tasks have changed periodically. (I tried to keep up with rebasing them, but I don't have a way to test them because of the privileged task issue). Hopefully they are useful for whoever picks this up in the future. When running locally I did also hit an issue with the gpu, so this patch may be necessary:

diff --git a/testing/mozharness/configs/android/androidx86_7_0.py b/testing/mozharness/configs/android/androidx86_7_0.py
index 518cc9b348b2f..748afbac4f39e 100644
--- a/testing/mozharness/configs/android/androidx86_7_0.py
+++ b/testing/mozharness/configs/android/androidx86_7_0.py
@@ -18,7 +18,7 @@ config = {
         ] """,
     "emulator_avd_name": "test-1",
     "emulator_process_name": "qemu-system-x86_64",
-    "emulator_extra_args": "-gpu on -skip-adb-auth -verbose -show-kernel -ranchu -selinux permissive -memory 3072 -cores 4",
+    "emulator_extra_args": "-gpu swiftshader_indirect -skip-adb-auth -verbose -show-kernel -ranchu -selinux permissive -memory 3072 -cores 4",
     "exes": {
         'adb': '%(abs_work_dir)s/android-sdk-linux/platform-tools/adb',
     },

And depending on how the privileged container thing ends up working, the generate-profile task might need a 'privileged: true' worker attribute.

We currently build with PGO on android-api-16, and would like to also
build with PGO for android-aarch64. Fortunately we can use the same
profile data for both, so it makes sense to generate this profile from
an android-x86 build for performance reasons (a ~5-minute runtime
instead of a ~30+ minute runtime). Additionally, tihs will make it so we
are no longer dependent on the arm emulator for PGO builds, which may be
going away in the future.

This is the 'run' task for Android PGO. All PGO-optimized Android builds
can use the profile data from this task.

Depends on D66652

Depends on D66653

The 90 second timeout matches the timeout used for desktop PGO in
build/pgo/index.html. The android emulator PGO had the timeout bumped
due to the slow performance of the arm emulator, but with the x86
emulator we no longer need the extended timeout.

Depends on D66654

We recently (bug 1631671) had trouble allowing for the old android 4.3 emulator used by pgo. I'm sure we can work around it, but it would be nice if this bug could move forward, so we don't have such a wide range of environments to be supported.

It looks like this might be blocked on bug 1545497? Can anyone confirm?

(In reply to Geoff Brown [:gbrown] from comment #10)

It looks like this might be blocked on bug 1545497? Can anyone confirm?

Yes, it is.

Mike won't be working on this. If the patches finally get a review and land as-is, we can assign him since it was his work, but better to not have an assignee who won't be pushing progress forward.