Running mach wpt for web-platform-tests on mobile the org.mozilla.geckoview.test_runner application gets killed during the start-up process due to the following errors. Maybe the app is untrusted and that needs to be fixed here?

To actually get that far you will have to apply the patch from bug 1748506 or wait until it's fixed (hopefully later today).

01-05 13:02:48.991  1665  3501 I ActivityManager: Start proc 4715:org.mozilla.geckoview.test_runner/u0a65 for activity org.mozilla.geckoview.test_runner/.TestRunnerActivity
01-05 13:02:49.007  4715  4715 I MultiDex: VM with version 2.1.0 has multidex support
01-05 13:02:49.007  4715  4715 I MultiDex: Installing application
01-05 13:02:49.007  4715  4715 I MultiDex: VM has multidex support, MultiDex support library is disabled.
01-05 13:02:49.016  1336  1990 D gralloc_ranchu: gralloc_alloc: Creating ashmem region of size 4096000
01-05 13:02:49.043  4715  4715 D GeckoThread: State changed to LAUNCHED
01-05 13:02:49.043  4715  4730 I GeckoThread: preparing to run Gecko
01-05 13:02:49.044  4715  4730 D GeckoThread: env var: MOZ_CRASHREPORTER=1
01-05 13:02:49.044  4715  4730 D GeckoThread: env var: MOZ_CRASHREPORTER_NO_REPORT=1
01-05 13:02:49.044  4715  4730 D GeckoThread: env var: MOZ_CRASHREPORTER_SHUTDOWN=1
01-05 13:02:49.044  4715  4730 D GeckoThread: env var: MOZ_HIDE_RESULTS_TABLE=1
01-05 13:02:49.044  4715  4730 D GeckoThread: env var: MOZ_IN_AUTOMATION=1
01-05 13:02:49.044  4715  4730 D GeckoThread: env var: MOZ_LOG=signaling:3,mtransport:4,DataChannel:4,jsep:4
01-05 13:02:49.044  4715  4730 D GeckoThread: env var: R_LOG_LEVEL=6
01-05 13:02:49.044  4715  4730 D GeckoThread: env var: R_LOG_DESTINATION=stderr
01-05 13:02:49.044  4715  4730 D GeckoThread: env var: R_LOG_VERBOSE=1
01-05 13:02:49.044  4715  4730 D GeckoThread: env var: MOZ_PROCESS_LOG=/tmp/tmpk_9611oypidlog
01-05 13:02:49.044  4715  4730 D GeckoThread: env var: MOZ_DISABLE_NONLOCAL_CONNECTIONS=1
01-05 13:02:49.044  4715  4730 D GeckoThread: env var: STYLO_THREADS=1
01-05 13:02:49.044  4715  4730 D GeckoThread: env var: MOZ_WEBRENDER=0
01-05 13:02:49.048  2050  2128 D EGL_emulation: eglMakeCurrent: 0x79e0088470a0: ver 3 0 (tinfo 0x79e00880c3a0)
01-05 13:02:49.054  4715  4730 D GeckoThread: State changed to MOZGLUE_READY
01-05 13:02:49.054  4715  4715 D GeckoRuntime: Lifecycle: onCreate
01-05 13:02:49.067  4715  4730 I GeckoLoader: Library base=/data/app/org.mozilla.geckoview.test_runner-1/lib/x86_64
01-05 13:02:49.068  4715  4730 W Settings: Setting animator_duration_scale has moved from android.provider.Settings.System to android.provider.Settings.Global, returning read-only global URI.
01-05 13:02:49.069  4715  4730 I GeckoLoader: Library base=/data/app/org.mozilla.geckoview.test_runner-1/lib/x86_64
01-05 13:02:49.070  4715  4730 E GeckoLibLoad: Load sqlite start
01-05 13:02:49.072  4715  4730 E GeckoLibLoad: Load sqlite done
01-05 13:02:49.072  4715  4730 I GeckoLoader: Library base=/data/app/org.mozilla.geckoview.test_runner-1/lib/x86_64
01-05 13:02:49.072  4715  4730 E GeckoLibLoad: Load nss start
01-05 13:02:49.072  4715  4730 E GeckoLibLoad: Load nss done
01-05 13:02:49.072  4715  4730 I GeckoLoader: Library base=/data/app/org.mozilla.geckoview.test_runner-1/lib/x86_64
01-05 13:02:49.079  4715  4715 I art     : Rejecting re-init on previously-failed class java.lang.Class<androidx.core.view.ViewCompat$OnUnhandledKeyEventListenerWrapper>: java.lang.NoClassDefFoundError: Failed resolution of: Landroid/view/View$OnUnhandledKeyEventListener;
01-05 13:02:49.079  4715  4715 I art     :   at boolean androidx.core.view.ViewCompat.isAttachedToWindow(android.view.View) (ViewCompat.java:3049)
01-05 13:02:49.079  4715  4715 I art     :   at void org.mozilla.geckoview.GeckoView.setSession(org.mozilla.geckoview.GeckoSession) (GeckoView.java:457)
01-05 13:02:49.079  4715  4715 I art     :   at void org.mozilla.geckoview.test_runner.TestRunnerActivity.onCreate(android.os.Bundle) (TestRunnerActivity.java:442)
01-05 13:02:49.079  4715  4715 I art     :   at void android.app.Activity.performCreate(android.os.Bundle) (Activity.java:6662)
01-05 13:02:49.079  4715  4715 I art     :   at void android.app.Instrumentation.callActivityOnCreate(android.app.Activity, android.os.Bundle) (Instrumentation.java:1118)
01-05 13:02:49.079  4715  4715 I art     :   at android.app.Activity android.app.ActivityThread.performLaunchActivity(android.app.ActivityThread$ActivityClientRecord, android.content.Intent) (ActivityThread.java:2599)
01-05 13:02:49.079  4715  4715 I art     :   at void android.app.ActivityThread.handleLaunchActivity(android.app.ActivityThread$ActivityClientRecord, android.content.Intent, java.lang.String) (ActivityThread.java:2707)
01-05 13:02:49.079  4715  4715 I art     :   at void android.app.ActivityThread.-wrap12(android.app.ActivityThread, android.app.ActivityThread$ActivityClientRecord, android.content.Intent, java.lang.String) (ActivityThread.java:-1)
01-05 13:02:49.079  4715  4715 I art     :   at void android.app.ActivityThread$H.handleMessage(android.os.Message) (ActivityThread.java:1460)
01-05 13:02:49.079  4715  4715 I art     :   at void android.os.Handler.dispatchMessage(android.os.Message) (Handler.java:102)
01-05 13:02:49.079  4715  4715 I art     :   at void android.os.Looper.loop() (Looper.java:154)
01-05 13:02:49.079  4715  4715 I art     :   at void android.app.ActivityThread.main(java.lang.String[]) (ActivityThread.java:6077)
01-05 13:02:49.079  4715  4715 I art     :   at java.lang.Object java.lang.reflect.Method.invoke!(java.lang.Object, java.lang.Object[]) (Method.java:-2)
01-05 13:02:49.079  4715  4715 I art     :   at void com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run() (ZygoteInit.java:866)
01-05 13:02:49.079  4715  4715 I art     :   at void com.android.internal.os.ZygoteInit.main(java.lang.String[]) (ZygoteInit.java:756)
01-05 13:02:49.079  4715  4715 I art     : Caused by: java.lang.ClassNotFoundException: Didn't find class "android.view.View$OnUnhandledKeyEventListener" on path: DexPathList[[zip file "/system/framework/android.test.runner.jar", zip file "/data/app/org.mozilla.geckoview.test_runner-1/base.apk"],nativeLibraryDirectories=[/data/app/org.mozilla.geckoview.test_runner-1/lib/x86_64, /data/app/org.mozilla.geckoview.test_runner-1/base.apk!/lib/x86_64, /system/lib64, /vendor/lib64]]
01-05 13:02:49.079  4715  4715 I art     :   at java.lang.Class dalvik.system.BaseDexClassLoader.findClass(java.lang.String) (BaseDexClassLoader.java:56)
01-05 13:02:49.079  4715  4715 I art     :   at java.lang.Class java.lang.ClassLoader.loadClass(java.lang.String, boolean) (ClassLoader.java:380)
01-05 13:02:49.079  4715  4715 I art     :   at java.lang.Class java.lang.ClassLoader.loadClass(java.lang.String) (ClassLoader.java:312)
01-05 13:02:49.079  4715  4715 I art     :   at boolean androidx.core.view.ViewCompat.isAttachedToWindow(android.view.View) (ViewCompat.java:3049)
01-05 13:02:49.079  4715  4715 I art     :   at void org.mozilla.geckoview.GeckoView.setSession(org.mozilla.geckoview.GeckoSession) (GeckoView.java:457)
01-05 13:02:49.079  4715  4715 I art     :   at void org.mozilla.geckoview.test_runner.TestRunnerActivity.onCreate(android.os.Bundle) (TestRunnerActivity.java:442)
01-05 13:02:49.079  4715  4715 I art     :   at void android.app.Activity.performCreate(android.os.Bundle) (Activity.java:6662)
01-05 13:02:49.079  4715  4715 I art     :   at void android.app.Instrumentation.callActivityOnCreate(android.app.Activity, android.os.Bundle) (Instrumentation.java:1118)
01-05 13:02:49.080  4715  4715 I art     :   at android.app.Activity android.app.ActivityThread.performLaunchActivity(android.app.ActivityThread$ActivityClientRecord, android.content.Intent) (ActivityThread.java:2599)
01-05 13:02:49.080  4715  4715 I art     :   at void android.app.ActivityThread.handleLaunchActivity(android.app.ActivityThread$ActivityClientRecord, android.content.Intent, java.lang.String) (ActivityThread.java:2707)
01-05 13:02:49.080  4715  4715 I art     :   at void android.app.ActivityThread.-wrap12(android.app.ActivityThread, android.app.ActivityThread$ActivityClientRecord, android.content.Intent, java.lang.String) (ActivityThread.java:-1)
01-05 13:02:49.080  4715  4715 I art     :   at void android.app.ActivityThread$H.handleMessage(android.os.Message) (ActivityThread.java:1460)
01-05 13:02:49.080  4715  4715 I art     :   at void android.os.Handler.dispatchMessage(android.os.Message) (Handler.java:102)
01-05 13:02:49.080  4715  4715 I art     :   at void android.os.Looper.loop() (Looper.java:154)
01-05 13:02:49.080  4715  4715 I art     :   at void android.app.ActivityThread.main(java.lang.String[]) (ActivityThread.java:6077)
01-05 13:02:49.080  4715  4715 I art     :   at java.lang.Object java.lang.reflect.Method.invoke!(java.lang.Object, java.lang.Object[]) (Method.java:-2)
01-05 13:02:49.080  4715  4715 I art     :   at void com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run() (ZygoteInit.java:866)
01-05 13:02:49.080  4715  4715 I art     :   at void com.android.internal.os.ZygoteInit.main(java.lang.String[]) (ZygoteInit.java:756)
01-05 13:02:49.080  4715  4715 I art     : 
01-05 13:02:49.101  4715  4730 E GeckoLibLoad: Loaded libs in 28.886970ms total, 10ms(30ms) user, 10ms(20ms) system, 0(0) faults
01-05 13:02:49.101  4715  4730 D GeckoThread: State changed to LIBS_READY
01-05 13:02:49.104  4715  4730 W GeckoThread: zerdatime 8100757 - runGecko
01-05 13:02:49.095  4715  4715 I Gecko   : type=1400 audit(0.0:37): avc: denied { write } for name="profile" dev="vdc" ino=168964 scontext=u:r:untrusted_app:s0:c512,c768 tcontext=u:object_r:shell_data_file:s0 tclass=dir permissive=1
01-05 13:02:49.095  4715  4715 I Gecko   : type=1400 audit(0.0:38): avc: denied { add_name } for name=".parentlock" scontext=u:r:untrusted_app:s0:c512,c768 tcontext=u:object_r:shell_data_file:s0 tclass=dir permissive=1
01-05 13:02:49.115  4715  4730 F libc    : Fatal signal 11 (SIGSEGV), code 1, fault addr 0x0 in tid 4730 (Gecko)
01-05 13:02:49.116  1294  1294 W         : debuggerd: handling request: pid=4715 uid=10065 gid=10065 tid=4730
01-05 13:02:49.095  4715  4715 I Gecko   : type=1400 audit(0.0:39): avc: denied { create } for name=".parentlock" scontext=u:r:untrusted_app:s0:c512,c768 tcontext=u:object_r:shell_data_file:s0:c512,c768 tclass=file permissive=1
01-05 13:02:49.105  4715  4715 I Gecko   : type=1400 audit(0.0:40): avc: denied { create } for name="lock" scontext=u:r:untrusted_app:s0:c512,c768 tcontext=u:object_r:shell_data_file:s0:c512,c768 tclass=lnk_file permissive=1
01-05 13:02:49.105  4715  4715 I Gecko   : type=1400 audit(0.0:41): avc: denied { create } for name="minidumps" scontext=u:r:untrusted_app:s0:c512,c768 tcontext=u:object_r:shell_data_file:s0:c512,c768 tclass=dir permissive=1
01-05 13:02:49.105  4715  4715 I Gecko   : type=1400 audit(0.0:42): avc: denied { write } for name="crashes" dev="vdc" ino=168976 scontext=u:r:untrusted_app:s0:c512,c768 tcontext=u:object_r:shell_data_file:s0:c512,c768 tclass=dir permissive=1
01-05 13:02:49.105  4715  4715 I Gecko   : type=1400 audit(0.0:43): avc: denied { add_name } for name="events" scontext=u:r:untrusted_app:s0:c512,c768 tcontext=u:object_r:shell_data_file:s0:c512,c768 tclass=dir permissive=1
01-05 13:02:49.105  4715  4715 I Gecko   : type=1400 audit(0.0:44): avc: denied { remove_name } for name="lock" dev="vdc" ino=168974 scontext=u:r:untrusted_app:s0:c512,c768 tcontext=u:object_r:shell_data_file:s0 tclass=dir permissive=1
01-05 13:02:49.105  4715  4715 I Gecko   : type=1400 audit(0.0:45): avc: denied { unlink } for name="lock" dev="vdc" ino=168974 scontext=u:r:untrusted_app:s0:c512,c768 tcontext=u:object_r:shell_data_file:s0:c512,c768 tclass=lnk_file permissive=1
01-05 13:02:49.175  4734  4734 E DEBUG   : unexpected waitpid response: n=4730, status=00000b00
01-05 13:02:49.175  4734  4734 E         : debuggerd: timed out waiting for signal
01-05 13:02:49.176  4734  4734 E         : debuggerd: ptrace detach from 4730 failed: No such process
01-05 13:02:49.176  4734  4734 E         : debuggerd: failed to kill process 4715: No such process
01-05 13:02:49.177  1665  4735 W ActivityManager:   Force finishing activity org.mozilla.geckoview.test_runner/.TestRunnerActivity

The above is not actually the problem. After some more investigation I have seen that we get an expected crash with the following crash reason when running a debug (instead of an opt) build:

F MOZ_CRASH: Hit MOZ_CRASH(Missing chrome or resource URLs: resource://gre/modules/CrashService.jsm) at /builds/worker/checkouts/gecko/netwerk/base/nsNetUtil.cpp:3427

I've taken a look into the APK as found under obj/gradle/build/mobile/android/test_runner/outputs/apk/withGeckoBinaries/debug and the omni.ja file's components folder misses nearly all components compared to an artifact build as created by our CI. The only two components in that folder are HandlerService.js and TestInterface.js. It looks actually the same for a geckoview example apk.

Oh, and the crash is always reproducible since Firefox 92 when the fix on bug 1721627 got landed. Just run the following steps:

1. Run mach bootstrap and configure your build for a mobile artifact build + create the appropriate mozconfig
2. Run mach build
3. Run mach wpt testing/web-platform/tests/console/ with an emulator

NI? myself so I don't forget to check this out.

Is it reproducible on a non-artifact build?

This is due to the crash reporter being disabled in local builds on Android, per https://searchfox.org/mozilla-central/rev/84c1031f19b2cf8221da0850983f8d0269f6efd1/toolkit/moz.configure#2670-2673

Workaround: Add ac_add_options --enable-crashreporter to your mozconfig.

(In reply to Mike Hommey [:glandium] from comment #5)

This is due to the crash reporter being disabled in local builds on Android, per https://searchfox.org/mozilla-central/rev/84c1031f19b2cf8221da0850983f8d0269f6efd1/toolkit/moz.configure#2670-2673

To clarify, that makes a discrepancy with the artifacts, where the crashreporter is enabled.

(Note this is a general problem we have with artifact builds, where the local config doesn't necessarily match that of the artifacts)

(In reply to Mike Hommey [:glandium] from comment #5)

This is due to the crash reporter being disabled in local builds on Android, per https://searchfox.org/mozilla-central/rev/84c1031f19b2cf8221da0850983f8d0269f6efd1/toolkit/moz.configure#2670-2673

That is interesting. So given that the crash reporter is disabled in local artifact builds why do we try to actually load the CrashService.jsm? Is that triggered by the call to MOZ_CRASH?

Also note that I do not see a stacktrace printed to the Android log even with it being disabled.

Workaround: Add ac_add_options --enable-crashreporter to your mozconfig.

Ok, that makes it working. Thanks a lot!

But I still don't understand why we have nearly no files in the components folder of the omni.jar. Did I inspect the wrong apk file (see comment 1)?

Updating summary because it only affects startups under automation.

it's probably libxul.so that is trying to load CrashService.jsm, via @mozilla.org/crashservice;1.

But I still don't understand why we have nearly no files in the components folder of the omni.jar.

Because we don't seem to have a lot of components. https://searchfox.org/mozilla-central/search?q=components&path=mobile%2Fandroid%2Finstaller%2Fpackage-manifest.in&case=false&regexp=false

(In reply to Mike Hommey [:glandium] from comment #10)

it's probably libxul.so that is trying to load CrashService.jsm, via @mozilla.org/crashservice;1.

We shouldn't be trying to do that though. The MOZ_CRASH should give you a C++ stack, and on debug builds we'll also try to print a JS stack for this crash error if there's JS on the stack, which should help clarify where specifically the load is coming from.

(In reply to :Gijs (he/him) from comment #11)

(In reply to Mike Hommey [:glandium] from comment #10)

it's probably libxul.so that is trying to load CrashService.jsm, via @mozilla.org/crashservice;1.

We shouldn't be trying to do that though. The MOZ_CRASH should give you a C++ stack, and on debug builds we'll also try to print a JS stack for this crash error if there's JS on the stack, which should help clarify where specifically the load is coming from.

... looks like there's only one place where we load the crash service like that anyway, https://searchfox.org/mozilla-central/rev/682e5a82d403974bacb779c1db515962d946be48/ipc/glue/CrashReporterHost.cpp#128-132 - could try putting that in an #ifdef MOZ_CRASHREPORTER.

That would change nothing... the crash reporter is enabled when that code is built. That's the whole problem. Incidentally, that code does support the crash service not being there... but it's the jar loader that doesn't let it handle its error case.

(In reply to Mike Hommey [:glandium] from comment #13)

That would change nothing... the crash reporter is enabled when that code is built. That's the whole problem.

Then can we change the build system such that --enable-artifacts triggers using the same flags that we use to produce the binary artifacts for the creation of the non-compiled code, or enforce that the artifacts we download match the local build config? It makes no sense for the flags to diverge, and the number of ways in which that could go wrong is a much bigger set than just the jar loader's checks here. If compiled code's #ifdef FOO doesn't match the local-artifact-build-created AppConstants.FOO then you're going to be in trouble...

Nick, any thoughts on this?

Incidentally, that code does support the crash service not being there... but it's the jar loader that doesn't let it handle its error case.

Right - but there's no way for the jar loader to know if the consumer is going to deal with that error case. There just isn't. If there's a better way for us to detect us relying on files that we're not packaging/shipping, I'd love to hear it - but doing the checks in the actual channel loads seemed like the obvious choice... I've already exempted fetch and xhr requests because they tend to come with error handling, but I'm loath to add module imports to that list because the vast majority of the JS ones (which in turn is the vast majority of module loads) do not handle errors at all.

Anyway, the short-term solution might be to add an exception to the jar loader code to allow consumers to attempt to load CrashService.jsm - though I expect that'll eventually cause false positives for frontend code that assumes the crash reporter stuff is all there...

This is an interesting corner-case. The crash service is used by the CrashReporterHost even if crash reporting is disabled. That's because it can send a telemetry crash ping even if no crash report was generated. However to do that it would need both the CrashService.jsm and CrashManager.jsm modules which means that in practice we disable crash pings together with the rest of the crash reporting machinery.

(In reply to :Gijs (he/him) from comment #11)

We shouldn't be trying to do that though. The MOZ_CRASH should give you a C++ stack, and on debug builds we'll also try to print a JS stack for this crash error if there's JS on the stack, which should help clarify where specifically the load is coming from.

That's interesting. The C++ stack is not shown when I run the above mentioned mach wpt command for the artifact mobile build. Only debug builds show the empty JS stack. That means using an opt build you basically do not get any information right now why it's failing at all. I think that this is something we should also fix while having a way to easily reproduce such a crash.

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+1] from comment #16)

(In reply to :Gijs (he/him) from comment #11)

We shouldn't be trying to do that though. The MOZ_CRASH should give you a C++ stack, and on debug builds we'll also try to print a JS stack for this crash error if there's JS on the stack, which should help clarify where specifically the load is coming from.

That's interesting. The C++ stack is not shown when I run the above mentioned mach wpt command for the artifact mobile build. Only debug builds show the empty JS stack. That means using an opt build you basically do not get any information right now why it's failing at all. I think that this is something we should also fix while having a way to easily reproduce such a crash.

mobile logging is weird, and without a crashreporter I don't know what's happening to dump files. But this is just using MOZ_CRASH - that prints stacks on infra, on debug builds, though again I think they might be in the adb log or some other strange place because mobile. It's really not a problem with this specific code - if something is going wrong, it'd be wrong for all MOZ_CRASH invocations. So I don't think this bug is the right place to figure it out.

I'm also wondering why we're hitting this code at all - AIUI it means a child process crashed. So why is the child process crashing? I guess if you build artifact locally with --enable-crashreporter perhaps you can find the answer to that question...

Set release status flags based on info from the regressing bug 1721627

(In reply to :Gijs (he/him) from comment #17)

I'm also wondering why we're hitting this code at all - AIUI it means a child process crashed. So why is the child process crashing? I guess if you build artifact locally with --enable-crashreporter perhaps you can find the answer to that question...

Sadly not. When the crash reporter is enabled I don't hit this crash and wpt tests run fine as expected.

Looks like a duplicate of bug 1728226. Would be nice to have a fix :)

A work-around for local development (also mentioned in this bug) is to add ac_add_options --enable-crashreporter to the .mozconfig file.

(In reply to :Gijs (he/him) from comment #14)

(In reply to Mike Hommey [:glandium] from comment #13)

That would change nothing... the crash reporter is enabled when that code is built. That's the whole problem.

Then can we change the build system such that --enable-artifacts triggers using the same flags that we use to produce the binary artifacts for the creation of the non-compiled code, or enforce that the artifacts we download match the local build config? It makes no sense for the flags to diverge, and the number of ways in which that could go wrong is a much bigger set than just the jar loader's checks here. If compiled code's #ifdef FOO doesn't match the local-artifact-build-created AppConstants.FOO then you're going to be in trouble...

Nick, any thoughts on this?

Yes, I certainly do Have Thoughts. In the short term, we absolutely want the default artifact configuration to match the upstream configuration. At one time, this was tricky w.r.t. Android, but now it does not appear to be so: there's a conscious choice to disable the crash reporter for developer builds targeting Android at https://searchfox.org/mozilla-central/rev/618f9970972adc5a21194d39d690ec0865f26024/toolkit/moz.configure#2673. We can make that take into account artifact builds as well; that's the immediate thing to be done.

In the longer term, we have many places where we'd like the local artifact build to be able to do more things with fine-grained parts of the consumed automation build. For example, GeckoView would like to consume some things generated from IDL files, which isn't easy to arrange right now in an artifact build: https://bugzilla.mozilla.org/show_bug.cgi?id=1580919 and https://bugzilla.mozilla.org/show_bug.cgi?id=1710463. I'm not aware of any obstacle to producing additional files in automation, placing them in dist/... for artifact builds to find, and acting upon them within an artifact build, but doubtless there are subtle issues and races to be discovered. We might be able to do that for flags such as --enable-crashreporter that impact both native and non-native code: we could have artifact builds verify that the automation config.status is "compatible" in various ways. But I haven't tried anything in this direction so I'm not sure if it's truly possible, let alone advisable.