User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0

Steps to reproduce:

I build a custom Android firmware that includes Firefox as a system app (installed to /system/app). I download the apk from here: https://download.mozilla.org/?product=fennec-latest&os=android&lang=multi, and I use the following Android.mk for Firefox:

LOCAL_PATH := $(call my-dir)
include $(CLEAR_VARS)
LOCAL_MODULE_TAGS := optional
LOCAL_MODULE := Firefox
LOCAL_SRC_FILES := Firefox.apk
LOCAL_MODULE_CLASS := APPS
LOCAL_MODULE_SUFFIX := $(COMMON_ANDROID_PACKAGE_SUFFIX)
LOCAL_CERTIFICATE := PRESIGNED
LOCAL_PREBUILT_JNI_LIBS := $(addprefix @,$(filter lib/%,$(shell zipinfo -1 "$(LOCAL_PATH)/$(LOCAL_SRC_FILES)")))
include $(BUILD_PREBUILT)

When I start Firefox, it is stuck at the white screen, and I see a repeating stacktrace in the logcat.

Additional information:

1. If I then install the same apk using adb install (it gets installed as a user application, masking the system one), Firefox starts working.
2. If I remove LOCAL_PREBUILT_JNI_LIBS from Android.mk, it doesn't affect anything.
3. Firefox 65.0.1 used to work, even without LOCAL_PREBUILT_JNI_LIBS.
4. The log message mentions /system/app/Firefox/lib/arm/libmozglue.so. I indeed don't have /system/app/Firefox/lib/ directory, but the libs are stored in the APK ZIP uncompressed (after the Android build system plays with it), and this approach used to work with 65.0.1.

Actual results:

Firefox is stuck at the white screen, logcat is flooded with the repeating stacktrace (attached).

Expected results:

Firefox runs and works when installed to /system/app.

Maxim sorry for the delay here. Did you find a solution to this issue in the meantime?

(In reply to David Bolter [:davidb] (NeedInfo me for attention) from comment #1)

Maxim sorry for the delay here. Did you find a solution to this issue in the meantime?

I don't have a solution so far. All I have is an ugly workaround: I put the libs next to the apk, so my /system/app/Firefox contains Firefox.apk and lib/arm/*. It allows Firefox to start, but it's not a solution:

1. Libs consume space twice: in the apk and in the filesystem.
2. Firefox should be able to use libs from the apk stored uncompressed. It used to work with an older version of FIrefox.

Hi, thank you for reporting, regarding this issue, the devices we are using for testing have the original operating system, with no alteration of the original code. The product is dedicated and written to work correctly and adapt to the official Android versions and not with those which are altered. Sorry for the inconvenience.
I will mark the ticket as WORKS FOR ME.

From your message, it looks like you didn't get what the bug is. Please read the description, and if anything is not clear, ask me the details you need.

the devices we are using for testing have the original operating system, with no alteration of the original code.

This is simply not true. Any device on the market has vendor's modifications to original Android code.

Next thing, this bug is not about installing Firefox on some device (with original or custom firmware, whatever). This bug is about having Firefox installed out of the box (as part of firmware). Your statement is just not applicable.

One more thing is that this bug is a regression.

I will mark the ticket as WORKS FOR ME.

This resolution is not correct. "WORKS FOR ME" means you followed the steps to reproduce and couldn't reproduce the bug. You didn't even try to repeat my steps.

What I want to say is that it feels for me as Firefox stopped using Android APIs (in this case, assumptions about APK format) correctly in some version, and it prevents it from loading the shared libraries. If my assumption is correct, this is an issue in Firefox that should be fixed, even if it doesn't affect some cases like installing Firefox as normal application.

This is the way Chrome is installed on most Android phones out of the box. This bug prevents Firefox from doing the same — from being installed out of the box. In my opinion, it's a huge limitation comparing to the competitor.

Hi, I understand what this bug is about and I'm sorry that you feel the other way, but we don't test on custom Android firmware that includes Firefox as a system app, only on builds from Play Store.
WORKSFORME - is the term used in the domain when we try the steps and we cannot reproduce with our own environment which sometimes doesn't match the reporter.
I'll NI on Petru, a Fennec developer, for an extra opinion.

Hey Maxim,

Sorry for any misunderstandings but as my colleagues said we only offer support for official systems and official app releases given that any third party customization are beyond our power to test and control.
That being said, unofficial support for our power users is something that we try to accommodate when possible.

Regarding the issue at hand, indeed it seems like we supported this in the past and we now have a recent regression so I'll reopen this ticket.

I tested on a rooted device by just copying the apk in /system/app/firefox
My findings:
Latest release would give

java.lang.Exception: Error loading sqlite libraries
at org.mozilla.gecko.mozglue.GeckoLoader.loadSQLiteLibsNative(Native Method)
at org.mozilla.gecko.mozglue.GeckoLoader.loadSQLiteLibs(GeckoLoader.java:200)
at org.mozilla.gecko.GeckoThread.loadGeckoLibs(GeckoThread.java:265)
at org.mozilla.gecko.GeckoThread.initGeckoEnvironment(GeckoThread.java:285)
at org.mozilla.gecko.GeckoThread.run(GeckoThread.java:439)

@Maxim is this also what you're seeing?

Version 65.0.0 arm64-v8a would give the same

java.lang.Exception: Error loading sqlite libraries
at org.mozilla.gecko.mozglue.GeckoLoader.loadSQLiteLibsNative(Native Method)
at org.mozilla.gecko.mozglue.GeckoLoader.loadSQLiteLibs(GeckoLoader.java:199)
at org.mozilla.gecko.GeckoThread.loadGeckoLibs(GeckoThread.java:264)
at org.mozilla.gecko.GeckoThread.initGeckoEnvironment(GeckoThread.java:287)
at org.mozilla.gecko.GeckoThread.run(GeckoThread.java:415)

Version 64.0.2 arm64-v8a would give

E/GeckoLibLoad: Load sqlite start
--------- beginning of crash
2019-08-08 13:43:45.298 16520-16549/? A/libc: Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0 in tid 16549 (Gecko), pid 16520 (mozilla.firefox)
...
A/DEBUG: signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0
A/DEBUG: Cause: null pointer dereference

Version 61.0 would give

E/GeckoApp: An error occurred during restore, switching to backup file
org.mozilla.gecko.GeckoApp$SessionRestoreException: Could not read from session file
at org.mozilla.gecko.GeckoApp.restoreSessionTabs(GeckoApp.java:1603)
at org.mozilla.gecko.GeckoApp.access$200(GeckoApp.java:110)
at org.mozilla.gecko.GeckoApp$9.run(GeckoApp.java:1148)
at android.os.Handler.handleCallback(Handler.java:873)
at android.os.Handler.dispatchMessage(Handler.java:99)
at android.os.Looper.loop(Looper.java:214)
at org.mozilla.gecko.util.GeckoBackgroundThread.run(GeckoBackgroundThread.java:43)
E/GeckoApp: An error occurred during restore
org.mozilla.gecko.GeckoApp$SessionRestoreException: Could not read from session file
at org.mozilla.gecko.GeckoApp.restoreSessionTabs(GeckoApp.java:1603)
at org.mozilla.gecko.GeckoApp.access$200(GeckoApp.java:110)
at org.mozilla.gecko.GeckoApp$9.run(GeckoApp.java:1166)
at android.os.Handler.handleCallback(Handler.java:873)
at android.os.Handler.dispatchMessage(Handler.java:99)
at android.os.Looper.loop(Looper.java:214)
at org.mozilla.gecko.util.GeckoBackgroundThread.run(GeckoBackgroundThread.java:43)

So this needs deeper investigations.

Dissecting the issue, the described situation is the expected behavior for apps containing .so files just copied to /system/app (if the individual apps don't themselves have a way of extracting their .so libs at runtime)
This is because it is only at install time that the .so libs are extracted from the apk and put in the appropriate places.

You could overcome this issue in two ways:

• "preinstall" an older version of the app and update it afterwards from Playstore.
• add a second script for extracting the .so files, something like https://groups.google.com/d/msg/android-platform/4kRyiFofQkA/qvjlVEPwt2QJ

Bugbug thinks this bug is a regression, but please revert this change in case of error.

We stopped compressing libxul in Fennec 65 (bug 1486524) as part of the work to support multi-architecture ("fat") GeckoView AARs.

Is the /system/app/Firefox directory writable? As Petru wrote, I think Fennec needs to extract some files on first run. We're unlikely to fix this issue in Fennec, but we should consider ways to support this for new apps that use GeckoView.

I tested on a rooted device by just copying the apk in /system/app/firefox

Just copying the apk to /system/app won't do, because the build system of Android modifies the zip file to store the libs uncompressed. If your apk stores some *.so files compressed, it shouldn't work, from what I know.

@Maxim is this also what you're seeing?

Yes, the same stacktrace (you can see it in the log attached to the first post).

the described situation is the expected behavior for apps containing .so files just copied to /system/app

Might be, but I'm not just copying the apk, it is done by Android build system in the correct way, uncompressing the *.so files.

This is because it is only at install time that the .so libs are extracted from the apk and put in the appropriate places.

It happens for applications installed to /data/app, but system apps stores the libs uncompressed inside the apk, and the dynamic loader can load them from there (I don't know the details of this mechanism, so I can be wrong).

add a second script for extracting the .so files

That's the workaround I'm using now, I'm putting the libs next to the apk, but it leads to double consumption of disk space.

Is the /system/app/Firefox directory writable?

No, it's /system, it's mounted ro.

As Petru wrote, I think Fennec needs to extract some files on first run.

Could you explain what has changed, so that Firefox started extracting files on the first run? I see the bug you referred to, but it's not clear how stopping compressing libxul leads to the necessity to extract files on the first run.

Could you explain what has changed, so that Firefox started extracting files on the first run? I see the bug you referred to, but it's not clear how stopping compressing libxul leads to the necessity to extract files on the first run.

I'm not sure. The libxul compression bug 1486524 seemed like it might be related since it changed in the same Fennec release that stopped working.

Unfortunately, it's very unlikely we will spend time to fix this bug since Fennec is now in ESR maintenance mode. If this problem also affects Firefox Preview (aka "Fenix"), then we might spend more time on it.

https://play.google.com/store/apps/details?id=org.mozilla.fenix