When a RemoteSettings-based blocklist was introduced, all Android tests were skipped in bug 1257565.

In bug 1706391, I re-enabled most tests to maintain a degree of test coverage, bu the patch stack got backed out for failures unrelated to my changes.
By the looks of it, it seems that the gfx blocklist isn't working on Android (or the tests are broken). I'm relanding the patches with the tests disabled again as before, but it's probably worth investigating why the gfx blocklist tests are failing on Android.

From https://treeherder.mozilla.org/jobs?repo=autoland&revision=ccd2f9b9c513ad8dca7cfbd3c6dc7fc4d200f7a0&selectedTaskRun=dMYxlgAuSFeR2a3WEaWrkA.0 :

TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_Equal_DriverNew.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_Equal_DriverNew.js | checkBlacklist - [checkBlacklist : 66] 1 == 3
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_Equal_OK.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_Equal_OK.js | checkBlacklist - [checkBlacklist : 53] 1 == 3
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_GTE_OK.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_GTE_OK.js | checkBlacklist - [checkBlacklist : 53] 1 == 3
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_No_Comparison.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_No_Comparison.js | checkBlacklist - [checkBlacklist : 49] 1 == 4
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_OK.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_OK.js | checkBlacklist - [checkBlacklist : 54] 1 == 3
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_Version.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_Version.js | checkBlocklist - [checkBlocklist : 60] 1 == 3
Return code: 1 

Oddly, I do recall running all tests (even with --verify) locally, in the x64 Android emulator (opt build, artifacts) without failures.

On try, the xpcshell tests on opt builds haven't failed.

So it might be a debug-only failure...

It's a debug-only failure because all test_gfxBlacklist* tests skip when nsIGfxInfoDebug is unavailable. Consequently the tests are only checking anything at all in debug builds:

• "We can't do anything if we can't spoof the stuff we need.", e.g. at https://searchfox.org/mozilla-central/rev/c1ec9ecbbc7eac698923ffd18c8594aa3e2e9da0/toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_GTE_OK.js#13-17
• #ifdef DEBUG - https://searchfox.org/mozilla-central/rev/c1ec9ecbbc7eac698923ffd18c8594aa3e2e9da0/widget/GfxInfoBase.h#35-38

The overall skip-if = os == "android" predates the migration to the RemoteSettings blocklist, e.g. test_gfxBlacklist_GTE_OK.js (the XML blocklist variant) (along with all tests in the directory) were disabled on Android in https://searchfox.org/mozilla-central/diff/d292ee73f107c441f68ae61730d8aed7c794dc14/toolkit/mozapps/extensions/test/xpcshell/xpcshell-shared.ini (7 oct 2014). The introduction of the skip-if did not mean that Android tests were running before, the mentioned xpcshell_android.ini file was introduced in bug 823703 and expanded later (without most of the tests from mozapps) (22 dec 2012). The test_gfxBlacklist_GTE_OK.js test was introduced in https://hg.mozilla.org/mozilla-central/rev/80c4d7317c29 (20 jan 2011).

I think that gfx blocklisting is entirely broken on Android because of the following changes:

• 2 nov 2012: https://hg.mozilla.org/mozilla-central/rev/f710f9f6f7e#l3.63 changed the os from DRIVER_OS_ANDROID (i.e. OperatingSystem::Android these days) to mOS, which was uninitialized, i.e. equivalent to "unknown" (these days: OperatingSystem::Unknown)
  • The change landed without unit tests for Android: https://bugzilla.mozilla.org/show_bug.cgi?id=806369#c25
• 8 jun 2016: https://hg.mozilla.org/mozilla-central/rev/af65533093de changed the logic, to require the OS of the blacklist entry to match. Except for macOS and Windows, where "unknown" is treated as matching.
  • .... but since Android unconditionally specified "unknown", the blacklist isn't working any more on Android.
  • this went undetected because unit tests for the blocklist tests were never run on Android CI.

After fixing that most of the affected tests started to pass. There are two more test failures, I'll look into that to see if it's something that I can fix.

The remote gfx blocklist for Android was broken since 2016 by
https://hg.mozilla.org/mozilla-central/rev/af65533093de as explained at
https://bugzilla.mozilla.org/show_bug.cgi?id=1714673#c2

This patch fixes the issue and re-enables the disabled tests.

test_gfxBlacklist_Equal_DriverNew.js fails on Android because the
meaning of "All" was changed by bug 1274635 to mean "All Windows"
instead of "All OSs", while the test relied on mock data in
test_gfxBlacklist.json that contains entries for "os": "All".

To fix the failures, add entries to test_gfxBlacklist.json based on
these existing "os": "All" entries, with driver version ranges chosen
to as closely match the original entry (greater/lower/equal compared to
6 (from gfxInfo.spoofDriverVersion("6")) in the test file).

The values for spoofVendorID/spoofDeviceID should match entries in
test_gfxBlacklist_AllOS.json. These entries were added in
https://hg.mozilla.org/mozilla-central/rev/21263fbfdcc2 (bug 1274152) but
inadvertently not updated (because the test was disabled).
Fixing that enables the test to use the expected entries.

Several test expectations were no longer correct for Android and have
been updated as well.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=9fbbdcc329ca8e69f53fc75452be15c175fb4c06

https://hg.mozilla.org/mozilla-central/rev/acf24b0a8c0c
https://hg.mozilla.org/mozilla-central/rev/65846ff92d38
https://hg.mozilla.org/mozilla-central/rev/51a6ac083fbe

The patch landed in nightly and beta is affected.
:robwu, is this bug important enough to require an uplift?
If not please set status_beta to wontfix.

For more information, please visit auto_nag documentation.