As of now all the wdspec tests that we are running in CI are done on desktop platforms, without any support for Android:

https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=wdspec

That means that the Android support that we are shipping for geckodriver is basically untested. The only pro side is that we have browsertime jobs actually utilizing the code path.

Nevertheless browsertime jobs seem to set some special Firefox preferences like devtools.debugger.remote-enabled=true, which make testing possible in Fenix. But that pref is not set by default in geckodriver, and as such normal users of Selenium/geckodriver will hit bug 1698968. For them it's not possible to test Fenix at all unless the pref is set via the session capabilities.

Joel, would there be enough bandwidth for e.g one Android platform to run wdspec tests? I think that we would also be happy in just running it on mozilla-central for opt.

I think we can do this specifically on emulators. As for hardware that is even more restricted than Apple, the more we can fine tune what we run, the better the chance we can squeeze it in. Emulators can run whatever you want to run!

I assume that all emulators and also the hardware are working as rooted devices? If that is the case is there a way to have unrooted support somewhere? That's actually missed most right now, and causes the main problem in geckodriver for Android.

I know the phones are rooted, I don't know about the emulators, let me redirect to :aerickson to comment if the emulators are rooted.

Emulators are rooted. Most of hardware devices are also rooted. We have 4 S7 Galaxy phones that are unrooted (rooted, but the rooting is disabled).

https://firefox-ci-tc.services.mozilla.com/provisioners/proj-autophone

worker-types are:
gecko-t-bitbar-gw-unit-s7 (2 devices)
gecko-t-bitbar-gw-perf-s7 (2 devices)

Andy

(In reply to Andrew Erickson [:aerickson] from comment #4)

Emulators are rooted. Most of hardware devices are also rooted. We have 4 S7 Galaxy phones that are unrooted (rooted, but the rooting is disabled).

Thanks Andy! I think going forward by using emulators might be the way to go then. I will have a look at it once dependencies are fixed.

Maybe we can also get bug 1708105 easily fixed to not have to disable that many wdspec tests for Android because user prompts cannot be handled.

James, could you maybe give us some hints in where to get started with wptrunner to get Android support added? Basically what we would have to do when creating a new session is to set the androidPackage capability. So it has to be passed-through from wptrunner. Could you foresee anything else that would be necessary?

We can reuse most of the implementation from Firefox, we just need to
ensure that the corect android options end up in the capabilities.

Not all platforms, notably mobile, support this, so we shouldn't
depend on it working in order for tests to run.

I put up a couple of patches that are enough to get things to run at least a little bit locally. I am still seeing some errors e.g. openTab not supported in the test browser. Maybe we want to default to a different package for these tests? That's a little tricky at the moment since we are setting the default value in the frontend (mach_commands.py or tools/wpt/run.py). I think we'd need to let it be passed in as None by default, and have the firefox_android.py take care of setting it to the right value depending on the test type, where ever it's used.

Thanks a lot James! That's pretty helpful when we would be able to run wdspec tests locally at least. We are aware that a lot is still not working yet including the WebDriver:NewWindow command, which is used for openTab. But that shouldn't stop us from landing changes to wptrunner. Once we can run locally it will be easier to fix.

I think that we should be fine in just using the default, which I assume is GeckoView_example? If we would like to run for Fenix we would have to create the jobs somewhere else anyway.

Bug 1708105 shouldn't block us from getting the attached patches on this bug landed. It's more the other way around given that they are helping us to better identify broken areas in wdspec that we have to focus on.

https://hg.mozilla.org/mozilla-central/rev/c9dafde8b9ee
https://hg.mozilla.org/mozilla-central/rev/d594570f12d1

(In reply to Web Platform Test Sync Bot (Matrix: #interop:mozilla.org) from comment #14)

Created web-platform-tests PR
https://github.com/web-platform-tests/wpt/pull/29949 for changes under
testing/web-platform/tests

James, can you please check why this PR doesn't get merged?

Turns out this was easier as expected. Lets count it for our M1 fixed bugs.

Will need to file follow up bugs to file to start running wdspec tests on Android on central.
As I understand the bugs that landed here get us closer to that goal, but we're still not running the tests.

We cannot run these tests on central yet due to various reasons, which are all marked as blocking bug 1723923. Nothing more needs to be done here.

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+1] from comment #20)

We cannot run these tests on central yet due to various reasons, which are all marked as blocking bug 1723923. Nothing more needs to be done here.

Actually this bug is BiDi only. I filed bug 1729409 for getting wdspec support added for Android.