Now that bug 1496627 lets us run the android wpt tests locally, and Bug 1495430 (and friends) added support to run them in CI, we need to get them greened up to a point where they can be turned on to run in CI per push, and developers can trust the results to know if new patches break them. Splitting the tests into several chunks, only a handful of tests are failing per chunk, and a few chunks actually apper to be fully passing: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed,busted,exception,success,retry,running,pending,runnable&tier=3&tochange=955a7fd1a807357faa79336b16418c41541be24e&fromchange=0ce62298c460df7dccc0d074c73999f13685b11b&group_state=expanded At first glance, some failures seem to be harness failures (we'll need to fix these), some are timeouts (emulator too slow for the default timeout values, we can maybe just increase the timeout for mobile tests), and some are actually unexpected passes (Fennec supports some webkit prefixed CSS, IIRC, making us pass those tests unexpectedly). So far all of these tests (ci and local) have been done exclusively with the x86 android emulator, which is much faster than the various ARM emulators. This greening up work will probably be only done with the x86 emulator. It's unclear to me what the plan is for running these tests on the ARM emulators, but I'd hope the only real difference is that we'd need to bump the default timeout value even higher. I'm not planning on dealing with that in this bug, though.

For the timeout value the thing to do is to set the timeout_multiplier value in [1]. We currently just use the function for desktop firefox, which may be conservative for mobile, even on x86. The first thing to do however might be to just update the expectation values for mobile and see how it looks. There's a script to fetch logs from treeherder in testing/web-platform/update/fetchlogs.py although I use the Rust application at [2]; with the latter you should be able to run something like fetchlogs try 955a7fd1a807357faa79336b16418c41541be24e --out-dir logs --log-type wptreport And then to set the expectation data to match the results, do something like ./mach wpt-update logs/* Although that will probably overwrite the default values in some cases (because without non-mobile results it can't understand that these are android-specific). So at some point we might need a try run with all the platforms. And to support reftests you might just be able to import the MarionetteReftestRunner in fennec.py and edit the dictionary at [3] to add {"reftest": MarionetteReftestRunner} or something. [1] https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/fennec.py#102 [2] https://github.com/jgraham/fetchlogs [3] https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/fennec.py#27

That's most of the wpt directories... There are still a few directories that could use bugs: These ones (and maybe others) seem to be timing out intermittently: payment-request picture-in-picture I guess they're right up against the timeout threshold, and if we bumped that a bit they might more consistently pass? These ones have lots of failures within them, and could probably be split up further to be more manageable: css encoding html intersection-observer

Some of these are probably just platform differences (eg device orientation matters on a phone but less so on the desktop) while others look like actual failures. Rather than working through them all now, just disable the ones that fail, so we can maybe turn the fennec wpt tests on while fixing and re-enabling these skipped tests.

Depends on D13012

With those patches applied, the try run looks like https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&group_state=expanded&revision=64890a86f79fd81da0e0fd4f8c266ab36dad8d86

For what it's worth, the wasm abort test should have been fixed properly by bug 1505632 which landed a few days ago. Can it be removed from the exclusion list?

Pushed by james@hoppipolla.co.uk: https://hg.mozilla.org/integration/autoland/rev/268c6832d2be Update Fennec wpt expectations r=jgraham

Pushed by james@hoppipolla.co.uk: https://hg.mozilla.org/integration/autoland/rev/cda49d66b3cc Skip some especially flaky fennec wpt tests r=jgraham

https://hg.mozilla.org/mozilla-central/rev/268c6832d2be https://hg.mozilla.org/mozilla-central/rev/cda49d66b3cc

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/14263 for changes under testing/web-platform/tests

Upstream PR merged

(In reply to Pulsebot from comment #13) > Pushed by wkocher@mozilla.com: > https://hg.mozilla.org/integration/mozilla-inbound/rev/8d5c855aedff > Update annotation references for content-security-policy/navigate-to/ r=me > a=testonly This got backed out because it included unrelated changes, and it had the wrong bug number attached to it. Oops. (In reply to Benjamin Bouvier [:bbouvier] from comment #7) > For what it's worth, the wasm abort test should have been fixed properly by > bug 1505632 which landed a few days ago. Can it be removed from the > exclusion list? Those patches landed before I could try addressing this, can you file a followup bug for that change?

Backout by shindli@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/431c208f6787 Backed out changeset 8d5c855aedff per developer's backout on IRC on a CLOSED TREE

Pushed by wkocher@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/1233b674e1cd Update the referenced bug for skipped fennec wpt user-timing test r=me a=test-only

Pushed by wkocher@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/5b5f5accfcae Update the referenced bug for skipped fennec wpt service-worker tests r=me a=testonly

https://hg.mozilla.org/mozilla-central/rev/1233b674e1cd https://hg.mozilla.org/mozilla-central/rev/5b5f5accfcae