Recently I see various issues when running mozilla tests on Firefox/Wayland. Also we need to somehow enable wayland backed on mozilla try tests.

Looks like there's problem with focus:
19:22.18 INFO Error: Unable to restore focus, expect failures and timeouts.

https://searchfox.org/mozilla-central/rev/153feabebc2d13bb4c29ef8adf104ec1ebd246ae/testing/mochitest/tests/SimpleTest/TestRunner.js#326

I managed to run it after all, there's a first try run on Wayland backend with some random tests selected:
https://treeherder.mozilla.org/jobs?repo=try&revision=2a4d6b19d3cfce3dea771d327872df38d8da906e&selectedTaskRun=H9L5ZIltScC_Wo8TEuY33A.0

With some tweaks I'm able to run mochitest locally. The dom directory looks pretty good:

mochitest-browser

Ran 19618 checks (19378 subtests, 240 tests)
Expected results: 19569
Skipped: 42 tests
Unexpected results: 7
subtest: 7 (7 fail)

mochitest-chrome

Ran 25235 checks (24880 subtests, 355 tests)
Expected results: 25206
Skipped: 18 tests
Unexpected results: 10
subtest: 10 (10 fail)

mochitest-plain

Ran 250114 checks (245268 subtests, 4846 tests)
Expected results: 249848
Skipped: 236 tests
Unexpected results: 21
subtest: 21 (20 fail, 1 pass)

There are particular failing tests:

mochitest-browser

dom/credentialmanagement/tests/browser/browser_active_document.js
dom/events/test/browser_mouse_enterleave_switch_tab.js
dom/html/test/browser_fullscreen-contextmenu-esc.js
dom/html/test/browser_fullscreen-newtab.js

mochitest-chrome

dom/tests/mochitest/chrome/test_clipboard_events_chrome.html
dom/tests/mochitest/chrome/test_queryCaretRect.html
dom/tests/mochitest/general/test_innerScreen.xhtml
dom/media/test/chrome/test_accumulated_play_time.html

mochitest-plain

dom/events/test/test_bug422132.html
dom/events/test/pointerevents/test_getCoalescedEvents.html
dom/html/test/test_fullscreen-api.html
dom/svg/test/test_bounds.html
dom/svg/test/test_getCTM.html
dom/tests/mochitest/bugs/test_sizetocontent_clamp.html
dom/tests/mochitest/general/test_interfaces_secureContext.html
dom/tests/mochitest/pointerlock/test_pointerlock-api.html
dom/media/test/test_eme_pssh_in_moof.html
dom/media/webrtc/tests/mochitests/test_peerConnection_basicAudioNATRelay.html
dom/media/webrtc/tests/mochitests/test_peerConnection_basicAudioNATRelayTCP.html
dom/media/webrtc/tests/mochitests/test_peerConnection_basicAudioNATSrflx.html
dom/media/webrtc/tests/mochitests/test_peerConnection_basicAudioNoisyUDPBlock.html
dom/media/webrtc/tests/mochitests/test_peerConnection_stats_relayProtocol.html
dom/media/webrtc/tests/mochitests/test_getUserMedia_audioConstraints.html
dom/canvas/test/webgl-mochitest/ensure-exts/test_EXT_texture_filter_anisotropic.html

Andres, is it possible to set a pref key to run the tests? Something like the --env we use to turn on Wayland. We'd need to set widget.wayland.test-workarounds.enabled to true for the testsuite.
Thanks.

Looks like we can run some Wayland tests even on such old infrastructure:

https://treeherder.mozilla.org/jobs?repo=try&revision=46391665343ef368d36cfaca067ce67e7f3cd703&selectedTaskRun=UiPFyOfgRjWz8oNXF4ELUg.0

I'll look at it to have at least some non-failing tests there which we can use for basic testing.

Looks like we can run some Wayland tests even on such old infrastructure:

Pretty cool! One more thing you could try would be adding disabling the Wayland vsync to widget.wayland.test-workarounds.enabled. The Mutter version in Ubuntu 18.04 is 3.28, which didn't get the patches necessary to make it run. I requested a backport[1], but so far it hasn't happened IIUC. So there is a chance that a couple of tests don't succeed because that Mutter bug.

1: https://bugs.launchpad.net/ubuntu/+source/mutter/+bug/1905330

New try looks much better:
https://treeherder.mozilla.org/jobs?repo=try&revision=91f63487ca15b59fb6f750ab43c619de70a958a1

Not via the command line at the moment (there's a bug on file somewhere), but you could add a user_pref to this file and it should get picked up by all major test harnesses:
https://searchfox.org/mozilla-central/source/testing/profiles/base/user.js

(see other user.js files in the profiles dir if you want to target a specific harness or need an example of the format)

Last try, I added wayland test variants:

https://treeherder.mozilla.org/jobs?repo=try&revision=9adc77fe97f383c57859d932aec989987d8ba918&selectedTaskRun=blRy6KvfQCusoYcSdk4XvQ.0

Andrew, do you think this approach is OK? If so I'll submit patches.
Thanks.

Yes, the approach is correct! I might have a few review requests, but that's better left to phabricator so feel free to submit and I can bring them up there. Thanks!

Also note we can land your variant without actually running it on trunk just yet, to make it easier for folks to run on try without needing to block on greening up the suites.

According to Andrew the Wayland tests can be run as:

$ ./mach try fuzzy --full -q "'wayland"

(when Bug 1708482 lands).

btw. the testsuite can be run with latest KWin compositor which supports xdg-activation. Mutter is expected to support it in 41.3 version.

(In reply to Martin Stránský [:stransky] (ni? me) from comment #14)

btw. the testsuite can be run with latest KWin compositor which supports xdg-activation. Mutter is expected to support it in 41.3 version.

I managed to bribe Jonas to make another Fedora 35 Mutter update (also because of other patches). So in case you want to try it over the next weeks, it should be ready soon: https://bodhi.fedoraproject.org/updates/FEDORA-2021-339e7bf6ef

Great, Thanks! Will look at it tomorrow.

(In reply to Robert Mader [:rmader] from comment #15)

(In reply to Martin Stránský [:stransky] (ni? me) from comment #14)

btw. the testsuite can be run with latest KWin compositor which supports xdg-activation. Mutter is expected to support it in 41.3 version.

I managed to bribe Jonas to make another Fedora 35 Mutter update (also because of other patches). So in case you want to try it over the next weeks, it should be ready soon: https://bodhi.fedoraproject.org/updates/FEDORA-2021-339e7bf6ef

xdg-activation & mutter-41.2-2.fc35.x86_64 seems to be working well, I can run the testsuite now.

I'm going to spin it again and run another round of tests on Ubuntu 22.04. Will post details later.