User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:69.0) Gecko/20100101 Firefox/69.0

Steps to reproduce:

Running GeckoView Webextension tests:
./mach mochitest -f plain --app org.mozilla.geckoview.test mobile/android/components/extensions/test/mochitest --verify

with some tests disabled by skip-if = !is_fennec && verify in mobile/android/components/extensions/test/mochitest/mochitests.ini

Actual results:

Tests marked with skip-if should not run, but they do.

Geoff, is is_fennec defined for geckoview junit tests mochitests? Perhaps we need to change (some or all) these is_fennec annotations to os == 'android' or toolkit == 'android'. Is toolkit == 'android' true in the geckoview junit tests?

is_fennec is defined for mochitest only, I believe -- not geckoview junit. geckoview junit tests use junit annotations in the .kt files, which are quite different.

But this bug is about mochitest, and I expect the problem is with verify, not is_fennec.

I believe the verify annotations were added to test against whether a test was being run in a verify job (in TV/TVg/TVwpt in continuous integration). Running locally with --verify may be different.

(In reply to Geoff Brown [:gbrown] from comment #2)

I believe the verify annotations were added to test against whether a test was being run in a verify job (in TV/TVg/TVwpt in continuous integration).

Confirmed - see bug 1441990.

I think I agree with the bug -- running locally with --verify should skip tests annotated with skip-if=verify -- but I want to think about that.

Test manifest annotations like "skip-if = verify" are normally handled in mozharness:
When a TV task finds a modified test that needs verification, but that test is skipped
in verify mode, mozharness discards the test from the list of tests to be run.
When running tests locally (mach mochitest, etc), these annotations should also be
respected, and are currently handled in wpt and reftest harnesses, but not in
mochitest or xpcshell -- rectified by this patch. mochitest and xpcshell had
neglected to set mozinfo["verify"] when running in verify mode -- easily corrected.
It should be noted that when running tests locally and a single test is requested,
most test harnesses run the requested test even if it is skip-annotated. Thus,
"mach test <test> --verify" will continue to run tests annotated "skip-if = verify"
and this patch only changes the mochitest/xpcshell behavior of verify-skipped
tests when run with "mach test <directory> --verify" -- a long-running test mode
with complex logging never used in continuous integration.

Backed out for failures on on runxpcshelltests.py

backout: https://hg.mozilla.org/integration/autoland/rev/96b2970ebf26b20058b3a445498fb3af018dc145

push: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&revision=4afd3ec36843ae75ee9b09e524a7aa7846045c96&selectedJob=259991711

failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=259991711&repo=autoland&lineNumber=50815

[task 2019-08-05T20:29:01.159Z] 20:29:01 INFO - ===================== 48 passed, 3 skipped in 3.93 seconds =====================
[task 2019-08-05T20:29:01.728Z] 20:29:01 INFO - /builds/worker/workspace/build/src/testing/xpcshell/selftest.py
[task 2019-08-05T20:29:01.729Z] 20:29:01 INFO - ============================= test session starts ==============================
[task 2019-08-05T20:29:01.729Z] 20:29:01 INFO - platform linux2 -- Python 2.7.9, pytest-3.6.2, py-1.5.4, pluggy-0.6.0 -- /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/obj-firefox-8yIyzR8r-2.7/bin/python
[task 2019-08-05T20:29:01.729Z] 20:29:01 INFO - rootdir: /builds/worker/workspace/build/src, inifile: /builds/worker/workspace/build/src/config/mozunit/mozunit/pytest.ini
[task 2019-08-05T20:29:01.729Z] 20:29:01 INFO - collecting ... collected 56 items
[task 2019-08-05T20:29:01.729Z] 20:29:01 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTaskRunNextTest TEST-UNEXPECTED-FAIL
[task 2019-08-05T20:29:01.729Z] 20:29:01 WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTaskSkip TEST-UNEXPECTED-FAIL

I see the issue: xpcshell self.options is available on Android only. I'll work around that to work on desktop.

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=8c8d133cd0759e37e47513374cfe8c9d5b5148aa

https://hg.mozilla.org/mozilla-central/rev/38a106b2bf3d