We made a try push [1] with a patch that updated a mochitest [2]. If I understand correctly, test-verify should've ran and verified this test, but it failed to pick-up that it'd been updated (via [3]): ... INFO - retry: Calling get_automationrelevance with args: (), kwargs: {}, attempt #1 INFO - retry: Calling _urlopen with args: (), kwargs: {'url': 'https://hg.mozilla.org/try/json-automationrelevance/69beda17be479fad950a2bb7c2c9c64808dd1e06'}, attempt #1 INFO - 0a2ab5ec7bc9 Bug 1423011 - Part 1: Allow APZ to async-scroll the layout viewport. r?botond INFO - ae59e265fb43 Bug 1423011 - Part 2: Add gtests. r?botond INFO - 7dddb5ab185d Bug 1423011 - Part 3: Add mochitests. r?botond INFO - 66572af458b8 Bug 1423011 - Part 4: Fix for overflow-x:hidden test failures in the mochitest test_wheel_default_action.html. r?botond INFO - 69beda17be47 try: -b do -p all -u all -t none INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/mochitest/tests/mochitest.ini INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/mochitest/chrome/chrome.ini INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/mochitest/browser/browser-chrome.ini INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/mochitest/a11y/a11y.ini INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/xpcshell/tests/xpcshell.ini INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/reftest/tests/layout/reftests/reftest.list INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/reftest/tests/testing/crashtest/crashtests.list INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/jsreftest/tests/jstests.list INFO - Running post-action listener: set_extra_try_arguments INFO - Running post-action listener: setup_coverage_tools ... We have similar pushes where the latest commit was the one to update the mochitest, and test-verify picked up the test change in those (e.g. [4]). [1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=69beda17be479fad950a2bb7c2c9c64808dd1e06 [2] https://hg.mozilla.org/try/rev/7dddb5ab185d3aa3ca1e8201f632d3fb8b5b0a8c [3] https://taskcluster-artifacts.net/Gj3S49uuS6GlMc4lO45QZQ/0/public/logs/live_backing.log [4] https://treeherder.mozilla.org/#/jobs?repo=try&revision=d7455ac0ac9cc4646db9e06e2d37a85798998806

Reproduced with debugging, apparently regardless of order of commits: https://treeherder.mozilla.org/#/jobs?repo=try&revision=d250857a5c774a3b0bb84ecc027c26061ff17456 https://treeherder.mozilla.org/#/jobs?repo=try&revision=74f68df28e6251138a1c1f9fb5216d0fc1790404

According to the debugging on try, gfx/layers/apz/test/mochitest/test_group_zoom.html is considered for verification in every TV run. Note that test_group_zoom.html is annotated: skip-if = (toolkit != 'android') # only android supports zoom so only Android runs are relevant.

According to the debugging on try, for Android, gfx/layers/apz/test/mochitest/test_group_zoom.html is skipped from verification because we hit the continue here: https://dxr.mozilla.org/mozilla-central/rev/085cdfb90903d4985f0de1dc7786522d9fb45596/testing/mozharness/mozharness/mozilla/testing/per_test_base.py#98 So this Android TV task seems to think it is running with --gpu-required? That seems wrong... Maybe :jmaher happens to understand this better?

> So this Android TV task seems to think it is running with --gpu-required? Or not... [task 2018-07-09T21:07:55.735Z] 21:07:55 INFO - 'gpu_required': False,

False vs 'False'? https://treeherder.mozilla.org/#/jobs?repo=try&revision=c92227983ff4ae7719e548dd45aaa61f7b479055

(In reply to Geoff Brown [:gbrown] from comment #5) > False vs 'False'? > > https://treeherder.mozilla.org/#/ > jobs?repo=try&revision=c92227983ff4ae7719e548dd45aaa61f7b479055 https://treeherder.mozilla.org/#/jobs?repo=try&revision=eb64967814ce9b64fcb3cdb468f765c27075a765

See try push in comment 6, which successfully ran the test verification on Android (although the test failed verification). The inconsistency in gpu_required has, I think, resulted in skipping all non-gpu tests for Android test verification. I see very few tests verified on Android on recent mozilla-central pushes. Oops. Do you have suggestions for additional testing scenarios?

(In reply to Geoff Brown [:gbrown] from comment #7) > (although the test failed verification). (That part is expected and is due to the local patches applies in that push.)

Comment on attachment 8990878 [details] [diff] [review] make gpu_required consistent between desktop and android Review of attachment 8990878 [details] [diff] [review]: ----------------------------------------------------------------- oh, I like a boolean False instead of a string False, good find. I wonder if this is why many of the [TV-bf notfound] tests are not being run: https://bugzilla.mozilla.org/buglist.cgi?resolution=---&resolution=FIXED&resolution=INVALID&resolution=WONTFIX&resolution=DUPLICATE&resolution=WORKSFORME&status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=[TV-bf%20notfound&list_id=14219633

Sanity check: https://treeherder.mozilla.org/#/jobs?repo=try&revision=113a9e146a8b2d35dc90a66e6882352ed6f572d9 (vs https://treeherder.mozilla.org/#/jobs?repo=try&revision=c153a8ea7fd4968b92d388583c11c3fe04aa0b92 before patch) ...looks okay to me.

Pushed by gbrown@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/33a1ccfa793e Ensure test-verify knows when it is running in gpu mode; r=jmaher

(In reply to Geoff Brown [:gbrown] from comment #10) > Sanity check: > https://treeherder.mozilla.org/#/ > jobs?repo=try&revision=113a9e146a8b2d35dc90a66e6882352ed6f572d9 > (vs > https://treeherder.mozilla.org/#/ > jobs?repo=try&revision=c153a8ea7fd4968b92d388583c11c3fe04aa0b92 before patch) > > ...looks okay to me. If I'm not mistaken, the Android TV didn't run a few files: ... INFO - Per-test run found test netwerk/test/unit/test_reopen.js (xpcshell) INFO - Per-test run found test dom/animation/test/crashtests/1216842-5.html (crashtest) INFO - Per-test run found test netwerk/test/unit/test_simple.js (xpcshell) INFO - Per-test run found test dom/animation/test/crashtests/1216842-1.html (crashtest) INFO - Per-test run found test dom/animation/test/crashtests/1216842-4.html (crashtest) INFO - Per-test run found test dom/animation/test/crashtests/1216842-2.html (crashtest) INFO - Per-test run found test dom/canvas/test/test_hitregion_canvas.html (plain) INFO - Per-test run found test dom/animation/test/crashtests/1244595-1.html (crashtest) INFO - Per-test run found test dom/canvas/test/test_2d.composite.canvas.multiply.html (plain) INFO - Per-test run found test caps/tests/mochitest/browser_checkloaduri.js (browser-chrome) INFO - Per-test run found test dom/canvas/test/test_2d.composite.canvas.source-in.html (plain) INFO - Per-test run found test dom/canvas/test/test_2d_composite_canvaspattern_setTransform.html (plain) INFO - Per-test run found test dom/animation/test/crashtests/1216842-3.html (crashtest) INFO - Per-test run found test dom/canvas/test/test_2d.composite.canvas.source-out.html (plain) INFO - Per-test run found test editor/libeditor/crashtests/448329-1.html (crashtest) INFO - Per-test run found test testing/mochitest/tests/Harness_sanity/test_sanity.html (plain) INFO - Per-test run found test dom/canvas/test/test_2d.composite.canvas.screen.html (plain) INFO - Per-test run found test accessible/tests/mochitest/treeupdate/test_ariaowns.html (a11y) INFO - Per-test run found test dom/animation/test/crashtests/1239889-1.html (crashtest) ... (via https://taskcluster-artifacts.net/M6Mw02Z3TOKkQMaUzQfxZQ/1/public/logs/live_backing.log) No mention of the following: devtools/client/debugger/new/test/mochitest/browser_dbg-chrome-debugging.js dom/canvas/test/test_2d.composite.canvas.xor.html dom/canvas/test/test_offscreencanvas_basic_webgl.html dom/canvas/test/test_offscreencanvas_many.html dom/canvas/test/test_offscreencanvas_neuter.html toolkit/components/places/tests/browser/browser_redirect.js toolkit/components/places/tests/browser/browser_settitle.js toolkit/components/places/tests/browser/browser_visituri.js widget/tests/test_bug343416.xul

(In reply to Kashav Madan [:kashav] from comment #13) > If I'm not mistaken, the Android TV didn't run a few files: Thanks for checking, but I'm pretty sure these are all expected... > No mention of the following: > > devtools/client/debugger/new/test/mochitest/browser_dbg-chrome-debugging.js devtools doesn't run on Android. > dom/canvas/test/test_2d.composite.canvas.xor.html disabled > dom/canvas/test/test_offscreencanvas_basic_webgl.html > dom/canvas/test/test_offscreencanvas_many.html > dom/canvas/test/test_offscreencanvas_neuter.html Explicitly skipped everywhere: skip-if = 1 > toolkit/components/places/tests/browser/browser_redirect.js > toolkit/components/places/tests/browser/browser_settitle.js > toolkit/components/places/tests/browser/browser_visituri.js browser-chrome doesn't run on Android. > widget/tests/test_bug343416.xul Explicitly skipped on Android: skip-if = (os == 'android') || (debug)

(In reply to Geoff Brown [:gbrown] from comment #14) > Thanks for checking, but I'm pretty sure these are all expected... Oops, you're right, thanks!