:jmaher has run TV-bf on several tests and found some of them failed to run the requested test. Let's investigate. https://bugzilla.mozilla.org/buglist.cgi?list_id=14280605&resolution=---&status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=TV-bf%20notfound

For bug 1475601, the reported failure is TEST-UNEXPECTED-TIMEOUT | /WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.html?1001-2000 | Derived key of type name: (anything) hash: (anything) length: (anything) using (anything), with (anything) - Test timed out TV-bf with testPath: testing/web-platform/tests/WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.html fails: https://treeherder.mozilla.org/logviewer.html#?job_id=193790258&repo=try&lineNumber=515 Per-test run could not find requested web-platform test 'testing/web-platform/tests/WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.html' There is no such file as 'testing/web-platform/tests/WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.html' in the tree, so this isn't an issue for regular TVw. I think the html is generated from testing/web-platform/tests/WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.js. It is possible to run TV-bf on testing/web-platform/tests/WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.js (although it times out!): https://treeherder.mozilla.org/logviewer.html#?job_id=193790163&repo=try The js and html files are related in MANIFEST.json...I wonder if that could be used? "WebCryptoAPI/derive_bits_keys/hkdf.https.any.js": [ [ "/WebCryptoAPI/derive_bits_keys/hkdf.https.any.html?1-1000", {} ], [ "/WebCryptoAPI/derive_bits_keys/hkdf.https.any.html?1001-2000", {} ], [ "/WebCryptoAPI/derive_bits_keys/hkdf.https.any.html?2001-3000", {} ], [ "/WebCryptoAPI/derive_bits_keys/hkdf.https.any.html?3001-last", {} ], [ "/WebCryptoAPI/derive_bits_keys/hkdf.https.any.worker.html?1-1000", {} ], [ "/WebCryptoAPI/derive_bits_keys/hkdf.https.any.worker.html?1001-2000", {} ],

interesting idea for using manifest.json; can we access that from the decision task?

We use it in mozharness currently at: https://dxr.mozilla.org/mozilla-central/rev/4e56a2f51ad739ca52046723448f3129a58f1666/testing/mozharness/mozharness/mozilla/testing/per_test_base.py#149 but we are missing out on this relationship. I think there would be some difficulties in accessing this from the decision task: want to keep the decision task quite lightweight, etc.

Several TV-bf:notfound bugs are for desktop dom/canvas tests; those have likely been addressed by bug 1476318. I verified these can now be run in TV-bf: bug 1474113 dom/canvas/test/webgl-conf/generated/test_2_conformance__ogles__GL__swizzlers__swizzlers_089_to_096.html https://treeherder.mozilla.org/logviewer.html#?job_id=193938141&repo=try bug 1474415 dom/canvas/test/webgl-conf/generated/test_2_conformance__ogles__GL__swizzlers__swizzlers_105_to_112.html https://treeherder.mozilla.org/logviewer.html#?job_id=193950722&repo=try bug 1475296 dom/canvas/test/webgl-conf/generated/test_2_conformance__ogles__GL__abs__abs_001_to_006.html https://treeherder.mozilla.org/logviewer.html#?job_id=193950806&repo=try bug 1475812 dom/canvas/test/webgl-conf/generated/test_conformance__glsl__bugs__conditional-discard-in-loop.html https://treeherder.mozilla.org/logviewer.html#?job_id=193951206&repo=try bug 1481750 dom/canvas/test/webgl-conf/generated/test_2_conformance__ogles__GL__exp2__exp2_009_to_012.html https://treeherder.mozilla.org/logviewer.html#?job_id=193950895&repo=try :jmaher -- What, if anything, should I do about the "TV-bf notfound" whiteboard entries for these bugs? TV-bf passes on all of them now, against a current m-c revision.

I suggest [tv-bf nowfound], then we can convert those, or if we want, just remove the [tv-bf notfound] whiteboard tag entirely.

OK - will update such bugs with "nowfound".

Similarly, this Android test is okay now, perhaps fixed in bug 1474393: bug 1474230 mobile/android/tests/browser/chrome/test_media_playback.html https://treeherder.mozilla.org/logviewer.html#?job_id=193961149&repo=try

Android jsreftest and mochitest-media were not being run -- addressed in bug 1483421.

Consider dom/canvas/crashtests/896047-1.html on win7/debug, from bug 1474751. Here we see that it can be verified in TV, but not TVg: https://treeherder.mozilla.org/#/jobs?repo=try&revision=83a38a93900d222607434edc30ece4b89809ff21 but backfill schedules only TVg-bf, where it does not run: https://treeherder.mozilla.org/#/jobs?repo=try&revision=b0b1a746c8149afe5867a1152f1bfea36ba54897 I suppose that's because of: https://dxr.mozilla.org/mozilla-central/rev/4e56a2f51ad739ca52046723448f3129a58f1666/taskcluster/taskgraph/actions/backfill.py#125-131 :jmaher - Would you happen to know what to do here?

This is interesting and showing cracks in our loose file matching we do to determine gpu or regular test type. We match files for test-verify proper here: https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/util/perfile.py#33 that looks like this: elif type.startswith('test-verify-gpu') or type.startswith('test-coverage-gpu'): file_patterns = ['**/*webgl*/**/test_*', '**/dom/canvas/**/test_*', '**/gfx/tests/**/test_*', '**/devtools/canvasdebugger/**/browser_*', '**/reftest*/**'] elif type.startswith('test-verify') or type.startswith('test-coverage'): file_patterns = ['**/test_*', '**/browser_*', '**/crashtest*/**', 'js/src/tests/test/**', 'js/src/tests/non262/**', 'js/src/tests/test262/**'] which is different than what we use for backfill: if (not is_wpttest) and \ ('gpu' in task.task['metadata']['name'] or 'webgl' in task.task['metadata']['name'] or 'canvas' in tp or 'gfx/tests' in tp or ('reftest' in tp and 'jsreftest' not in tp)): gpu_required = True I think we should make the file patterns the same and do a slightly more expensive matching. We use mozpackmatch (from mozpack.path) in perfile.py, I assume we could do the same in backfill.py since both have code in taskcluster/taskgraph. Ideally we could find a way to refactor the code to do a single call to determine if it is a test and what type of job to run in. We might want to wait until more edge cases are resolved for that, or take a stab at it now.

there are only 3 bugs left in the query of tests that were not found: https://bugzilla.mozilla.org/buglist.cgi?resolution=---&resolution=FIXED&resolution=INVALID&resolution=WONTFIX&resolution=DUPLICATE&resolution=WORKSFORME&status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=%5BTV-bf%20notfound&list_id=14366405 one is a crashtest, and the other 2 are the same test with different urlparams for a wpt test. :gbrown, are these ok to run, or do you think that these are not worth the time to fix them and we can resolve this bug:)

I think it is important to review the INCOMPLETE bugs as well, which are not insignificant https://bugzilla.mozilla.org/buglist.cgi?list_id=14366810&status_whiteboard_type=allwordssubstr&status_whiteboard=%5BTV-bf%20notfound&resolution=---&resolution=FIXED&resolution=INVALID&resolution=WONTFIX&resolution=INACTIVE&resolution=DUPLICATE&resolution=WORKSFORME&resolution=INCOMPLETE&query_format=advanced So, still working away at this as time allows.

I found several wpt bugs had been marked notfound but without any obvious cause. I suspect the entered testPath may have been incorrect. It usually needs to be "testing/web-platform/tests" + the path reported in the bug, trimmed of any ?... parameters. Perhaps we can make the backfill action smarter to avoid this sort of problem. Until then, I verify the path with 'mach test-info --show-info <file-name>'.

Most of the remaining known problems are in toolkit/components/extensions/test. I wonder if this issue is related to the TestResolver (or manifestparser?) problem seen in bug 1498636.