Open
Bug 1488850
Opened 6 years ago
Updated 2 years ago
Test verification TV and TVw fail because of unknown test file paths
Categories
(Testing :: Code Coverage, enhancement)
Tracking
(Not tracked)
NEW
People
(Reporter: gabriel-v, Unassigned)
References
(Blocks 1 open bug)
Details
This happens when 'mach try coverage' adds a path for a test that is not registered with TV or TVw. This happens, for example, with paths from the python test suite. This causes instant task failure for TV and TVw.
Should we avoid scheduling TV and TVw tasks when running mach try coverage? Or should we stop failing the task when we find a unknown test file?
Failing try push made with 'mach try coverage': https://treeherder.mozilla.org/#/jobs?repo=try&revision=3429ad812ce4267273bfe7934b4c1737be947cc6&selectedJob=196918179
Error message:
Per-test run could not find requested test 'tools/tryselect/test/test_tasks.py'
Reporter | ||
Comment 1•6 years ago
|
||
Here's another example: I added enough paths to 'mach try fuzzy' in such a way that TV doesn't recognize a WPT, and TVw doesn't recognize a reftest, so they both fail.
This try push is based off of central directly, made with 'mach try fuzzy': https://treeherder.mozilla.org/#/jobs?repo=try&revision=4556485e5ad31c0042e24436d2a5e76f6362a1db
Summary: Test verification tasks fail when run by 'mach try coverage' because of unknown test file paths → Test verification TV and TVw fail because of unknown test file paths
Reporter | ||
Comment 2•6 years ago
|
||
This is introduced by bug 1482413, https://hg.mozilla.org/mozilla-central/rev/71af60e17c8b
Geoff: Can we turn those 'self.fatal(...)' lines into warnings? Or is there a smarter way of fixing this?
Flags: needinfo?(gbrown)
Updated•6 years ago
|
Flags: needinfo?(gbrown)
Comment 3•6 years ago
|
||
In bug 1482413, we specifically wanted to fail when test-verify is requested for an invalid path. A particular use case we were thinking of was running backfill (TV-bf), manually entering a path, and making a typo. If we s/fatal/warning/, those typos will produce green runs -- which do nothing.
Comment 4•6 years ago
|
||
Maybe we could add a flag to indicate a 'mach try coverage' run and behave differently in that case? Or is there a way for 'mach try coverage' to avoid requesting TV for unsupported test types?
Comment 5•6 years ago
|
||
(In reply to Geoff Brown [:gbrown] from comment #4)
> Maybe we could add a flag to indicate a 'mach try coverage' run and behave
> differently in that case? Or is there a way for 'mach try coverage' to avoid
> requesting TV for unsupported test types?
Is there a way to tell which test types are not supported by TV or TVwpt?
Flags: needinfo?(gbrown)
Comment 6•6 years ago
|
||
I think there's no file or api providing that information.
Currently TV/TVg support mochitests (including all flavors and subsuites), reftests (including crashtests and jsreftests), and xpcshell-tests. TVw supports all web-platform tests.
This bit of code defines the relationship:
https://dxr.mozilla.org/mozilla-central/rev/d6ac0dc85306b753c1d1d5f3084f73f0d1286b86/testing/mozharness/mozharness/mozilla/testing/per_test_base.py#254-259
I read that as:
- for TVw, files of interest are found in the wpt manifest
- for TVg/TV, files of interest are found in the manifest listed at
https://dxr.mozilla.org/mozilla-central/rev/d6ac0dc85306b753c1d1d5f3084f73f0d1286b86/testing/mozharness/mozharness/mozilla/testing/per_test_base.py#45-50
- files in 'gpu' or 'webgl' suites should run in TVg, else TV
- everything else is unsupported
Flags: needinfo?(gbrown)
Updated•2 years ago
|
Severity: normal → S3
You need to log in
before you can comment on or make changes to this bug.
Description
•