test-verify-wpt's errorsummary.log don't contain the group names that run in the task
Categories
(Testing :: web-platform-tests, defect, P3)
Tracking
(Not tracked)
People
(Reporter: marco, Unassigned)
References
(Blocks 1 open bug)
Details
Instead, they all contain "/".
E.g. in https://firefox-ci-tc.services.mozilla.com/tasks/eMMu2QpPSwmLDTvlMBAdNg, we have multiple errorsummary.log files (https://firefoxci.taskcluster-artifacts.net/eMMu2QpPSwmLDTvlMBAdNg/0/public/test_info//wpt-test1_errorsummary.log, https://firefoxci.taskcluster-artifacts.net/eMMu2QpPSwmLDTvlMBAdNg/0/public/test_info//wpt-test2_errorsummary.log, https://firefoxci.taskcluster-artifacts.net/eMMu2QpPSwmLDTvlMBAdNg/0/public/test_info//wpt-test3_errorsummary.log, https://firefoxci.taskcluster-artifacts.net/eMMu2QpPSwmLDTvlMBAdNg/0/public/test_info//wpt-test4_errorsummary.log, https://firefoxci.taskcluster-artifacts.net/eMMu2QpPSwmLDTvlMBAdNg/0/public/test_info//wpt-test5_errorsummary.log), which look like:
{"groups": ["/"], "action": "test_groups", "line": 15}
{"group": "/", "status": "OK", "duration": 4184, "action": "group_result", "line": 79}
Comment 1•2 years ago
|
||
James, can you please provide some info for this one, thank you!
Comment 2•2 years ago
|
||
Is this still a problem?
Generally wptrunner doesn't know what the right group name for a test is; that logic got moved to the task selection. It could guess based on the directory name, but that might get out of sync with any changes in the test selection logic. So maybe the right solution here is for mozharness to pass in the correct group name for each test in the verify run via a groups JSON file, like we do for full test runs.
Reporter | ||
Comment 3•2 years ago
|
||
Yes it seems to still be happening, e.g. https://firefox-ci-tc.services.mozilla.com/tasks/OnW8lgw2R0yDREyG6ggz0w/runs/0/logs/public/test_info/wpt-test1_errorsummary.log and https://firefox-ci-tc.services.mozilla.com/tasks/ExobDrDeQw22DOh5Nzq2ZA/runs/0/logs/public/test_info/wpt-test11_errorsummary.log.
The first has lines such as {"groups": ["reftest"], "action": "test_groups", "line": 33}
, the second has lines such as {"groups": ["testharness"], "action": "test_groups", "line": 33}
.
Description
•