Structured logs are supposed to generate a JSON line like (added formatting for clarity):

{"tests": {
    groupName1: [test11, test12, ...],
    groupName2: [test21, test22, ...],
    ....
}}

This does not always happen. For example, in

https://firefoxci.taskcluster-artifacts.net/FKJFB_HET0q1gTy-Pzow_w/0/public/test_info//wpt_raw.log

the following test can not be found in any group

/content-security-policy/media-src/media-src-7_1.html

ActiveData can provide fresh instances of the same problem:

https://activedata.allizom.org/tools/query.html#query_id=47cp3IPY

Requesting clarification of the problem scope.

Example pulled from the activedata query above:
test: /2dcontext/compositing/2d.composite.canvas.copy.html
task: https://firefox-ci-tc.services.mozilla.com/tasks/PcVMnSSwR_au9vJIjzP3pQ
treeherder: https://treeherder.mozilla.org/#/jobs?repo=try&revision=68e8ca06e5c0d174e5744aaf588d730459552102&searchStr=test-android-em-7.0-x86_64%2Fdebug-geckoview-web-platform-tests-backlog-e10s-4
raw log: https://firefoxci.taskcluster-artifacts.net/PcVMnSSwR_au9vJIjzP3pQ/0/public/test_info//wpt_raw.log
live log: https://firefoxci.taskcluster-artifacts.net/PcVMnSSwR_au9vJIjzP3pQ/0/public/logs/live_backing.log

This is what I see in the raw logs:

{"source": "web-platform-tests", "test": "/2dcontext/compositing/2d.composite.canvas.copy.html", "thread": "MainThread", "time": 1587383012642, "action": "test_start", "pid": 1113}
{"status": "SKIP", "source": "web-platform-tests", "test": "/2dcontext/compositing/2d.composite.canvas.copy.html", "thread": "MainThread", "time": 1587383012642, "action": "test_end", "pid": 1113}

I am not certain what the I am expected to see vs what I actually see recorded in the raw logs.

Addendum to the above message: the portion that handles the construction of the log message is https://searchfox.org/mozilla-central/source/testing/mozbase/mozlog/mozlog/structuredlog.py#281-290 which is a component of mozlog, not web-platform-tests.

But :ahal changed the component back to web-platform-tests, so the assumption held here is that this issue is due to how web-platform-tests is sending the components to be assembled into a log message, hence it is an issue with the web-platform-test harness.

Is this assumption correct?

Yes, I believe this is an issue in WPT.

Though if this is only happening to SKIP tests, then it likely isn't a huge deal. Possibly even WONTFIX. We should verify if there are any instances of non-skip tests where there is no group.

Note the example test from comment 0 is also skipped:
https://searchfox.org/mozilla-central/source/testing/web-platform/meta/content-security-policy/media-src/media-src-7_1.html.ini#2

(In reply to Andrew Halberstadt [:ahal] from comment #5)

Note the example test from comment 0 is also skipped:
https://searchfox.org/mozilla-central/source/testing/web-platform/meta/content-security-policy/media-src/media-src-7_1.html.ini#2

My observations so far confirm what you've seen - the query only returns tests that appear to have been skipped on web-platform-tests.

:marco I assume this s OK: After removing the skips, and a variety of crashes, all tests have a group.

https://activedata.allizom.org/tools/query.html#query_id=bTD8OPxQ

I think the bug is still valid (ideally skipped tests still have a group), I just assume it's not super important for the smart scheduling project.

There's only one problem that we need to handle: currently mozci would consider a skipped group as a passing group.

The severity field is not set for this bug.
:jgraham, could you have a look please?

For more information, please visit auto_nag documentation.

(In reply to Marco Castelluccio [:marco] from comment #9)

There's only one problem that we need to handle: currently mozci would consider a skipped group as a passing group.

I thought this meant the skipped groups would be considered as passing in ActiveData, but it isn't actually the case, so this bug is not needed at all for smart scheduling (bug 1632242 is, but not for WPT since WPT currently doesn't output skipped groups in the "groups" section of errorsummary.log).

:jgraham on Matrix [edited to use permalinks]:

https://searchfox.org/mozilla-central/rev/61fceb7c0729773f544a9656f474e36cd636e5ea/testing/web-platform/tests/tools/wptrunner/wptrunner/wptrunner.py#242 we pass in tests and at https://searchfox.org/mozilla-central/rev/61fceb7c0729773f544a9656f474e36cd636e5ea/testing/web-platform/tests/tools/wptrunner/wptrunner/wptrunner.py#276 we use disabled_tests
If you want to change things, just make disabled_tests = test_loader.disabled_tests[test_type] as a variable, make the first call suite_start(tests + disabled_tests) and change the second to use the new variable
Or something like that