The fission team is currently triaging their tests in preparation for enabling fission on mozilla-central. They would like to be able to see a snapshot of all the tests that are disabled with fission across all the various platforms and configurations. This is basically what Test Informant used to do.

I think we could write an ActiveData recipe that grabs this information (the old test-informant used to use AD already). But Joel also mentioned that :gbrown might have something (or be working on something) that might help..

Geoff, could you explain what Joel was talking about? Do you know if it would help for this scenario?

Yes, we are specifically looking to query for the skip-if(s) and fail-if(s) with fission, split by bug component, and collected in a json format.

I have been reviving the work in bug 1404472, in hopes of eventually providing a meaningful summary of the tests associated with a group of bugzilla components; I think it might be beneficial if a bz triage owner could track data like "how many tests are in my components?", "how many of those tests are skipped?" etc. See bug 1519171 for a better idea of my motivations.

The work-in-progress patch on bug 1404472 generates a report of the test manifests associated with each bugzilla component and includes a count of tests found in each manifest and the number of "skip-if" annotations in ini manifests (does not handle reftests). The report looks like:

{
  "Testing::BrowserTest": [
    {
      "manifest": "testing/modules/tests/browser/browser.ini", 
      "skipped": 0, 
      "tests": 1
    }
  ], 
  "Testing::Firefox UI Tests": [
    {
      "manifest": "testing/firefox-ui/tests/functional/keyboard_shortcuts/manifest.ini", 
      "skipped": 0, 
      "tests": 1
    }, 
    {
      "manifest": "testing/firefox-ui/tests/functional/private_browsing/manifest.ini", 
      "skipped": 0, 
      "tests": 1
    }, 
    {
      "manifest": "testing/firefox-ui/tests/functional/safebrowsing/manifest.ini", 
      "skipped": 1, 
      "tests": 3
    }, 
 ...

I think it could be easily modified to report annotations containing "fission" in ini files, but I don't have a solution for reftests.

(In reply to Geoff Brown [:gbrown] from comment #3)

The work-in-progress patch on bug 1404472 generates a report of the test manifests associated with each bugzilla component and includes a count of tests found in each manifest and the number of "skip-if" annotations in ini manifests (does not handle reftests). The report looks like:
...
I think it could be easily modified to report annotations containing "fission" in ini files

That sounds like exactly what we want (well, except that we also need to take fail-ifs into account, but I don't think that will make things much harder).

but I don't have a solution for reftests.

We can leave reftests as a separate issue for now. The current focus is on mochitests only.

Usage: mach test-info report [<src path(s)>] [--components <bz component>,<bz component>,...]

How's this?

Nice! I love when requests are already solved ;). Thanks Geoff.

Would it be reasonable to block on bug 1404472 and then use this bug to add the ability to filter on specific configurations in a more robust manner (e.g to avoid picking up strings like !fission)?

Sure!

Hi Geoff,
When should we expect bug 1404472 to land so it can be used to build up and add filter on top for tracking fission skipped tests?

I expect to request review tomorrow and land as soon as possible.

One problem is that --filter-values=fission will pick up !fission.

So, e.g:

[test_foo.js]
skip-if = !fission

would get counted. Probably not a huge deal for now as I doubt many tests are annotated with !fission yet. Simple solution would be to allow this value to be a regex.

Filtering on the regex '(?<!!)fission' would be ideal-ish, yes.

I'm landing changes in bug 1572794 to support that sort of regex.

Then we can generate a fission test report with something like

mach test-info report --show-tests --show-summary --filter-keys=fail-if,skip-if --filter-values='(?<!!)fission'

But talking to Neha yesterday, she said 'it requires downloading the tree with every push and running that query. It would be really helpful if we had this info as part of the build metadata'. :ahal -- Do you have an idea of how to implement that? Might it be appropriate to add that on to a task like source-test-file-metadata-bugzilla-components?

Hi Neha, please define "build metadata" (I think it might be the wrong term here). I mentioned in an IRC chat that we could set up a task that runs this command and uploads the output as an artifact. Then your scripts could download these artifacts as needed. Is that what you're looking for?

Yes, an artifact that we could download from each m-c build will help so we don't have to pull the tree and run the query for each build.

Depends on D43513

Depends on D43514

https://hg.mozilla.org/mozilla-central/rev/a2339aac248c
https://hg.mozilla.org/mozilla-central/rev/7d7cdf191fe7
https://hg.mozilla.org/mozilla-central/rev/8f93f148e4fa

Andrew, I'm trying to query for these artifacts with the API in order to gather them together for a graph and am not having much luck. I'm guessing the artifact path I'm using is wrong, but based on the url I see for the artifact at https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=cae93ef1993e02a136ef64d974856071b905997f&selectedJob=264337756 I'm not sure what else it should be. Here is the query I'm trying https://activedata.allizom.org/tools/query.html#query_id=jA_4qkgn:

{
	"sort":{"repo.push.date":"desc"},
	"limit":10,
	"from":"task.task.artifacts",
	"where":{"eq":{"name":"public/test-info-fission.json"}},
	"select":["task.artifacts.url","repo"],
	"format":"list"
}

Do you have any ideas how to get this list, or know who else I could ask?

I created an index for this task, so use the taskcluster index instead. To get the latest:
https://index.taskcluster.net/v1/task/gecko.v2.mozilla-central.latest.source.test-info-fission/artifacts/public/test-info-fission.json

Or by revision:
https://index.taskcluster.net/v1/task/gecko.v2.mozilla-central.revision.cae93ef1993e02a136ef64d974856071b905997f.source.test-info-fission/artifacts/public/test-info-fission.json

Take a look at the index browser for other ways to find it, like by date.

https://hg.mozilla.org/releases/mozilla-beta/rev/a2339aac248c
https://hg.mozilla.org/releases/mozilla-beta/rev/7d7cdf191fe7
https://hg.mozilla.org/releases/mozilla-beta/rev/8f93f148e4fa

(In reply to Andrew Halberstadt [:ahal] from comment #24)

I created an index for this task, so use the taskcluster index instead. To get the latest:
https://index.taskcluster.net/v1/task/gecko.v2.mozilla-central.latest.source.test-info-fission/artifacts/public/test-info-fission.json

Or by revision:
https://index.taskcluster.net/v1/task/gecko.v2.mozilla-central.revision.cae93ef1993e02a136ef64d974856071b905997f.source.test-info-fission/artifacts/public/test-info-fission.json

Take a look at the index browser for other ways to find it, like by date.

For context, Andrew and I discussed how to collect these artifacts for a graph and decided that we'd like a new index that returns the latest artifact on a given day in order to avoid having to querying active data or scraping the tc index site to get the full timestamp of a push. He made a patch to do this in Bug 1578510.

Neha: is there a desire to somehow backfill these jobs so that we can show data from july/august in a dashboard, or should it start only once this data started being collected (Sept 2)? I'm not sure if it'd be possible to do through taskcluster or if we'd want to archive the existing data from the spreadsheet as JSON, but if it's something you want to show we could check out the options.

When we build the graphs, the July/August data should be visible too. As far as I understood, it wasn't possible to query such past data, which is why we were keeping the spreadsheet. If it is possible to backfill this data via taskcluster, then that's great. Otherwise, we can use kmag's spreadsheet data to do a one-time fill for the past numbers.

(In reply to Neha Kochar [:neha] from comment #28)

When we build the graphs, the July/August data should be visible too. As far as I understood, it wasn't possible to query such past data, which is why we were keeping the spreadsheet. If it is possible to backfill this data via taskcluster, then that's great. Otherwise, we can use kmag's spreadsheet data to do a one-time fill for the past numbers.

Kris, is it possible to get a copy of your daily spreadsheet data, but as JSON matching the structure of test-info-fission.json? We can hardcode those daily measurements up until around Sept 2 and then start gathering from taskcluster after that.

(In reply to Brian Grinstead [:bgrins] from comment #29)

(In reply to Neha Kochar [:neha] from comment #28)

When we build the graphs, the July/August data should be visible too. As far as I understood, it wasn't possible to query such past data, which is why we were keeping the spreadsheet. If it is possible to backfill this data via taskcluster, then that's great. Otherwise, we can use kmag's spreadsheet data to do a one-time fill for the past numbers.

Kris, is it possible to get a copy of your daily spreadsheet data, but as JSON matching the structure of test-info-fission.json? We can hardcode those daily measurements up until around Sept 2 and then start gathering from taskcluster after that.

Nevermind, I ended up figuring out how to export individual csvs and then manually converted into the right json format.