Joel Maher ( :jmaher ) (UTC -8) Reporter
	Description • 10 years ago

We have so many disabled tests on various platforms and configurations.  Right now, we just dream of green runs when we land a patch.  This means the focus is not on complete test coverage, instead it is on running what looks good.

Knowing that, it is important for us to track the tests we have and what their expected state is.

While I don't expect this to be a simple this, this shouldn't be a 6 month project.

I envision something like this:
hg clone m-c
build
for config in ['linux', 'linux&debug', 'e10s', 'linux&e10s', 'b2g', ...]:
  tests = parseManifest(mozinfo=config)
  sqlinsert(config, tests.total, tests.active)


The exact coding for this could be much more complicated or at least more complete.

We could have a json api to query the stats, likewise a simple graph/webui.

Please comment in here to outline what details we need to account for.

	Jonathan Griffin (:jgriffin)
	Comment 1 • 10 years ago

Adding Armen, since he may help out with the implementation after this is scoped well enough.

	Gregory Szorc [:gps]
	Comment 2 • 10 years ago

The skipped tests are something I'd like to capture as part of the metadata emitted by the build system (currently all-tests.json). I would eventually like to see --run-skipped (or similar) options on the mach testing commands to allow people to force run certain tests without having to update manifests.

We kinda/sorta get this for free if skipped tests aren't ignored by the build system. I think we only do that for xpcshell now, but I saw a bug to do that for mochitests too.

	u459114
	Comment 3 • 10 years ago

Please also check bug 1029348 comment 6

	Andrew Halberstadt [:ahal] Assignee
	Comment 5 • 10 years ago

I'm working on this here:
https://github.com/ahal/test-informant

	Andrew Halberstadt [:ahal] Assignee
	Comment 6 • 10 years ago

Here's an early version of what the report will look like:
http://people.mozilla.org/~ahalberstadt/report.html

Note that the data that populated it was from inbound, and the urls are hardcoded to mozilla-central.. but you get the idea. I'll also be adding more suites and platforms in the near future. For now, any feedback on what kind of data you'd like to see would be appreciated.

	Jonathan Griffin (:jgriffin)
	Comment 7 • 10 years ago

This looks great!  I think this is good for a rev 1.  Making the div wider for the tests might be nice, so that they don't wrap.

	Andrew Halberstadt [:ahal] Assignee
	Comment 8 • 10 years ago

Yeah, good point. I guess I've been testing on my high-res desktop monitor. I updated report at the same url to be wider, made it scale a bit better with smaller resolutions and added some new mochitest flavours. The percentages for browser-chrome and devtools are skewed, that's a known issue.

	Andrew Halberstadt [:ahal] Assignee
	Comment 9 • 10 years ago

Note: this is more or less done. I'll resolve it once the process of generating and posting reports is fully automated. This is blocked on some IT bugs.