I have a prototype based on TestResolver.

$ ./mach test-report netwerk Test manifest: /home/gbrown/src/netwerk/test/reftest/reftest.list 4 tests, 0 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms Test manifest: /home/gbrown/src/netwerk/test/browser/browser.ini 6 tests, 2 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms Test manifest: /home/gbrown/src/netwerk/test/unit/xpcshell.ini 308 tests, 19 skipped on one or more platforms, 2 skipped on all platforms, 3 failed on one or more platforms Test manifest: /home/gbrown/src/netwerk/test/mochitests/mochitest.ini 16 tests, 0 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms Test manifest: /home/gbrown/src/netwerk/test/httpserver/test/xpcshell.ini 29 tests, 1 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms Test manifest: /home/gbrown/src/netwerk/test/crashtests/crashtests.list 4 tests, 0 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms Test manifest: /home/gbrown/src/netwerk/cookie/test/unit/xpcshell.ini 7 tests, 0 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms Test manifest: /home/gbrown/src/netwerk/test/unit_ipc/xpcshell.ini 43 tests, 43 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms Test manifest: /home/gbrown/src/netwerk/cookie/test/browser/browser.ini 1 tests, 0 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms Total: 418 tests, 65 skipped on one or more platforms, 2 skipped on all platforms, 3 failed on one or more platforms Known limitations: - doesn't understand reftest annotations - doesn't count reftests "properly" Also, it would be nice to generate a report for a bugzilla component (list of components for an owner); I don't know how to do that.

ideally I would like to have a dashboard that does this- in fact davehunt has a dashboard that queries activedata and generates some related information: http://fxtestr.us-west-2.elasticbeanstalk.com/unittest?branch=autoland&path=netwerk&since=today-1week It might have different limitations based on the data/availability. Overall this would be useful- especially if you could query further: ./mach test-report netwerk --verbose <lists the tests and related skipped configs> ./mach test-report netwerk --verbose --suite=xpcshell <only lists xpcshell related tests> the 4 different data points here are interesting: Total: 418 tests, 65 skipped on one or more platforms, 2 skipped on all platforms, 3 failed on one or more platforms I am curious what the failed status is? is that a different item in a manifest?

(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #2) Thanks for the feedback -- much appreciated. > I am curious what the failed status is? is that a different item in a > manifest? This is the difference between skip-if and fail-if. With proper support for reftests, it would get more complicated: fuzzy-if, random-if, etc...not sure how important all these distinctions are.

This version support --component: $ ./mach test-report --component Core::Layout Test manifest: /home/gbrown/src/layout/reftests/backgrounds/reftest.list 216 tests, 0 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms Test manifest: /home/gbrown/src/layout/reftests/css-blending/reftest.list 120 tests, 0 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms Test manifest: /home/gbrown/src/layout/reftests/border-dotted/reftest.list 12 tests, 0 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms ... The implementation is naive / brute-force-ish and performance is not great, but not too bad either: a few seconds for component lookup on the top src dir, in my experience.

this seems useful to maybe include in |./mach test-info| ? I would like to get these stats easier, and applying a patch to get them is not ideal :)

Now provided as a subcommand of test-info: mach test-info report [--component product::component] [--verbose] I'm not comfortable checking this in until I add proper reftest manifest support. Sorry.

Pushed by gbrown@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/d76bbec5d855 Add mach subcommand for per-component test-info report; r=jmaher