Now that running mochitests plain in headless mode is supported by gecko we could run many of the tests in parallel (locally and in automation). I'm not very familiar with the mochitests harness, so I'm hoping someone else can fill in what needs to be done here on the mochitests side of things.

this looks like a great idea, if we are running in headless, are there options for disabling tests on non-headless? maybe on certain configs?

(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #1) > this looks like a great idea, if we are running in headless, are there > options for disabling tests on non-headless? maybe on certain configs? There's the 'headless' variable supported in the test .ini files. There are definitely some tests that we want to run in both modes and I'm not quite sure the best way to decide what tests run in each mode. I've got a few ideas, but it's lunchtime, so I'll update the bug in a bit here.

I'm thinking we first need to do some experiments to see how big improvement this would be on test run time. Then we can try to find a balance of still getting adequate code coverage and test run time. If it helps a lot, we may want to choose a more aggressive approach below. The possible ways we could migrate: * Switch all tests to headless - would lose code coverage of native widget code * Flag folders or suites of tests to run headless - decent amount of work to sort through test folders * Flag individual tests to run at headless - tons of work to sort through tests * Programmatically determine if tests is headless - i.e. does the test trigger potentially problematic headless code. Not sure this is really feasible as most tests trigger some portion headless code, but could looked into further * New tests default to headless, flag to run non-headless - no immediate improvement With all the above approaches we could run all tests in normal mode at certain frequencies (e.g. nightly) to still get full test coverage.

I definitely like the non-headless by default, we could annotate all existing manifests with: [DEFAULT] headless = false and push on module owners to do the triage of their own tests. We'd also want a command line flag that overrides whatever is defined in the manifest. For tests that should run in both, we could create a second manifest, e.g: mochitest.ini [test_foo.js] [test_bar.js] mochitest-headless.ini [include:mochitest.ini] headless = true (or something like that)

*meant to say "headless by default"

I think we should be working towards headless by default (assuming it works in the majority of cases), as headless is friendlier to development experience and faster to execute. If we need to run entire suites in "traditional" mode, we can certainly do that. We can also have CI run tests in both flavors as needed. Obviously this would be a big change. So annotating all manifests with "headless = false" so the change of default behavior is essentially a no-op sounds prudent. We could follow up with a bulk removal of this annotation to switch things to headless. i.e. treat the "policy" change and enacting that change as separate steps.

(In reply to Brendan Dahl [:bdahl] from comment #3) > * Programmatically determine if tests is headless - i.e. does the test > trigger potentially problematic headless code. Not sure this is really > feasible as most tests trigger some portion headless code, but could looked > into further What do you consider "potentially problematic" here? The headless code provides a separate widget backend, so the biggest concern would seem to be tests that are explicitly testing widget features, right? I probably can't enumerate the set of things that covers off the top of my head, but certainly anything doing reftest-style checks of rendering would be an issue, and anything attempting to test OS integration might be an issue (depending on the specific feature, not everything goes through widget). Judging by the contents of widget/headless: https://dxr.mozilla.org/mozilla-central/source/widget/headless It seems like: * clipboard * key bindings * native theming * screen info * nsISound Would be things to avoid running in headless mode.

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #7) > What do you consider "potentially problematic" here? The headless code > provides a separate widget backend, so the biggest concern would seem to be > tests that are explicitly testing widget features, right? Yes, anything exercising emulated behavior should be avoided (or tested in both modes). Your list looks good as far as the general things. For reftets, most would probably be fine to run headlessly unless they are trying to specifically test a platforms gfx backend.

for windows we run reftests on gpu instances specifically due to Direct2D, will that work fine in headless?