When running all tests (ie: no APP= environment variable), it seems that the test-agent is running the tests before outputting anything. We can see that because we see various console.log (eg: "no icon suite" in the homescreen tests). This does not happen when running only tests for a specific app.

Etienne told me, and I verified, that "APP=communications" triggers the same behavior. Therefore, my new guess is that when the test-agent needs several sandboxes, we have this issue.

I've noticed that it's displayed in the same time in the html reporter in the test-agent webapp and on the console. So this could be a problem with the reporting to mocha from the sandbox.

new finding ! The latest sandbox results are displayed incrementally as expected, but only the previous ones are displayed in one block.

I added some logs in [1]. It seems we get the "end" events for all previous sandboxes only when the _last_ sandbox is starting. However this is probably the end of my quest, I don't exactly understand what is supposed to send the "end" event and why it doesn't work. Hopefully James will either fix this or help me to move forward. I think it is important to fix this because it will help finding the other problems. [1] https://github.com/mozilla-b2g/gaia/blob/master/test_apps/test-agent/common/vendor/test-agent/test-agent.js#L1908

I think this is probably all intentional behavior... we need to do some refactoring of how we handle files in test agent... What I would like to do is: - merge alot of the proxy work into mocha-json-proxy (used in marionette-js-runner) - run single files at a time in their own sandbox - write custom gaia reporter for mocha (or attempt to fix problems with a single existing mocha reporter with multiple test runner results At a high level this would let us deal with tests at a "per-file" level which has a ton of other benefits. This is my long-term vision depending on time constants and other details we might attempt a smaller fix first. Keeping the needinfo so I am reminded of this but hopefully this is enough context for the moment.

I put back the needinfo because you actually removed it ;)

This was not easy to spot, but the patch is a lot simpler than what I anticipated. With this change, we now see the tests as they're run, instead of waiting for the last one. I tried also running from the UI tests from different apps, this looks to work as expected ;) Here is the Gaia PR with these changes so that we see that everything still works: https://github.com/mozilla-b2g/gaia/pull/16344 Yuren, James, the first to review is the winner ;) This will be very useful to understand why the test-agent does not finish sometimes, next step is to turn on the Spec reporter instead of Min.

Comment on attachment 8376850 [details] github PR now I see output from console for |make test-agent-test| when running tests, great job Julien!!

master: 05fe37782169c114ca373394036960a712063590