with the exception of windows, test_timeout.html fails pretty consistently when it comes to running a single directory. On windows test_success_without_recognition_service.html fails. The pattern, these are the last test in the directory to run. Here is a view on treeherder from a try push: https://treeherder.mozilla.org/#/jobs?repo=try&revision=72b7fdf2885a&filter-searchStr=mochitest-3 and I was able to reproduce this locally on a linux64 environment with a local build of inbound by running: ./mach mochitest-plain dom/media/webspeech/recognition/ ^ note: this reproduced about 50% of the time locally. Typically when we are running tests in a runbydir fashion there are 3 reasons for failure: 1) the test depends on a startup state which hasn't happened yet because we are running it much closer to the startup of the browser 2) the test depends on features from other tests that used to run before it 3) the test doesn't clean up and depends on the browser running for a longer duration to do the cleanup in this case, I am calling out #3 as the culprit. Now to find a developer that knows anything about this.

this is the commit history for test_timeout.html: https://hg.mozilla.org/mozilla-central/filelog/ba44099cbd07/dom/media/webspeech/recognition/test/test_timeout.html

what is odd is that windows xp debug doesn't crash at all- maybe that is the best OS ever?

:smaug, can you take a look at this? I would prefer not to disable the entire dom/media/webspeech set of tests to solve this problem. Quite possibly you know who could be a better point of contact.

./mach mochitest-plain dom/media/webspeech/recognition/ works fine here, 64bit Fedora 20. Run that several times. which crash are you talking about? I see some shutdown issue, coming from webspeech/synth (how from that directory?), which indeed could hint about 3). Andre, could you test this locally?

(In reply to Olli Pettay [:smaug] from comment #4) > which crash are you talking about? I see some shutdown issue, coming from > webspeech/synth I mean in your tryserver logs.

there are a few errors in the logs, mostly all the mochitest-3 failures are in recognition or synth (in the cases where the recognition stuff might not run all the tests due to platform restrictions in the manifest) I really don't understand all the errors, but I would be happy to help where possible. The crashes appear to happen when we are shutting down, what I saw locally is that we hung there and then the automation would kill it causing a crash.

(In reply to Olli Pettay [:smaug] from comment #4) > ./mach mochitest-plain dom/media/webspeech/recognition/ works fine here, > 64bit Fedora 20. > Run that several times. > > which crash are you talking about? I see some shutdown issue, coming from > webspeech/synth > (how from that directory?), which indeed could hint about 3). > > > Andre, could you test this locally? I had no problem in Ubuntu 14.04 64 bits. I also ran it several times and never had any issue.

I have been trying to find which test are problematic, for linux, it is [test_timeout.html]. here is a try server showing the run with a linux mochitest-3 crash: https://treeherder.mozilla.org/#/jobs?repo=try&revision=52e268ea47ae&filter-searchStr=linux windows is a different beast. In opt mode we get crashes. It appears to be the last test in the list: https://treeherder.mozilla.org/#/jobs?repo=try&revision=0da7845ceb4a&filter-searchStr=win https://treeherder.mozilla.org/#/jobs?repo=try&revision=d5ffffa8ae90&filter-searchStr=win https://treeherder.mozilla.org/#/jobs?repo=try&revision=27163f70b0d0&filter-searchStr=win you can see in each of these runs I keep getting a different crash- the thing in common is the crashing test is always the last test ran!

thanks for giving this a try locally. it seems there are 2 issues here (at least identified so far): * linux (at least 32 bit) crashes on test_timeout.html * windows* opt crashes on the last test to run. hmm, speaking of crashes- these might not be crashes of the browser, maybe forced crashes by the test harness due to failure to terminate the browser after completing the test. That would make a lot more sense. Is there something in the webspeech/recognition code that might take a while to unregister or cleanup? we could be locked on it when trying to shutdown right after using it.

kdavis, could you take a look at this?

:kaustabh, is this still a problem?

A push was made to try with these tests enabled, and we got fairly green results : https://treeherder.mozilla.org/#/jobs?repo=try&revision=351ba2066462