Doing a quick polling of the Android 4.0 talos failure rates on m-c over the last week: tp4m_nochrome: 19.0% trobocheck2: 5.1% trobopan: 12.5% tspaint: 36.6% tsvgx: 9.3% Anecdotally, these numbers look reasonably sane based on experience. And as they show, the only suite that even comes remotely close to meeting standards is trobocheck2. Also, these don't include "blue" disconnect jobs, just outright fatal failures. The numbers would be much higher were they included. The common failures are noted in this bug's deps. In the mean time, Android 4.0 talos is being hidden on all trees until they can be made to meet standards.

Can we get a recount? Ouija shows quite different numbers, unless we are counting retries: http://54.215.155.53/android_failures.html?platform=android4.0, for 2014-11-03 - 2014-12-08, reports: Status tp4m trobocheck2 trobopan troboprovider tspaint tsvgx Green 188 184 183 186 157 177 Orange 0 0 0 0 0 0 Red 1 3 2 0 10 2 Blue 28 26 26 34 164 41 Are we counting retries? I notice that bug 913870 is a dep, but that affects all panda jobs and I thought all parties were ignoring those mozpool-ish failures. (All the same, I agree there is something bad happening to tspaint.)

This bug was filed in October. No, retries weren't being counted. I haven't been watching since, as doing so should fall on whoever owns the tests. If they're green enough to meet standards now, we can unhide. Again, *owners* should be taking responsibility for such things :)

Ryan, great- lets unhide these. I am not aware of the process of how to toggle jobs in the land of treeherder. lets resolve this today.

Everything but tspaint is unhidden. I'll leave it for you to decide if you want to leave this open until it's fixed or not.

It was fixed by being shut off in bug 1141656, so I removed the exclusion which hasn't been excluding anything for several weeks.