Now that we have a number of the crashes fixed, it is time to revisit the Unit Tests. I ran two sets of tests: 1. Using the default try device assignments in production across the 3 servers autophone-{1,2,3}: <https://treeherder.mozilla.org/#/jobs?repo=try&revision=4d48cb6b54df52b6d083921a58e61aacdc56fc66&filter-tier=1&filter-tier=2&filter-tier=3&group_state=expanded> 2. Using autophone-4 and the nexus 5 and pixel devices attached there: <https://treeherder.allizom.org/#/jobs?repo=mozilla-central&revision=e6b3498a39b94616ba36798fe0b71a3090b1b14c&filter-searchStr=autophone&group_state=expanded> #2 is much worse that #1. It may be due to the fact that #2 has all of the tests running on a single host while #1 spreads them out across 3 different hosts. It could also be a function of the different devices being used. Apart from time outs, crashes are pretty common. We already have a few bugs filed which I'll block this bug. 17 application crashed [@ libc.so + 0x2173c 16 application crashed [@ libc.so + 0x48484 1 application crashed [@ libc.so + 0x490a4 1 application crashed [@ libart.so + 0x313064 1 application crashed [@ boot.oat + 0x56d2cc Another issue appears to be related to a browser hang where we end up exceeding 330 seconds then have to try SIGABRT to kill the browser: org.mozilla.fennec_aurora still alive after SIGABRT: waiting... Then we have the Timed out while waiting for websocket/process bridge startup. But this appears to be common between the allizom and try runs with the same number of errors. Then we have the various test time outs. If bwu's and snorp's teams can fix the [@ libc.so + 0x2173c] and [@ libc.so + 0x48484] crashes that will help. I will: * investigate if multiple unit tests running on a host cause server start up issues and if I can improve the situation. * enable the devices which previously been used for production unit tests to run unit tests on try so that we can get results for the devices which will actually run in production. Suggestions welcome.

fyi, I handle autophone-4 separately by applying local patches. This patch will allow me to run our normal unittests on each of the idle devices. This will also increase the number of unittests running simultaneously on autophone-{1..3} and allow me to judge the interference between multiple instances of the test runners and local web servers started by the unit tests.

Seems autophone had stalled consuming pulse messages sometime after I submitted my try job. Now that I've applied the patch and restarted the servers, additional jobs for the new devices have been created.

gbrown is fixing an issue with killing orphaned xpcshell and ssltunnel processes. The try runs: https://treeherder.mozilla.org/#/jobs?repo=try&revision=6bc6ceb4adce778f89a5e8d03de202a79a4c8f86 https://treeherder.allizom.org/#/jobs?repo=try&revision=6bc6ceb4adce778f89a5e8d03de202a79a4c8f86 both show marked improvement. The remaining issues in the unit tests appear very solvable and we should make good progress once bug 1403501 lands.

This patch achieves full coverage by adding whatever devices are necessary. It also includes 10 spare devices of each type on top of the requirements for full coverage. This provides us the ability to recover as devices wear out and have to be recycled. If we don't wish to add this many devices we can decide on which tests are required for each device type and cut down on the coverage. device type #devices required nexus 4 20 nexus 5 10 nexus 6p 11 pixel 4 follow up patches will trim the manifests to match our current inventory until the new devices are available.

device/test/repo assignments.

This patch disables the unit tests on beta and release until we can confirm they are green there.

This patch disables the spares that are planned.

This patch disables the new devices until they have been acquired.

This patch disables Mdm1 until bug 1408241 is fixed or until we disable the failing test.

I forgot to call out that the initial patch also renames the devices so that we no longer have single digit names... nexus-5-1 is now nexus-5-01. This prevent some confusion in matching device names in scripts as well as displays the devices in a more logical sorted order. This does require that I update the mysql database for phonedash to change the names when we deploy the patches.

Comment on attachment 8918294 [details] [diff] [review] bug-1403283-01-full-coverage-v1.patch Review of attachment 8918294 [details] [diff] [review]: ----------------------------------------------------------------- many changes here, some rubber stamping, much real reviews.

Comment on attachment 8918296 [details] [diff] [review] bug-1403283-02-disable-beta-release-v1.patch Review of attachment 8918296 [details] [diff] [review]: ----------------------------------------------------------------- do we get much value out of mozilla-release? the volume is low there and do bugs get filed? Possibly we can reduce some complexity by ignoring that.

Comment on attachment 8918298 [details] [diff] [review] bug-1403283-04-disable-new-v1.patch Review of attachment 8918298 [details] [diff] [review]: ----------------------------------------------------------------- could we make it simpler to not have to comment out each line, maybe change the filename or something? or use a [default] section at the top

Comment on attachment 8918299 [details] [diff] [review] bug-1403283-05-disable-Mdm1-v1.patch Review of attachment 8918299 [details] [diff] [review]: ----------------------------------------------------------------- looks good

(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #13) > > do we get much value out of mozilla-release? the volume is low there and do > bugs get filed? Possibly we can reduce some complexity by ignoring that. Not much value. I don't think we have ever filed a bug from release but I do think it is important that we test what is shipping "just in case". (In reply to Joel Maher ( :jmaher) (UTC-5) from comment #14) > > could we make it simpler to not have to comment out each line, maybe change > the filename or something? or use a [default] section at the top I suppose we could do something better. For me, the autophone-4 data was just to have a place to hang my hat (devices, tests) while I calculated how to distribute the devices. A quick find and replace takes just seconds while coming up with an alternative approach would take longer to design, code and test. With our plans on revamping how Autophone handles devices, tests and repos I really don't want to invest any time in improving something that has a limited lifetime anyway.

as a note, we don't do anything with the mozilla-release data in perfherder, maybe we run some unittests on there only?

Lets let snorp decide if he finds any value in release. We won't be enabling unit tests there for a while anyway. Attaching a screen shot of release phonedash since june 1 for reference.

https://github.com/mozilla/autophone/commit/611d574024d073090c2744fbdd83cba72d9fdc96 https://github.com/mozilla/autophone/commit/4ec828fdd47674f9ce4986e5b4da6cecebc170d2 https://github.com/mozilla/autophone/commit/ad3570430ab9714fc9864d71c777a62534cbf531 https://github.com/mozilla/autophone/commit/ad191462945ae0430044b7fe170eabda91429d08 https://github.com/mozilla/autophone/commit/eac051edea05e3aa7524a53825618435947b205b deploying now.