a few things to fix in talos tests coming into autophone: 1) talostest.py supports iterations, we only do 1, lets fix that. The simple fix is to hardcode it to a forloop of size 1. The real fix is to fix the code to handle the break/continue statements properly- most likely a lot of code will be removed :) 2) robotest.py does iterations (4 currently) and we print the numbers in the json, but only report the summarized value as the first value in the list. We should take into account the other values and show a real summarized value. 3) robotest.py will fail if we miss a single data point, I would like to allow for 1 failure, and have a redo automatically. This will allow us to continue on and collect many data points. 4) consider dropping the first value from the data series in all tests- maybe this is good for a followup bug, but upon initial data analysis, the first data point seemed to be abnormally high compared to the remaining points. This might include increasing the iterations/cycles to 6, dropping the first 1 and keeping the remaining 5.

made all the modifications, a brief summary: * tp4m/tsvgx - 6 cycles (instead of 4), ignore the first * tp4m/tsvgx - talostest.py code iterations loop removed * rck3 - 6 iterations (instead of 4), ignore the first * rck3 - summarize the data from each iteration into the perfherder suite, then take the geometric mean of the last 5 data points to come up with a number for rck3. I suspect we could do this cleaner, ideally someone else looking at this patch will spot something obvious that I am overlooking.

Comment on attachment 8676933 [details] https://github.com/mozilla/autophone/pull/45 I would really like to talk more about what is going on in collecting the measurements in logcat and storing them in the PerfherderSuite. It is difficult to tell what is doing on here.

Comment on attachment 8676933 [details] https://github.com/mozilla/autophone/pull/45 updated the code with review comments and tested: https://treeherder.allizom.org/#/jobs?repo=mozilla-inbound&filter-searchStr=autophone&fromchange=9f16e62baa3a

Comment on attachment 8676933 [details] https://github.com/mozilla/autophone/pull/45 r+, looks good.

landed: https://github.com/mozilla/autophone/commit/52e5fa4b8622e3a1358d9bc7ec9f0ab960f76128