Joel Maher ( :jmaher ) (UTC -8) Reporter
	Description • 9 years ago

this is a hard issue to pin down given that there is some noise in the test and it looks like we have a few points where we went up slightly, making a cumulative regression of 2% (keep in mind it looks like we had a slight win the day before, so this regression is a bit more amplified).

Here is a graph showing the regression:
https://treeherder.mozilla.org/perf.html#/graphs?timerange=1209600&series=[mozilla-inbound,e9be3e94b1d2eb1dc606b5af74a2b47d890998f5,1]&series=[mozilla-inbound,d302a19fe1f655de7f9db1928a97bda4b3274568,0]&series=[mozilla-inbound,878b980d0b7f428578bca3666fc80f130d0709bf,0]&series=[mozilla-inbound,abb6d4529112e132dc9d28a6a531f33acad51aaa,0]&series=[mozilla-inbound,331915616eb45cfdc3870bdae1f0474d27541351,0]&series=[mozilla-inbound,31bd26f64638a428fd20cf5c6e4e41891e9f7847,0]&zoom=1438300744058.4958,1438714039476.323,1428.7443299224415,2660.6283878934555

I focused on linux32, and did some retriggers on the 'other' job:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&fromchange=5ba973a43c12&tochange=62561970f611&filter-searchStr=ubuntu%20hw%2012.04%20mozilla-inbound%20talos%20other

you can look at the results there and see we started with a range of 2065-2095 at the bottom of the list and ended with a range of 2121-2146.  This is roughly a bump in the reported value of 50 which would be 2.5%.

Now, who is to blame?!?  

Let me annotate the ranges here for clarity:
62561970f611: 2121-2146
94185637c0f3: 2119-2148
2f16fb18314a: 2119-2152*
8ad982618f06: 2113-2138*
0364858aed56: 2101-2124*
947359cbc153: 2066-2094
5ba973a43c12: 2067-2095

I would call out 2 suspect revisions as a starting point here:
30 points 0364858aed56: https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=0364858aed56
15 points 8ad982618f06: https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=8ad982618f06

I suspect more data points would yield some clarity- let me trigger a bunch and see if it helps.

In the meantime, maybe we can speculate on what might be the reason for this, and then determine if it is something to live with, fix, or backout.

	Joel Maher ( :jmaher ) (UTC -8) Reporter
	Comment 1 • 9 years ago

update with more data:
0364858aed56: 2098 - 2131
8ad982618f06: 2106 - 2138
2f16fb18314a: 2111 - 2152 (waiting on 6 more data points- quite a backlog)

I am thinking that 8ad982618f06 is not causing regressions and that 0364858aed56 and 2f16fb18314a are the causes.

	Avi Halachmi (:avih)
	Comment 2 • 9 years ago

(In reply to Joel Maher (:jmaher) from comment #1)
> update with more data:
> 0364858aed56: 2098 - 2131
> ...

min/max are susceptible to outliers, and in  general they don't matter at all in any of the performance calculations (other than affecting stddev and mean).

Probably better to post the values as mean +- stddev.

	Joel Maher ( :jmaher ) (UTC -8) Reporter
	Comment 3 • 9 years ago

hard to do that in my head, but it would be more useful

	Avi Halachmi (:avih)
	Comment 4 • 9 years ago

(In reply to Joel Maher (:jmaher) from comment #3)
> hard to do that in my head, but it would be more useful

Use this bookmarklet.