Closed
Bug 1192210
Opened 9 years ago
Closed 9 years ago
2% session restore regression on all platforms around august 3rd
Categories
(Testing :: Talos, defect)
Testing
Talos
Tracking
(Not tracked)
RESOLVED
WONTFIX
People
(Reporter: jmaher, Unassigned)
References
Details
(Keywords: perf, regression, Whiteboard: [talos_regression])
Attachments
(1 file)
(deleted),
text/html
|
Details |
this is a hard issue to pin down given that there is some noise in the test and it looks like we have a few points where we went up slightly, making a cumulative regression of 2% (keep in mind it looks like we had a slight win the day before, so this regression is a bit more amplified).
Here is a graph showing the regression:
https://treeherder.mozilla.org/perf.html#/graphs?timerange=1209600&series=[mozilla-inbound,e9be3e94b1d2eb1dc606b5af74a2b47d890998f5,1]&series=[mozilla-inbound,d302a19fe1f655de7f9db1928a97bda4b3274568,0]&series=[mozilla-inbound,878b980d0b7f428578bca3666fc80f130d0709bf,0]&series=[mozilla-inbound,abb6d4529112e132dc9d28a6a531f33acad51aaa,0]&series=[mozilla-inbound,331915616eb45cfdc3870bdae1f0474d27541351,0]&series=[mozilla-inbound,31bd26f64638a428fd20cf5c6e4e41891e9f7847,0]&zoom=1438300744058.4958,1438714039476.323,1428.7443299224415,2660.6283878934555
I focused on linux32, and did some retriggers on the 'other' job:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&fromchange=5ba973a43c12&tochange=62561970f611&filter-searchStr=ubuntu%20hw%2012.04%20mozilla-inbound%20talos%20other
you can look at the results there and see we started with a range of 2065-2095 at the bottom of the list and ended with a range of 2121-2146. This is roughly a bump in the reported value of 50 which would be 2.5%.
Now, who is to blame?!?
Let me annotate the ranges here for clarity:
62561970f611: 2121-2146
94185637c0f3: 2119-2148
2f16fb18314a: 2119-2152*
8ad982618f06: 2113-2138*
0364858aed56: 2101-2124*
947359cbc153: 2066-2094
5ba973a43c12: 2067-2095
I would call out 2 suspect revisions as a starting point here:
30 points 0364858aed56: https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=0364858aed56
15 points 8ad982618f06: https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=8ad982618f06
I suspect more data points would yield some clarity- let me trigger a bunch and see if it helps.
In the meantime, maybe we can speculate on what might be the reason for this, and then determine if it is something to live with, fix, or backout.
Reporter | ||
Comment 1•9 years ago
|
||
update with more data:
0364858aed56: 2098 - 2131
8ad982618f06: 2106 - 2138
2f16fb18314a: 2111 - 2152 (waiting on 6 more data points- quite a backlog)
I am thinking that 8ad982618f06 is not causing regressions and that 0364858aed56 and 2f16fb18314a are the causes.
Comment 2•9 years ago
|
||
(In reply to Joel Maher (:jmaher) from comment #1)
> update with more data:
> 0364858aed56: 2098 - 2131
> ...
min/max are susceptible to outliers, and in general they don't matter at all in any of the performance calculations (other than affecting stddev and mean).
Probably better to post the values as mean +- stddev.
Reporter | ||
Comment 3•9 years ago
|
||
hard to do that in my head, but it would be more useful
Comment 4•9 years ago
|
||
(In reply to Joel Maher (:jmaher) from comment #3)
> hard to do that in my head, but it would be more useful
Use this bookmarklet.
Reporter | ||
Updated•9 years ago
|
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → WONTFIX
You need to log in
before you can comment on or make changes to this bug.
Description
•