3.37 - 25.65% allrecipes/instagram/youtube-watch/youtube-watch ContentfulSpeedIndex (android-hw-g5-7-0-arm7-api-16-shippable, android-hw-p2-8-0-android-aarch64-shippable) regression on push 4951412ca28ee33b50f4f8af722d01703feb7099 (Tue September 15 2020)
Categories
(Remote Protocol :: Marionette, defect)
Tracking
(firefox82 affected)
Tracking | Status | |
---|---|---|
firefox82 | --- | affected |
People
(Reporter: alexandrui, Unassigned)
References
(Regression)
Details
(Keywords: perf, perf-alert, regression)
Perfherder has detected a browsertime performance regression from push 4951412ca28ee33b50f4f8af722d01703feb7099. As author of one of the patches included in that push, we need your help to address this regression.
Regressions:
26% instagram ContentfulSpeedIndex android-hw-p2-8-0-android-aarch64-shippable opt cold 815.46 -> 1,024.58
15% allrecipes ContentfulSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,532.17 -> 2,910.08
12% allrecipes ContentfulSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable opt cold 2,586.54 -> 2,887.00
4% youtube-watch LastVisualChange android-hw-g5-7-0-arm7-api-16-shippable opt warm 1,968.33 -> 2,051.83
3% youtube-watch loadtime android-hw-g5-7-0-arm7-api-16-shippable opt warm 1,595.79 -> 1,649.50
Improvements:
23% google-maps ContentfulSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable opt cold 1,454.25 -> 1,121.33
3% allrecipes SpeedIndex android-hw-p2-8-0-android-aarch64-shippable opt warm 1,145.58 -> 1,112.33
3% allrecipes loadtime android-hw-p2-8-0-android-aarch64-shippable opt warm 1,923.83 -> 1,869.17
2% allrecipes PerceptualSpeedIndex android-hw-p2-8-0-android-aarch64-shippable opt warm 989.50 -> 965.58
Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.
For more information on performance sheriffing please see our FAQ.
Reporter | ||
Updated•4 years ago
|
Comment 1•4 years ago
|
||
Browsertime doesn't use the geckodriver / Marionette navigation commands but just a call to Execute Async Script
. So I don't see how this could have been influenced browsertime. Andrew, do you have any ideas?
Comment 2•4 years ago
|
||
This is quite interesting.
I've just starting looking at the impact of this change via a modified Browsertime that actually uses webdriver navigation.
While it's running nicely locally, so far in CI this change is failing for me.
Let me see if I can find out why this is happening by profiling.
Comment 3•4 years ago
|
||
But ultimately Bug 1612831 is a change to the test harness to make it more closely follow the user experience.
This regression wouldn't impact users -- it's making our tests more realistic.
Reporter | ||
Comment 5•4 years ago
|
||
I can't manage to retrigger vismet on a different instance of the test. Sparky, could you please do 5 retriggers on this range?
Updated•4 years ago
|
Comment 6•4 years ago
|
||
:alexandrui, we have docs for this here: https://wiki.mozilla.org/TestEngineering/Performance/Raptor/Browsertime#Retriggering_Browsertime_Visual_Metrics_Tasks
I've retriggered the tasks.
Comment 7•4 years ago
|
||
Any updates here? Greg, can you please have a look for the retriggered tasks?
Comment 8•4 years ago
|
||
It's a legitimate change, not an outlier. I think we should resolve this as WONTFIX/INVALID like the other alert we were looking at.
Videos from this patch: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/OEtXXru9T9CoCb8ALEOHVg/runs/0/artifacts/public/test_info/browsertime-results.tgz
Videos from previous patch (without regression): https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/LxPM3DwjRG2YpYdCryy1xQ/runs/0/artifacts/public/test_info/browsertime-results.tgz
Comment 9•4 years ago
|
||
I don't see any issues in the videos but I agree that this should be resolved at WONTFIX/INVALID.
Comment 10•4 years ago
|
||
Thank you both for checking!
Updated•2 years ago
|
Description
•