Closed Bug 1663668 Opened 4 years ago Closed 4 years ago

0.72 - 4.28% cnn PerceptualSpeedIndex / cnn SpeedIndex / ebay-kleinanzeigen-search LastVisualChange / ebay-kleinanzeigen-search fcp (android-hw-g5-7-0-arm7-api-16-shippable) regression on push 4bce303c45e740a5636fa88edfbbe41b6cd2e35e (Thu August 27 2020)

Categories

(Testing :: geckodriver, defect)

Firefox 82
defect

Tracking

(firefox-esr68 unaffected, firefox-esr78 unaffected, firefox80 unaffected, firefox81 unaffected, firefox82 affected)

RESOLVED INVALID
Tracking Status
firefox-esr68 --- unaffected
firefox-esr78 --- unaffected
firefox80 --- unaffected
firefox81 --- unaffected
firefox82 --- affected

People

(Reporter: Bebe, Unassigned)

References

(Regression)

Details

(Keywords: perf, perf-alert, regression)

Perfherder has detected a browsertime performance regression from push 4bce303c45e740a5636fa88edfbbe41b6cd2e35e. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

4% ebay-kleinanzeigen-search LastVisualChange android-hw-g5-7-0-arm7-api-16-shippable opt cold 3,930.48 -> 4,098.83
4% cnn PerceptualSpeedIndex android-hw-g5-7-0-arm7-api-16-shippable opt cold 4,818.87 -> 5,002.00
3% cnn SpeedIndex android-hw-g5-7-0-arm7-api-16-shippable opt cold 5,377.43 -> 5,562.17
1% ebay-kleinanzeigen-search fcp android-hw-g5-7-0-arm7-api-16-shippable opt cold 1,231.02 -> 1,239.83

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

Component: Performance → geckodriver
Flags: needinfo?(hskupin)

This change cannot make any difference. The pref isn't used for a while in Firefox and even if it would, it's only related to Firefox upgrades. Are you sure that you have the correct range or that the numbers are correct?

Flags: needinfo?(hskupin) → needinfo?(fstrugariu)

Did some re triggers and backfills to have a better confirmation.

leaving the NI? open

Regressed by: 1583504

Set release status flags based on info from the regressing bug 1583504

Florin, any update here?

In that case there is clearly something wrong with the performance tests. As I said the removed code isn't in use for a long time, and on central there is no single use of it anymore:

https://searchfox.org/mozilla-central/search?q=extensions.showMismatchUI&path=

Greg, any idea what this could be?

Flags: needinfo?(gmierz2)

It looks like the regression detection failed here and that your patch didn't cause the regression here. Have a look at this graph: https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=1&series=autoland,2473343,1,13&timerange=2592000&zoom=1597693949000,1600266039250,876.2777777777778,1625.7222222222222

From there I think it's clear that a patch on August 26th caused the regression, not your patch. :bebe can you bisect further to get the true culprit?

Flags: needinfo?(gmierz2) → needinfo?(fstrugariu)

you are right Opened new bug:

Bug 1667974 - 0.72 - 4.53% cnn PerceptualSpeedIndex / cnn SpeedIndex / ebay-kleinanzeigen-search LastVisualChange / ebay-kleinanzeigen-search fcp (android-hw-g5-7-0-arm7-api-16-shippable) regression on push f8cda577ac2f0eb74b5707f5f8140d656994628f (Wed August 26 2020)

Status: NEW → RESOLVED
Closed: 4 years ago
Flags: needinfo?(fstrugariu)
Resolution: --- → INVALID
You need to log in before you can comment on or make changes to this bug.