Closed Bug 1767626 Opened 3 years ago Closed 3 years ago

447.57 - 3.64% cnn-nav.world LastVisualChange / reddit ContentfulSpeedIndex + 60 more (Android, Linux, OSX, Windows) regression on Mon May 2 2022

Categories

(Testing :: Raptor, defect, P2)

Product:
Testing
Component:
Raptor
Version:
Firefox 102
Type:
defect

Tracking

(firefox102 fixed)

Status:
RESOLVED FIXED
Milestone:
102 Branch
Tracking Flags:
Tracking Status
firefox102 --- fixed

People

(Reporter: alexandrui, Assigned: sparky)

References

(Regression)

Details

(Keywords: perf, perf-alert, regression)

Attachments

(1 file)

Bug 1767626 - Accept a 5 pixel error when comparing duplicate frames. r?#perftest
(deleted), text/x-phabricator-request
Details

Perfherder has detected a browsertime performance regression from push 7bbdcfc5d92dd30fa5eb7b603643c6cfb1d4fa6f. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio Test Platform Options Absolute values (old vs new)
448% cnn-nav.world LastVisualChange windows10-64-shippable-qr cold fission webrender 1,197.17 -> 6,555.38
418% cnn-nav.world LastVisualChange macosx1015-64-shippable-qr cold fission webrender 1,015.00 -> 5,260.00
323% cnn-nav.world LastVisualChange linux1804-64-shippable-qr cold fission webrender 1,150.00 -> 4,866.67
321% cnn-nav.landing LastVisualChange windows10-64-shippable-qr cold fission webrender 2,521.04 -> 10,616.71
321% cnn-nav.landing LastVisualChange windows10-64-shippable-qr cold fission webrender 2,524.54 -> 10,625.12
247% cnn-nav.landing LastVisualChange linux1804-64-shippable-qr cold fission webrender 2,860.00 -> 9,931.67
152% ebay ContentfulSpeedIndex linux1804-64-shippable-qr fission warm webrender 129.42 -> 325.92
150% ebay ContentfulSpeedIndex macosx1015-64-shippable-qr fission warm webrender 127.33 -> 318.00
102% ebay ContentfulSpeedIndex windows10-64-shippable-qr fission warm webrender 212.00 -> 428.08
85% reddit-billgates-post-2.hot LastVisualChange macosx1015-64-shippable-qr cold fission webrender 1,088.33 -> 2,018.33
... ... ... ... ...
5% netflix ContentfulSpeedIndex linux1804-64-shippable-qr cold fission webrender 920.58 -> 969.58
5% linkedin ContentfulSpeedIndex windows10-64-shippable-qr cold fission webrender 2,317.96 -> 2,422.17
4% google-docs-canvas ContentfulSpeedIndex windows10-64-shippable-qr cold fission webrender 1,006.12 -> 1,050.67
4% linkedin ContentfulSpeedIndex linux1804-64-shippable-qr cold fission webrender 2,209.12 -> 2,293.75
4% reddit ContentfulSpeedIndex android-hw-g5-7-0-arm7-shippable-qr warm webrender 2,267.67 -> 2,350.17

Improvements:

Ratio Test Platform Options Absolute values (old vs new)
61% welcome ContentfulSpeedIndex windows10-64-shippable-qr fission warm webrender 368.43 -> 143.83
60% welcome ContentfulSpeedIndex linux1804-64-shippable-qr fission warm webrender 388.69 -> 157.42
49% facebook-nav.friends ContentfulSpeedIndex windows10-64-shippable-qr cold fission webrender 1,156.54 -> 586.38
44% welcome ContentfulSpeedIndex linux1804-64-shippable-qr cold fission webrender 521.54 -> 292.50
34% welcome ContentfulSpeedIndex windows10-64-shippable-qr cold fission webrender 438.50 -> 289.42
... ... ... ... ...
2% google-maps LastVisualChange android-hw-p2-8-0-android-aarch64-shippable-qr warm webrender 751.54 -> 735.50

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a sheriff to do that for you.

For more information on performance sheriffing please see our FAQ.

Flags: needinfo?(gmierz2)
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → DUPLICATE

I've re-openened since it's not a duplicate of the other bug (even though the regressing bug is the same).

The lastvisualchange changes are concerning to me - I found that they are caused by very small differences in pixel counts and that this change should resolve it: https://github.com/sitespeedio/browsertime/pull/1780

Leaving the ni? open for myself.

Status: RESOLVED → REOPENED
Resolution: DUPLICATE → ---
Severity: -- → S2
Flags: needinfo?(gmierz2)
Priority: -- → P2
Assignee: nobody → gmierz2

This patch updates browsertime to include a patch that accepts a 5 pixel error (instead of 0) when comparing the frames at the end of the recording to remove duplicate ones. With this change, we should no longer over-estimate the LastVisualChange.

Has Regression Range: --- → yes
Pushed by gmierz2@outlook.com: https://hg.mozilla.org/integration/autoland/rev/8c8dc11475fa Accept a 5 pixel error when comparing duplicate frames. r=perftest-reviewers,AlexandruIonescu

https://hg.mozilla.org/mozilla-central/rev/8c8dc11475fa

Status: REOPENED → RESOLVED
Closed: 3 years ago → 3 years ago
status-firefox102: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 102 Branch

(In reply to Pulsebot from comment #4)

Pushed by gmierz2@outlook.com:
https://hg.mozilla.org/integration/autoland/rev/8c8dc11475fa
Accept a 5 pixel error when comparing duplicate frames.
r=perftest-reviewers,AlexandruIonescu

== Change summary for alert #34081 (as of Sat, 07 May 2022 18:54:47 GMT) ==

Improvements:

Ratio Test Platform Options Absolute values (old vs new)
9% reddit-billgates-post-2.billg ContentfulSpeedIndex linux1804-64-shippable-qr cold fission webrender 744.67 -> 675.08

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=34081

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: