Perfherder has detected a browsertime performance regression from push cfc0f48add0096b2a5ebdf045a31316eb4de1998. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Improvements:

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

Set release status flags based on info from the regressing bug 1689729

Is there a way to get a profile of the hardware running this test? I don't have access to this particular hardware.

yes just build gecko profiles.

I triggered the jobs for you you can find the -p
https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&selectedTaskRun=RWJgAMxzQLCaSgWaieY8pw.0&tier=1%2C2%2C3&searchStr=android%2C7.0%2Cmotog5%2Cshippable%2Copt%2Cbrowsertime%2Cperformance%2Ctests%2Con%2Candroid%2Cwith%2Cwebrender%2Cenabled%2Ctest-android-hw-g5-7-0-arm7-api-16-shippable%2Fopt-browsertime-tp6m-live-geckoview-cnn-ampstories-wr-e10s%2Ccnn-amp&tochange=cfc0f48add0096b2a5ebdf045a31316eb4de1998&fromchange=0c4662217a8d30dcbd59d17b5db39ecda2ef2926

The numbers here look pretty bad. Could we consider backing out bug 1689729 from at least beta87 next week?

Bug 1689729 is only in Nightly so far. Do you mean not let it ride to beta? I think that would be a good idea for now.

Oh, I see - that is what you're saying. Yes, let's do that.

Thanks, tracking so it's on my radar for the merge on Monday.

Rather than tracking this bug, can you just back out https://hg.mozilla.org/mozilla-central/rev/cfc0f48add00?

You mean back it out now on central, or back it out on beta after the merge? I'm using the blocker flag on this bug as a reminder for myself to do the latter on Monday.

Oh, I didn't realize that was what you were planning. I guess let's start with that.

Florin, I can't seem to run any test-android-hw-g5-7-0-arm7-api-16-shippable/opt-browsertime-tp6m-live-geckoview-cnn-ampstories-wr-e10s jobs when I push to try - is that expected?

Backed out for 87:
https://hg.mozilla.org/releases/mozilla-beta/rev/6f195189b558607afe45dafbde29a4912bc1bee6

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #10)

Oh, I didn't realize that was what you were planning. I guess let's start with that.

Florin, I can't seem to run any test-android-hw-g5-7-0-arm7-api-16-shippable/opt-browsertime-tp6m-live-geckoview-cnn-ampstories-wr-e10s jobs when I push to try - is that expected?

Please try the following: ./mach try --full -q "'android-hw-g5 'live-geckoview-cnn". The --full is necessary to run tests against the limited pool of Android hardware.

Hmmm - that resulted in mach: error: unrecognized arguments for try auto: '--full', '-q', ''android-hw-g5 'live-geckoview-cnn'

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #13)

Hmmm - that resulted in mach: error: unrecognized arguments for try auto: '--full', '-q', ''android-hw-g5 'live-geckoview-cnn'

Sorry, I think I was just missing the fuzzy argument. My environment is not currently working so I'm unable to test and confirm. I would suggest trying ./mach try fuzzy --full -q "'android-hw-g5 'live-geckoview-cnn" but if that's not working try using ./mach try fuzzy --full and looking for the appropriate jobs.

Hey :keeler let me know if this helps. Also, this bug needs a priority and assignee as we're tracking it for 88.

That definitely helped - thank you!
If I'm reading https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&selected=2931227,1318252415&series=try,2931227,1,13&timerange=1209600 correctly, bug 1694542 should address this.

The graphs are a bit noisy, but it looks like bug 1694542 has helped - is there a way to confirm this?

The change that introduced this, Bug 1689729, is now in Beta 88.

I'm measuring a pageload performance regression between Fenix Beta 87 and Fenix Beta 88.

Is it possible to re-run the tests that triggered the initial alert?

On a broad live pageload test, I can measure a ~7% drop in visual metric performance from fenix_beta_2021_03_15 (gv87) to fenix_beta_2021_03_24 (gv88) that results from the revision that triggered this regression alert.

https://docs.google.com/spreadsheets/d/18qCiz3SReDgDPwhbYfuDrbnBK1030FuVWGBHWwdgCFY/edit#gid=1689990234&range=AG87:AO87

Unless we can find a way to mitigate this, I don't think we want to ship this to Fenix users because of the performance impact for new users.

I took a look over the graphs and did not saw any significant improvements.

As you can see from these graphs there are no performance fixes/alerts after the 04Feb alert
https://treeherder.mozilla.org/perfherder/graphs?timerange=5184000&series=autoland,2891209,1,13
https://treeherder.mozilla.org/perfherder/graphs?timerange=5184000&series=autoland,2890593,1,13
https://treeherder.mozilla.org/perfherder/graphs?timerange=5184000&series=autoland,2891207,1,13

I will let you guys decide if we should accept this regression or not

Might be best to hold this back until the rest of the changes in this area have landed.

Bug 1689729 backed out from Beta again for 88.
https://hg.mozilla.org/releases/mozilla-beta/rev/8f1fac2611a4

Pascal, I guess we should back this out of beta again for 89.

Dana, where do we stand here for 90?

The initial patch does not backout cleanly to beta, Dana could you provide an update patch to back it out? Thanks

Because this change is affecting a couple of performance tests (android startup, android pageload), can we consider putting it behind a pref or backing it out of nightly until we have the improvement patches ready?

Rather than repeatedly backing out a patch that doesn't backout cleanly or using a pref, I'd rather just revert the specific changes that caused the performance issue until the rest of the code is ready.

Bug 1689729 moved some certificate verification operations to the socket thread
using synchronous runnables. Unfortunately this caused a performance regression
that can't be addressed until all certificate verification operations that
involve NSS certificate resources happen on the socket thread. Until then, this
patch reverts that behavior.

https://hg.mozilla.org/mozilla-central/rev/783b7ceb7ff9

The patch landed in nightly and beta is affected.
:keeler, is this bug important enough to require an uplift?
If not please set status_beta to wontfix.

For more information, please visit auto_nag documentation.

Comment on attachment 9221691 [details]
Bug 1691898 - revert the parts of bug 1689729 that caused a performance regression r?bbeurdouche

Beta/Release Uplift Approval Request

• User impact if declined: Performance regression.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This patch reverts to previous behavior, so it shouldn't be risky.
• String changes made/needed: none

Comment on attachment 9221691 [details]
Bug 1691898 - revert the parts of bug 1689729 that caused a performance regression r?bbeurdouche

Approved for 89 beta 14, thanks.

https://hg.mozilla.org/releases/mozilla-beta/rev/79ea72e65255