Raptor has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=0280084a067e50e6f8ac4076ba05665f61ab1c84

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

272% raptor-tp6-google-firefox loadtime macosx1014-64-shippable opt 184.29 -> 684.92
245% raptor-tp6-google-firefox loadtime windows10-64-shippable-qr opt 125.25 -> 431.71
240% raptor-tp6-google-firefox loadtime linux64-shippable-qr opt 139.46 -> 474.58
239% raptor-tp6-google-firefox loadtime linux64-shippable-qr opt 140.06 -> 474.42
235% raptor-tp6-google-firefox loadtime windows7-32-shippable opt 129.38 -> 433.92
232% raptor-tp6-google-firefox loadtime windows10-64-shippable opt 127.71 -> 423.50
228% raptor-tp6-google-firefox loadtime linux64-shippable opt 141.00 -> 462.08
167% raptor-tp6-google-firefox macosx1014-64-shippable opt 160.44 -> 428.23
129% raptor-tp6-google-firefox fcp macosx1014-64-shippable opt 166.04 -> 380.04
126% raptor-tp6-google-firefox windows10-64-shippable-qr opt 117.36 -> 265.47
120% raptor-tp6-google-firefox windows10-64-shippable opt 119.75 -> 263.06
116% raptor-tp6-google-firefox windows7-32-shippable opt 123.41 -> 267.04
113% raptor-tp6-google-firefox linux64-shippable-qr opt 128.93 -> 275.21
106% raptor-tp6-google-firefox linux64-shippable opt 130.92 -> 269.36
87% raptor-tp6-google-firefox fcp windows10-64-shippable-qr opt 127.92 -> 238.75
84% raptor-tp6-google-firefox fcp windows10-64-shippable opt 128.96 -> 236.79
78% raptor-tp6-google-firefox fcp linux64-shippable-qr opt 136.88 -> 244.08
77% raptor-tp6-google-firefox fcp windows7-32-shippable opt 134.83 -> 238.25
71% raptor-tp6-google-firefox fcp linux64-shippable opt 137.83 -> 235.67
70% raptor-tp6-youtube-firefox-cold fcp linux64-shippable-qr opt 536.79 -> 913.08
70% raptor-tp6-youtube-firefox-cold fcp linux64-shippable opt 558.75 -> 948.25
58% raptor-tp6-youtube-firefox-cold fcp windows7-32-shippable opt 572.79 -> 905.08
45% raptor-tp6-youtube-firefox-cold fcp macosx1014-64-shippable opt 853.96 -> 1,237.42
45% raptor-tp6-facebook-firefox loadtime windows10-64-shippable-qr opt 293.00 -> 424.50
44% raptor-tp6-facebook-firefox loadtime windows7-32-shippable opt 289.62 -> 418.38
43% raptor-tp6-facebook-firefox macosx1014-64-shippable opt 456.10 -> 654.30
42% raptor-tp6-youtube-firefox-cold windows7-32-shippable opt 653.35 -> 925.98
38% raptor-tp6-facebook-firefox windows7-32-shippable opt 228.79 -> 316.14
38% raptor-tp6-facebook-firefox loadtime linux64-shippable opt 293.71 -> 404.71
37% raptor-tp6-facebook-firefox loadtime linux64-shippable-qr opt 305.42 -> 419.29
37% raptor-tp6-facebook-firefox windows10-64-shippable-qr opt 232.14 -> 317.15
35% raptor-tp6-facebook-firefox loadtime macosx1014-64-shippable opt 948.67 -> 1,284.58
30% raptor-tp6-facebook-firefox linux64-shippable opt 231.77 -> 300.64
28% raptor-tp6-facebook-firefox linux64-shippable-qr opt 244.42 -> 312.02
25% raptor-tp6-facebook-firefox fcp macosx1014-64-shippable opt 430.21 -> 539.12
16% raptor-tp6-facebook-firefox fcp windows7-32-shippable opt 261.12 -> 304.17
16% raptor-tp6-youtube-firefox-cold fcp windows7-32-shippable opt 511.88 -> 591.67
16% raptor-tp6-facebook-firefox fcp windows10-64-shippable opt 263.21 -> 304.17
14% raptor-tp6-facebook-firefox fcp windows10-64-shippable-qr opt 265.62 -> 303.38
8% raptor-tp6-facebook-firefox fcp linux64-shippable opt 264.67 -> 285.71
6% raptor-tp6-facebook-firefox fcp linux64-shippable-qr opt 279.75 -> 296.58
6% raptor-tp6-facebook-firefox fcp linux64-shippable-qr opt 280.15 -> 296.71
2% raptor-tp6-youtube-firefox fcp linux64-shippable opt 735.62 -> 752.29

Improvements:

28% raptor-tp6-facebook-firefox-cold fcp macosx1014-64-shippable opt 1,035.75 -> 743.25
26% raptor-tp6-amazon-firefox-cold loadtime macosx1014-64-shippable opt 2,141.50 -> 1,577.92
18% raptor-tp6-facebook-firefox-cold macosx1014-64-shippable opt 1,049.03 -> 855.19
18% raptor-tp6-facebook-firefox-cold fcp linux64-shippable-qr opt 601.21 -> 491.50
18% raptor-tp6-facebook-firefox-cold loadtime windows10-64-shippable-qr opt 1,160.71 -> 952.75
17% raptor-tp6-facebook-firefox-cold loadtime windows7-32-shippable opt 1,168.92 -> 966.83
17% raptor-tp6-amazon-firefox-cold macosx1014-64-shippable opt 1,353.14 -> 1,123.70
16% raptor-tp6-facebook-firefox-cold loadtime linux64-shippable-qr opt 1,165.42 -> 979.33
14% raptor-tp6-facebook-firefox-cold loadtime windows10-64-shippable opt 1,131.50 -> 976.25
7% raptor-tp6-facebook-firefox-cold loadtime macosx1014-64-shippable opt 1,680.04 -> 1,565.50

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=21580

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a Treeherder page showing the Raptor jobs in a pushlog format.

To learn more about the regressing test(s) or reproducing them, please see: https://wiki.mozilla.org/Performance_sheriffing/Raptor

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Performance_sheriffing/Talos/RegressionBugsHandling

This change is because of the new recordings.

In these recordings we added:

• deterministicjs
• Upstream certs
• mitm 4.0.4

I would say this regressions are expected based on previous testing when introducing these framework changes

try build with Upstream chert activated (aka. no outbound connections)

https://treeherder.mozilla.org/#/jobs?repo=try&revision=7569cff3b25702847243e53d6e816717f9ffd2ae

try build with mitm2.0.2

https://treeherder.mozilla.org/#/jobs?repo=try&revision=aa8ab1cc7853ddf76e4b0d1a2686f3ecb57df5d5
Note: Build on 2.0.2 is failing becauso of recording compatibility

It looks like we're still connecting to the upstream servers because we only disable this behaviour for netflix:
https://searchfox.org/mozilla-central/search?q=playback_upstream_cert&path=
You can also see the invocation of mitmproxy in logs (search for "Starting mitmproxy playback using command")
https://taskcluster-artifacts.net/DjfhzbbvTMutPhm8YdXTRw/0/public/logs/live_backing.log

When I run this locally I can disconnect my network after raptor has started and it will hang when making the TLS connection to facebook.com.

I don't see any cases of protocol downgrade in the facebook.com playback.
But from looking at the stored protocols (e.g. facebook.json) we'll have to be aware that they are, as expected, for region-specific domains (e.g. "www.google.ro": "HTTP/2.0").

As we have upstream certs activated to almost all desktop tests in https://hg.mozilla.org/mozilla-central/rev/6bcdd2b94604
Desktop tests should be fully on proxy replay and no outbound connections

As this is a framework change we won't fix this regressions