Move bytecode cache testing into its own test
Categories
(Testing :: Raptor, defect, P1)
Tracking
(firefox104 fixed)
| Tracking | Status | |
|---|---|---|
| firefox104 | --- | fixed |
People
(Reporter: sparky, Assigned: sparky)
References
Details
(Keywords: perf-alert)
Attachments
(1 file)
|
(deleted),
text/x-phabricator-request
|
Details |
We recently enabled a pref that would prepopulate the bytecode cache before the warm pageload. This has caused an increase in variance for a few of our tests, and has increased task run-time by at most 8.3 minutes in all of our pageload tests. This bug is for moving the test outside of the main pageload tests and running them in their own browsertime-tp6-bytecode tests. We'll only run essential tests there at tier 2.
|
Assignee | |
Comment 1•2 years ago
|
||
This patch adds a new set of tests (essential only) to run in CI at tier 2. A command line argument is added to turn bytecode cache testing on in warm pageloads. This also disables the testing in our primary pageload tests to fix the variance and increases in runtime.
|
||
Comment 3•2 years ago
|
||
| bugherder | ||
|
||
Comment 4•2 years ago
|
||
== Change summary for alert #34833 (as of Fri, 15 Jul 2022 05:29:15 GMT) ==
Regressions:
| Ratio | Test | Platform | Options | Absolute values (old vs new) |
|---|---|---|---|---|
| 20% | nytimes fcp | linux1804-64-shippable-qr | fission warm webrender | 77.33 -> 93.04 |
| 4% | nytimes loadtime | linux1804-64-shippable-qr | fission warm webrender | 1,069.94 -> 1,113.17 |
Improvements:
| Ratio | Test | Platform | Options | Absolute values (old vs new) |
|---|---|---|---|---|
| 67% | nytimes FirstVisualChange | linux1804-64-shippable-qr | fission warm webrender | 140.00 -> 46.67 |
| 50% | nytimes ContentfulSpeedIndex | linux1804-64-shippable-qr | fission warm webrender | 353.12 -> 175.50 |
| 12% | nytimes PerceptualSpeedIndex | linux1804-64-shippable-qr | fission warm webrender | 414.08 -> 363.25 |
For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=34833
|
|
||
Updated•2 years ago
|
|
||
Comment 5•2 years ago
|
||
This change caused a ~30% regression, not just increased variance, on some Android warm page load tests on AWFY:
https://treeherder.mozilla.org/perfherder/graphs?series=autoland,4366742,1,13&timerange=2592000
Is that expected? Are the "regressed" results the new baselines? (I understand this is a test change, not a real regression.)
|
|
Assignee | |
Comment 6•2 years ago
|
||
Yes, this is expected so this is a new (or previous) baseline. It brought the metrics back to what they were before this change was made: https://phabricator.services.mozilla.com/D145803
For the A51, we don't have enough history to see what it was before the change.
Description
•