Closed Bug 1681175 Opened 4 years ago Closed 4 years ago

accept new Ubuntu 18.04 gecko-t-linux-talos worker performance

Categories

(Testing :: Performance, task, P2)

Product:
Testing
Component:
Performance
Version:
Default
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: aerickson, Unassigned)

References

Details

Attachments

(1 file)

linux-outlier-summary.7z
(deleted), application/x-7z-compressed
Details

Tracking the testing done to qualify the new gecko-t-linux-talos workers running 18.04 running on the HP Moonshot hardware.

update on preliminary testing: the 18.04 workers are slower than the existing 16.04 workers

I've compared the puppet configuration for the new 1804 pool to the existing 1604 workers and the 1804 Dockerfile and I couldn't find any further sources of slowness.

:gmierz has said that if we can ensure the workers variance is low, the slower performance is OK (and there are some test scripts we can to use to verify).

Flags: needinfo?(gmierz2)
Whiteboard: [perftest:triage]
Severity: -- → S2
Priority: -- → P2
Whiteboard: [perftest:triage]

:aerickson, if you can get me a couple try runs (one with 18 and another with 16) then I can do the comparisons with the scripts I have in January. I'd need to have all the concerning tests retriggered at least 10 times in each of those pushes.

Flags: needinfo?(gmierz2)

Each task was rerun via custom action 10 times with downstream tasks. Thanks. :)

1804 run:
https://treeherder.mozilla.org/jobs?repo=try&revision=37b18d1324ec467708983b04ed0fcd6b79a274e5&group_state=expanded

1604 run: https://treeherder.mozilla.org/jobs?repo=try&revision=fcb1edb0055cfdc9f61473297d532dd405b61087&group_state=expanded

Flags: needinfo?(gmierz2)

I've downloaded all the data, I should have some results for us today or tomorrow.

Attached file linux-outlier-summary.7z (deleted) —

Initial pass for today doesn't look good.

Total outliers for 1604: 2876
Total outliers for 1804: 3902

Total outliers for 1604 (webrender): 2494
Total outliers for 1804 (webrender): 3306

It looks like there's a 32-35% increase in outliers. I've attached some files which summarize what I've found.

I need to run this through Levene's test before I can say if this is problematic or not - I should have results for that tomorrow.

Here are the results I got from Levene's test which tests for equal variance. A p-value of < 0.05 indicates a significant difference in variance (i.e. variances are not equal). Alone, Levene's test doesn't tell us if noise increased or decreased. I used the ratio between the normalized standard-deviation of 1604 data versus 1804 data and transformed this metric to a percentage (percentage for increases = 100 * (val^-1) - 100).

Below, I've provided some more detail that shows which tests are noisier or more stable.

The good news is that there are some metrics that benefit greatly from the change. For non-webrender, we see an average decrease of 66.6% in the noise for 78 tests/metrics. For webrender, we see a decrease of ~99% (to be clear, this means that the variance was cut in half).

That said, there's bad news, and that's that a majority of tests (33% higher than the number of good tests) that had a significant difference in the variance which showed an increase in noise. The increases in noise were ~62%, and ~50% for non-webrender, and webrender respectively.

:aerickson, there's something that I think is skewing our results and that's that the 1804 pool of machines is small. In the past, we have shown that pooling tests to a small set of machines actually increases the variance in our metrics for Linux. I have a feeling we may be hitting the same thing here since the 1804 pool is very small when we compare it to the size of the 1604 pool.

It's clear that there will be a large number of tests which will have an increase in noise but also other tests which will decrease in noise. Because we're seeing improvements as well as issues (majority), I'm not 100% sure of how we should proceed here. I think it's important that we start testing 1804 since it's more current but increasing the variance means that we might start missing regressions or triggering too many alerts.

:davehunt/:aerickson, what are your thoughts on this? Do you have any questions I could answer with the data we have?

Non-webrender:

Summary:
Average noise increases: 61.9 %
Average noise decreases: 66.6 %
Bad tests (107):
    browsertime-tp6-firefox-twitter-e10s-loadtimenocondprof-cold
    browsertime-tp6-firefox-twitter-e10s-fcpnocondprof-warm
    browsertime-tp6-firefox-twitter-e10s-fnbpaintnocondprof-warm
    browsertime-tp6-firefox-twitter-e10s-SpeedIndexnocondprof-cold
    browsertime-tp6-firefox-twitter-e10s-LastVisualChangenocondprof-cold
    browsertime-tp6-firefox-twitter-e10s-ContentfulSpeedIndexnocondprof-cold
    browsertime-tp6-firefox-twitter-e10s-PerceptualSpeedIndexnocondprof-cold
    browsertime-tp6-firefox-twitter-e10s-FirstVisualChangenocondprof-warm
    browsertime-tp6-firefox-twitch-e10s-fcpnocondprof-cold
    browsertime-tp6-firefox-twitch-e10s-fnbpaintnocondprof-cold
    browsertime-tp6-firefox-twitch-e10s-dcfnocondprof-warm
    browsertime-tp6-firefox-twitch-e10s-fcpnocondprof-warm
    browsertime-tp6-firefox-twitch-e10s-fnbpaintnocondprof-warm
    browsertime-tp6-firefox-twitch-e10s-loadtimenocondprof-warm
    browsertime-tp6-firefox-twitch-e10s-LastVisualChangenocondprof-cold
    browsertime-tp6-firefox-twitch-e10s-ContentfulSpeedIndexnocondprof-cold
    browsertime-tp6-firefox-twitch-e10s-SpeedIndexnocondprof-warm
    browsertime-tp6-firefox-twitch-e10s-FirstVisualChangenocondprof-warm
    browsertime-tp6-firefox-twitch-e10s-ContentfulSpeedIndexnocondprof-warm
    browsertime-tp6-firefox-twitch-e10s-PerceptualSpeedIndexnocondprof-warm
    browsertime-tp6-firefox-twitter-fis-e10s-dcfnocondprof-fission-cold
    browsertime-tp6-firefox-twitter-fis-e10s-loadtimenocondprof-fission-cold
    browsertime-tp6-firefox-twitter-fis-e10s-dcfnocondprof-fission-warm
    browsertime-tp6-firefox-twitter-fis-e10s-fcpnocondprof-fission-warm
    browsertime-tp6-firefox-twitter-fis-e10s-fnbpaintnocondprof-fission-warm
    browsertime-tp6-firefox-twitter-fis-e10s-SpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-twitter-fis-e10s-LastVisualChangenocondprof-fission-cold
    browsertime-tp6-firefox-twitter-fis-e10s-ContentfulSpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-twitter-fis-e10s-PerceptualSpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-google-mail-fis-e10s-dcfnocondprof-fission-warm
    browsertime-tp6-firefox-google-mail-fis-e10s-fcpnocondprof-fission-warm
    browsertime-tp6-firefox-google-mail-fis-e10s-fnbpaintnocondprof-fission-warm
    browsertime-tp6-firefox-google-mail-fis-e10s-loadtimenocondprof-fission-warm
    browsertime-tp6-firefox-google-mail-fis-e10s-FirstVisualChangenocondprof-fission-cold
    browsertime-tp6-firefox-google-mail-fis-e10s-LastVisualChangenocondprof-fission-cold
    browsertime-tp6-firefox-google-mail-fis-e10s-ContentfulSpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-google-mail-fis-e10s-SpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-google-mail-fis-e10s-FirstVisualChangenocondprof-fission-warm
    browsertime-tp6-firefox-google-mail-fis-e10s-PerceptualSpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-twitch-fis-e10s-dcfnocondprof-fission-warm
    browsertime-tp6-firefox-twitch-fis-e10s-fcpnocondprof-fission-warm
    browsertime-tp6-firefox-twitch-fis-e10s-fnbpaintnocondprof-fission-warm
    browsertime-tp6-firefox-twitch-fis-e10s-loadtimenocondprof-fission-warm
    browsertime-tp6-firefox-twitch-fis-e10s-LastVisualChangenocondprof-fission-cold
    browsertime-tp6-firefox-twitch-fis-e10s-ContentfulSpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-twitch-fis-e10s-SpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-twitch-fis-e10s-ContentfulSpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-twitch-fis-e10s-PerceptualSpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-google-slides-fis-e10s-loadtimenocondprof-fission-cold
    browsertime-tp6-firefox-google-slides-fis-e10s-LastVisualChangenocondprof-fission-cold
    browsertime-tp6-firefox-tumblr-fis-e10s-dcfnocondprof-fission-cold
    browsertime-tp6-firefox-tumblr-fis-e10s-fcpnocondprof-fission-cold
    browsertime-tp6-firefox-tumblr-fis-e10s-fnbpaintnocondprof-fission-cold
    browsertime-tp6-firefox-tumblr-fis-e10s-loadtimenocondprof-fission-cold
    browsertime-tp6-firefox-tumblr-fis-e10s-dcfnocondprof-fission-warm
    browsertime-tp6-firefox-tumblr-fis-e10s-fcpnocondprof-fission-warm
    browsertime-tp6-firefox-tumblr-fis-e10s-loadtimenocondprof-fission-warm
    browsertime-tp6-firefox-tumblr-fis-e10s-SpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-tumblr-fis-e10s-ContentfulSpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-tumblr-fis-e10s-PerceptualSpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-google-slides-e10s-dcfnocondprof-cold
    browsertime-tp6-firefox-google-slides-e10s-fcpnocondprof-cold
    browsertime-tp6-firefox-google-slides-e10s-fnbpaintnocondprof-cold
    browsertime-tp6-firefox-google-slides-e10s-loadtimenocondprof-cold
    browsertime-tp6-firefox-google-slides-e10s-dcfnocondprof-warm
    browsertime-tp6-firefox-google-slides-e10s-loadtimenocondprof-warm
    browsertime-tp6-firefox-google-slides-e10s-SpeedIndexnocondprof-cold
    browsertime-tp6-firefox-google-slides-e10s-FirstVisualChangenocondprof-cold
    browsertime-tp6-firefox-google-slides-e10s-LastVisualChangenocondprof-cold
    browsertime-tp6-firefox-google-slides-e10s-ContentfulSpeedIndexnocondprof-cold
    browsertime-tp6-firefox-google-slides-e10s-PerceptualSpeedIndexnocondprof-cold
    browsertime-tp6-firefox-google-slides-e10s-SpeedIndexnocondprof-warm
    browsertime-tp6-firefox-google-slides-e10s-LastVisualChangenocondprof-warm
    browsertime-tp6-firefox-google-slides-e10s-PerceptualSpeedIndexnocondprof-warm
    browsertime-tp6-firefox-google-mail-e10s-loadtimenocondprof-cold
    browsertime-tp6-firefox-google-mail-e10s-dcfnocondprof-warm
    browsertime-tp6-firefox-google-mail-e10s-loadtimenocondprof-warm
    browsertime-tp6-firefox-google-mail-e10s-SpeedIndexnocondprof-cold
    browsertime-tp6-firefox-google-mail-e10s-ContentfulSpeedIndexnocondprof-cold
    browsertime-tp6-firefox-amazon-fis-e10s-fcpnocondprof-fission-warm
    browsertime-tp6-firefox-amazon-fis-e10s-fnbpaintnocondprof-fission-warm
    browsertime-tp6-firefox-amazon-fis-e10s-loadtimenocondprof-fission-cold
    browsertime-tp6-firefox-amazon-fis-e10s-LastVisualChangenocondprof-fission-cold
    browsertime-tp6-firefox-tumblr-e10s-fcpnocondprof-warm
    browsertime-tp6-firefox-tumblr-e10s-fnbpaintnocondprof-warm
    browsertime-tp6-firefox-tumblr-e10s-SpeedIndexnocondprof-warm
    browsertime-tp6-firefox-tumblr-e10s-FirstVisualChangenocondprof-warm
    browsertime-tp6-firefox-tumblr-e10s-PerceptualSpeedIndexnocondprof-warm
    browsertime-tp6-firefox-amazon-e10s-loadtimenocondprof-cold
    browsertime-tp6-firefox-amazon-e10s-LastVisualChangenocondprof-cold
    browsertime-tp6-firefox-amazon-e10s-LastVisualChangenocondprof-warm
    browsertime-tp6-firefox-imgur-e10s-dcfnocondprof-warm
    browsertime-tp6-firefox-imgur-e10s-fnbpaintnocondprof-warm
    browsertime-tp6-firefox-imgur-fis-e10s-fcpnocondprof-fission-cold
    browsertime-tp6-firefox-imgur-fis-e10s-fnbpaintnocondprof-fission-cold
    browsertime-tp6-firefox-imgur-fis-e10s-dcfnocondprof-fission-warm
    browsertime-tp6-firefox-imgur-fis-e10s-fcpnocondprof-fission-warm
    browsertime-tp6-firefox-imgur-fis-e10s-fnbpaintnocondprof-fission-warm
    browsertime-tp6-firefox-imgur-fis-e10s-loadtimenocondprof-fission-warm
    browsertime-tp6-firefox-imgur-fis-e10s-SpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-imgur-fis-e10s-FirstVisualChangenocondprof-fission-cold
    browsertime-tp6-firefox-imgur-fis-e10s-LastVisualChangenocondprof-fission-cold
    browsertime-tp6-firefox-imgur-fis-e10s-PerceptualSpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-imgur-fis-e10s-SpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-imgur-fis-e10s-LastVisualChangenocondprof-fission-warm
    browsertime-tp6-firefox-imgur-fis-e10s-ContentfulSpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-imgur-fis-e10s-PerceptualSpeedIndexnocondprof-fission-warm
Good tests (74):
    browsertime-tp6-firefox-twitter-e10s-fcpnocondprof-cold
    browsertime-tp6-firefox-twitter-e10s-fnbpaintnocondprof-cold
    browsertime-tp6-firefox-twitter-e10s-dcfnocondprof-warm
    browsertime-tp6-firefox-twitter-e10s-SpeedIndexnocondprof-warm
    browsertime-tp6-firefox-twitch-e10s-dcfnocondprof-cold
    browsertime-tp6-firefox-twitch-e10s-loadtimenocondprof-cold
    browsertime-tp6-firefox-twitch-e10s-LastVisualChangenocondprof-warm
    browsertime-tp6-firefox-twitter-fis-e10s-fcpnocondprof-fission-cold
    browsertime-tp6-firefox-twitter-fis-e10s-fnbpaintnocondprof-fission-cold
    browsertime-tp6-firefox-twitter-fis-e10s-FirstVisualChangenocondprof-fission-cold
    browsertime-tp6-firefox-twitter-fis-e10s-PerceptualSpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-google-mail-fis-e10s-dcfnocondprof-fission-cold
    browsertime-tp6-firefox-google-mail-fis-e10s-fcpnocondprof-fission-cold
    browsertime-tp6-firefox-google-mail-fis-e10s-fnbpaintnocondprof-fission-cold
    browsertime-tp6-firefox-google-mail-fis-e10s-LastVisualChangenocondprof-fission-warm
    browsertime-tp6-firefox-google-mail-fis-e10s-ContentfulSpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-google-mail-fis-e10s-videoRecordingStartnocondprof-fission-warm
    browsertime-tp6-firefox-twitch-fis-e10s-fcpnocondprof-fission-cold
    browsertime-tp6-firefox-twitch-fis-e10s-fnbpaintnocondprof-fission-cold
    browsertime-tp6-firefox-twitch-fis-e10s-loadtimenocondprof-fission-cold
    browsertime-tp6-firefox-twitch-fis-e10s-FirstVisualChangenocondprof-fission-cold
    browsertime-tp6-firefox-twitch-fis-e10s-PerceptualSpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-twitch-fis-e10s-LastVisualChangenocondprof-fission-warm
    browsertime-tp6-firefox-google-slides-fis-e10s-dcfnocondprof-fission-cold
    browsertime-tp6-firefox-google-slides-fis-e10s-dcfnocondprof-fission-warm
    browsertime-tp6-firefox-google-slides-fis-e10s-loadtimenocondprof-fission-warm
    browsertime-tp6-firefox-google-slides-fis-e10s-SpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-google-slides-fis-e10s-FirstVisualChangenocondprof-fission-cold
    browsertime-tp6-firefox-google-slides-fis-e10s-PerceptualSpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-google-slides-fis-e10s-SpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-google-slides-fis-e10s-LastVisualChangenocondprof-fission-warm
    browsertime-tp6-firefox-google-slides-fis-e10s-ContentfulSpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-google-slides-fis-e10s-PerceptualSpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-tumblr-fis-e10s-SpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-tumblr-fis-e10s-FirstVisualChangenocondprof-fission-warm
    browsertime-tp6-firefox-tumblr-fis-e10s-ContentfulSpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-google-mail-e10s-fcpnocondprof-cold
    browsertime-tp6-firefox-google-mail-e10s-fnbpaintnocondprof-cold
    browsertime-tp6-firefox-google-mail-e10s-FirstVisualChangenocondprof-cold
    browsertime-tp6-firefox-google-mail-e10s-PerceptualSpeedIndexnocondprof-cold
    browsertime-tp6-firefox-google-mail-e10s-LastVisualChangenocondprof-warm
    browsertime-tp6-firefox-amazon-fis-e10s-dcfnocondprof-fission-cold
    browsertime-tp6-firefox-amazon-fis-e10s-fcpnocondprof-fission-cold
    browsertime-tp6-firefox-amazon-fis-e10s-fnbpaintnocondprof-fission-cold
    browsertime-tp6-firefox-amazon-fis-e10s-SpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-amazon-fis-e10s-FirstVisualChangenocondprof-fission-cold
    browsertime-tp6-firefox-amazon-fis-e10s-ContentfulSpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-amazon-fis-e10s-PerceptualSpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-amazon-fis-e10s-SpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-amazon-fis-e10s-ContentfulSpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-amazon-fis-e10s-PerceptualSpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-tumblr-e10s-fcpnocondprof-cold
    browsertime-tp6-firefox-tumblr-e10s-fnbpaintnocondprof-cold
    browsertime-tp6-firefox-tumblr-e10s-loadtimenocondprof-warm
    browsertime-tp6-firefox-tumblr-e10s-SpeedIndexnocondprof-cold
    browsertime-tp6-firefox-tumblr-e10s-FirstVisualChangenocondprof-cold
    browsertime-tp6-firefox-tumblr-e10s-ContentfulSpeedIndexnocondprof-cold
    browsertime-tp6-firefox-tumblr-e10s-PerceptualSpeedIndexnocondprof-cold
    browsertime-tp6-firefox-tumblr-e10s-LastVisualChangenocondprof-warm
    browsertime-tp6-firefox-amazon-e10s-dcfnocondprof-cold
    browsertime-tp6-firefox-amazon-e10s-fcpnocondprof-cold
    browsertime-tp6-firefox-amazon-e10s-fnbpaintnocondprof-cold
    browsertime-tp6-firefox-amazon-e10s-SpeedIndexnocondprof-cold
    browsertime-tp6-firefox-amazon-e10s-FirstVisualChangenocondprof-cold
    browsertime-tp6-firefox-amazon-e10s-ContentfulSpeedIndexnocondprof-cold
    browsertime-tp6-firefox-imgur-e10s-dcfnocondprof-cold
    browsertime-tp6-firefox-imgur-e10s-loadtimenocondprof-warm
    browsertime-tp6-firefox-imgur-e10s-FirstVisualChangenocondprof-cold
    browsertime-tp6-firefox-imgur-e10s-LastVisualChangenocondprof-cold
    browsertime-tp6-firefox-imgur-e10s-ContentfulSpeedIndexnocondprof-cold
    browsertime-tp6-firefox-imgur-e10s-PerceptualSpeedIndexnocondprof-cold
    browsertime-tp6-firefox-imgur-e10s-LastVisualChangenocondprof-warm
    browsertime-tp6-firefox-imgur-e10s-ContentfulSpeedIndexnocondprof-warm

With webrender:

Summary:
Average noise increases: 50.1 %
Average noise decreases: 99.1 %
Bad tests (107):
    browsertime-tp6-firefox-twitter-e10s-loadtimenocondprof-cold-webrender
    browsertime-tp6-firefox-twitter-e10s-fcpnocondprof-webrender-warm
    browsertime-tp6-firefox-twitter-e10s-fnbpaintnocondprof-webrender-warm
    browsertime-tp6-firefox-twitter-e10s-SpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-twitter-e10s-LastVisualChangenocondprof-cold-webrender
    browsertime-tp6-firefox-twitter-e10s-ContentfulSpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-twitter-e10s-PerceptualSpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-twitch-e10s-fcpnocondprof-cold-webrender
    browsertime-tp6-firefox-twitch-e10s-fnbpaintnocondprof-cold-webrender
    browsertime-tp6-firefox-twitch-e10s-fcpnocondprof-webrender-warm
    browsertime-tp6-firefox-twitch-e10s-LastVisualChangenocondprof-cold-webrender
    browsertime-tp6-firefox-twitch-e10s-ContentfulSpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-twitch-e10s-FirstVisualChangenocondprof-webrender-warm
    browsertime-tp6-firefox-twitch-e10s-LastVisualChangenocondprof-webrender-warm
    browsertime-tp6-firefox-twitch-e10s-ContentfulSpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-twitter-fis-e10s-dcfnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-twitter-fis-e10s-loadtimenocondprof-fission-cold-webrender
    browsertime-tp6-firefox-twitter-fis-e10s-fcpnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-twitter-fis-e10s-fnbpaintnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-twitter-fis-e10s-SpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-twitter-fis-e10s-LastVisualChangenocondprof-fission-cold-webrender
    browsertime-tp6-firefox-twitter-fis-e10s-ContentfulSpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-twitter-fis-e10s-FirstVisualChangenocondprof-fission-webrender-warm
    browsertime-tp6-firefox-twitter-fis-e10s-PerceptualSpeedIndexnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-google-mail-fis-e10s-dcfnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-google-mail-fis-e10s-loadtimenocondprof-fission-webrender-warm
    browsertime-tp6-firefox-google-mail-fis-e10s-FirstVisualChangenocondprof-fission-webrender-warm
    browsertime-tp6-firefox-google-mail-fis-e10s-videoRecordingStartnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-twitch-fis-e10s-fcpnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-twitch-fis-e10s-loadtimenocondprof-fission-webrender-warm
    browsertime-tp6-firefox-twitch-fis-e10s-LastVisualChangenocondprof-fission-cold-webrender
    browsertime-tp6-firefox-twitch-fis-e10s-LastVisualChangenocondprof-fission-webrender-warm
    browsertime-tp6-firefox-twitch-fis-e10s-videoRecordingStartnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-google-slides-fis-e10s-fcpnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-google-slides-fis-e10s-fnbpaintnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-google-slides-fis-e10s-ContentfulSpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-google-slides-fis-e10s-SpeedIndexnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-google-slides-fis-e10s-FirstVisualChangenocondprof-fission-webrender-warm
    browsertime-tp6-firefox-tumblr-fis-e10s-dcfnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-tumblr-fis-e10s-fcpnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-tumblr-fis-e10s-fnbpaintnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-tumblr-fis-e10s-loadtimenocondprof-fission-cold-webrender
    browsertime-tp6-firefox-tumblr-fis-e10s-loadtimenocondprof-fission-webrender-warm
    browsertime-tp6-firefox-tumblr-fis-e10s-SpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-tumblr-fis-e10s-FirstVisualChangenocondprof-fission-cold-webrender
    browsertime-tp6-firefox-tumblr-fis-e10s-LastVisualChangenocondprof-fission-cold-webrender
    browsertime-tp6-firefox-tumblr-fis-e10s-ContentfulSpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-tumblr-fis-e10s-PerceptualSpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-tumblr-fis-e10s-videoRecordingStartnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-google-slides-e10s-dcfnocondprof-cold-webrender
    browsertime-tp6-firefox-google-slides-e10s-fcpnocondprof-cold-webrender
    browsertime-tp6-firefox-google-slides-e10s-fnbpaintnocondprof-cold-webrender
    browsertime-tp6-firefox-google-slides-e10s-ContentfulSpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-google-slides-e10s-LastVisualChangenocondprof-webrender-warm
    browsertime-tp6-firefox-google-mail-e10s-dcfnocondprof-cold-webrender
    browsertime-tp6-firefox-google-mail-e10s-loadtimenocondprof-cold-webrender
    browsertime-tp6-firefox-google-mail-e10s-loadtimenocondprof-webrender-warm
    browsertime-tp6-firefox-google-mail-e10s-SpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-google-mail-e10s-FirstVisualChangenocondprof-cold-webrender
    browsertime-tp6-firefox-google-mail-e10s-ContentfulSpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-google-mail-e10s-PerceptualSpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-google-mail-e10s-SpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-google-mail-e10s-FirstVisualChangenocondprof-webrender-warm
    browsertime-tp6-firefox-google-mail-e10s-ContentfulSpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-google-mail-e10s-videoRecordingStartnocondprof-webrender-warm
    browsertime-tp6-firefox-google-mail-e10s-PerceptualSpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-amazon-fis-e10s-dcfnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-amazon-fis-e10s-fcpnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-amazon-fis-e10s-fnbpaintnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-amazon-fis-e10s-loadtimenocondprof-fission-webrender-warm
    browsertime-tp6-firefox-amazon-fis-e10s-loadtimenocondprof-fission-cold-webrender
    browsertime-tp6-firefox-amazon-fis-e10s-LastVisualChangenocondprof-fission-cold-webrender
    browsertime-tp6-firefox-amazon-fis-e10s-SpeedIndexnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-amazon-fis-e10s-FirstVisualChangenocondprof-fission-webrender-warm
    browsertime-tp6-firefox-amazon-fis-e10s-LastVisualChangenocondprof-fission-webrender-warm
    browsertime-tp6-firefox-amazon-fis-e10s-ContentfulSpeedIndexnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-amazon-fis-e10s-PerceptualSpeedIndexnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-tumblr-e10s-dcfnocondprof-cold-webrender
    browsertime-tp6-firefox-tumblr-e10s-dcfnocondprof-webrender-warm
    browsertime-tp6-firefox-tumblr-e10s-fcpnocondprof-webrender-warm
    browsertime-tp6-firefox-tumblr-e10s-ContentfulSpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-tumblr-e10s-PerceptualSpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-amazon-e10s-dcfnocondprof-webrender-warm
    browsertime-tp6-firefox-amazon-e10s-fcpnocondprof-webrender-warm
    browsertime-tp6-firefox-amazon-e10s-fnbpaintnocondprof-webrender-warm
    browsertime-tp6-firefox-amazon-e10s-loadtimenocondprof-webrender-warm
    browsertime-tp6-firefox-amazon-e10s-loadtimenocondprof-cold-webrender
    browsertime-tp6-firefox-amazon-e10s-LastVisualChangenocondprof-cold-webrender
    browsertime-tp6-firefox-amazon-e10s-SpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-amazon-e10s-FirstVisualChangenocondprof-webrender-warm
    browsertime-tp6-firefox-amazon-e10s-LastVisualChangenocondprof-webrender-warm
    browsertime-tp6-firefox-amazon-e10s-ContentfulSpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-amazon-e10s-PerceptualSpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-imgur-e10s-dcfnocondprof-webrender-warm
    browsertime-tp6-firefox-imgur-e10s-SpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-imgur-e10s-PerceptualSpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-imgur-e10s-SpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-imgur-e10s-FirstVisualChangenocondprof-webrender-warm
    browsertime-tp6-firefox-imgur-e10s-LastVisualChangenocondprof-webrender-warm
    browsertime-tp6-firefox-imgur-e10s-PerceptualSpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-imgur-fis-e10s-fcpnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-imgur-fis-e10s-fnbpaintnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-imgur-fis-e10s-FirstVisualChangenocondprof-fission-cold-webrender
    browsertime-tp6-firefox-imgur-fis-e10s-SpeedIndexnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-imgur-fis-e10s-FirstVisualChangenocondprof-fission-webrender-warm
    browsertime-tp6-firefox-imgur-fis-e10s-LastVisualChangenocondprof-fission-webrender-warm
    browsertime-tp6-firefox-imgur-fis-e10s-PerceptualSpeedIndexnocondprof-fission-webrender-warm
Good tests (78):
    browsertime-tp6-firefox-twitter-e10s-fcpnocondprof-cold-webrender
    browsertime-tp6-firefox-twitter-e10s-fnbpaintnocondprof-cold-webrender
    browsertime-tp6-firefox-twitter-e10s-loadtimenocondprof-webrender-warm
    browsertime-tp6-firefox-twitter-e10s-FirstVisualChangenocondprof-cold-webrender
    browsertime-tp6-firefox-twitter-e10s-videoRecordingStartnocondprof-cold-webrender
    browsertime-tp6-firefox-twitch-e10s-dcfnocondprof-webrender-warm
    browsertime-tp6-firefox-twitch-e10s-PerceptualSpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-twitch-e10s-SpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-twitter-fis-e10s-fcpnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-twitter-fis-e10s-fnbpaintnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-twitter-fis-e10s-loadtimenocondprof-fission-webrender-warm
    browsertime-tp6-firefox-twitter-fis-e10s-FirstVisualChangenocondprof-fission-cold-webrender
    browsertime-tp6-firefox-twitter-fis-e10s-videoRecordingStartnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-twitter-fis-e10s-ContentfulSpeedIndexnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-google-mail-fis-e10s-dcfnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-google-mail-fis-e10s-fcpnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-google-mail-fis-e10s-fnbpaintnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-google-mail-fis-e10s-loadtimenocondprof-fission-cold-webrender
    browsertime-tp6-firefox-google-mail-fis-e10s-SpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-google-mail-fis-e10s-FirstVisualChangenocondprof-fission-cold-webrender
    browsertime-tp6-firefox-google-mail-fis-e10s-LastVisualChangenocondprof-fission-cold-webrender
    browsertime-tp6-firefox-google-mail-fis-e10s-ContentfulSpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-google-mail-fis-e10s-videoRecordingStartnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-google-mail-fis-e10s-PerceptualSpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-google-mail-fis-e10s-LastVisualChangenocondprof-fission-webrender-warm
    browsertime-tp6-firefox-twitch-fis-e10s-fcpnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-twitch-fis-e10s-fnbpaintnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-twitch-fis-e10s-loadtimenocondprof-fission-cold-webrender
    browsertime-tp6-firefox-twitch-fis-e10s-dcfnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-twitch-fis-e10s-FirstVisualChangenocondprof-fission-cold-webrender
    browsertime-tp6-firefox-twitch-fis-e10s-videoRecordingStartnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-twitch-fis-e10s-PerceptualSpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-twitch-fis-e10s-SpeedIndexnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-google-slides-fis-e10s-dcfnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-google-slides-fis-e10s-loadtimenocondprof-fission-cold-webrender
    browsertime-tp6-firefox-google-slides-fis-e10s-dcfnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-google-slides-fis-e10s-loadtimenocondprof-fission-webrender-warm
    browsertime-tp6-firefox-google-slides-fis-e10s-SpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-google-slides-fis-e10s-FirstVisualChangenocondprof-fission-cold-webrender
    browsertime-tp6-firefox-google-slides-fis-e10s-LastVisualChangenocondprof-fission-cold-webrender
    browsertime-tp6-firefox-google-slides-fis-e10s-PerceptualSpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-tumblr-fis-e10s-videoRecordingStartnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-google-slides-e10s-loadtimenocondprof-cold-webrender
    browsertime-tp6-firefox-google-slides-e10s-LastVisualChangenocondprof-cold-webrender
    browsertime-tp6-firefox-google-slides-e10s-videoRecordingStartnocondprof-cold-webrender
    browsertime-tp6-firefox-google-slides-e10s-SpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-google-slides-e10s-PerceptualSpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-google-mail-e10s-fcpnocondprof-cold-webrender
    browsertime-tp6-firefox-google-mail-e10s-fnbpaintnocondprof-cold-webrender
    browsertime-tp6-firefox-google-mail-e10s-dcfnocondprof-webrender-warm
    browsertime-tp6-firefox-google-mail-e10s-videoRecordingStartnocondprof-cold-webrender
    browsertime-tp6-firefox-amazon-fis-e10s-dcfnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-amazon-fis-e10s-SpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-amazon-fis-e10s-FirstVisualChangenocondprof-fission-cold-webrender
    browsertime-tp6-firefox-amazon-fis-e10s-ContentfulSpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-amazon-fis-e10s-videoRecordingStartnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-amazon-fis-e10s-PerceptualSpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-tumblr-e10s-FirstVisualChangenocondprof-cold-webrender
    browsertime-tp6-firefox-tumblr-e10s-ContentfulSpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-tumblr-e10s-PerceptualSpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-tumblr-e10s-LastVisualChangenocondprof-webrender-warm
    browsertime-tp6-firefox-amazon-e10s-dcfnocondprof-cold-webrender
    browsertime-tp6-firefox-amazon-e10s-fcpnocondprof-cold-webrender
    browsertime-tp6-firefox-amazon-e10s-fnbpaintnocondprof-cold-webrender
    browsertime-tp6-firefox-amazon-e10s-SpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-amazon-e10s-FirstVisualChangenocondprof-cold-webrender
    browsertime-tp6-firefox-amazon-e10s-ContentfulSpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-amazon-e10s-videoRecordingStartnocondprof-cold-webrender
    browsertime-tp6-firefox-amazon-e10s-PerceptualSpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-imgur-e10s-dcfnocondprof-cold-webrender
    browsertime-tp6-firefox-imgur-e10s-LastVisualChangenocondprof-cold-webrender
    browsertime-tp6-firefox-imgur-e10s-ContentfulSpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-imgur-fis-e10s-loadtimenocondprof-fission-webrender-warm
    browsertime-tp6-firefox-imgur-fis-e10s-LastVisualChangenocondprof-fission-cold-webrender
    browsertime-tp6-firefox-imgur-fis-e10s-ContentfulSpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-imgur-fis-e10s-videoRecordingStartnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-imgur-fis-e10s-PerceptualSpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-imgur-fis-e10s-videoRecordingStartnocondprof-fission-webrender-warm
Flags: needinfo?(gmierz2)
Flags: needinfo?(dave.hunt)
Flags: needinfo?(aerickson)

this is a great analysis- in some cases we just rebase our numbers and write the difference up as OS/hardware change. Since this is the same hardware, I wonder if there is something else going on with the OS installation (background processes, disk/cpu settings, graphics settings, etc.) which are giving us different results.

as we are on 18.04, I imagine we will be upgrading to 20.04 before too long for unittests- hopefully we can find what changes to make at the OS level so this is a simple upgrade.

I guess if we are running out of tweaks to make and the numbers are not great- what is the risk of switching? what is the risk of not switching?

Ubuntu 16.04 is unsupported in April of 2021. We'd like to not be using it at that point.

We're currently at ~180 (177) workers in the talos pool (up from 90 before the holidays, increased to make things smooth over break). MDC2 is closing down, so we're going to lose half of those hosts in mid-feb (mid-march at the absolute last, swept-clean date for MDC2 is 4/30).

Current 1604/1804 split:
MDC1:
12 1804
~90 1604
MDC2:
0 1804
~90 1604

:jmaher, if we did only have MDC1 with 90 workers, would 45 1604 talos workers be enough to handle load or do we need 90?

:sparky, how many hosts would reveal if the problems are solved? 45? 90?

What's the plan if 1804 proves to not be usable? Try 20.04?

I'd like to avoid having to create 1804 nodes in MDC2 as they're going away, but if we need the capacity during the migration (before the MDC2 closure) I guess we have no choice. I guess we should spin up more 1804 nodes and see if the problem is resolved.

Flags: needinfo?(jmaher)
Flags: needinfo?(gmierz2)
Flags: needinfo?(aerickson)

90 seems to be the minimum, otherwise we get complaints about backlogs. If we go lower than 90, I would like to be as little as possible (i.e. >=80). We have some flexibility now, could we do more 18.04 nodes in mdc1 as we have capacity of 16.04 in mdc2 ?

Flags: needinfo?(jmaher)

OK. Yeah, we can do them in MDC1 (if we're somewhat confident we're going to 1804 that makes sense).

:sparky, how many more 1804 hosts should I create and rerun tests on?

Are there any patterns for which metrics fall into bad/good? Given the focus on visual metrics, how do we look if we exclude non-visual metrics?

Flags: needinfo?(dave.hunt)

:aerickson, before answering your question about number of machines. I'm going to see how it looks when we only look at visual metrics. (Leaving ni? open for myself).

Looking at only vismet results, we're seeing a similar pattern. The number of bad tests is 23% higher than the number of good tests with an average increase in noise for the bad ones at 38-43% and a decrease in noise by 36-40%. It's really a case of "win some, lose some" here.

To get a fairer comparison, we would need to have pools of equal size for 1604 and 1804 - having 30 machines each would do the trick. That said, it's still possible that this won't help with the results and that we'll see the same thing.

However, I wonder if we should accept these changes in variance? We'll be working on improving our regression detection to be less sensitive to noise but more sensitive to trend changes which will help a lot here. :davehunt, what do you think?

For non-webrender:

Summary:
Average noise increases: 1.4364776438980529
Average noise decreases: 1.362573188732328
Bad tests (34):
browsertime-tp6-firefox-twitter-e10s-SpeedIndexnocondprof-cold
    browsertime-tp6-firefox-twitter-e10s-ContentfulSpeedIndexnocondprof-cold
    browsertime-tp6-firefox-twitter-e10s-PerceptualSpeedIndexnocondprof-cold
    browsertime-tp6-firefox-twitch-e10s-ContentfulSpeedIndexnocondprof-cold
    browsertime-tp6-firefox-twitch-e10s-SpeedIndexnocondprof-warm
    browsertime-tp6-firefox-twitch-e10s-ContentfulSpeedIndexnocondprof-warm
    browsertime-tp6-firefox-twitch-e10s-PerceptualSpeedIndexnocondprof-warm
    browsertime-tp6-firefox-twitter-fis-e10s-SpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-twitter-fis-e10s-ContentfulSpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-twitter-fis-e10s-PerceptualSpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-google-mail-fis-e10s-ContentfulSpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-google-mail-fis-e10s-SpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-google-mail-fis-e10s-PerceptualSpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-twitch-fis-e10s-ContentfulSpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-twitch-fis-e10s-SpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-twitch-fis-e10s-ContentfulSpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-twitch-fis-e10s-PerceptualSpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-tumblr-fis-e10s-SpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-tumblr-fis-e10s-ContentfulSpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-tumblr-fis-e10s-PerceptualSpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-google-slides-e10s-SpeedIndexnocondprof-cold
    browsertime-tp6-firefox-google-slides-e10s-ContentfulSpeedIndexnocondprof-cold
    browsertime-tp6-firefox-google-slides-e10s-PerceptualSpeedIndexnocondprof-cold
    browsertime-tp6-firefox-google-slides-e10s-SpeedIndexnocondprof-warm
    browsertime-tp6-firefox-google-slides-e10s-PerceptualSpeedIndexnocondprof-warm
    browsertime-tp6-firefox-google-mail-e10s-SpeedIndexnocondprof-cold
    browsertime-tp6-firefox-google-mail-e10s-ContentfulSpeedIndexnocondprof-cold
    browsertime-tp6-firefox-tumblr-e10s-SpeedIndexnocondprof-warm
    browsertime-tp6-firefox-tumblr-e10s-PerceptualSpeedIndexnocondprof-warm
    browsertime-tp6-firefox-imgur-fis-e10s-SpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-imgur-fis-e10s-PerceptualSpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-imgur-fis-e10s-SpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-imgur-fis-e10s-ContentfulSpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-imgur-fis-e10s-PerceptualSpeedIndexnocondprof-fission-warm
Good tests (27):
browsertime-tp6-firefox-twitter-e10s-SpeedIndexnocondprof-warm
    browsertime-tp6-firefox-twitter-fis-e10s-PerceptualSpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-google-mail-fis-e10s-ContentfulSpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-twitch-fis-e10s-PerceptualSpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-google-slides-fis-e10s-SpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-google-slides-fis-e10s-PerceptualSpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-google-slides-fis-e10s-SpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-google-slides-fis-e10s-ContentfulSpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-google-slides-fis-e10s-PerceptualSpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-tumblr-fis-e10s-SpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-tumblr-fis-e10s-ContentfulSpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-google-mail-e10s-PerceptualSpeedIndexnocondprof-cold
    browsertime-tp6-firefox-amazon-fis-e10s-SpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-amazon-fis-e10s-ContentfulSpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-amazon-fis-e10s-PerceptualSpeedIndexnocondprof-fission-cold
    browsertime-tp6-firefox-amazon-fis-e10s-SpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-amazon-fis-e10s-ContentfulSpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-amazon-fis-e10s-PerceptualSpeedIndexnocondprof-fission-warm
    browsertime-tp6-firefox-tumblr-e10s-SpeedIndexnocondprof-cold
    browsertime-tp6-firefox-tumblr-e10s-ContentfulSpeedIndexnocondprof-cold
    browsertime-tp6-firefox-tumblr-e10s-PerceptualSpeedIndexnocondprof-cold
    browsertime-tp6-firefox-amazon-e10s-SpeedIndexnocondprof-cold
    browsertime-tp6-firefox-amazon-e10s-ContentfulSpeedIndexnocondprof-cold
    browsertime-tp6-firefox-imgur-e10s-ContentfulSpeedIndexnocondprof-cold
    browsertime-tp6-firefox-imgur-e10s-PerceptualSpeedIndexnocondprof-cold
    browsertime-tp6-firefox-imgur-e10s-ContentfulSpeedIndexnocondprof-warm
    browsertime-tp6-firefox-imgur-e10s-PerceptualSpeedIndexnocondprof-warm

For webrender:

Average noise increases: 1.38286339170011
Average noise decreases: 1.4067248192688313
Bad tests (34):
browsertime-tp6-firefox-twitter-e10s-SpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-twitter-e10s-ContentfulSpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-twitter-e10s-PerceptualSpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-twitch-e10s-ContentfulSpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-twitch-e10s-ContentfulSpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-twitter-fis-e10s-SpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-twitter-fis-e10s-ContentfulSpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-twitter-fis-e10s-PerceptualSpeedIndexnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-google-slides-fis-e10s-ContentfulSpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-google-slides-fis-e10s-SpeedIndexnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-tumblr-fis-e10s-SpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-tumblr-fis-e10s-ContentfulSpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-tumblr-fis-e10s-PerceptualSpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-google-slides-e10s-ContentfulSpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-google-mail-e10s-SpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-google-mail-e10s-ContentfulSpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-google-mail-e10s-PerceptualSpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-google-mail-e10s-SpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-google-mail-e10s-ContentfulSpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-google-mail-e10s-PerceptualSpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-amazon-fis-e10s-SpeedIndexnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-amazon-fis-e10s-ContentfulSpeedIndexnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-amazon-fis-e10s-PerceptualSpeedIndexnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-tumblr-e10s-ContentfulSpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-tumblr-e10s-PerceptualSpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-amazon-e10s-SpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-amazon-e10s-ContentfulSpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-amazon-e10s-PerceptualSpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-imgur-e10s-SpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-imgur-e10s-PerceptualSpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-imgur-e10s-SpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-imgur-e10s-PerceptualSpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-imgur-fis-e10s-SpeedIndexnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-imgur-fis-e10s-PerceptualSpeedIndexnocondprof-fission-webrender-warm
Good tests (23):
browsertime-tp6-firefox-twitch-e10s-PerceptualSpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-twitch-e10s-SpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-twitter-fis-e10s-ContentfulSpeedIndexnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-google-mail-fis-e10s-SpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-google-mail-fis-e10s-ContentfulSpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-google-mail-fis-e10s-PerceptualSpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-twitch-fis-e10s-PerceptualSpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-twitch-fis-e10s-SpeedIndexnocondprof-fission-webrender-warm
    browsertime-tp6-firefox-google-slides-fis-e10s-SpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-google-slides-fis-e10s-PerceptualSpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-google-slides-e10s-SpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-google-slides-e10s-PerceptualSpeedIndexnocondprof-webrender-warm
    browsertime-tp6-firefox-amazon-fis-e10s-SpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-amazon-fis-e10s-ContentfulSpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-amazon-fis-e10s-PerceptualSpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-tumblr-e10s-ContentfulSpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-tumblr-e10s-PerceptualSpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-amazon-e10s-SpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-amazon-e10s-ContentfulSpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-amazon-e10s-PerceptualSpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-imgur-e10s-ContentfulSpeedIndexnocondprof-cold-webrender
    browsertime-tp6-firefox-imgur-fis-e10s-ContentfulSpeedIndexnocondprof-fission-cold-webrender
    browsertime-tp6-firefox-imgur-fis-e10s-PerceptualSpeedIndexnocondprof-fission-cold-webrender
Flags: needinfo?(gmierz2) → needinfo?(dave.hunt)

I'll spin up another 18 nodes (to get us to 30), retest, and report back.

(In reply to Greg Mierzwinski [:sparky] from comment #13)

However, I wonder if we should accept these changes in variance? We'll be working on improving our regression detection to be less sensitive to noise but more sensitive to trend changes which will help a lot here. :davehunt, what do you think?

I don't see that we have alternatives to accepting the changes if we've done all we reasonably can to minimise them. Let's see what the results look like with additional nodes, but I think we should be prepared to move forward. I'd be interested in seeing the impact of upgrading to 20.04. How do we decide which OS version to use?

Flags: needinfo?(dave.hunt)

How do we decide which OS version to use?

Linux telemetry is problematic, but I think we should try to use a distro that many users use. Ideally, that distro/release would be in support with the vendor also.

Here are the latest test results (with the 30 node 1804 cluster):

1804:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=91aa6f2b886e981841ac94640db92daa0028ccf1

1604:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f3d8a751c8915643d38aabe6386c23f53539759f

Flags: needinfo?(gmierz2)

Any updates?

Revisiting my comments in https://bugzilla.mozilla.org/show_bug.cgi?id=1681175#c8, we're going to lose half of the current talos workers mid-Feb (delayable to mid-March at the very latest) due to the MDC2 closure. Ideally we'd have some time to let the worker pool change ride the trees and avoid uplifts.

We should accept these changes as is. As I mentioned above, our upcoming work on regression detection should counter the increases in variance so we're not concerned about it.

Flags: needinfo?(gmierz2)
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: