We are seeing some performance spikes when tests are running on macosx1014-64-shippable during weekend time.

We identified this as a weekend run and we think it's caused by a low load on the devices.

as an example see alerts like

30-32 August:
https://treeherder.mozilla.org/perf.html#/alerts?id=22899
https://treeherder.mozilla.org/perf.html#/alerts?id=22906

17-20 August
https://treeherder.mozilla.org/perf.html#/alerts?id=22520
https://treeherder.mozilla.org/perf.html#/alerts?id=22532

Can we investigate these spikes and maybe can come up with a solution?

7-9 August
https://treeherder.mozilla.org/perf.html#/alerts?id=22990
https://treeherder.mozilla.org/perf.html#/alerts?id=23011

Anthony are you aware of any performance impact on macosx1014-64-shippable during the weekend period?

As you can see our tests are more stable during weekend periods.

The only explanation for this behavior is less load on the hw that is used to run the tests.

7-8 sep, 14-15 sep and basically all the latest weekends.
https://treeherder.mozilla.org/perf.html#/alerts?id=23055
https://treeherder.mozilla.org/perf.html#/alerts?id=23081

:fubar maybe you can help with some info on this topic:

We are searching for any info that could hep determine this difference in performance.

Do you have some information about:
Load on these machines during weekend/week days ?
Distribution of tests during the weekends?
Do we use a smaller group of machines during the weekend?
CPU temp statistics on these nodes?

Dave, can you help Florin out? The size of the pool doesn't change, and load during the week shouldn't matter as we're not running multiple jobs on a given node. However, overall test load in Taskcluster does drop off on the weekends.

I'll start collecting cpu temp and usage metrics, and I'll research if we can check for thermal throttling.

We don't have anything changing on the machine OS or environment from one day to the next. So the only change is the tasks that run. Because we have more tasks executed on week days, it is likely a symptom of continuous task execution during the week days that is causing the difference with the week ends when the workers are often idle. I think some of those symptoms could be:

1. temperature (like you asked about; thermal throttling)
2. OS services performing maintenance/etc outside of task runs (instead of during them) when the machine is idle or right after start-up (perhaps the tasks on the weekend are executed after the machine has booted and sat idle for an hour .. and so reached a stable idle state).
3. network performance affecting cpu load (perhaps some processes are waiting/etc because of slower network performance and are using cpu cycles)

Also, are these the same tests and same tested code being compared? I'm assuming these are the same tests but testing the changing code. Perhaps this is showing something about that code state at those times and not the tests? (long-shot, but still)

We do allow Apple's services to run and so those could be performing different checks during the weekends. However, if that were true then I think other professional users of MacOS would report the issue.

21-22, 28-29 Sept, 5-6, 12-13 Oct:
https://treeherder.mozilla.org/perf.html#/alerts?id=23371
https://treeherder.mozilla.org/perf.html#/alerts?id=23462

21-22, 28-29 Sept, 5-6, 12-13, 19-20 Oct:
https://treeherder.mozilla.org/perf.html#/alerts?id=23503
https://treeherder.mozilla.org/perf.html#/alerts?id=23511

One thing that I noticed when trying to make the Raptor pageload tests wait for the OS to be idle (Bug 1589356) (rather than a fixed duration) is that the Mac's seem to be in a state with a lot of disk activity.

[task 2019-10-21T19:40:30.858Z] 19:40:30     INFO -  raptor-control-server Info: CPU use: 22.3%
[task 2019-10-21T19:40:30.859Z] 19:40:30     INFO -  raptor-control-server Info: disk reads: 5283
[task 2019-10-21T19:40:30.859Z] 19:40:30     INFO -  raptor-control-server Info: disk writes: 873
[task 2019-10-21T19:40:30.859Z] 19:40:30     INFO -  raptor-control-server Info: AJC - is_os_idle: false

https://taskcluster-artifacts.net/cKXQK9cKR463C6tYT89Ytg/0/public/logs/live_backing.log

Note the high disk reads (I haven't seen this on other platforms).

One hypothesis is that the previous test is affecting the current one.

Perhaps the tasks are scheduled too closely together?

I'll see if I can find out which processes are responsible for all that disk access.

== Change summary for alert #23647 (as of Fri, 01 Nov 2019 06:24:00 GMT) ==

Regressions:

14% raptor-tp6-imdb-firefox fcp macosx1014-64-shippable opt 461.13 -> 527.38

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=23647

== Change summary for alert #23680 (as of Mon, 04 Nov 2019 19:29:37 GMT) ==

Regressions:

28% raptor-tp6-ebay-firefox fcp macosx1014-64-shippable opt 482.06 -> 618.75
25% raptor-tp6-ebay-firefox macosx1014-64-shippable opt 542.15 -> 676.10
20% raptor-tp6-binast-instagram-firefox macosx1014-64-shippable opt 332.40 -> 400.18
20% raptor-tp6-ebay-firefox loadtime macosx1014-64-shippable opt 645.42 -> 773.62
9% raptor-tp6-binast-instagram-firefox loadtime macosx1014-64-shippable opt 680.12 -> 742.92

Improvements:

28% raptor-tp6-binast-instagram-firefox fcp macosx1014-64-shippable opt 295.25 -> 212.50
27% raptor-tp6-ebay-firefox fcp macosx1014-64-shippable opt 620.54 -> 455.29
24% raptor-tp6-ebay-firefox macosx1014-64-shippable opt 680.17 -> 516.30
24% raptor-tp6-paypal-firefox loadtime macosx1014-64-shippable opt 500.54 -> 381.79
22% raptor-tp6-instagram-firefox fcp macosx1014-64-shippable opt 287.17 -> 222.92
22% raptor-tp6-paypal-firefox loadtime macosx1014-64-shippable opt 491.58 -> 382.96
22% raptor-tp6-binast-instagram-firefox macosx1014-64-shippable opt 400.23 -> 312.70
20% raptor-tp6-ebay-firefox loadtime macosx1014-64-shippable opt 776.08 -> 619.29
19% raptor-tp6-bing-firefox macosx1014-64-shippable opt 277.22 -> 224.16
19% raptor-tp6-bing-firefox loadtime macosx1014-64-shippable opt 274.42 -> 222.54
18% raptor-tp6-instagram-firefox macosx1014-64-shippable opt 412.33 -> 336.41
18% raptor-tp6-bing-firefox fcp macosx1014-64-shippable opt 294.92 -> 240.67
17% raptor-tp6-apple-firefox loadtime macosx1014-64-shippable opt 692.33 -> 577.38
15% raptor-tp6-paypal-firefox macosx1014-64-shippable opt 384.04 -> 326.96
14% raptor-tp6-paypal-firefox macosx1014-64-shippable opt 381.84 -> 328.99
14% raptor-tp6-google-mail-firefox fcp macosx1014-64-shippable opt 442.23 -> 381.58
14% raptor-tp6-imdb-firefox loadtime macosx1014-64-shippable opt 1,253.08 -> 1,082.08
13% raptor-tp6-google-mail-firefox macosx1014-64-shippable opt 490.13 -> 428.17
12% raptor-tp6-instagram-firefox loadtime macosx1014-64-shippable opt 899.38 -> 790.38
12% raptor-tp6-instagram-firefox loadtime macosx1014-64-shippable opt 909.88 -> 801.42
12% raptor-tp6-paypal-firefox fcp macosx1014-64-shippable opt 360.06 -> 318.38
11% raptor-tp6-apple-firefox macosx1014-64-shippable opt 571.84 -> 507.60
11% raptor-tp6-reddit-firefox fcp macosx1014-64-shippable opt 399.21 -> 355.50
11% raptor-tp6-binast-instagram-firefox loadtime macosx1014-64-shippable opt 738.42 -> 658.12

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=23680

== Change summary for alert #23979 (as of Fri, 15 Nov 2019 14:15:51 GMT) ==

Regressions:

27% raptor-tp6-binast-instagram-firefox fcp macosx1014-64-shippable opt 224.69 -> 285.21
20% raptor-tp6-binast-instagram-firefox macosx1014-64-shippable opt 321.13 -> 386.53

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=23979

== Change summary for alert #24112 (as of Sun, 24 Nov 2019 07:41:49 GMT) ==

Regressions:

31% raptor-tp6-ebay-firefox fcp macosx1014-64-shippable opt 468.94 -> 612.75
27% raptor-tp6-ebay-firefox macosx1014-64-shippable opt 528.24 -> 668.77
23% raptor-tp6-instagram-firefox fcp macosx1014-64-shippable opt 241.46 -> 296.75
22% raptor-tp6-paypal-firefox loadtime macosx1014-64-shippable opt 404.17 -> 495.04
22% raptor-tp6-ebay-firefox loadtime macosx1014-64-shippable opt 628.23 -> 763.54
19% raptor-tp6-instagram-firefox macosx1014-64-shippable opt 353.57 -> 420.09
14% raptor-tp6-imdb-firefox loadtime macosx1014-64-shippable opt 1,062.23 -> 1,215.33
14% raptor-tp6-imdb-firefox macosx1014-64-shippable opt 561.01 -> 639.62
14% raptor-tp6-paypal-firefox macosx1014-64-shippable opt 332.86 -> 379.50
14% raptor-tp6-imdb-firefox fcp macosx1014-64-shippable opt 466.60 -> 530.50

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=24112

== Change summary for alert #24571 (as of Sat, 21 Dec 2019 09:36:43 GMT) ==

Regressions:

41% raptor-tp6-binast-instagram-firefox fcp macosx1014-64-shippable opt 202.17 -> 285.46
29% raptor-tp6-binast-instagram-firefox macosx1014-64-shippable opt 295.65 -> 380.99
7% raptor-tp6-binast-instagram-firefox loadtime macosx1014-64-shippable opt 633.21 -> 680.62

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=24571

== Change summary for alert #24564 (as of Fri, 20 Dec 2019 17:57:46 GMT) ==

Improvements:

18% raptor-tp6-binast-instagram-firefox macosx1014-64-shippable opt 363.82 -> 297.24
9% raptor-tp6-binast-instagram-firefox loadtime macosx1014-64-shippable opt 671.54 -> 610.92

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=24564

== Change summary for alert #25518 (as of Mon, 30 Mar 2020 22:17:31 GMT) ==

Improvements:

19% raptor-tp6-binast-instagram-firefox fcp macosx1014-64-shippable opt 290.77 -> 234.17
16% raptor-tp6-binast-instagram-firefox macosx1014-64-shippable opt 386.18 -> 326.04
14% raptor-tp6-outlook-firefox-cold fcp macosx1014-64-shippable opt 324.83 -> 280.42
12% raptor-tp6-outlook-firefox-cold macosx1014-64-shippable opt 438.42 -> 384.68
11% raptor-tp6-google-firefox-cold loadtime macosx1014-64-shippable opt 1,745.25 -> 1,553.42
9% raptor-tp6-google-firefox-cold macosx1014-64-shippable opt 626.52 -> 568.72

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=25518

== Change summary for alert #25505 (as of Fri, 27 Mar 2020 18:27:39 GMT) ==

Improvements:

14% raptor-tp6-bing-firefox-cold loadtime macosx1014-64-shippable opt 564.55 -> 486.17
14% raptor-tp6-bing-firefox-cold macosx1014-64-shippable opt 410.21 -> 354.13
12% raptor-tp6-bing-firefox-cold fcp macosx1014-64-shippable opt 393.92 -> 346.67

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=25505

== Change summary for alert #25717 (as of Sat, 25 Apr 2020 22:01:55 GMT) ==

Regressions:

23% raptor-tp6-bing-firefox-cold loadtime macosx1014-64-shippable opt 491.71 -> 606.33
19% raptor-tp6-binast-instagram-firefox fcp macosx1014-64-shippable opt 255.02 -> 303.79
15% raptor-tp6-binast-instagram-firefox macosx1014-64-shippable opt 346.26 -> 397.22
14% raptor-tp6-microsoft-firefox-cold loadtime macosx1014-64-shippable opt 1,127.12 -> 1,286.50
11% raptor-tp6-microsoft-firefox-cold macosx1014-64-shippable opt 758.05 -> 841.06

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=25717

== Change summary for alert #25861 (as of Thu, 07 May 2020 22:21:29 GMT) ==

Improvements:

13% raptor-tp6-google-mail-firefox-cold macosx1014-64-shippable opt 424.44 -> 370.15
12% raptor-tp6-google-mail-firefox-cold fcp macosx1014-64-shippable opt 359.62 -> 315.33

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=25861

== Change summary for alert #25890 (as of Mon, 11 May 2020 04:50:57 GMT) ==

Regressions:

15% raptor-tp6-outlook-firefox-cold macosx1014-64-shippable opt 381.68 -> 440.49
13% raptor-tp6-outlook-firefox-cold fcp macosx1014-64-shippable opt 290.96 -> 329.08
11% raptor-tp6-microsoft-firefox-cold macosx1014-64-shippable opt 739.27 -> 818.11
10% raptor-tp6-microsoft-firefox-cold fcp macosx1014-64-shippable opt 680.75 -> 746.17

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=25890

== Change summary for alert #25917 (as of Wed, 13 May 2020 11:20:45 GMT) ==

Regressions:

15% raptor-tp6-microsoft-firefox-cold loadtime macosx1014-64-shippable opt 1,110.08 -> 1,277.50
12% raptor-tp6-instagram-firefox-cold macosx1014-64-shippable opt 572.42 -> 642.19
12% raptor-tp6-google-firefox-cold loadtime macosx1014-64-shippable opt 1,619.75 -> 1,806.50
11% raptor-tp6-microsoft-firefox-cold macosx1014-64-shippable opt 758.14 -> 843.84
10% raptor-tp6-microsoft-firefox-cold fcp macosx1014-64-shippable opt 692.79 -> 764.33

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=25917

== Change summary for alert #25985 (as of Mon, 18 May 2020 16:50:10 GMT) ==

Improvements:

14% raptor-tp6-google-mail-firefox-cold macosx1014-64-shippable opt 428.10 -> 366.76
14% raptor-tp6-google-mail-firefox-cold fcp macosx1014-64-shippable opt 362.54 -> 311.58
10% raptor-tp6-google-mail-firefox-cold loadtime macosx1014-64-shippable opt 960.75 -> 869.17

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=25985

== Change summary for alert #25992 (as of Tue, 19 May 2020 11:48:29 GMT) ==

Regressions:

16% raptor-tp6-google-mail-firefox-cold macosx1014-64-shippable opt 370.19 -> 429.35
15% raptor-tp6-google-mail-firefox-cold fcp macosx1014-64-shippable opt 314.77 -> 363.00
10% raptor-tp6-google-mail-firefox-cold loadtime macosx1014-64-shippable opt 875.92 -> 967.67

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=25992

== Change summary for alert #26010 (as of Wed, 20 May 2020 11:50:34 GMT) ==

Improvements:

9% raptor-tp6-apple-firefox-cold loadtime macosx1014-64-shippable opt 1,820.67 -> 1,663.58

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=26010

:bebe no comments since May, do we still see these spikes?

Nope things look peaty stable now closing this bug and will reopen it if we see any other issues.

Great, clearing needinfos.