When looking at any of the Speedometer2 tests that contain "async" in their name: https://arewefastyet.com/#machine=35&view=single&suite=speedometer-misc (win64) https://arewefastyet.com/#machine=17&view=single&suite=speedometer-misc (win32) https://arewefastyet.com/#machine=36&view=single&suite=speedometer-misc It appears that the performance on Firefox is very flaky and has a lot of variance, while the performance on Chromium is very stable. This suggests a bug on our side. I haven't profiler these tests, nor investigated: first ideas which come to mind are GC or the event loop handling.

This is basically bug 1366803. A number of bugs were fixed to run stuff during idle time, but there's more to be done here.

FWIW, we don't have enough idle time to run GC/CC during idle period when executing Speedometer. I'm planning to try to increase NS_INTERSLICE_GC_DELAY and kICCIntersliceDelay to see if that would give a bit better chance to use idle time, but I doubt that will help much.

I hope nursery allocated strings and the GC scheduler will help.. Do you know what's triggering the GCs on Speedometer?

mostly GC alloc, and also some CC_WAITING

Should we dupe it against bug 1366803?

I have been using this patch to mark the async test periods as a UserTiming area in the profiler UI. It is helpful for profiling this period exclusively.

Markus noticed that the backbone tests seem to run into issues where a timer that the test schedules may end up running at within the async period: http://bit.ly/2tOFUwU The AWFY graphs seem to suggest that this test is a source of noise in the benchmark for us. This might explain why.

(In reply to :Ehsan Akhgari (needinfo please, extremely long backlog) from comment #7) > Markus noticed that the backbone tests seem to run into issues where a timer > that the test schedules may end up running at within the async period: > http://bit.ly/2tOFUwU > > The AWFY graphs seem to suggest that this test is a source of noise in the > benchmark for us. This might explain why. Note, chrome current clamps all setTimeout() calls to a minimum of 1ms. So a setTimeout(f, 0) in chrome will run later than for us. I'm not sure we want to do that, but perhaps it helps them here.

Tagging this with [qf] as async tests on speedometer still have a lot of variance: https://arewefastyet.com/#machine=35&view=breakdown&suite=speedometer-misc

I ran these tests locally through the raptor framework, and yes, the async tests show a relative standard deviation that is many times that of the sync tests. As a quick experiment, I implemented a 1ms clamp on the timeout intervals (https://phabricator.services.mozilla.com/D10946). I believe this is correctly implemented. However when running with this change I did not see a significant reduction in variance on the async tests.