Out of the box, browsertime supports cold pageload tests. This ticket tracks extending to warm pageload tests. The way this works in browsertime is that one runs a Selenium Node.js script to drive the additional loads.

nalexander and acreskey both have versions of this.

Tarek: it's probably worth implementing this so that we can run more cycles, more quickly, while we align the two harnesses. (Otherwise we wait a long time for each cold load run to start.)

Locally this runs through a Node.js script, like:

// From https://github.com/acreskeyMoz/browsertime_tools/blob/ecde1ad8f55fdbc6be95188481951f05a07aaa85/reload.js.
module.exports = async function(context, commands) {
    let url = context.options.perftest.url;
    let reloads = context.options.perftest.reloads;
    await commands.measure.start(url, 'cold load');

    // According to the docs you need to prepend a dummy parameter to test the same page in browsertime:
    // https://www.sitespeed.io/documentation/sitespeed.io/scripting/
    let dummy = url.indexOf('?') == -1 ? '/?dummy' : '&dummy';
    for (let count = 0; count < reloads; count++) {
        await commands.wait.byTime(3000);
        await commands.measure.start(url + dummy + count, 'reload' + count);
    }
    return true;
};

Then we'll do something like

bin/browsertime.js ... warmload.js --perftest.url='https://example.com' --perftest.reloads=5

I haven't tested this locally in a long, long time so you'll have to investigate.

Notes:

1. Raptor has browser_cycles and page_cycles. This will be page_cycles.
2. Bonus points for handling both browser and page cycles.
3. You can differentiate coldload from warmload with test.get('cold').
4. I expect that result processing will need to accommodate differently shaped browsertime.json due to only having one Browsertime invocation with multiple results in the single run.

I believe that the dummy parameters interfere with Necko caching, but I'm not aware of what the best option is here. acreskey, do you have an opinion on what we should do for warmload testing?

I'm not sure why the sitespeed.io documentation says "testing the same site multiple times will break".

I had started off with the dummy url method but I'm sure it does break Necko caching and it does indeed produce a differently-shaped browsertime.json which isn't great.

So I went to a method that's closer to what Denis was using:
(Transcribed here with the automation variables)

module.exports = async function(context, commands) {
    let url = context.options.perftest.url;
    let reloads = context.options.perftest.reloads;
    
    await commands.navigate('https://www.example.com');
    await commands.wait.byTime(30000); // magic number
    
    // Initial cold load and wait
    await commands.navigate(url);
    await commands.wait.byTime(10000); // magic number

    // Warm loads
    for (let count = 0; count < reloads; count++) {
        await commands.wait.byTime(3000);  // magic number
        await commands.measure.start(url);
    }
    return true;
};

This one is throwing away the results of the cold load.

The browsertime.json shape is identical to the cold load, so the process.py and R scripts all work out of the box.

Support warmload tests under --browsertime

Nick, here's a minimal diff that runs multiple page cycles and grabs them in the results - seems to work. I think we can have a single bt script for both cold/warm tests, but we should talk about it before I go further here.

try runs

with bt: https://treeherder.mozilla.org/#/jobs?repo=try&revision=e9b16091ee8c3243920789c5c7b7fbf2d0809134
without bt: https://treeherder.mozilla.org/#/jobs?repo=try&revision=8c6d146a67ce9c52b2ae2de7a38ba49006572d99

https://hg.mozilla.org/mozilla-central/rev/9eb8f5738e31

This landed! Clearing NI.

I must have made a mistake in Comment 3 -- although that script works for warm loads the shape of the json is quite different.
The cold load format has all the iterations within the first element's browserscripts.
This warm load format creates new root elements for each warm load.

Denis, did you happen to have a warm load script that gives browsertime.json in the same shape as the default cold loads?
I was experimenting with the measure command but wasn't successful.