There are some things which we can do to make our test set up faster. * Bug 1212993 - Reduce hidden Buildbot lag * Bug 1209112 - Virtualenv cache always gets clobbered * Switch to a checkout model for running tests of Gecko ** This would remove the download_and_extract action of Mozharness ** This would reduce packaging and upload cost on the build side ** This would reduce download and extracting cost on the test side Metrics: * Bug 1212500 will give us timings for Buildbot steps * We need timings for runner (pre-Buildbot - outside of logs) ** I believe this is reported to influxDB ** Not all platforms have runner support * Determine end-of-job to next job cost ** This could be determine by listening to the non-normalize Buildbot pulse exchange There is more things to file but I will file this for now.

* Move more things that are currently in pypi into the tree (e.g. blobberuploader, mozinstall) - this seems like low-hanging fruit

(In reply to James Graham [:jgraham] from comment #1) > * Move more things that are currently in pypi into the tree (e.g. > blobberuploader, mozinstall) - this seems like low-hanging fruit Filed: Bug 1213052 - Analyze what we could improve in the virtualenv action

Updated summary: * https://bugzil.la/1209112 - Virtualenv cache always gets clobbered * Switch to a checkout model for running tests * https://bugzil.la/1213052 - Analyze what we could improve in the virtualenv action * https://bugzil.la/1220108 - Switch to wheels? * https://bugzil.la/1213040 - Improve Windows file removal * Reduce the number of Buildbot steps (do them within Mozharness)