Summary

Investigate the feasibility of creating a pre-packaged virtualenv for each platform, such that it will be downloaded and decompressed by the test machine instead.

Data

Each chunk per push uses up anywhere between 1-3 minutes setting up the virtual environment.

Provided that the pre-packaged virtualenv archive is of reasonable size, a case could be made that it is faster to download the archive, decompress on the machine then activate the virtualenv.

Challenges

Several challenges:

• how to upload generated virtualenv archive to storage?
• how to rewrite harness to download and use archived virtualenv?
• when to regenerate the virtualenv?

:catlee - I have spent the last week tackling the problem, and I've run into some difficulties. I am hoping you can point me in the right direction.

So far, I have done the following:

• defined a new task in toolchain
• define artifacts to be generated
• defined a Dockerfile to be used for virtualenv generation
• defined a script to install virtualenv and other dependencies
• defined a script to run pip install packages

I've had reasonable success with a proof of concept: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=8038de9f4b6c7d48e9eda57470b762093fa523b5

Where I'm running into trouble is as follows:

In my Dockerfile, I add topsrcdir/testing to the image. This is to permit access to files like testing/config/mozbase_requirements.txt, which will be consumed with pip install -r testing/config/mozbase_requirements.txt. But despite building the docker image locally and confirming that topsrcdir/testing is available in the image, in the CI environment it cannot be accessed.

This means I am unable to generate a useful virtualenv for tests. One workaround is to define manually the packages required, but this is unsightly and unmaintainable.

For reference, this is the patch (lots of other experimental attempts in there as well): https://hg.mozilla.org/try/rev/b636499640c187fd4a8fe90f49f2b71919beb154

Any pointers and/or help would be appreciated.

Sorry, I'm not sure how to help.

Let's see in H1 how we can improve this.

Could we install all the needed packages directly in the Docker image used by the test tasks?

(In reply to Marco Castelluccio [:marco] from comment #4)

Could we install all the needed packages directly in the Docker image used by the test tasks?

That's an idea and while it won't provide the same amount of gains as prepping a virtualenv package, it is probably the easier solution at the expense of increasing the size of the docker image somewhat.

As an aside, I do have ideas for the other task in task efficiencies (bug 1598055) to reduce the size of the docker image by a good amount, so whatever increase in size we see from preinstalling and generating the virtualenv in the docker image can likely be offset.

(In reply to Edwin Takahashi (:egao) from comment #5)

(In reply to Marco Castelluccio [:marco] from comment #4)

Could we install all the needed packages directly in the Docker image used by the test tasks?

That's an idea and while it won't provide the same amount of gains as prepping a virtualenv package, it is probably the easier solution at the expense of increasing the size of the docker image somewhat.

Why do you think a pre-packaged virtual env would be faster? I was thinking the Docker solution would actually be more or less the same in terms of performance.

I've thought of this problem few times in the past.
I've never understood why pip installation and activation (even with a cache) would be so slow but it is.

(In reply to Marco Castelluccio [:marco] from comment #6)

(In reply to Edwin Takahashi (:egao) from comment #5)

(In reply to Marco Castelluccio [:marco] from comment #4)

Could we install all the needed packages directly in the Docker image used by the test tasks?

That's an idea and while it won't provide the same amount of gains as prepping a virtualenv package, it is probably the easier solution at the expense of increasing the size of the docker image somewhat.

Why do you think a pre-packaged virtual env would be faster? I was thinking the Docker solution would actually be more or less the same in terms of performance.

I would expect either approach would be roughly the same.

Docker or not; that would only deal with one platform.

(In reply to Marco Castelluccio [:marco] from comment #6)

(In reply to Edwin Takahashi (:egao) from comment #5)

(In reply to Marco Castelluccio [:marco] from comment #4)

Could we install all the needed packages directly in the Docker image used by the test tasks?

That's an idea and while it won't provide the same amount of gains as prepping a virtualenv package, it is probably the easier solution at the expense of increasing the size of the docker image somewhat.

Why do you think a pre-packaged virtual env would be faster? I was thinking the Docker solution would actually be more or less the same in terms of performance.

Looping back into this topic, I don't think it will be faster to install the packages into the docker image necessarily. It is easier to do that, though, since I don't have to create a new task, populate the virtualenv then upload it as artifact.

As :armenzg said though, that would only deal with one platform, so for Windows and macOS the virtualenv would still need to be populated.

I think I can take this task on in H2 2020 provided the goals align with company-wide goals and I get manager approval.

The bug assignee didn't login in Bugzilla in the last 7 months.
:ahal, could you have a look please?
For more information, please visit auto_nag documentation.

Skimming through the bug, it's not clear how anyone would pick this up again. I think the idea might still have merit, but the original motiviations for it are unclear. Going to resolve and we can file a new bug if we think this approach warrants another look.