We may run into cases where we need a new virtualenv module, like bug 907895, where we need to land a patch on every branch (including aurora, beta, release) to avoid bustage. If the mozharness scripts got their list of modules from the in-tree mozharness config, we wouldn't need to do this. http://mxr.mozilla.org/mozilla-central/source/testing/config/mozharness_config.py

I kind of like an in-tree/in-test-zip requirements.txt, or a master mozbase setup.py or install script that either dynamically or statically adds the setup.py's from the subdirs. IMO we might as well install all of the available mozbase modules every time to make things simpler. In-tree virtualenv_modules works too.

mozbase does have a master install script: https://github.com/mozilla/mozbase/blob/master/setup_development.py fwiw

In-tree requirements.txt files FTW. We want as much information as possible to live in the tree. We ideally want in-tree installs to "pin" package versions so we run with a consistent set of package versions no matter when we run the job.

(In reply to Jeff Hammel [:jhammel] from comment #2) > mozbase does have a master install script: > https://github.com/mozilla/mozbase/blob/master/setup_development.py fwiw I wouldn't mind having a mozilla-specific install_mozbase() mozharness method that calls this. It would have to live outside virtualenv_modules, but assuming this installs everything from the test zip, I think that would be sufficient. (In reply to Gregory Szorc [:gps] from comment #3) > In-tree requirements.txt files FTW. We want as much information as possible > to live in the tree. We ideally want in-tree installs to "pin" package > versions so we run with a consistent set of package versions no matter when > we run the job. I haven't yet figured out how to get the requirements.txt to install from disk rather than trying to hit pypi or whatever --find-links urls, but if we figure it out, great.

> I wouldn't mind having a mozilla-specific install_mozbase() mozharness method > that calls this. It would have to live outside virtualenv_modules, but > assuming this installs everything from the test zip, I think that would be > sufficient. This seems like the least fragile way to handle it, as long as setup_development.py always installs things in dependent order, so that we don't end up trying to hit pypi.

setup_development does follow dependency order; i'm not 100% sure it doesn't hit net/pypi, but shouldnt be hard to fix. TBH, I'm not necessarily for or against this vs a requirements file.

--no-deps is an option to work around hitting pypi if needed.

Installing packages in the right order is a tangential issue (see bug 879765). I proposed a two-pass install algorithm using --no-deps in https://bugzilla.mozilla.org/show_bug.cgi?id=879765#c7. This issue is more about having the ability to require packages on a per branch basis. A requirements.txt has the advantage of allowing us to do this for non-mozbase modules, so my vote would be for that. While calling setup_development.py would solve the immediate issue, it doesn't let us say "only try to install psutil on mozilla-central"..

Makes sense; we'll undoubtedly have the case at some point where we need package X (from in-tree sources) in some trees but not others.

This patch implements support for ahal's idea of a 2-pass install of pip requirements files, so that package order isn't important.

Gecko patch with Marionette and mozbase requirements files, plus a change to package them into tests.zip

Comment on attachment 799122 [details] [diff] [review] In-tree reuqirements files for marionette and mozbase, Could you do Try run with this after landing/merging the mozharness patch? And/or an ash run. I'll also take a build peer review. Asking 'cause I'm somewhat stamping here.

Comment on attachment 799122 [details] [diff] [review] In-tree reuqirements files for marionette and mozbase, Sure, asking for gps' review too. I guess the safest thing to do here is land on ash/ash-mozharness, so I'll do that.

I just landed this on ash/ash-mozharness: https://tbpl.mozilla.org/?tree=Ash&rev=3d2c44a9933c

Missed mozprofile in the requirements file; running again on ash: https://tbpl.mozilla.org/?tree=Ash&rev=0d4ae6057ef5

Comment on attachment 800271 [details] [diff] [review] In-tree reuqirements files for marionette and mozbase, Carry r+ forward.

ash looks good; landing the gecko part of this: https://hg.mozilla.org/integration/mozilla-inbound/rev/8b4a47eb1217

Cool, so does this mean bug 879765 is fixed (for both lists of modules and requirements.txt files)?

merged to production: http://hg.mozilla.org/build/mozharness/rev/312852646d08

(In reply to Andrew Halberstadt [:ahal] from comment #20) > Cool, so does this mean bug 879765 is fixed (for both lists of modules and > requirements.txt files)? This only affects requirements files, not static lists of modules.