Currently when we use the in-tree mozbase packages we need to be careful that we install all the packages in the right order. E.g if we install mozcrash before we install mozlog, pip install will grab mozlog from puppetagain instead of the test package. While we can just control the order, it will be very easy to add new cross-dependencies to mozbase without realizing it could cause unexpected behaviour in mozharness. I can think of two ways to fix this: A) Give create_virtualenv the option to pass in --no-deps to pip and leave it up to the mozharness script to define all the dependencies via "virtualenv_modules". At least this way if dependencies change mozharness will fail rather than silently do something else. B) Modify create_virtualenv to search the PATH + sibling directories for dependencies before resorting to puppetagain. This has the advantage of not breaking backwards compatibility.

If mozbase's setup_development.py were a pip-friendly setup.py, we might be able to completely bypass this. I'm not sure if that's easily doable.

If we create a generic self.install_mozbase_from_path() method, maybe it belongs in a MozillaVirtualenvMixin that inherits VirtualenvMixin and adds this extra method. Then we can replace VirtualenvMixin inheritance with MozillaVirtualenvMixin inheritance and we don't increment the number of mixins we inherit. (Maybe I'm worrying about the wrong things.)

(In reply to Aki Sasaki [:aki] from comment #1) > If mozbase's setup_development.py were a pip-friendly setup.py, we might be > able to completely bypass this. > > I'm not sure if that's easily doable. See https://bugzilla.mozilla.org/show_bug.cgi?id=736086 ; in short, a setup.py could accomplish this, but this doesn't work (AFAIK, still) with `python setup.py develop`: https://bitbucket.org/tarek/distribute/issue/177/setuppy-develop-doesnt-support-package_dir .

For documentation, see http://docs.python.org/2/distutils/setupscript.html#listing-whole-packages

My personal recommendation is to do something like: 1. have https://github.com/mozilla/mozbase/blob/master/setup_development.py have a flag to create a pip compatible requirements.txt file (see http://www.pip-installer.org/en/latest/logic.html#requirements-file-format ) that will point to the various mozbase packages in installation order, since setup_development.py already has the logic to unroll dependencies: https://github.com/mozilla/mozbase/blob/master/setup_development.py#L110 . 2. I think we already add setup_development.py to tests.zip: http://mxr.mozilla.org/mozilla-central/source/testing/mozbase/Makefile.in#30 ; if not it can be added 3. mozharness invokes setup_development.py to generate the unrolled requirements; then in the next step mozharness creates the virtualenv with the requirements.txt A few caveats with this approach mostly dealing with how complicated it is: A. This approach assumes that there is a reason not to just run setup_development.py from the virtualenv; of course, if this is doable, this IMHO would be the easiest B. Step 1 assumes that the requirements.txt resolves dependencies in order; not sure if this is the default case or not or if they can be resolved holistically based on how the file is written, etc There are certainly other workable approaches, for example: - if setup.py is usable only for installation (i.e. `python setup.py install`, never `python setup.py develop`) then a setup.py could be written a la comment 3 for this purpose (not sure about dep resolution here, for that matter; might have to get "creative", e.g. generation on versionbump.py invocation) - .pth files could be used which bypasses dep resolution; this is what is done in mozilla-central's virtualenv; however, this is probably a non-starter since it will not (among other things) install console_scripts (etc) which IIRC are used in mozharness scripts already - building dependency resolution/unrolling into mozharness or into a (e.g.) standalone package (my preference) or a script/package that could be used in mozharness tests to verify config files to ensure compatability (and variations on the theme) Obviously there are a plethora of solutions possible. IMHO, the best long term solution is, if pip doesn't already support "what we need to do" (in abstract; my thinking being for the time being installing all packages in a directory though other interpretations are possible) then fixing upstream there and reaping the benefit, though of course this is not amenable to a timetable. I'm open to thoughts on these or other suggestions; I guess I'm not sure the problem space or restrictions.

We could just have a mozbase/setup.py that installs all its subpackages in order from disk. We can also call setup_development.py from the virtualenv python. Not sure why the python setup.py develop, or multiple-step operation is needed.

Maybe it's not really important, but it seems if we can solve this in a way that doesn't require listing packages in order anywhere at all would be best. In theory we could have a cyclic dependency in mozbase in which case depending on listing modules in the right order won't work at all. One solution that just occurred to me is to have create_virtualenv do a two-pass install. First it installs all listed modules with the --no-deps flag, then it goes back and installs them a second time without --no-deps. This second pass will only install the dependencies that weren't explicitly defined by the virtualenv_modules or virtualenv_requirements variables.

I should note that the above solution has the benefit of being generic, this problem isn't really mozbase specific after all..

> In theory we could have a cyclic dependency in mozbase in which > case depending on listing modules in the right order won't work at > all. AIUI, setuptools/distribute doesn't handle cyclic dependencies anyway

I'm going to guess we already ruled out making mozbase a single python package?

(In reply to Aki Sasaki [:aki] from comment #10) > I'm going to guess we already ruled out making mozbase a single python > package? FWIW, I'd rather not. We've certainly had the discussion before, and I don't necessarily think anything is ruled out. I'm generally against large utility packages: IMHO, they lose both general usefulness as well as tending to lose atomicity and modularity. Pragmatically speaking, however, the importance of MHO depends on the intended purpose of such software and my intentions and desires seem to be less important or unimportant from others' POV and if the purpose of mozbase is solely to support in-tree test harnesses, as some have advocated, then none of this matters. For that matter, IMHO a better long term plan, again depending on intention but herein my interpretation thereof that is implied, would be the consolidation of mozilla tree-related python into a single package spanning mozbase, mach, mozbuild, etc that would live in m-c. Further down the road to pragmatism, this is out of scope for this bug, I think, is an issue that has been turned down before but that I don't think anyone would block on revisiting if it offered practical gain, and has the considerable overhead (AIUI) of changing all import paths to what are currently mozbase packages. Again, for the record and not to reraise the issue, but I acting alone would instead move in the opposite direction and have a repository per mozbase package, as is the standard practice for python development (and which would at least change if not fix the problem after a fashion).

Bug 908356 landed a change that gets us half way there. It adds the option to do the two pass install per-item. This helps us if we are using a requirements.txt, but not if we have a list of modules. I think the 'two_pass' flag from bug 908356 should be moved from a member of a module in self._virtualenv_modules to a parameter of the 'create_virtualenv' function. If set to True here (which I think should be default) it would do a two pass install for all modules, whether they come from a list or a requirements.txt.