Closed Bug 1712133 Opened 3 years ago Closed 3 years ago

Allow the centralized package management system to source packages from PyPI

Categories

(Firefox Build System :: Mach Core, enhancement, P2)

enhancement

Tracking

(firefox91 fixed)

RESOLVED FIXED
91 Branch
Tracking Status
firefox91 --- fixed

People

(Reporter: mhentges, Assigned: mhentges)

References

(Blocks 1 open bug)

Details

Attachments

(13 files)

(deleted), text/x-phabricator-request
Details
(deleted), text/x-phabricator-request
Details
(deleted), text/x-phabricator-request
Details
(deleted), text/x-phabricator-request
Details
(deleted), text/x-phabricator-request
Details
(deleted), text/x-phabricator-request
Details
(deleted), text/x-phabricator-request
Details
(deleted), text/x-phabricator-request
Details
(deleted), text/x-phabricator-request
Details
(deleted), text/x-phabricator-request
Details
(deleted), text/x-phabricator-request
Details
(deleted), text/x-phabricator-request
Details
(deleted), text/x-phabricator-request
Details

Add a new "dependency type" (PyPI) when parsing dependency definitions.

Priority: -- → P2
Assignee: nobody → mhentges
Status: NEW → ASSIGNED

We had split up init from init_py3 because mach had
traditionally been invoked by either Python 2 or Python 3, and
the two couldn't share the same virtualenv.

Now that the same context isn't shared by both Python 2 and 3
3
(developers always use Python 3, and the remaining Python 2
usages are CI jobs that never reuse the objdir with Python 3),
We can centralize on a single default virtualenv.

I've called this "common" instead of "init" to clarify its
existing position as the virtualenv that's used by many different
commands. As we associate virtualenvs with requirement definitions,
it'll also make the file less confusing: it's a "common" requirement
definition as opposed to an "init" one.

This was originally set up so that tests wouldn't "create a new
virtualenv for no reason." However, virtual environments now will have
different packages installed, and therefore the separation is necessary.

So, for the virtual environment used for builds (regular or for tests):

  • We want it to be able to reuse the build venv, if it already exists.
  • We don't want to pollute a pytest virtualenv with build-specific
    packages.

Depends on D115635

Attachment #9222845 - Attachment description: WIP: Bug 1712133: Rename default virtualenv to "common" → Bug 1712133: Rename default virtualenv to "common"
Attachment #9222851 - Attachment description: WIP: Bug 1712133: Remove build VIRTUALENV_NAME customization → Bug 1712133: Remove build VIRTUALENV_NAME customization

Accidentally missing a return in a code path could mean that
handle_package(...) would accidentally do an action and
raise the "Unknown action" error.

Additionally, it simplifies the code a bit.

Depends on D115922

Child handle_package(...) invocations don't need to modify
sitecustomize.py, so don't pass it to them.

Depends on D115923

The pypi action uses pip to fetch a package and its dependencies

Depends on D115924

Keywords: leave-open
Pushed by mhentges@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/e1cd8757104d Rename default virtualenv to "common" r=ahal https://hg.mozilla.org/integration/autoland/rev/2e37a3ce90c2 Remove build VIRTUALENV_NAME customization r=glandium https://hg.mozilla.org/integration/autoland/rev/965304eb5799 Use if/else chain instead of early-return in handle_package r=ahal https://hg.mozilla.org/integration/autoland/rev/7e8e5f4c8e73 Simplify virtualenv "sitecustomize" writing r=ahal

Last year, we stopped vendoring Python packages that have native
code. Since we have only had pure-python packages since, the
Windows-specific qualifier (or excluder in the case of !windows)
hasn't been needed.

I don't foresee us need it again, but if anything we can peel it
back from hg history if this assumption is incorrect.

The mach_bootstrap:search_path() implementation is out of
date compared to virtualenv.py.

Since python2:, python3: and optional: packages are no
longer valid virtualenv requirement actions, they can be removed.

This has two benefits:

  1. handle_package() becomes more clear - rather than referring to
    action and package with array index numbers, we now give
    them real names. The benefit here is also shown in up_to_date().
  2. This makes the top-level parser for packages() less opinionated
    about sub-formats: if an action has a nested structure, it should
    have the flexibility to define what it looks like.

Depends on D117468

Having separate <name>.pth files in the virtual environments
isn't providing an advantage. We can simplify configuration
by putting all pth adjustments into a single file: mach.pth.

Depends on D117708

The mozbase modules were being unconditionally added to the
sys.path regardless of the Mach command being run, so there isn't
much value keeping them in a separate file. Besides, all other
source module paths are described in common_virtualenv_packages,
why is mozbase special?

In the future, we're going to want to make improvements here (such as:
there's a difference between informing mach of first-party code
versus defining which third_party vendored packages should be in scope,
and that workflow difference should be represented in-code).
It's useful to peel out the existing, less useful abstraction before
we can build a stronger one.

Depends on D115925

Check that all expected pypi packages are installed
when checking if a virtualenv is up-to-date.

Depends on D117711

Pushed by mhentges@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/4a326b86ad24 Synchronize virtualenv requirement parsing logic r=ahal https://hg.mozilla.org/integration/autoland/rev/a705e6a13d96 Remove "windows" action support from virtualenvs r=ahal

Specifying a filename for sys.path modifications is
no longer necessary - if a path needs to be on the
Python sys.path, then just mark it with pth: for
it to go into a .pth file in the virtualenv.

Keywords: leave-open
Pushed by mhentges@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/f439c1345fd7 Make virtualenv package parsing more specific r=ahal https://hg.mozilla.org/integration/autoland/rev/030a489e51f5 Remove "pth" name customization r=ahal https://hg.mozilla.org/integration/autoland/rev/258461c83794 Inline `testing/mozbase/packages.txt` contents r=ahal https://hg.mozilla.org/integration/autoland/rev/6a5c661cafe7 Adds "pypi" action to virtualenv `handle_package()` r=ahal https://hg.mozilla.org/integration/autoland/rev/06df3dc28c58 Validate that pypi dependencies are installed r=ahal
Pushed by thunderbird@calypsoblue.org: https://hg.mozilla.org/comm-central/rev/ca8a42b08051 Remove filename prefix from virtualenv pths r=rjl

Depends on D122893

Pushed by mhentges@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/f49440e40bfd Add missing space to pypi-optional error message r=ahal
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: