Allow the centralized package management system to source packages from PyPI
Categories
(Firefox Build System :: Mach Core, enhancement, P2)
Tracking
(firefox91 fixed)
Tracking | Status | |
---|---|---|
firefox91 | --- | fixed |
People
(Reporter: mhentges, Assigned: mhentges)
References
(Blocks 1 open bug)
Details
Attachments
(13 files)
(deleted),
text/x-phabricator-request
|
Details | |
(deleted),
text/x-phabricator-request
|
Details | |
(deleted),
text/x-phabricator-request
|
Details | |
(deleted),
text/x-phabricator-request
|
Details | |
(deleted),
text/x-phabricator-request
|
Details | |
(deleted),
text/x-phabricator-request
|
Details | |
(deleted),
text/x-phabricator-request
|
Details | |
(deleted),
text/x-phabricator-request
|
Details | |
(deleted),
text/x-phabricator-request
|
Details | |
(deleted),
text/x-phabricator-request
|
Details | |
(deleted),
text/x-phabricator-request
|
Details | |
(deleted),
text/x-phabricator-request
|
Details | |
(deleted),
text/x-phabricator-request
|
Details |
Add a new "dependency type" (PyPI) when parsing dependency definitions.
Assignee | ||
Updated•3 years ago
|
Assignee | ||
Updated•3 years ago
|
Assignee | ||
Comment 1•3 years ago
|
||
We had split up init
from init_py3
because mach
had
traditionally been invoked by either Python 2 or Python 3, and
the two couldn't share the same virtualenv.
Now that the same context isn't shared by both Python 2 and 3
3
(developers always use Python 3, and the remaining Python 2
usages are CI jobs that never reuse the objdir with Python 3),
We can centralize on a single default virtualenv.
I've called this "common" instead of "init" to clarify its
existing position as the virtualenv that's used by many different
commands. As we associate virtualenvs with requirement definitions,
it'll also make the file less confusing: it's a "common" requirement
definition as opposed to an "init" one.
Assignee | ||
Comment 2•3 years ago
|
||
This was originally set up so that tests wouldn't "create a new
virtualenv
for no reason." However, virtual environments now will have
different packages installed, and therefore the separation is necessary.
So, for the virtual environment used for builds (regular or for tests):
- We want it to be able to reuse the build venv, if it already exists.
- We don't want to pollute a
pytest
virtualenv with build-specific
packages.
Depends on D115635
Updated•3 years ago
|
Updated•3 years ago
|
Assignee | ||
Comment 3•3 years ago
|
||
Accidentally missing a return
in a code path could mean that
handle_package(...)
would accidentally do an action and
raise the "Unknown action" error.
Additionally, it simplifies the code a bit.
Depends on D115922
Assignee | ||
Comment 4•3 years ago
|
||
Child handle_package(...)
invocations don't need to modify
sitecustomize.py
, so don't pass it to them.
Depends on D115923
Assignee | ||
Comment 5•3 years ago
|
||
The pypi
action uses pip
to fetch a package and its dependencies
Depends on D115924
Assignee | ||
Updated•3 years ago
|
Comment 7•3 years ago
|
||
bugherder |
Assignee | ||
Comment 8•3 years ago
|
||
Last year, we stopped vendoring Python packages that have native
code. Since we have only had pure-python packages since, the
Windows-specific qualifier (or excluder in the case of !windows
)
hasn't been needed.
I don't foresee us need it again, but if anything we can peel it
back from hg
history if this assumption is incorrect.
Assignee | ||
Comment 9•3 years ago
|
||
The mach_bootstrap:search_path()
implementation is out of
date compared to virtualenv.py
.
Since python2:
, python3:
and optional:
packages are no
longer valid virtualenv requirement actions, they can be removed.
Assignee | ||
Comment 10•3 years ago
|
||
This has two benefits:
handle_package()
becomes more clear - rather than referring to
action
andpackage
with array index numbers, we now give
them real names. The benefit here is also shown inup_to_date()
.- This makes the top-level parser for
packages()
less opinionated
about sub-formats: if an action has a nested structure, it should
have the flexibility to define what it looks like.
Depends on D117468
Assignee | ||
Comment 11•3 years ago
|
||
Having separate <name>.pth
files in the virtual environments
isn't providing an advantage. We can simplify configuration
by putting all pth
adjustments into a single file: mach.pth
.
Depends on D117708
Assignee | ||
Comment 12•3 years ago
|
||
The mozbase
modules were being unconditionally added to the
sys.path
regardless of the Mach command being run, so there isn't
much value keeping them in a separate file. Besides, all other
source module paths are described in common_virtualenv_packages
,
why is mozbase
special?
In the future, we're going to want to make improvements here (such as:
there's a difference between informing mach of first-party code
versus defining which third_party vendored packages should be in scope,
and that workflow difference should be represented in-code).
It's useful to peel out the existing, less useful abstraction before
we can build a stronger one.
Depends on D115925
Assignee | ||
Comment 13•3 years ago
|
||
Check that all expected pypi packages are installed
when checking if a virtualenv is up-to-date.
Depends on D117711
Comment 14•3 years ago
|
||
Assignee | ||
Comment 15•3 years ago
|
||
Specifying a filename for sys.path
modifications is
no longer necessary - if a path needs to be on the
Python sys.path
, then just mark it with pth:
for
it to go into a .pth
file in the virtualenv.
Comment 16•3 years ago
|
||
bugherder |
Assignee | ||
Updated•3 years ago
|
Comment 17•3 years ago
|
||
Comment 18•3 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/f439c1345fd7
https://hg.mozilla.org/mozilla-central/rev/030a489e51f5
https://hg.mozilla.org/mozilla-central/rev/258461c83794
https://hg.mozilla.org/mozilla-central/rev/6a5c661cafe7
https://hg.mozilla.org/mozilla-central/rev/06df3dc28c58
Comment 19•3 years ago
|
||
Assignee | ||
Comment 20•3 years ago
|
||
Depends on D122893
Comment 21•3 years ago
|
||
Comment 22•3 years ago
|
||
bugherder |
Description
•