Open Bug 1768001 Opened 3 years ago Updated 3 years ago

Improve lockfile generation performance: remove setup.py dependencies

Categories

(Firefox Build System :: Mach Core, enhancement, P3)

enhancement

Tracking

(Not tracked)

People

(Reporter: mhentges, Unassigned)

References

(Depends on 1 open bug)

Details

Each lockfile takes ~15s to create, but most of this time is spinning up a Python interpreter to invoke setup.py for a few of our packages:

  • third_party/python/gyp (third party)
  • third_party/python/dlmanager (third party)
  • testing/web-platform/tests/tools/wptrunner (first party)
  • testing/web-platform/tests/tools/wptserve (first party)
  • testing/web-platform/tests/tools/third_party/mod_pywebsocket (third party)

We should do the following:

  1. Convert our first-party packages to use pyproject.toml instead of setup.py: this allows them to still be trivially modified, yet far cheaper for poetry to check dependency information/etc (reading TOML is cheaper than interpreting a Python file, how strange is that? 🤔)
  2. Process dist info for our third party packages
    • As part of this, we can tweak our Poetry integration (python_lockfile.py) to no longer copy setup.py packages before processing them.
  3. Add a test to python/mach/ that verifies that no site has a setup.py package with outdated dist information.

By allowing Poetry to run without having to invoke any setup.py files, we should be able to drop per-lockfile generation time to ~1s each, which is very decent.

Severity: -- → S3
Priority: -- → P3
You need to log in before you can comment on or make changes to this bug.