Closed Bug 1758584 Opened 3 years ago Closed 3 years ago

Raise error if active site is marked as out-of-date

Categories

(Firefox Build System :: Mach Core, enhancement)

enhancement

Tracking

(firefox100 fixed)

RESOLVED FIXED
100 Branch
Tracking Status
firefox100 --- fixed

People

(Reporter: mhentges, Assigned: mhentges)

References

Details

Attachments

(3 files)

There's some cases in CI where the "system python" is used, and the build site is activated multiple times.
Confusingly, the build site is recreated each time, and this causes a Windows-specific error in CI (Access is denied: '$topsrcdir\\_virtualenvs\\build\\Scripts\\python.exe').

Let's guard against this in a platform-specific way, then resolve the issue.

Assignee: nobody → mhentges
Status: NEW → ASSIGNED

Unless either the site's requirements or its on-disk virtualenv
changes, it shouldn't become out-of-date.
This will also catch situations where sites are incorrectly
being marked out-of-date due to sys.path mismanagement.

So far, we've been using virtualenv's activate_this.py script.
However, unlike earlier expectations, it adds its sys.path additions
to the front, not the back! This breaks our prioritization
requirements, such as:

  • When using any package from the system environment, import all
    possible
    packages from the system to avoid compatibility issues.
  • Use vendored packages instead of virtualenv-installed packages
    wherever possible, because it more-closely matches developer
    expectations ("why is this package vendored if it's not used?")

Define an activate_virtualenv() function that replicates the logic
of activate_this.py [1], except for three differences:

  • Don't modify sys.real_prefix, since it's a non-standard change
  • Only add seen-with-venv-module paths to the sys.path ($prefix,
    $prefix/.../$site_packages_dir) - don't do the paths in-between.
  • And, of course, append instead of prepend sys.path entries.

As an aside, this is one of the few remaining blockers from allowing
us to fully embrace venv instead of virtualenv - the last piece is
waiting on the fix for bug 1697833 to propagate.

[1]
https://github.com/pypa/virtualenv/blob/20.7.2/src/virtualenv/activation/python/activate_this.py

Depends on D140578

We want _pthfile_lines() to consistently and accurately represent a
virtualenv's sys.path modifications.

However, this becomes tricky when comparing expected pthfile_lines
//before// activating a virtualenv versus //afterwards//, especially
since Python implicitly adds some paths (such as the path to
site-packages).

The current solution made this scrubbing only happen if we were
pip install-ing into the site. Unfortunately, it //doesn't// work for
the "use system Python for the build site", because:

  • Pre-activation result: site-packages isn't added, because we aren't
    using it
  • Post-activation result, implicitly-added site-packages isn't
    scrubbed, because scrubbing only happened for
    SitePackagesSource.VENV

Stepping back, the solution here is:

  • pthfile_lines only represents Mach modifications //on top// of
    implicit virtualenv behaviour: since site-packages is always added
    by default, it shouldn't be in our explicit pthfile_lines.
  • The only time when implicit sys.path additions throws us off is when
    SitePackagesSource.SYSTEM: so, move the scrubbing to happen in that
    case instead.

Finally refactor the "conditional deprioritization" comment to be more
useful and more accurate: we've implemented the "nontrivial complexity"
of purging site-packages, and the other piece about nothing being
pip install-ed feels self-explanatory enough.

Depends on D140579

Attachment #9266924 - Attachment description: WIP: Bug 1758584: Fix virtualenv-path scrubbing from command _pthfile_lines() → WIP: Bug 1758584: Fix virtualenv-path scrubbing from command
Attachment #9266922 - Attachment description: WIP: Bug 1758584: Raise error if active site is out-of-date → Bug 1758584: Raise error if active site is out-of-date
Attachment #9266923 - Attachment description: WIP: Bug 1758584: Add in-proc venv activation paths to the end of `sys.path` → Bug 1758584: Add in-proc venv activation paths to the end of `sys.path`
Attachment #9266924 - Attachment description: WIP: Bug 1758584: Fix virtualenv-path scrubbing from command → Bug 1758584: Fix virtualenv-path scrubbing from command
Attachment #9266924 - Attachment description: Bug 1758584: Fix virtualenv-path scrubbing from command → Bug 1758584: Fix virtualenv-path scrubbing from command _pthfile_lines
Pushed by mhentges@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/d0c262934162 Raise error if active site is out-of-date r=ahal https://hg.mozilla.org/integration/autoland/rev/1337faa66745 Add in-proc venv activation paths to the end of `sys.path` r=ahal https://hg.mozilla.org/integration/autoland/rev/1883182d614f Fix virtualenv-path scrubbing from command _pthfile_lines r=ahal
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → 100 Branch
Regressions: 1759125
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: