Closed Bug 1717645 Opened 3 years ago Closed 3 years ago

Virtualenv "up-to-date" logic is not resilient to setup failures

Categories

(Firefox Build System :: Mach Core, defect)

Product:
Firefox Build System
Component:
Mach Core
Type:
defect

Tracking

(firefox92 fixed)

Status:
RESOLVED FIXED
Milestone:
92 Branch
Tracking Flags:
Tracking Status
firefox92 --- fixed

People

(Reporter: mhentges, Assigned: mhentges)

References

Details

Attachments

(3 files)

Bug 1717645: Resolve nested virtualenv requirements up-front
(deleted), text/x-phabricator-request
Details
Bug 1717645: Check virtualenv pths are up-to-date
(deleted), text/x-phabricator-request
Details
Bug 1717645: Fix up_to_date for Python 3.6
(deleted), text/x-phabricator-request
Details

In Gijs' bug here, a failure setting up pth files caused an original failure, but follow-up ./mach invocations happily used the broken virtualenv.

The virtualenv "up-to-date" logic should more intelligently handle failures with the venv.

Assignee: nobody → mhentges
Status: NEW → ASSIGNED

This simplifies consumer logic, since they get the parsed list of pypi
and pth requirements, as well as the list of input files that were
parsed.

One benefit of this simplification is that we no longer
recursively create VirtualenvManagers.

Note that mach_bootstrap cannot (yet) take advantage
of ParseMachEnvRequirements because of a dependency cycle:

  • mach_bootstrap must set up the sys.path to import
    ParseMachEnvRequirements.
  • mach_bootstrap would want ParseMachEnvRequirements to
    determine which paths to add to the sys.path.

It's possible for a virtualenv to have an incorrect list of directories
to add to the sys.path, such as the following cases:

  • Its creation got cancelled halfway through
  • The list of pths changed in a new revision
  • It got modified by an external tool

By validating the list of provided pths against the list of required
pths, we can ensure that the virtualenv is more dependably up-to-date.

Depends on D119685

Python 3.6 doesn't support the capture_output option, so use
stdout=subprocess.PIPE instead.

Depends on D119686

Pushed by mhentges@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/370e02f73923 Resolve nested virtualenv requirements up-front r=ahal https://hg.mozilla.org/integration/autoland/rev/16b0fb30aff2 Check virtualenv pths are up-to-date r=ahal https://hg.mozilla.org/integration/autoland/rev/82dc0771b98b Fix up_to_date for Python 3.6 r=ahal
Backout by csabou@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/7b7a049fb3e3 Backed out 7 changesets (bug 1717645, bug 1717051, bug 1720215) for busting the gecko decision task. CLOSED TREE DONTBUILD
Pushed by mhentges@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/7864e8241d77 Resolve nested virtualenv requirements up-front r=ahal https://hg.mozilla.org/integration/autoland/rev/eab19a103b56 Check virtualenv pths are up-to-date r=ahal https://hg.mozilla.org/integration/autoland/rev/e43fb9c6cd0f Fix up_to_date for Python 3.6 r=ahal
Regressions: 1721522

Backed out for causing win debug build bustages

Backout link: https://hg.mozilla.org/integration/autoland/rev/d716918916ac1a4993a616f07d501f9b64be467d

Push with failures

Failure log

Flags: needinfo?(mhentges)
Flags: needinfo?(mhentges)
Pushed by mhentges@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/61b5c99a7ad2 Resolve nested virtualenv requirements up-front r=ahal https://hg.mozilla.org/integration/autoland/rev/ea6ab9ecdc1b Check virtualenv pths are up-to-date r=ahal https://hg.mozilla.org/integration/autoland/rev/766ea0a1adf1 Fix up_to_date for Python 3.6 r=ahal

Backed out per glandium's request, 7 changesets (Bug 1720215, Bug 1717645, Bug 1717051) for causing build bustage and conflicting backout

Log: https://treeherder.mozilla.org/logviewer?job_id=346816196&repo=autoland&lineNumber=987

Backout: https://hg.mozilla.org/integration/autoland/rev/791ec1c34fb986a24cfcda7c4d98af416299a024

Flags: needinfo?(mhentges)

Changes from my Windows machine weren't synchronized to my Linux one before I re-attempted landing :(
Thanks for the backout 👍

Flags: needinfo?(mhentges)
Pushed by mhentges@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/13d016bcc12c Resolve nested virtualenv requirements up-front r=ahal https://hg.mozilla.org/integration/autoland/rev/719348d541d8 Check virtualenv pths are up-to-date r=ahal https://hg.mozilla.org/integration/autoland/rev/3a844a148590 Fix up_to_date for Python 3.6 r=ahal

https://hg.mozilla.org/mozilla-central/rev/13d016bcc12c
https://hg.mozilla.org/mozilla-central/rev/719348d541d8
https://hg.mozilla.org/mozilla-central/rev/3a844a148590

Status: ASSIGNED → RESOLVED
Closed: 3 years ago
status-firefox92: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 92 Branch
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: