Closed Bug 1732228 Opened 3 years ago Closed 2 years ago

Intermittent Can't call install() without mozinstall!

Categories

(Firefox Build System :: Task Configuration, defect, P5)

Product:
Firefox Build System
Component:
Task Configuration
Type:
defect

Tracking

(firefox113 fixed)

Status:
RESOLVED FIXED
Milestone:
113 Branch
Tracking Flags:
Tracking Status
firefox113 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: ahochheiden)

References

Details

(Keywords: intermittent-failure, Whiteboard: [stockwell disable-recommended])

Attachments

(3 files)

Bug 1732228 - Upgrade the wheel for `pip` used by `mozharness` to version `23.0.1` r?#build
(deleted), text/x-phabricator-request
Details
Bug 1732228 - Add the default pip to the venv so we can use that to install the specific version of pip from the in-tree wheel r?#build
(deleted), text/x-phabricator-request
Details
Bug 1732228 - Bump `pip` to version `23.0.1` to maintain consistency with the wheel used by `mozharness` r?#build
(deleted), text/x-phabricator-request
Details

Filed by: nerli [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=352470327&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Mqj7CPbQSTGF1nGxmErqUg/runs/0/artifacts/public/logs/live_backing.log

[task 2021-09-23T03:35:27.163Z] 03:35:27     INFO - [mozharness: 2021-09-23 03:35:27.163098Z] Finished create-virtualenv step (success)
[task 2021-09-23T03:35:27.163Z] 03:35:27     INFO - [mozharness: 2021-09-23 03:35:27.163704Z] Running start-pulseaudio step.
[task 2021-09-23T03:35:27.163Z] 03:35:27     INFO - Running pre-action listener: _resource_record_pre_action
[task 2021-09-23T03:35:27.164Z] 03:35:27     INFO - Running main action method: start_pulseaudio
[task 2021-09-23T03:35:27.164Z] 03:35:27     INFO - Running post-action listener: _resource_record_post_action
[task 2021-09-23T03:35:27.164Z] 03:35:27     INFO - [mozharness: 2021-09-23 03:35:27.164395Z] Finished start-pulseaudio step (success)
[task 2021-09-23T03:35:27.164Z] 03:35:27     INFO - [mozharness: 2021-09-23 03:35:27.164484Z] Running install step.
[task 2021-09-23T03:35:27.164Z] 03:35:27     INFO - Running pre-action listener: _resource_record_pre_action
[task 2021-09-23T03:35:27.164Z] 03:35:27     INFO - Running main action method: install
[task 2021-09-23T03:35:27.164Z] 03:35:27     INFO - Getting output from command: ['/opt/worker/tasks/task_163236801295463/build/venv/bin/pip', 'list', '--format', 'freeze']
[task 2021-09-23T03:35:27.165Z] 03:35:27     INFO - Copy/paste: /opt/worker/tasks/task_163236801295463/build/venv/bin/pip list --format freeze
[task 2021-09-23T03:35:27.575Z] 03:35:27     INFO - Reading from file tmpfile_stdout
[task 2021-09-23T03:35:27.575Z] 03:35:27     INFO - Reading from file tmpfile_stderr
[task 2021-09-23T03:35:27.576Z] 03:35:27    FATAL - Can't call install() without mozinstall!
[task 2021-09-23T03:35:27.576Z] 03:35:27    FATAL - Did you run with --create-virtualenv? Is mozinstall in virtualenv_modules?
[task 2021-09-23T03:35:27.576Z] 03:35:27    FATAL - Running post_fatal callback...
[task 2021-09-23T03:35:27.576Z] 03:35:27    FATAL - Exiting -1
[task 2021-09-23T03:35:27.576Z] 03:35:27     INFO - Running post-action listener: _resource_record_post_action
[task 2021-09-23T03:35:27.576Z] 03:35:27     INFO - [mozharness: 2021-09-23 03:35:27.576759Z] Finished install step (failed)
[task 2021-09-23T03:35:27.576Z] 03:35:27     INFO - Running post-run listener: _resource_record_post_run
[taskcluster 2021-09-23T03:35:27.667Z]    Exit Code: 255
[taskcluster 2021-09-23T03:35:27.667Z]    User Time: 1m19.531768s
[taskcluster 2021-09-23T03:35:27.667Z]  Kernel Time: 19.597983s
[taskcluster 2021-09-23T03:35:27.667Z]    Wall Time: 1m41.78791s
[taskcluster 2021-09-23T03:35:27.667Z]       Result: FAILED
[taskcluster 2021-09-23T03:35:27.668Z] === Task Finished ===
[taskcluster 2021-09-23T03:35:27.668Z] Task Duration: 1m41.790652s
[taskcluster 2021-09-23T03:35:27.764Z] Uploading artifact public/logs/localconfig.json from file logs/localconfig.json with content encoding "gzip", mime type "application/json" and expiry 2022-09-23T02:53:52.099Z
[taskcluster 2021-09-23T03:35:27.965Z] Uploading artifact public/test_info/resource-usage.json from file build/blobber_upload_dir/resource-usage.json with content encoding "gzip", mime type "application/json" and expiry 2022-09-23T02:53:52.099Z
[taskcluster 2021-09-23T03:35:28.074Z] Uploading redirect artifact public/logs/live.log to URL https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Mqj7CPbQSTGF1nGxmErqUg/runs/0/artifacts/public%2Flogs%2Flive_backing.log with mime type "text/plain; charset=utf-8" and expiry 2022-09-23T02:53:52.099Z
[taskcluster:error] exit status 255
Component: Mozbase → Task Configuration
Product: Testing → Firefox Build System

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → INCOMPLETE

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=361960994&repo=autoland&lineNumber=1612

Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Status: REOPENED → RESOLVED
Closed: 3 years ago → 3 years ago
Resolution: --- → INCOMPLETE

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=365844924&repo=mozilla-central

Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---

(In reply to Treeherder Bug Filer from comment #7)

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=365844924&repo=mozilla-central

[task 2022-01-28T00:55:21.289Z] 00:55:21     INFO -    Stored in directory: C:\Users\task_164332779897322\AppData\Local\Temp\pip-ephem-wheel-cache-q7cik4vc\wheels\12\fa\59\7827417f37ea6742dc72bc9bb1c0b93eac4de745a84d3bd0d2
[task 2022-01-28T00:55:21.289Z] 00:55:21     INFO -    WARNING: Built wheel for mozinfo is invalid: Wheel has unexpected file name: expected 'mozinfo', got 'UNKNOWN'
[task 2022-01-28T00:55:21.290Z] 00:55:21     INFO -    Building wheel for mozInstall (setup.py): started
[task 2022-01-28T00:55:21.647Z] 00:55:21     INFO -    Building wheel for mozInstall (setup.py): finished with status 'done'
[task 2022-01-28T00:55:21.664Z] 00:55:21     INFO -    Created wheel for mozInstall: filename=UNKNOWN-0.0.0-py3-none-any.whl size=982 sha256=79f7f31fb55721ff0d9c5c1060b3853bbf6dd1ddc7be3a26992c7cf985f1ff6a
[task 2022-01-28T00:55:21.665Z] 00:55:21     INFO -    Stored in directory: C:\Users\task_164332779897322\AppData\Local\Temp\pip-ephem-wheel-cache-q7cik4vc\wheels\58\9d\9a\990110bbe984cb7f88bdf68811cc1d9bb85d1039aa372a845a
[task 2022-01-28T00:55:21.665Z] 00:55:21     INFO -    WARNING: Built wheel for mozInstall is invalid: Wheel has unexpected file name: expected 'mozinstall', got 'UNKNOWN'

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Status: REOPENED → RESOLVED
Closed: 3 years ago → 3 years ago
Resolution: --- → INCOMPLETE

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=377642580&repo=autoland

Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Status: REOPENED → RESOLVED
Closed: 3 years ago → 2 years ago
Resolution: --- → INCOMPLETE

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=383370280&repo=try

Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---

this is happening quite frequently on windows 11 tests. I see this on wpt a lot, and it seems that the UNKNOWN==0.0.0 version that we end up with is a key, also the missing packages are ones that come from the tree (i.e. downloaded test package) vs from pypi.

comparing a passing retrigger and a failure, pip and setuptools are the same version, the downloaded bytes are the same for the test package.

I wonder if this is a specific VM that is bad? Not sure what could be wrong on the VM.

pip == 21.2.3
setuptools == 51.2.0

these are hardcoded in tree:
https://searchfox.org/mozilla-central/search?q=21.2.3-py3-none-any.whl&path=&case=false&regexp=false

setuptools has 67.3.2 and pip is up to 23.0 now. I read that upgrading these might solve problems.

This has spiked on win 11 workers after enabling tests in Bug 1816568. Please also have a look over Bug 1818693 and Bug 1752965.

Flags: needinfo?(jmaher)

:ahochheiden, fyi, this is the bug related to mozinstall and a potential fix with the updated pip/setuptools.

Flags: needinfo?(jmaher) → needinfo?(ahochheiden)

both Bug 1818693 and Bug 1752965 have the same problems where we see in pip freeze: UNKNOWN: 0.0.0, from what I can tell this is the same root cause as this bug and ideally a fix for this bug will fix those.

Assignee: nobody → ahochheiden
  • We can't use the pip from source that's in-tree, since that is no longer a supported entry point of pip.
  • We can't use the pip wheel to install itself, since that is no longer supported either.
  • We can't create the venv with the default pip, because that fails on Windows10 in CI.
  • We can create the venv with --without-pip and then run ensurepip --default-pip to get a version of pip in all contexts that can install the wheel of pip that's in-tree.

Depends on D171176

Attachment #9320175 - Attachment description: Bug 1732228 - Upgrade `pip` and `setuptools` r?#build → Bug 1732228 - Upgrade the wheels for `pip` and `setuptools` used by `mozharness` r?#build
Attachment #9320176 - Attachment description: WIP: Bug 1732228 - Add the default pip to the venv so we can use that to install the specific version of pip from the in-tree wheel #?build → Bug 1732228 - Add the default pip to the venv so we can use that to install the specific version of pip from the in-tree wheel r?#build

That stack should resolve it.

Flags: needinfo?(ahochheiden)
Pushed by ahochheiden@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/61a521ac78e9 Upgrade the wheels for `pip` and `setuptools` used by `mozharness` r=firefox-build-system-reviewers,glandium https://hg.mozilla.org/integration/autoland/rev/fe7621bd33fe Add the default pip to the venv so we can use that to install the specific version of pip from the in-tree wheel r=firefox-build-system-reviewers,glandium

Backed out for causing mda failures on Android.

Flags: needinfo?(ahochheiden)

This should work now, as of bug 1804514.

Flags: needinfo?(ahochheiden)
Pushed by ahochheiden@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/37b904639717 Upgrade the wheels for `pip` and `setuptools` used by `mozharness` r=firefox-build-system-reviewers,glandium https://hg.mozilla.org/integration/autoland/rev/675f6a2134ac Add the default pip to the venv so we can use that to install the specific version of pip from the in-tree wheel r=firefox-build-system-reviewers,glandium

Backed out 4 changesets (Bug 1734402, Bug 1732228) for build bustages when upgrade pip-23.0.1-py3-none-any.whl.
Backout link
Push with failures
Failure Log

Flags: needinfo?(ahochheiden)

That was weird. The .whl files were 0 bytes on Phabricator. I've re-uploaded the patches now and they have the correct file size. I'll reattempt to land this.

Flags: needinfo?(ahochheiden)
Pushed by ahochheiden@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/ad9c880233ed Upgrade the wheels for `pip` and `setuptools` used by `mozharness` r=firefox-build-system-reviewers,glandium https://hg.mozilla.org/integration/autoland/rev/266c417563fe Add the default pip to the venv so we can use that to install the specific version of pip from the in-tree wheel r=firefox-build-system-reviewers,glandium

(In reply to Alex Hochheiden [:ahochheiden] from comment #59)

That was weird. The .whl files were 0 bytes on Phabricator. I've re-uploaded the patches now and they have the correct file size. I'll reattempt to land this.

See bug 1709608

Backed out 4 changesets (Bug 1732228, Bug 1734402) for multiple browsertime and talos failures.
Backout link
Push with failures <--> cnn <--> damp-inspector
cnn Failure Log
damp-inspector Failure Log

Flags: needinfo?(ahochheiden)
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]

There have been 99 total failures in the last 7 days, recent failure log.
Affected platforms are:

  • windows10-64-shippable-qr
  • windows11-32-2009-mingwclang-qr
  • windows11-32-2009-qr
  • windows11-32-2009-shippable-qr
  • windows11-64-2009-asan-qr
  • windows11-64-2009-mingwclang-qr
  • windows11-64-2009-qr
  • windows11-64-2009-shippable-qr
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]

There have been 73 total failures in the last 7 days, recent failure log.
Affected platforms are:

  • macosx1015-64-qr
  • windows11-32-2009-mingwclang-qr
  • windows11-32-2009-qr
  • windows11-64-2009-asan-qr
  • windows11-64-2009-mingwclang-qr
  • windows11-64-2009-qr
  • windows11-64-2009-shippable-qr
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]
Attachment #9320175 - Attachment description: Bug 1732228 - Upgrade the wheels for `pip` and `setuptools` used by `mozharness` r?#build → Bug 1732228 - Upgrade the wheel for `pip` used by `mozharness` r?#build
Attachment #9320175 - Attachment description: Bug 1732228 - Upgrade the wheel for `pip` used by `mozharness` r?#build → Bug 1732228 - Upgrade the wheel for `pip` used by `mozharness` to version `23.0.1` r?#build
Pushed by ahochheiden@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/c63a01f11f45 Upgrade the wheel for `pip` used by `mozharness` to version `23.0.1` r=firefox-build-system-reviewers,glandium https://hg.mozilla.org/integration/autoland/rev/705d211ab4df Add the default pip to the venv so we can use that to install the specific version of pip from the in-tree wheel r=firefox-build-system-reviewers,glandium https://hg.mozilla.org/integration/autoland/rev/ae965ed71f59 Bump `pip` to version `23.0.1` to maintain consistency with the wheel used by `mozharness` r=firefox-build-system-reviewers,glandium

https://hg.mozilla.org/mozilla-central/rev/c63a01f11f45
https://hg.mozilla.org/mozilla-central/rev/705d211ab4df
https://hg.mozilla.org/mozilla-central/rev/ae965ed71f59

Status: REOPENED → RESOLVED
Closed: 2 years ago → 2 years ago
status-firefox113: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 113 Branch
Regressions: 1825755
Duplicate of this bug: 1752965
Flags: needinfo?(ahochheiden)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: