Filed by: mlaza [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=375034654&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/IpX0FURASHyklmTHWIvo9A/runs/0/artifacts/public/logs/live_backing.log

[task 2022-04-19T19:12:21.512Z] python/mozbuild/mozbuild/test/configure/test_toolchain_configure.py::RustTest::test_rust_target PASSED
[task 2022-04-19T19:12:21.512Z] 
[task 2022-04-19T19:12:21.512Z] =============================== warnings summary ===============================
[task 2022-04-19T19:12:21.512Z] python/mozbuild/mozbuild/test/configure/test_toolchain_configure.py: 562 warnings
[task 2022-04-19T19:12:21.512Z]   /builds/worker/checkouts/gecko/python/mozbuild/mozbuild/preprocessor.py:863: DeprecationWarning: 'U' mode is deprecated
[task 2022-04-19T19:12:21.512Z]     args = io.open(args, "rU", encoding="utf-8")
[task 2022-04-19T19:12:21.512Z] 
[task 2022-04-19T19:12:21.512Z] -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
[task 2022-04-19T19:12:21.512Z] ================= 54 passed, 562 warnings in 201.81s (0:03:21) =================
[task 2022-04-19T19:12:21.512Z] 
[task 2022-04-19T19:12:21.513Z] 
[task 2022-04-19T19:12:21.513Z] Tests Completed: 100%|██████████| 77/77 [03:36<00:00, 48.19s/Test]
[task 2022-04-19T19:12:21.513Z]                                                                   
[task 2022-04-19T19:12:21.513Z] /builds/worker/checkouts/gecko/python/mozbuild/mozbuild/test/test_vendor.py
[task 2022-04-19T19:12:21.513Z] ============================= test session starts ==============================
[task 2022-04-19T19:12:21.513Z] platform linux -- Python 3.6.9, pytest-7.0.1, pluggy-1.0.0 -- /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/python-test/bin/python
[task 2022-04-19T19:12:21.513Z] rootdir: /builds/worker/checkouts/gecko, configfile: config/mozunit/mozunit/pytest.ini
[task 2022-04-19T19:12:21.513Z] collecting ... collected 1 item
[task 2022-04-19T19:12:21.513Z] 
[task 2022-04-19T19:12:21.513Z] python/mozbuild/mozbuild/test/test_vendor.py::test_up_to_date_vendor TEST-UNEXPECTED-FAIL
[task 2022-04-19T19:12:21.513Z] 
[task 2022-04-19T19:12:21.513Z] =================================== FAILURES ===================================
[task 2022-04-19T19:12:21.513Z] ____________________________ test_up_to_date_vendor ____________________________
[task 2022-04-19T19:12:21.514Z] python/mozbuild/mozbuild/test/test_vendor.py:31: in test_up_to_date_vendor
[task 2022-04-19T19:12:21.514Z]     vendor.vendor()
[task 2022-04-19T19:12:21.514Z] python/mozbuild/mozbuild/vendor/vendor_python.py:36: in vendor
[task 2022-04-19T19:12:21.514Z]     lockfiles = poetry.generate_lockfiles(do_update=False)
[task 2022-04-19T19:12:21.514Z] python/mach/mach/python_lockfile.py:68: in generate_lockfiles
[task 2022-04-19T19:12:21.514Z]     self._run_poetry(["lock"] + (["--no-update"] if not do_update else []))
[task 2022-04-19T19:12:21.514Z] python/mach/mach/python_lockfile.py:78: in _run_poetry
[task 2022-04-19T19:12:21.515Z]     [sys.executable, "-m", "poetry"] + args, cwd=self._work_dir
[task 2022-04-19T19:12:21.515Z] /usr/lib/python3.6/subprocess.py:311: in check_call
[task 2022-04-19T19:12:21.515Z]     raise CalledProcessError(retcode, cmd)
[task 2022-04-19T19:12:21.515Z] E   subprocess.CalledProcessError: Command '['/builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/python-test/bin/python', '-m', 'poetry', 'lock', '--no-update']' returned non-zero exit status 1.
[task 2022-04-19T19:12:21.515Z] ----------------------------- Captured stdout call -----------------------------
[task 2022-04-19T19:12:21.515Z] Resolving dependencies...
[task 2022-04-19T19:12:21.516Z] ----------------------------- Captured stderr call -----------------------------
[task 2022-04-19T19:12:21.516Z] 
[task 2022-04-19T19:12:21.516Z]   PackageNotFound
[task 2022-04-19T19:12:21.516Z] 
[task 2022-04-19T19:12:21.516Z]   Package wheel (0.37.0) not found.
[task 2022-04-19T19:12:21.516Z] 
[task 2022-04-19T19:12:21.517Z]   at ~/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/python-test/lib/python3.6/site-packages/poetry/repositories/pool.py:152 in package
[task 2022-04-19T19:12:21.517Z]       148│                     self._packages.append(package)
[task 2022-04-19T19:12:21.517Z]       149│
[task 2022-04-19T19:12:21.517Z]       150│                     return package
[task 2022-04-19T19:12:21.517Z]       151│
[task 2022-04-19T19:12:21.517Z]     → 152│         raise PackageNotFound(f"Package {name} ({version}) not found.")
[task 2022-04-19T19:12:21.518Z]       153│
[task 2022-04-19T19:12:21.518Z]       154│     def find_packages(self, dependency: "Dependency") -> List["Package"]:
[task 2022-04-19T19:12:21.518Z]       155│         repository = dependency.source_name
[task 2022-04-19T19:12:21.518Z]       156│         if repository is not None:
[task 2022-04-19T19:12:21.518Z] ============================== 1 failed in 4.05s ===============================
[task 2022-04-19T19:12:21.518Z] 
[task 2022-04-19T19:12:21.519Z] 
[task 2022-04-19T19:12:21.519Z] Tests Completed: 100%|██████████| 77/77 [03:36<00:00,  2.81s/Test]Return code from mach python-test: 1
[task 2022-04-19T19:12:21.542Z] 
[taskcluster 2022-04-19 19:12:21.893Z] === Task Finished ===
[taskcluster 2022-04-19 19:12:21.893Z] Unsuccessful task run with exit code: 1 completed in 564.557 seconds

This is permafailing now: https://treeherder.mozilla.org/jobs?repo=autoland&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception%2Crunnable&searchStr=linux%2C18.04%2Cx64%2Copt%2Cpython%2C3%2Cunit%2Ctests%2Csource-test-python-mozbuild-linux1804-64%2Fopt-py3%2Cmbu&group_state=expanded&fromchange=a616ef9e2944fdd74bcba857c172c95bd84b78f8&tochange=d7f24c995ed5df9ca9ca018f8f86b8c98374dcc7&selectedTaskRun=Y3VufJGiTPCPVmHg0ZMCGg.0
This is now failing even where the job was green before: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception%2Crunnable&searchStr=py3&revision=68ec3d16cb3ffcd223849b31e9a9c06d40a5083e&selectedTaskRun=Tr6dGUFRTkKh1lNaVSRTNQ.0

Hi Andrew! Can you please take a look at this?
Thank you!
btw, there is a conversation about this on Element in build channel.

Same problem reported upstream today.

I dug into this quite a bit but wasn't able to figure out the root cause. It doesn't occur on every OS or OS version. I can repro on Ubuntu 18.04 but not Ubuntu 20.04. I don't think any change on our end caused this, so I'm leaning towards sitting on this for a day or two and seeing what conclusions upstream comes to.

It looks like this was a breaking change on pypi's end. There is a fix pushed for Poetry and a backport to Poetry 1.1.14.

It doesn't look like a Poetry 1.2 release is being rushed out the door, not that it matters, since we're tied to a specific release (1.2.0a2) and upgrading past that version requires Python 3.7 as our minimum supported version (I am actively working on that).

In the meantime we could disable this test on the hosts/version combos it fails on, and add a check for the same hosts/versions inside ./mach vendor python and gracefully exit early, so people don't try vendoring on those platforms.

https://hg.mozilla.org/mozilla-central/rev/27095572983d

This is permafailing on OS X: https://treeherder.mozilla.org/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception%2Crunnable&searchStr=os%2Cx%2C10.15%2Copt%2Cpython%2C3%2Cunit%2Ctests%2Csource-test-python-mozbuild-macosx1015-64%2Fopt-py3%2Cmbu&revision=f745c54f85264261404d0be6dd14627788ff6485&selectedTaskRun=YrGePWBBTZ6eUsh9fpUSkg.0

There have been 130 failure in the last 7 days: https://treeherder.mozilla.org/intermittent-failures/bugdetails?startday=2022-07-07&endday=2022-07-14&tree=trunk&bug=1765416

Happens on:
-linux1804-64 opt
-macosx1015-64 opt
-windows10-64 opt

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=384335287&repo=autoland&lineNumber=2295

Hmm, yeah we need to temporarily disable test_vendor.py. I'm not sure why some platforms weren't impacted earlier, but all platforms are failing now. The good news is that poetry did push their 1.2.0b3 pre-release so once I get the Python 3.7 minimum version bump out we should be able to resolve this pretty quick.

I did test downgrading to poetry 1.1.14 (which has the fix for this), but it has a different problem where some dependencies aren't handled correctly. So for now, we can't upgrade or downgrade, to resolve this. The only immediate solution is ripping out poetry completely, but I'd rather not do that unless a long-term blocker pops up for the Python 3.7 version bump.

Andreea, could you please disable the test for all platforms and I will review?

Test can be reenabled after the migration to Python 3.7+ (bug 1734402) because
the fixed Poetry version (1.2.0) has that as minimum requirement.

Thanks Sebastian!

https://hg.mozilla.org/mozilla-central/rev/9aca18c79f83

Unfortunately the temporary fix added in bug 1782785 does not work when ./mach vendor python is run in python-test, so these tests will have to stay disabled until the permanent fix can be rolled out with Python 3.7.

I took pity on the sheriffs and disabled this on ESR102 now too :-)
https://hg.mozilla.org/releases/mozilla-esr102/rev/0648fb6ffd0b841ae42a5a9f32538156501286dc