I am looking into Bug 1759450 & Bug 1770910 (the latter of which, although marked as duplicate, is more relevant as it has the error log that occurs), We get this error: (e.g MacOS)

21:08:09     INFO -    File "/Users/kshampur/wrkspc/mozilla-unified/third_party/python/virtualenv/virtualenv.py", line 55, in <module>
21:08:09     INFO -      run()
21:08:09     INFO -    File "/Users/kshampur/wrkspc/mozilla-unified/third_party/python/virtualenv/virtualenv.py", line 51, in run
21:08:09     INFO -      run_virtualenv()
21:08:09     INFO -    File "/Users/kshampur/wrkspc/mozilla-unified/third_party/python/virtualenv/virtualenv/__main__.py", line 18, in run
21:08:09     INFO -      session = cli_run(args, options, env)
21:08:09     INFO -    File "/Users/kshampur/wrkspc/mozilla-unified/third_party/python/virtualenv/virtualenv/run/__init__.py", line 30, in cli_run
21:08:09     INFO -      of_session = session_via_cli(args, options, setup_logging, env)
21:08:09     INFO -    File "/Users/kshampur/wrkspc/mozilla-unified/third_party/python/virtualenv/virtualenv/run/__init__.py", line 48, in session_via_cli
21:08:09     INFO -      parser, elements = build_parser(args, options, setup_logging, env)
21:08:09     INFO -    File "/Users/kshampur/wrkspc/mozilla-unified/third_party/python/virtualenv/virtualenv/run/__init__.py", line 69, in build_parser
21:08:09     INFO -      discover = get_discover(parser, args)
21:08:09     INFO -    File "/Users/kshampur/wrkspc/mozilla-unified/third_party/python/virtualenv/virtualenv/run/plugin/discovery.py", line 16, in get_discover
21:08:09     INFO -      discover_types = Discovery.entry_points_for("virtualenv.discovery")
21:08:09     INFO -    File "/Users/kshampur/wrkspc/mozilla-unified/third_party/python/virtualenv/virtualenv/run/plugin/base.py", line 14, in entry_points_for
21:08:09     INFO -      return OrderedDict((e.name, e.load()) for e in cls.entry_points().select(group=key))
21:08:09     INFO -    File "/Users/kshampur/wrkspc/mozilla-unified/third_party/python/virtualenv/virtualenv/run/plugin/base.py", line 19, in entry_points
21:08:09     INFO -      PluginLoader._ENTRY_POINTS = entry_points()
21:08:09     INFO -    File "/Users/kshampur/.pyenv/versions/3.10.4/lib/python3.10/importlib/metadata/__init__.py", line 1009, in entry_points
21:08:09     INFO -      return SelectableGroups.load(eps).select(**params)
21:08:09     INFO -    File "/Users/kshampur/.pyenv/versions/3.10.4/lib/python3.10/importlib/metadata/__init__.py", line 459, in load
21:08:09     INFO -      ordered = sorted(eps, key=by_group)
21:08:09     INFO -    File "/Users/kshampur/.pyenv/versions/3.10.4/lib/python3.10/importlib/metadata/__init__.py", line 1006, in <genexpr>
21:08:09     INFO -      eps = itertools.chain.from_iterable(
21:08:09     INFO -    File "/Users/kshampur/.pyenv/versions/3.10.4/lib/python3.10/importlib/metadata/_itertools.py", line 16, in unique_everseen
21:08:09     INFO -      k = key(element)
21:08:09     INFO -  AttributeError: 'PathDistribution' object has no attribute '_normalized_name'
21:08:09    ERROR - Return code: 1
21:08:09    ERROR - 1 not in success codes: [0]
21:08:09    ERROR - failures found while parsing output
21:08:09  WARNING - setting return code to 2
21:08:09    FATAL - Halting on failure while running ['/Users/kshampur/.mozbuild/srcdirs/mozilla-unified-bcd9b1125b72/_virtualenvs/mach/bin/python', '/Users/kshampur/wrkspc/mozilla-unified/third_party/python/virtualenv/virtualenv.py', '--always-copy', '/Users/kshampur/wrkspc/mozilla-unified/obj-aarch64-apple-darwin21.5.0/testing/raptor-venv']
21:08:10    FATAL - Running post_fatal callback...
21:08:10    FATAL - Exiting 2
21:08:10     INFO - Running post-action listener: _resource_record_post_action
21:08:10     INFO - Running post-action listener: _start_resource_monitoring
21:08:10    ERROR - Exception during post-action for create-virtualenv: Traceback (most recent call last):
21:08:10    ERROR -   File "/Users/kshampur/wrkspc/mozilla-unified/testing/mozharness/mozharness/base/script.py", line 2292, in run_action
21:08:10    ERROR -     self._possibly_run_method(method_name, error_if_missing=True)
21:08:11    ERROR -   File "/Users/kshampur/wrkspc/mozilla-unified/testing/mozharness/mozharness/base/script.py", line 2244, in _possibly_run_method
21:08:11    ERROR -     return getattr(self, method_name)()
21:08:11    ERROR -   File "/Users/kshampur/wrkspc/mozilla-unified/testing/mozharness/mozharness/mozilla/testing/raptor.py", line 1102, in create_virtualenv
21:08:11    ERROR -     super(Raptor, self).create_virtualenv(modules=modules)
21:08:11    ERROR -   File "/Users/kshampur/wrkspc/mozilla-unified/testing/mozharness/mozharness/base/python.py", line 514, in create_virtualenv
21:08:12    ERROR -     self.run_command(
21:08:12    ERROR -   File "/Users/kshampur/wrkspc/mozilla-unified/testing/mozharness/mozharness/base/script.py", line 1664, in run_command
21:08:12    ERROR -     self.fatal(
21:08:12    ERROR -   File "/Users/kshampur/wrkspc/mozilla-unified/testing/mozharness/mozharness/base/log.py", line 245, in fatal
21:08:12    ERROR -     self.log(message, level=FATAL, exit_code=exit_code)
21:08:12    ERROR -   File "/Users/kshampur/wrkspc/mozilla-unified/testing/mozharness/mozharness/base/log.py", line 126, in log
21:08:13    ERROR -     return self.log_obj.log_message(
21:08:13    ERROR -   File "/Users/kshampur/wrkspc/mozilla-unified/testing/mozharness/mozharness/base/log.py", line 617, in log_message
21:08:13    ERROR -     raise SystemExit(exit_code)
21:08:13    ERROR - SystemExit: 2
21:08:13    ERROR - 
21:08:14    ERROR - During handling of the above exception, another exception occurred:
21:08:14    ERROR - 
21:08:14    ERROR - Traceback (most recent call last):
21:08:14    ERROR -   File "/Users/kshampur/wrkspc/mozilla-unified/testing/mozharness/mozharness/base/script.py", line 2304, in run_action
21:08:14    ERROR -     method(action, success=success and self.return_code == 0)
21:08:14    ERROR -   File "/Users/kshampur/wrkspc/mozilla-unified/testing/mozharness/mozharness/base/python.py", line 688, in _start_resource_monitoring
21:08:15    ERROR -     self.activate_virtualenv()
21:08:15    ERROR -   File "/Users/kshampur/wrkspc/mozilla-unified/testing/mozharness/mozharness/base/python.py", line 615, in activate_virtualenv
21:08:15    ERROR -     exec(open(activate).read(), dict(__file__=activate))
21:08:15    ERROR - FileNotFoundError: [Errno 2] No such file or directory: '/Users/kshampur/wrkspc/mozilla-unified/obj-aarch64-apple-darwin21.5.0/testing/raptor-venv/bin/activate_this.py'
21:08:15     INFO - [mozharness: 2022-06-30 21:08:15.872295Z] Finished create-virtualenv step (failed)
21:08:15    FATAL - Aborting due to failure in post-action listener.
21:08:16    FATAL - Running post_fatal callback...
21:08:16    FATAL - Exiting -1
21:08:16     INFO - Running post-run listener: _resource_record_post_run

Of interest would be ``AttributeError: 'PathDistribution' object has no attribute '_normalized_name'

similarly, you can also find the same error on Fedora here (And with entire log). I don't know about Windows, but I would imagine it is the same deal.

After doing some searching, regarding: AttributeError: 'PathDistribution' object has no attribute '_normalized_name' I think the issue might be with some combination of:

• setuptools
• virtualenv
• backports.entry_points_selectable
• Importlib_metadata

But I am unsure how to best proceed.

I was hoping someone could help me debug this. One way I bypassed this (just to see if things could even work) was hardcoding needs_backport to True.

Obviously that is not the best way to deal with that, but it bypassed the PathDistribution error. (Also revealed another error where Raptor is installing an "older" wptserve-3.0 from https://pypi.pub.build.mozilla.org/pub/, which seems to be slightly behind what is found here. Probably outside the scope of this bug, but thought I'd mention it for the sake of completeness. The first stage is still dealing with PathDistribution error)

Regarding the PathDistrubition _normalized_name error: Seeing these people having similar issues seemed to indicate the issue might be with virtualenv/setuptools etc.
https://github.com/pypa/setuptools/issues/3102
https://github.com/pypa/setuptools/issues/3169
https://phabricator.wikimedia.org/T301690
https://github.com/saltstack/salt/issues/61819
https://github.com/pypa/pipenv/issues/4883

how to reproduce

On Mac I just set python to 3.10 with pyenv and clobber&rebuild artifact and ran a browsertime test e.g : ./mach raptor --browsertime -t twitter

On Fedora, well 3.10 ships by default, but the same above applies

The obvious workaround is just downgrading python to <3.10, but it would be nice to have Raptor be 3.10 ready

I think the problem is with our version of importlib_metadata since the code on GitHub at main has _normalized_name, but our vendored version does not.

Digging through the comments in one of your links:

https://github.com/pypa/setuptools/issues/3102#issuecomment-1039628434

The issue is that importlib_metadata is expected to exist at exactly one version, but now that there's a vendored copy in Setuptools, one could encounter multiple, perhaps incompatible versions. The Distribution._normalized_name property was added in importlib_metadata 4.3, so the presence of importlib_metadata 4.2 causes its MetadataPathFinder to be installed and thus for Distribution objects without _normalized_name property to be provided.

Which sounds correct, since that's what I'm seeing. We do have a version of importlib_metadata in tree that has _normalized_name, and I'm just going to assume that's coming from setuptools as per the comments in your links.

So the good news is I'm already working on upgrading the version of importlib_metadata towards resolving something else. I'm already bumping the version to 4.4 and as per the GitHub comment above, 4.3 or higher should be sufficient to fixing this problem.

The bad news is that I'm blocked from upgrading importlib_metadata because our version of poetry requires this specific version of importlib_metadata (As well as packaging, which I also plan to upgrade). I want to upgrade poetry to the newer version as well, but I unfortunately found that the version I want to upgrade to dropped support for Python 3.6. So I can't do that upgrade until I do the Python 3.7 minimum version bump.

I'm actively working towards making Python 3.7 the required minimum version, so hopefully this will be resolved sooner, rather than later.

Thanks for the follow-up Alex! That is great to hear this may be resolved with already in-progress patches. I look forward to this!

The specific error in comment 0 will be fixed by bug 1766497, which is pending some CI migrations.

Closing this, should be resolved now in Bug 1759450