Python 3.7 has some cool new features, some of which that I'm most excited about are:

• dataclasses
• postponed typing annotation evaluation

Python 3.6 is end-of-life'd on December 23 this year.

One question here is whether downstream Firefox builders (e.g. old Debian, Ubuntu, CentOS, etc) can provide Python 3.7 for the build, or if they're only running Python 3.6 with their own security patches. Investigation is needed.

Ubuntu's policy on stable release updates is to apply selected security patches rather than update to a new version.
Ubuntu 18.04, generally supported until April 2023, has Python 3.6.7.

That release is scheduled to continue receiving security updates for another 5 years (until 2028), but in theory this excludes applications like Firefox, for which the official recommendation is to install the snap package (which itself is built on a more recent base, so not affected by this problem).

Ubuntu 16.04 is already out of the equation for the same reason, i.e. we're not building Firefox there any longer.
When it was still built there (until April 2021), we had to carry a set of distro-patches to revert some Python 3.6 specifics (because 16.04 only had 3.5.1):

https://bazaar.launchpad.net/~mozillateam/firefox/firefox.xenial/view/head:/debian/patches/lower-python3-requirement.patch
https://bazaar.launchpad.net/~mozillateam/firefox/firefox.xenial/view/head:/debian/patches/python3-remove-variable-annotations.patch
https://bazaar.launchpad.net/~mozillateam/firefox/firefox.xenial/view/head:/debian/patches/python3-remove-fstrings.patch
https://bazaar.launchpad.net/~mozillateam/firefox/firefox.xenial/view/head:/debian/patches/python3-remove-pep487.patch

When the requirement is bumped to Python 3.7, we'll likely have to do something similar to continue building Firefox on Ubuntu 18.04 until April 2023.

Tracking impact to other distros:

• I received an email back from Martin Stránský from Red Hat, and it sounds like RHEL 8 is able to build Firefox with Python 3.8 because it's in "modules".

Hmm, that makes sense, thanks Olivier.
To confirm, it sounds like "Standard Support" vs "Extended Security Maintenance" handles applications like Firefox differently:

• In "Standard Support": Firefox is still built using packages associated with the current Ubuntu release, and is itself available as an official apt-installable package. Installing via snap is also allowed.
• In "Extended Security Maintenance", Firefox is no longer built or available as an official apt-installable package, but can only be installed via snap.
  • Unrelated, but I wonder what happens if the firefox package is installed in 18.04 when it ticks over to "Extended Security Maintenance", is it automatically uninstalled and replaced with the associated snap package?

Hmm, this will require some further pondering then, since you have to patch out all Python 3.7 usages, which I can understand to be annoying.
I bet the most impactful feature of Python 3.7 that will be used is dataclasses, which would require simple but boilerplate-y __init__() creation.
There's also the various new parameters and function changes that would require backports. Fortunately, they should mostly all "fail fast" (at call-time), but may not be trivial to fix.

One option here is that, with the recent improvements to Python dependency-management, we might be able to do something flexible like allow specific Mach commands to use newer Python versions, while still forcing Mach and the build to be compatible with Python 3.6.

I'm going to chew on this, thanks :)

There is some pressure to update wpt to 3.7+ because various dependencies (like the HTTP/3 server) are only releasing updates for 3.7+. I assume Ubuntu don't need to be able to run tests, so if we could update test jobs in CI to 3.7, that would be very welcome, even if the build itself continues to support 3.6 for another year.

I've created a ticket to track that here.
Note that, if you're bumping your vendored packages to be Python 3.7+, then you'll need to tweak Mach to not depend on them (moving them to a WPT-specific site/virtualenv).

Just as a further note: wpt is now in the position where we're having to patch third party libraries we've vendored in to get both Python 3.10 and 3.6 support, and has stopped taking almost any dependency updates because the ecosystem seems to be fully committed to 3.7+ as the targets. I think the local-only deployment of wpt, and lack of untrusted input, makes it less likely that any patched security issues in older library versions are realistically a problem, but it's at least uncomfortable.

Backed out for causing mda failures on Android.

• Backout link
• Push with failures
• Failure Log
• Failure line: Could not install python package: failed all attempts.

This should work now, as of bug 1804514.

Backed out 4 changesets (Bug 1734402, Bug 1732228) for build bustages when upgrade pip-23.0.1-py3-none-any.whl.
Backout link
Push with failures
Failure Log

That was weird. The .whl files were 0 bytes on Phabricator. I've re-uploaded the patches now and they have the correct file size. I'll reattempt to land this.

Backed out 4 changesets (Bug 1732228, Bug 1734402) for multiple browsertime and talos failures.
Backout link
Push with failures <--> cnn <--> damp-inspector
cnn Failure Log
damp-inspector Failure Log

See D171176

pip==23.0.1
setuptools==67.4.0
wheel==0.38.4

(wheel is not used by mozharness, but we'll just update it here for good measure)

Depends on D171628

https://hg.mozilla.org/mozilla-central/rev/fa1b65b06266
https://hg.mozilla.org/mozilla-central/rev/7c4f9b1462b2

Backed out for causing build bustages Python related

• Backout link
• Push with failures
• Failure Log
• Failure line: Exception: According to "pip check", the current Python environment has package-compatibility issues.
  [taskcluster:error] Uploading error artifact public/logs from file logs with message "Could not read directory '/var/opt/generic-worker/tasks/task_167815377605851/logs'", reason "file-missing-on-worker" and expiry 2024-03-06T04:25:24.001Z

https://hg.mozilla.org/mozilla-central/rev/bc631037df77

Comment on attachment 9321148 [details]
Bug 1734402 - Version bumps to pip, setuptools, and wheel to maintain consistency with the wheels used by mozharness r?#build

Revision D171629 was moved to bug 1732228. Setting attachment 9321148 [details] to obsolete.

• If this is not done, the next person to vendor will also make all of
  the changes in this commit, which is unnecessary noise.

Depends on D183449

Comment on attachment 9343621 [details]
Bug 1734402 - Regenerate the third_party/python lockfile with Python 3.8 as the minimum version r?#build

Revision D183450 was moved to bug 1843209. Setting attachment 9343621 [details] to obsolete.