Closed Bug 1330474 Opened 8 years ago Closed 6 years ago

Switch Treeherder from Python 2.7 to Python 3.6

Categories

(Tree Management :: Treeherder, defect, P2)

Product:
Tree Management
Component:
Treeherder
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: emorley, Assigned: emorley)

References

Details

Attachments

(2 files, 1 obsolete file)

Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/4668
(deleted), text/x-github-pull-request
Details
Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/4675
(deleted), text/x-github-pull-request
Details
The upcoming Django 1.11 release (April 2017) will be the last that supports Python 2.7. Django 2.0 (due December 2017) will only support Python 3.5+. See: https://docs.djangoproject.com/en/dev/faq/install/#what-python-version-can-i-use-with-django Now that we're no longer using Datasource (which was Python 2 only) virtually all (if not all) of our dependencies are Python 3 compatible. This will still be some work (and no rush until nearer the end of this year), but it's now actually within the realm of possibility.
As of today Django master is Python 3.x only. (Django 1.11a1 was released yesterday, the final release will be in ~2 months from the stable/1.11.x branch).
Hi I would really like to contribute in open source and specially in my domain i.e python.A little help would be highly appreciated. Thanks!
Hi Rupanshu - thank you for your interest in contributing! This hadn't been marked as a "good first bug" since it's probably a bit more of a tricky one, however if you still want to take a look, the best place to start is making sure all of the Python packages used by Treeherder are Python 3 compatible: https://pyup.io/account/repos/github/mozilla/treeherder/ (Click the "Python 3" button to show the compatibility column) Currently the following aren't marked as compatible with Python 3: * mozlog -> bug 1388019 filed for this * futures -> though this won't be needed for Python 3. * functools32 -> though this won't be needed for Python 3. * coreapi -> I've previously filed https://github.com/core-api/python-client/issues/130 * coreschema -> I've previously filedhttps://github.com/core-api/python-coreschema/issues/6 * itypes -> I've previously filed https://github.com/tomchristie/itypes/issues/6 * sphinxcontrib-httpdomain -> though this only really runs on Read The Docs, so isn't urgent. -> perhaps still file an issue here: https://bitbucket.org/birkenfeld/sphinx-contrib ? For the ones that aren't needed on Python 3, use the special requirements file notation to limit them to Python2: https://www.python.org/dev/peps/pep-0508/#environment-markers For the rest, I'd: 1) Check if their tests are already running against Python 3. 2) If so, just update the trove classifiers in setup.py to make it clear that they do support Python 3. 3) If not: * Make their tests run against Python 3 (for things on GitHub this will typically mean adding python 3 to .travis.yml; for mozlog this will mean depending on bug 1388013 and bug 1385381) * Fix any test failures * Add the trove classifiers the same as above Once all of that is complete, next steps will likely be running some of the Python 3 linters against the Treeherder codebase to see what needs changing there.
Depends on: 1388019
coreschema and core-api just needed trove classifier updates, so I've opened: https://github.com/core-api/python-client/pull/144 https://github.com/core-api/python-coreschema/pull/9 itypes didn't actually run its tests on Travis yet, so I've added a Travis config and the trove classifiers (since the tests already passed on Python 3): https://github.com/tomchristie/itypes/pull/8
Depends on: 1326247
Depends on: 1389592
For inspiration as to an initial lint check we could try, see bug 1390968.
Not that it blocks the work here, but I've opened an issue against Pyup.io to see if they could fix the "PY3: X" notations appearing against the packages that are not intended for use with Python 3 (and for which we've already added the conditional syntax to the requirements file), to avoid confusion: https://github.com/pyupio/pyup/issues/247
Some useful resources: https://eev.ee/blog/2016/07/31/python-faq-how-do-i-port-to-python-3/ http://python-future.org/
Summary: Switch Treeherder from Python 2.7 to Python 3.5+ → Switch Treeherder from Python 2.7 to Python 3.6
Blocks: 1426683
Depends on: 1428045
Django 1.11 is EOL on April 2020: https://www.djangoproject.com/download/#supported-versions
Depends on: 1447735
Depends on: 1428362
No longer depends on: 1447735
Depends on: 1451518
(In reply to Ed Morley [:emorley] from comment #7) > Some useful resources: Also: https://docs.djangoproject.com/en/1.11/topics/python3/
Depends on: 1451774
Depends on: 1370246
Depends on: 1452420
Depends on: 1453837
Depends on: 1453850
Depends on: 1453967
Summary: Switch Treeherder from Python 2.7 to Python 3.6 → Switch Treeherder from Python 2.7 to Python 3
Blocks: 1478059
No longer blocks: 1478059
Blocks: 1489212

There are now three bugs directly or indirectly blocked by this conversion work. Since it's only just over a year before Django 1.11 is EOL, I think we need to prioritise this soon - since we'll need to not only switch to Python 3, but also upgrade through 3 breaking change versions of Django to be on a supported release in that timeframe.

On the plus side, I believe this is now very close. I have a few WIP branches locally for some of the remaining issues, though hit some quirks wrt log parsing and line numbers not matching when using requests' decode_unicode=True in streaming mode.

Priority: P3 → P2
Depends on: 1526657
Depends on: 1526682
Depends on: 1526743
Depends on: 1526763
Depends on: 1526764
Depends on: 1526766
Blocks: 1526911
Depends on: 1527020

Ah so it turns out that Celery doesn't yet support Python 3.7 (fix on celery master, not yet released: https://github.com/celery/celery/issues/4500), so we'll need to use Python 3.6 instead for now. Once they support Python 3.7, and we've had a chance to figure out the celery upgrade issue in bug 1337717, we can then upgrade.

This incompatibility didn't show up during tests likely due to bug 1349362. It could be seen via Papertrail in the form of:

app/worker_store_pulse_jobs.4:   File "/app/.heroku/python/lib/python3.7/site-packages/celery/concurrency/prefork.py", line 20, in <module> 
app/worker_store_pulse_jobs.4:     from celery.concurrency.base import BasePool 
app/worker_store_pulse_jobs.4:   File "/app/.heroku/python/lib/python3.7/site-packages/celery/concurrency/base.py", line 21, in <module> 
app/worker_store_pulse_jobs.4:     from celery.utils import timer2 
app/worker_store_pulse_jobs.4:   File "/app/.heroku/python/lib/python3.7/site-packages/celery/utils/timer2.py", line 19 
app/worker_store_pulse_jobs.4:     from kombu.async.timer import Entry, Timer as Schedule, to_timestamp, logger 
app/worker_store_pulse_jobs.4:                    ^ 
app/worker_store_pulse_jobs.4: SyntaxError: invalid syntax
Blocks: 1527336
Assignee: nobody → emorley
Status: NEW → ASSIGNED
Depends on: 1528485
Depends on: 1528601
Summary: Switch Treeherder from Python 2.7 to Python 3 → Switch Treeherder from Python 2.7 to Python 3.6
Depends on: 1528710

A Python 3 compatible version of treeherder-client will be released in bug 1529191.

Blocks: 1529243

The switch to Python 3 was deployed to production at 08:12am UTC Thursday 21st.

No issues seen so far. Only notable change I've seen is a ~20% reduction in memory usage on certain worker dyno types, which is a welcome additional benefit :-)

Once this has been deployed for a bit longer, we'll remove Travis testing against Python 2, at which point this bug can be closed, and PRs opened for the bugs this one is marked as blocking.

Everything has been looking great on prod using Python 3 since last week.

Testing of Python 2 (for fallback purposes) removed in:
https://github.com/mozilla/treeherder/commit/a0d1f5fccfeb0a6cd6593d5017e4b70a8a191446

We're now fine to move on with the bugs that required we drop Python 2 support.

Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Blocks: 1530267
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: