Python 3 Support

Registered by Michael Terry

The aim is that we get to full Python 2.7+/3.4+ support in duplicity 0.8 and then cut off Python 2 support (move to Python 3 only) for 0.9.

We will use the http://python-future.org/futurize.html approach.

Blueprint information

Status:
Complete
Approver:
Kenneth Loafman
Priority:
High
Drafter:
Aaron Whitehouse
Direction:
Approved
Assignee:
Aaron Whitehouse
Definition:
Approved
Series goal:
Proposed for 0.8
Implementation:
Implemented
Milestone target:
milestone icon 0.8.00
Started by
Aaron Whitehouse
Completed by
Aaron Whitehouse

Whiteboard

I'm going to close this again now, as the Python 3 support should be a first-class citizen and anything that is not working now (e.g. backends) is best dealt with as a bug.

==
Moving this back a stage, as while we have Python 3 compatible code in 0.8-series with all tests also being run against this environment, some of the backends are not supported in Python 3, e.g.:
rfc822, used by imapbackend
Pyrax, which has been deprecated in favour of rackspacesdk, built on the OpenStack SDK (or libcloud).
we need to do a proper review and test each of the ported backends are working as expected (and ideally build this into our automated testing, though that is always a challenge with backends that need credentials).

We can remove Python 2 separately in the future.

Original draft Blueprint by Michael Terry (https://launchpad.net/~mterry).

=== Approach ===
Please see the various mailing list discussions about this:
https://<email address hidden>/msg04613.html
https://<email address hidden>/msg04614.html
https://lists.nongnu.org/archive/html/duplicity-talk/2015-07/msg00018.html
(and threads linked within).

The aim is that we get to full Python 2.7+/3.4+ support in duplicity 0.8 and then cut off Python 2 support (move to Python 3 only) for 0.9.

One of the big risks is that we are working on a Python 3 branch separate to a working Python 2 version and these get out of sync (as has happened in previous attempts). We therefore want to push all compatible changes back into trunk ASAP.

We will use the http://python-future.org/futurize.html approach.

The futurize tool spits out code looks and feels like Python 3, but runs on Python 2.7 with imports that make those Python 2 functions act like their Python 3 equivalents. We can then maintain just the one branch.

Then in 0.9 we can drop the compatibility imports.

One of the most time consuming, and least easy to automate, parts of supporting both Python 2 and 3 is string literals. This is because simple strings (e.g. a = "Hello") will be treated as bytes (e.g. encoded ASCII) in Python 2 and Unicode in Python 3. As we are trying to support both Python 2 and Python 3 for at least a transition period, we may end up with odd behaviour wherever we have an unadorned string. The versions of Python 2 and 3 we are targeting means that we can "adorn" strings with letters to indicate what type of string (u for Unicode or b for Bytes). An important preliminary step to Python 2/3 support is therefore for us to add these adornments to each and every string literal in the code base. See
https://blueprints.launchpad.net/duplicity/+spec/adorn-string-literals
for details on this preliminary task.

(?)

Work Items

Work items:
Investigate list of dependencies that still need Python 3 support. See: https://<email address hidden>/msg04613.html : DONE
mark string literals with u/b https://blueprints.launchpad.net/duplicity/+spec/adorn-string-literals : DONE
convert str() to unicode()/bytes() -- we can potentially futurize all str()s to bytes() and then chip away at converting bytes to (unicode) strings over time: DONE
http://python-future.org/futurize.html stage 1, which should still be valid Python 2 code: DONE
http://python-future.org/futurize.html stage 2. Set up tox etc for both Python 3 and Python 2 testing. fix remaining 2to3 exceptions (https://bazaar.launchpad.net/~duplicity-team/duplicity/0.8-series/view/head:/testing/test_code.py test_2to3) that are not done by futurize. All needs to happen in one go, with risk of falling out of sync with trunk: DONE

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.