Add Python 3 support to Cinder

Registered by Victor Stinner on 2015-06-11

It’s time to add Python 3 support to Cinder by generalizing the usage of the six module, in addition to the Python 2 support.

The goal is to make Cinder compatible with Python 3 at the end of the Liberty cycle.

For the rationale, the plan and work items, please see Nova Python 3 spec, since it's almost the same for Cinder:
http://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/adding-python34-support-to-nova.html

Summary of the plan:

* Write a few bunch of patches to fix the most obvious Python 3 syntax issues (dict.iteritems, 123L, reraise an exception, etc.) to allow to load unit tests
* Find a subset of unit tests which pass on Python 3.4 and add a py34 target in tox.ini to run them.
* Add a non-voting py34 check job running tox -e py34
* When the py34 check job becomes stable enough, make it voting. At this point, it's no more possible to add regressions on the Python 3 in the tests executed by tox -e py34
* Fix remaining tests one by one, each time add it to the py34 target of tox.ini

All Cinder dependencies are now compatible with Python 3! Latest changes related to dependencies:

* MySQL-python is not compatible with Python 3: it was replaced with PyMySQL, https://review.openstack.org/#/c/184369/
* rtslib-fb was not compatible with Python 3: Python 3 support was added in 2.1.56 by Victor Stinner

Python 3 patches for Cinder: https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:py3,n,z

For more information on Python 3, see the wiki page: https://wiki.openstack.org/wiki/Python3

Blueprint information

Status:
Complete
Approver:
Mike Perez
Priority:
Medium
Drafter:
Victor Stinner
Direction:
Approved
Assignee:
Victor Stinner
Definition:
Approved
Series goal:
Accepted for liberty
Implementation:
Implemented
Milestone target:
milestone icon 7.0.0
Started by
Mike Perez on 2015-06-23
Completed by
Mike Perez on 2015-08-15

Related branches

Sprints

Whiteboard

Gerrit topic: https://review.openstack.org/#q,topic:bp/cinder-python3,n,z

Addressed by: https://review.openstack.org/190153
    Replace basestring with six.string_types

Addressed by: https://review.openstack.org/185417
    Use six to fix imports on Python 3

Addressed by: https://review.openstack.org/190169
    Use six.reraise() for Python 3 compatibility

Addressed by: https://review.openstack.org/189720
    Get StringIO from six for Python 3 compatibility

Addressed by: https://review.openstack.org/185418
    Replace xrange() with six.moves.range()

Gerrit topic: https://review.openstack.org/#q,topic:py3,n,z

Addressed by: https://review.openstack.org/190173
    Replace it.next() with next(it) for py3 compat

Addressed by: https://review.openstack.org/190152
    Drop L suffix from long integers

Addressed by: https://review.openstack.org/191012
    san driver: don't use relative Python import

Addressed by: https://review.openstack.org/191019
    Fix Python 3 issues in utils

Addressed by: https://review.openstack.org/191063
    Replace dict.iteritems() with dict.items()

Addressed by: https://review.openstack.org/191064
    Replace dict(obj.iteritems() with dict(obj)

Addressed by: https://review.openstack.org/191065
    Rewrite code merging two dictionaries

Addressed by: https://review.openstack.org/191066
    Replace dit.itervalues() with dict.values()

Addressed by: https://review.openstack.org/192118
    tox: add a basic py34 venv running a single test

Addressed by: https://review.openstack.org/194560
    Fix tox -e py34

Addressed by: https://review.openstack.org/194587
    Fix test_backup unit tests on Python 3

Addressed by: https://review.openstack.org/194588
    Port test_backup_ceph and test_rbd to Python 3

Addressed by: https://review.openstack.org/194629
    Fix Python 3 issues in the blockbridge driver

Addressed by: https://review.openstack.org/194628
    Fix Python 3 issues in the swift backup driver

Addressed by: https://review.openstack.org/197049
    Port dothill to Python 3

Addressed by: https://review.openstack.org/197065
    Port hitachi driver to Python 3

Addressed by: https://review.openstack.org/197068
    Port huawei driver to Python 3

Addressed by: https://review.openstack.org/197070
    Port IBM driver to Python 3

Addressed by: https://review.openstack.org/197072
    Port test_db_api to Python 3

Addressed by: https://review.openstack.org/197075
    Port remotefs driver to Python 3

Addressed by: https://review.openstack.org/197095
    Port drbdmanagedrv driver to Python 3

Addressed by: https://review.openstack.org/197099
    Port image/glance.py to Python 3

Addressed by: https://review.openstack.org/197105
    Fix Python 3 issues in targets unit tests

Addressed by: https://review.openstack.org/197124
    Fix Python 3 issues in cmd

Addressed by: https://review.openstack.org/197127
    Fix Python 3 issues in HDS tests

Addressed by: https://review.openstack.org/197131
    Fix Python 3 issues in Windows tests

Addressed by: https://review.openstack.org/213817
    Port more unit tests to Python 3

Addressed by: https://review.openstack.org/214384
    Port test_nfs to Python 3

Addressed by: https://review.openstack.org/215752
    Port volume transfer to Python 3

Addressed by: https://review.openstack.org/215099
    Python 3 incompatible expression fix

Addressed by: https://review.openstack.org/217408
    py3: Port pure driver test to Python 3

Addressed by: https://review.openstack.org/217417
    Fix Python 3 issues in wsgi

Addressed by: https://review.openstack.org/215729
    Port test_volume to Python 3

Addressed by: https://review.openstack.org/229991
    Fix volume throttling to Python 3

Addressed by: https://review.openstack.org/229998
    emc vmac driver: use integer division for Python 3

Addressed by: https://review.openstack.org/230302
    Fix various Python 3 issues

Addressed by: https://review.openstack.org/230313
    Port scheduler host manager to Python 3

Addressed by: https://review.openstack.org/230503
    Port netapp SSC Cmode to Python 3

Addressed by: https://review.openstack.org/230504
    Port test_netapp to Python 3

Addressed by: https://review.openstack.org/230510
    Port test_quobyte to Python 3

Addressed by: https://review.openstack.org/231995
    py3: Run unit tests with ostestr on Python 3.4

Addressed by: https://review.openstack.org/232002
    Port test_srb to Python 3

Addressed by: https://review.openstack.org/232214
    Port WSGI tests to Python 3

Addressed by: https://review.openstack.org/232215
    Port API to Python 3

Addressed by: https://review.openstack.org/232216
    Port API types extra specs to Python 3

Addressed by: https://review.openstack.org/232218
    Port image cache to Python 3

Addressed by: https://review.openstack.org/232221
    Port EMC scaleio to Python 3

Addressed by: https://review.openstack.org/232227
    Port violin driver to Python 3

Addressed by: https://review.openstack.org/232228
    Port _transfer_data() to Python 3

Addressed by: https://review.openstack.org/232232
    Port targets test_iet_driver to Python 3

Addressed by: https://review.openstack.org/232233
    Port IBM flashsystem to Python 3

Addressed by: https://review.openstack.org/232242
    Port API admin action tests to Python 3

Addressed by: https://review.openstack.org/232249
    WIP: Fix Python 3 issues in EMC vnx client

Addressed by: https://review.openstack.org/234946
    Port Windows drivers to Python 3

Addressed by: https://review.openstack.org/234961
    Port SolidFire to Python3

Addressed by: https://review.openstack.org/242127
    Modify test_hp3par to support random hash

Addressed by: https://review.openstack.org/242128
    Port HP 3part driver to Python 3

Addressed by: https://review.openstack.org/242139
    Port xio driver to Python 3

Addressed by: https://review.openstack.org/242140
    Port vmware datastore to Python 3

Addressed by: https://review.openstack.org/242142
    Port test_tintri to Python 3

Addressed by: https://review.openstack.org/242148
    Port cinder.utils.monkey_patch() to Python 3

Addressed by: https://review.openstack.org/242053
    Port cinder.hacking to Python 3

Addressed by: https://review.openstack.org/242871
    Fix error handling in prophetstor driver

Addressed by: https://review.openstack.org/242872
    Port vzstorage to Python 3

Addressed by: https://review.openstack.org/242873
    Port zfssa driver to Python 3

Addressed by: https://review.openstack.org/242878
    Port IBM storewize_svc driver to Python 3

Addressed by: https://review.openstack.org/243051
    Port EMC Vmax to Python 3

Addressed by: https://review.openstack.org/249397
    Port key manager to Python 3

Addressed by: https://review.openstack.org/249398
    Port zonemanager to Python 3

Addressed by: https://review.openstack.org/249399
    Test middleware test_faults to Python 3

Addressed by: https://review.openstack.org/251576
    Revert "Port xio driver to Python 3"

Addressed by: https://review.openstack.org/251755
    xio: fix regression in authentication

Addressed by: https://review.openstack.org/252641
    Python 3: fix cinder.tests.unit.api.contrib.test_snapshot_{,un}manage

Addressed by: https://review.openstack.org/252642
    Python 3: fix cinder.tests.unit.volume.drivers.netapp.eseries.test_utils

Addressed by: https://review.openstack.org/252643
    Python 3: fix cinder.tests.unit.api.contrib.test_scheduler_hints

Addressed by: https://review.openstack.org/252644
    Python 3: Use oslo_serialization.jsonutils.loads instead of json.loads

Addressed by: https://review.openstack.org/252645
    Python 3: fix cinder.tests.unit.api.contrib.test_volume_{,un}manage

Addressed by: https://review.openstack.org/252646
    Python 3: fix cinder.tests.unit.api.contrib.test_cgsnapshots

Addressed by: https://review.openstack.org/252647
    Python3: fix cinder.tests.unit.api.contrib.test_volume_encryption_metadata

Addressed by: https://review.openstack.org/252648
    Python 3: fix cinder.tests.unit.api.v2.test_volumes

Addressed by: https://review.openstack.org/252649
    Python 3: fix cinder.tests.unit.api.contrib.test_snapshot_actions

Addressed by: https://review.openstack.org/254198
    Python 3: fix a lot of tests

Addressed by: https://review.openstack.org/262526
    Use six.moves.reduce instead of builtin reduce

Gerrit topic: https://review.openstack.org/#q,topic:bug/1268439,n,z

Addressed by: https://review.openstack.org/267656
    Fix: range() is not the same in py3.x and py2.x

Addressed by: https://review.openstack.org/272576
    Port objects unit tests to Python 3

Addressed by: https://review.openstack.org/277778
    Port backup drivers to Python 3

Addressed by: https://review.openstack.org/277835
    Port API v1 to Python 3

Addressed by: https://review.openstack.org/277841
    hacking: Fix false positive in C302 check

Addressed by: https://review.openstack.org/277861
    Port volume_type_encryption to Python 3

Addressed by: https://review.openstack.org/277983
    Port utils.safe_minidom_parse_string() to Python 3

Addressed by: https://review.openstack.org/277989
    py3: Fix usage of JSON in API contrib

Addressed by: https://review.openstack.org/280154
    Port hgst driver to Python 3

Gerrit topic: https://review.openstack.org/#q,topic:bug/1348818,n,z

Addressed by: https://review.openstack.org/280241
    Port test_emc_vnx to Python 3

Addressed by: https://review.openstack.org/280331
    Port netapp dataontap driver to Python 3

Addressed by: https://review.openstack.org/280335
    Fix last Python 3 issues in zonemanager

Addressed by: https://review.openstack.org/284308
    Enable all unit tests on Python 3.4

Addressed by: https://review.openstack.org/416154
    Replace unicode with six.text_type

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.