Add a asyncio executor to oslo.messaging

Registered by Flavio Percoco on 2013-12-04

The Enovance company is working actively on porting OpenStack on Python 3, see the status on the Python3 page. Eventlet dependency blocks the portage because it is not yet compatible with Python 3. Even if it would be available, we would like to get ride of eventlet for the reasons listed below.

We choose to start the work in the Olso Messaging project because its design allows to use different "executor drivers", a function to run a task asynchronously. Olso Messaging is a core component and so should be ported on Python 3 first to be able to port other components on Python 3.

The asyncio module is a new and modern asynchronous library which is part of the Python 3.4 standard library. The idea is to add a new asyncio executor to Olso Messaging. At the beginning, asyncio will reuse the greenlet event loop. If an OpenStack server doesn't use eventlet anymore, it will be possible to use an asyncio event loop.

Patches:

* openstack/requirements: https://review.openstack.org/#/c/70983/ Add a new dependency: trollius (Merged)
* Olso Messaging: https://review.openstack.org/#/c/71003/ Add an optional timeout parameter to Listener.poll
* Olso Messaging: https://review.openstack.org/#/c/70948/ Add a new asynchronous executor based on Trollius

--

Originial blueprint of Flavio Percoco:

Oslo messaging supports different type of executors. As of the time of this blueprint registration, there are 2 executors currently supported: Eventlet and Blocking.

This blueprint proposes the creation of an executor for asyncio[0] which is part of Python's 3.4 stdlib and has a package for Python 3.3

[0] http://docs.python.org/dev/library/asyncio.html

Victor Stinner: Flavio gave me the right to modify its blueprint. I created this blueprint before I noticed his blueprint:
https://blueprints.launchpad.net/oslo.messaging/+spec/asyncio

Blueprint information

Status:
Not started
Approver:
Mark McLoughlin
Priority:
High
Drafter:
Flavio Percoco
Direction:
Approved
Assignee:
Victor Stinner
Definition:
Approved
Series goal:
None
Implementation:
Deferred
Milestone target:
None

Related branches

Sprints

Whiteboard

Joshua Harlow: Why directly jump into asyncio (a 3.4/3.3(?) feature) when http://docs.python.org/dev/library/concurrent.futures.html exists in 3.2 (and has been backported to 2.6/2.7)? The executor concept there seems to be somewhat similar to the one in oslo.messaging.

I've even made an eventlet compatiblity layer @ https://github.com/stackforge/taskflow/blob/master/taskflow/utils/eventlet_utils.py that makes the same concept work with eventlet. Just a thought, would it be applicable?

--

I think we should go to asyncio because we already have py27 covered with eventlet and we could also add another executor for gevent (which has some work going on for the Py3K compat). The other reason is that we already have a gate for py33 and we won't have one for py32.

-- flaper87

--

I did a mockup of this here: https://github.com/markmc/oslo-incubator/blob/8509b8b/openstack/common/messaging/_executors/impl_tulip.py

and the background is discussed here: http://lists.openstack.org/pipermail/openstack-dev/2013-June/009934.html

-- @markmc

--

Hi, I created a duplicate blueprint because I didn't know this one:
https://blueprints.launchpad.net/oslo.messaging/+spec/asyncio

Full specification:
https://wiki.openstack.org/wiki/Oslo/blueprints/asyncio

I ported asyncio on Python 2.6/2.7, it's the new Trollius project:
https://bitbucket.org/enovance/trollius

Using the greenio module, it becomes possible to use the asyncio API with the greenlet event loop (greenio is an asyncio event loop which calls greenlet).

I just posted an implementation of the blueprint. The patchset is still a work-in-progress:
https://review.openstack.org/#/c/70948

Would it be possible to merge the two blueprints? I don't really care of the name, but this one is accepted, whereas mine is still a draft.

Update (2014-02-12): I merged my blueprint wit Flavio's blueprint. Thanks Flavio.

-- @haypo (victor Stinner, Enovance)

---

After some discussion, we're planning on adding a greenio executor in Juno - see https://blueprints.launchpad.net/oslo.messaging/+spec/asyncio-executor

A pure asyncio/trollius executor will come in a future release when we have an OpenStack service sufficently ported that it can use it.

-- @markmc

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.