Provide a AMQP 1.0 implementation for oslo.messaging

Registered by Flavio Percoco on 2013-12-17

We already have rabbit and qpid drivers
for earlier (and different!) versions of AMQP, the proposal would be
to add an additional driver for a _protocol_ not a particular broker.
(Both RabbitMQ and Qpid support AMQP 1.0 now).

By targeting a clear mapping on to a protocol, rather than a specific
implementation, we would simplify the task in the future for anyone
wishing to move to any other system that spoke AMQP 1.0. That would no
longer require a new driver, merely different configuration and
deployment. That would then allow openstack to more easily take
advantage of any emerging innovations in this space.

http://lists.openstack.org/pipermail/openstack-dev/2013-December/021596.html

Blueprint information

Status:
Complete
Approver:
Doug Hellmann
Priority:
Medium
Drafter:
Gordon Sim
Direction:
Approved
Assignee:
Gordon Sim
Definition:
Approved
Series goal:
Accepted for juno
Implementation:
Implemented
Milestone target:
milestone icon 1.4.0
Started by
Flavio Percoco on 2014-01-24
Completed by
Doug Hellmann on 2014-09-15

Related branches

Sprints

Whiteboard

The driver is being developed in this fork of oslo.messaging: https://github.com/FlaPer87/oslo.messaging/blob/gordon/oslo/messaging/_drivers/protocols/amqp/driver.py (updated link, driver working using intermediaries, branch with direct communication option coming very soon)

---

Removing from @icehouse for now - unless something changes dramatically, it's not going to make it -- @markmc

---

What would we need to do to get it reconsidered for icehouse in terms of timelines, process etc? It is an optional component, with existing stable alternatives, but getting it in early would allow for more 'real world' experimentation and feedback. Could it for example be included in some sort of 'preview', i.e. with its relative immaturity being clearly indicated?

---

The feature freeze for icehouse-3 is coming up. Changes need to be committed to the official oslo.messaging repository by 6 March 2014 in order to make the milestone. If it doesn't make icehouse, we can still work on adding it for juno-1, which would let you test with the other projects early in the juno cycle. -- dhellmann

---

Gerrit topic: https://review.openstack.org/#q,topic:bp/amqp10-driver-implementation,n,z

Addressed by: https://review.openstack.org/75815
    An initial implementation of a 1.0 based messaging driver

---

Spec proposed here: https://review.openstack.org/96729

Proposal to add pyngus: https://review.openstack.org/103616 *ABANDONED*

Main blocking issues for Juno are a) the availability of the proton C library in distros like Ubuntu and b) the availability of the proton python bindings on PyPI

Update 8/1/2014:

1) The proton python bindings are now available
2) The bindings are (still) dependent on the availability of the Proton C development library (needed when building the python extension)
3) Have moved the AMQP 1.0 unit tests and requirements into an optional tox testenv:

      tox -e amqp1

4) Cannot include python bindings in openstack/requirements.txt due to the lack of availability of the proton c development libraries on Ubuntu.
5) Have proposed a change to devstack which allows the Proton C dev libraries to be installed based on configuration (using the Qpid PPA if necessary):

https://review.openstack.org/#/c/109118/

This devstack change would need to be available in order to create a non-voting gate test of the new driver.

---
Update 8/25/2014:

Return from vacation :)

Working on a new non-voting check job for oslo.messaging that runs the AMQP 1.0 functional tests:

https://review.openstack.org/#/c/115752/

This job is dependent on the change I proposed to devstack that installs the prerequisite AMQP 1.0 libraries:

https://review.openstack.org/#/c/109118/

Unfortunately, this change was -1 due to the inclusion of the Apache Qpid PPA for ubuntu systems.

As a temporary measure I'm proposing that the non-voting check job only run on EPEL-based CI systems as the packages for the libraries are available via EPEL.

---
Update 9/3/2014:

Have updated the driver patch with feedback from Ilya.

Automated functional testing: https://review.openstack.org/#/c/115752/
 - This defines a CI check test that runs the amqp 1.0 functional tests in response to gerrit review on oslo.messaging.
 - Status: blocked on the following items:
   1) Ian's work to add Fedora 20 bare nodes to the CI pool: https://review.openstack.org/#/c/117397/
 - Next steps: follow up with Ian re: the availability of f20 nodes

devstack support: https://review.openstack.org/#/c/109118/
 - Adds a parameter that configures the AMQP 1.0 protocol for the rpc_backend
 - Status: blocked:
   1) none of the current CI systems support a broker that can support AMQP 1.0 yet:
      1a) qpid for centos6 is too old. I'm working with epel to see if we can get a more current version of qpidd available if possible.
   2) the proton libraries are not available on ubuntu
 - next steps: propose having devstack support of amqp 1.0 only for fedora/centos7 +qpidd in the short short term.

integration gate tests: TBD
- Requires devstack support first (I believe). Suffers from the same CI platform/packaging issues as devstack.

Documentation:
- Updating http://docs.openstack.org/icehouse/config-reference/content/configuring-rpc.html to add the new functionality. Patch coming shortly.

We have completed the majority of the work for this. There are some remaining tasks to handle during Kilo. - dhellmann

---
Update 9/16/2014 (kgiusti):

Devstack support:
Support for AMQP 1.0 has been added to Devstack! https://review.openstack.org/#/c/109118/

This permits us to write devstack-based integration tests, which is my next task.

Gate check test: https://review.openstack.org/#/c/115752/
The proposed functional gate test is still in review, but the proposed feature it depends on (bare f20 nodes) has just gotten 2 x +2.

User documentation: https://review.openstack.org/#/c/120860/
Needs review!
Instead of updating the config-reference first, I've created driver documentation in the oslo.messaging docs directory instead. Once this documentation get reviewed and accepted, I'll use it to update the config-reference.

(?)

Work Items

Work items:
Create AMQP 1.0 driver: DONE
Create functional tests of the new driver: DONE
Add documentation to the RPC section of the Openstack manual: INPROGRESS
Create non-voting gate test for new driver: INPROGRESS
Support AMQP1.0 in Devstack: DONE
Integration Gate tests: TODO

This blueprint contains Public information 
Everyone can see this information.