Allow bulk consumption of message from the message bus

Registered by John Herndon on 2013-12-17

Ceilometer is currently trying to handle extremely large numbers of notifications coming from oslo.messaging. We have found that it is much more efficient to send a batch of events to a storage backend than it is to handle them individually. We would like to add an interface to oslo.messaging that will build up a batch of N messages before sending them to the consumer. This interface would also accept an optional timeout, so after T seconds, even if N messages have not been received, the batch will be sent to the consumer anyway.

Implementation:
- After reviewing the current oslo.messaging code, I believe the best way to implement this feature is to create a new executor called impl_bulk. This executor will store the notifications in a batch until a threshold is reached, or until a timeout is hit, and will then pass them along to the dispatcher.

- The current implementation catches the requeue exception and will make sure the messages is requeued. This becomes more difficult if the events are stored in a batch. Our current idea is to have the invoked method return a list of messages that need to be requeued. If an exception is caught, the executor will likely need to requeue all messages.

Blueprint information

Status:
Complete
Approver:
Doug Hellmann
Priority:
Undefined
Drafter:
John Herndon
Direction:
Needs approval
Assignee:
John Herndon
Definition:
Obsolete
Series goal:
None
Implementation:
Unknown
Milestone target:
None
Completed by
John Herndon on 2014-01-10

Related branches

Sprints

Whiteboard

(thomasem) Some additional considerations would be things like what to do with failed messages, or messages that we want to retry. John has some details on bulk acknowledgement of messages which would directly support such a feature. An etherpad here for more detailed thoughts: https://etherpad.openstack.org/p/batch_consume_amqp

(herndon) After some consideration, I think we will implement everything inside of ceilometer as a first step, with the option of moving it into oslo if so desired.

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.