Redis Storage Driver

Registered by Allele Dev on 2013-07-26

Add a storage driver that understands how to work with Redis DBs. This will be our non-AGPL alternative to MongoDB (durability can be obtained via AOF + RDB).

Developing it outside the main repo is the best test for: 1) Our test suite structure 2) our plugins stuff. Then we can review and merge it later.

This driver will be for use cases where all of the following are true:

1. High-throughput needed (2K+ messages/sec per queue)
2. Lots of small messages (<= 1K)
3. Messages are short-lived (minutes, not hours)
4. High durability is optional (2 copies, RAM only)
5. HA is still very important

This is most likely useful only for worker pool use cases (classic producer-consumer).

TBD

1. Do such workloads require FIFO guarantee given a single producer? (probably not)
2. Do such workloads require once-only delivery for observers (not claiming messages)? (probably not)

NOTE: scaling a single queue and HA will be addressed in a separate blueprint (redis-pool)

Blueprint information

Status:
Complete
Approver:
Flavio Percoco
Priority:
Essential
Drafter:
Kurt Griffiths
Direction:
Approved
Assignee:
Prashanth Raghu
Definition:
Approved
Series goal:
Accepted for juno
Implementation:
Implemented
Milestone target:
milestone icon 2014.2
Started by
Prashanth Raghu on 2014-05-27
Completed by
Flavio Percoco on 2014-09-04

Related branches

Sprints

Whiteboard

Gerrit topic: https://review.openstack.org/#q,topic:bp/redis-storage-driver,n,z

Addressed by: https://review.openstack.org/95638
    Partially-Implements: bp/redis-storage-driver

Addressed by: https://review.openstack.org/95667
    Partially-Implements:bp/redis-storage-driver

Addressed by: https://review.openstack.org/96017
    Adding dependencies for redis support in marconi.

A few minor things remaining, so won't be done for j-2. Moving to j-3. (kgriffs)

Addressed by: https://review.openstack.org/97178
    Implements Queue,Message and Claims Controllers for Redis

Addressed by: https://review.openstack.org/117566
    Do not fail silently when given a bogus claim ID

Addressed by: https://review.openstack.org/118392
    Fix Redis message controller getting stuck in while loop

Addressed by: https://review.openstack.org/118393
    Implement claim and message expiration logic for Redis

Addressed by: https://review.openstack.org/118394
    Calculate stats dynamically instead of using side counters

Addressed by: https://review.openstack.org/118413
    Switch Redis driver modules over to oslo.utils.timeutils

Addressed by: https://review.openstack.org/118568
    Use oslo.utils.encodeutils instead of strutils

Gerrit topic: https://review.openstack.org/#q,topic:redis-storage-driver,n,z

Addressed by: https://review.openstack.org/118892
    Don't pass pipe when filtering messages in the Redis driver

Addressed by: https://review.openstack.org/119153
    In the Redis driver, only read the msg body when needed

Addressed by: https://review.openstack.org/119820
    Fix: Redis driver does not fall back to default port

Addressed by: https://review.openstack.org/119821
    Add missing flavors_controller method to Redis driver

Addressed by: https://review.openstack.org/120256
    Remove extraneous created_iso property

Addressed by: https://review.openstack.org/121045
     Redis driver does not failover to slave when master fails

Addressed by: https://review.openstack.org/121474
     Redis driver does not failover to slave when master fails

Addressed by: https://review.openstack.org/126948
    Add support for redis-sentinel

(?)

Work Items

Work items:
Controllers: DONE
Optimize _get_claim_info: DONE
Optimize _get_queue_info: DONE
Count expired messages using Lua or remove: DONE
Remove stats counters and calculate on the fly?: DONE
GC for Expired claims & messages: DONE

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.