Redis Storage Driver

Registered by Allele Dev

Add a storage driver that understands how to work with Redis DBs. This will be our non-AGPL alternative to MongoDB (durability can be obtained via AOF + RDB).

Developing it outside the main repo is the best test for: 1) Our test suite structure 2) our plugins stuff. Then we can review and merge it later.

This driver will be for use cases where all of the following are true:

1. High-throughput needed (2K+ messages/sec per queue)
2. Lots of small messages (<= 1K)
3. Messages are short-lived (minutes, not hours)
4. High durability is optional (2 copies, RAM only)
5. HA is still very important

This is most likely useful only for worker pool use cases (classic producer-consumer).


1. Do such workloads require FIFO guarantee given a single producer? (probably not)
2. Do such workloads require once-only delivery for observers (not claiming messages)? (probably not)

NOTE: scaling a single queue and HA will be addressed in a separate blueprint (redis-pool)

Blueprint information

Flavio Percoco
Kurt Griffiths
Prashanth Raghu
Series goal:
Accepted for juno
Milestone target:
milestone icon 2014.2
Started by
Prashanth Raghu
Completed by
Flavio Percoco

Related branches



Gerrit topic:,topic:bp/redis-storage-driver,n,z

Addressed by:
    Partially-Implements: bp/redis-storage-driver

Addressed by:

Addressed by:
    Adding dependencies for redis support in marconi.

A few minor things remaining, so won't be done for j-2. Moving to j-3. (kgriffs)

Addressed by:
    Implements Queue,Message and Claims Controllers for Redis

Addressed by:
    Do not fail silently when given a bogus claim ID

Addressed by:
    Fix Redis message controller getting stuck in while loop

Addressed by:
    Implement claim and message expiration logic for Redis

Addressed by:
    Calculate stats dynamically instead of using side counters

Addressed by:
    Switch Redis driver modules over to oslo.utils.timeutils

Addressed by:
    Use oslo.utils.encodeutils instead of strutils

Gerrit topic:,topic:redis-storage-driver,n,z

Addressed by:
    Don't pass pipe when filtering messages in the Redis driver

Addressed by:
    In the Redis driver, only read the msg body when needed

Addressed by:
    Fix: Redis driver does not fall back to default port

Addressed by:
    Add missing flavors_controller method to Redis driver

Addressed by:
    Remove extraneous created_iso property

Addressed by:
     Redis driver does not failover to slave when master fails

Addressed by:
     Redis driver does not failover to slave when master fails

Addressed by:
    Add support for redis-sentinel


Work Items

Work items:
Controllers: DONE
Optimize _get_claim_info: DONE
Optimize _get_queue_info: DONE
Count expired messages using Lua or remove: DONE
Remove stats counters and calculate on the fly?: DONE
GC for Expired claims & messages: DONE

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.