Redis Storage Driver
Add a storage driver that understands how to work with Redis DBs. This will be our non-AGPL alternative to MongoDB (durability can be obtained via AOF + RDB).
Developing it outside the main repo is the best test for: 1) Our test suite structure 2) our plugins stuff. Then we can review and merge it later.
This driver will be for use cases where all of the following are true:
1. High-throughput needed (2K+ messages/sec per queue)
2. Lots of small messages (<= 1K)
3. Messages are short-lived (minutes, not hours)
4. High durability is optional (2 copies, RAM only)
5. HA is still very important
This is most likely useful only for worker pool use cases (classic producer-consumer).
TBD
1. Do such workloads require FIFO guarantee given a single producer? (probably not)
2. Do such workloads require once-only delivery for observers (not claiming messages)? (probably not)
NOTE: scaling a single queue and HA will be addressed in a separate blueprint (redis-pool)
Blueprint information
- Status:
- Complete
- Approver:
- Flavio Percoco
- Priority:
- Essential
- Drafter:
- Kurt Griffiths
- Direction:
- Approved
- Assignee:
- Prashanth Raghu
- Definition:
- Approved
- Series goal:
- Accepted for juno
- Implementation:
-
Implemented
- Milestone target:
-
2014.2
- Started by
- Prashanth Raghu
- Completed by
- Flavio Percoco
Whiteboard
Gerrit topic: https:/
Addressed by: https:/
Partially-
Addressed by: https:/
Partially-
Addressed by: https:/
Adding dependencies for redis support in marconi.
A few minor things remaining, so won't be done for j-2. Moving to j-3. (kgriffs)
Addressed by: https:/
Implements Queue,Message and Claims Controllers for Redis
Addressed by: https:/
Do not fail silently when given a bogus claim ID
Addressed by: https:/
Fix Redis message controller getting stuck in while loop
Addressed by: https:/
Implement claim and message expiration logic for Redis
Addressed by: https:/
Calculate stats dynamically instead of using side counters
Addressed by: https:/
Switch Redis driver modules over to oslo.utils.
Addressed by: https:/
Use oslo.utils.
Gerrit topic: https:/
Addressed by: https:/
Don't pass pipe when filtering messages in the Redis driver
Addressed by: https:/
In the Redis driver, only read the msg body when needed
Addressed by: https:/
Fix: Redis driver does not fall back to default port
Addressed by: https:/
Add missing flavors_controller method to Redis driver
Addressed by: https:/
Remove extraneous created_iso property
Addressed by: https:/
Redis driver does not failover to slave when master fails
Addressed by: https:/
Redis driver does not failover to slave when master fails
Addressed by: https:/
Add support for redis-sentinel
Work Items
Work items:
Controllers: DONE
Optimize _get_claim_info: DONE
Optimize _get_queue_info: DONE
Count expired messages using Lua or remove: DONE
Remove stats counters and calculate on the fly?: DONE
GC for Expired claims & messages: DONE
Dependency tree
* Blueprints in grey have been implemented.
