Marconi: Scaling Storage
This is essential for scaling and operating a large deployment.
The goals for this blueprint are:
* find an approach that works
* prototype it
* demonstrate it
Some requirements:
- distribute load reasonably well amongst partitions
- handle migrations transparently
Some nice-to-haves:
- statelessness: might require an intelligent client + consistent hashing & rebalance
- simple: change as little as possible, add as little as possible
- easy to deploy
- easy to manage
Under consideration:
- nginx + luajit
- uwsgi fast routing
Disqualified:
- haproxy (no request cloning)
- pyrox (would take a lot of additional impl. to support our needs)
EDIT: previously, haproxy was under consideration. When it was realized that haproxy cannot duplicate requests, it was ruled out of consideration.
Read below for more details and brainstorming.
Blueprint information
- Status:
- Complete
- Approver:
- Flavio Percoco
- Priority:
- Not
- Drafter:
- Allele Dev
- Direction:
- Approved
- Assignee:
- Allele Dev
- Definition:
- Superseded
- Series goal:
- Accepted for icehouse
- Implementation:
-
Informational
- Milestone target:
-
icehouse-2
- Started by
- Completed by
- Allele Dev
Whiteboard
Gerrit topic: https:/
Addressed by: https:/
feat: marconi proxy (WIP)
Addressed by: https:/
feat: marconi proxy (v1, health)
Addressed by: https:/
feat: marconi-proxy forwarding
Gerrit topic: https:/
Gerrit topic: https:/
Addressed by: https:/
Add mongo driver for marconi proxy storage
Gerrit topic: https:/
Addressed by: https:/
proxy: mongodb storage driver
Gerrit topic: https:/
Addressed by: https:/
proxy: memory storage driver + tests + fixes
Addressed by: https:/
proxy: adds oslo.cache, config support, & stevedore
Addressed by: https:/
proxy: mirror structure of marconi queues + bootstrap
Addressed by: https:/
fix(proxy): forward marconi headers
Addressed by: https:/
test(proxy): fix proxy-transport tests
Addressed by: https:/
feat: separate config for queues and proxy
Addressed by: https:/
feat: split proxy API into admin and public apps
Gerrit topic: https:/
Addressed by: https:/
feat (proxy/admin): allow partition modifications
Addressed by: https:/
feat (proxy/admin): allow partition modifications
Gerrit topic: https:/
Addressed by: https:/
feat: async forwarding
Gerrit topic: https:/
Addressed by: https:/
feat: add logging to proxy
Gerrit topic: https:/
Addressed by: https:/
fix: do not duplicate queues across partitions
