Storage Sharding

Registered by Kurt Griffiths on 2013-10-08

Implement driver-agnostic storage sharding. Will work with any backend. Optional (disabled by default). Transport will call a storage "lookup" function instead of getting a direct reference to a driver object.

This will allow Marconi to support queue "flavors" with different performance/durability characteristics, as well as allowing Marconi to scale quite large.

http://grab.by/rsvA

Blueprint information

Status:
Complete
Approver:
Flavio Percoco
Priority:
Essential
Drafter:
None
Direction:
Approved
Assignee:
Allele Dev
Definition:
Approved
Series goal:
Accepted for icehouse
Implementation:
Implemented
Milestone target:
milestone icon 2014.1
Started by
Kurt Griffiths on 2013-10-08
Completed by
Allele Dev on 2013-12-09

Related branches

Sprints

Whiteboard

Gerrit topic: https://review.openstack.org/#q,topic:bp/storage-sharding,n,z

Addressed by: https://review.openstack.org/50437
    WIP

Addressed by: https://review.openstack.org/49550
    Use oslo.config directly instead of common.config

Addressed by: https://review.openstack.org/50456
    feat: storage sharding schema for marconi-queues

Addressed by: https://review.openstack.org/50701
    feat: split queues API into public/admin

Addressed by: https://review.openstack.org/50702
    feat: add shard management resource

Addressed by: https://review.openstack.org/50721
    feat: shards storage controller interface

Addressed by: https://review.openstack.org/50815
    feat: shards mongodb driver + tests

Addressed by: https://review.openstack.org/50998
    feat: integrate shard storage with transport

Addressed by: https://review.openstack.org/51705
    fix(queues): Global config used everywhere

Addressed by: https://review.openstack.org/51049
    Setup storage pipeline in the boostrap instead of driver base

Addressed by: https://review.openstack.org/51016
    chore: Remove GC cruft from storage driver base class

Addressed by: https://review.openstack.org/52160
    feat: add catalogue storage driver for queues

Gerrit topic: https://review.openstack.org/#q,topic:bp/remove-global-config,n,z

Gerrit topic: https://review.openstack.org/#q,topic:bug/1241686,n,z

Gerrit topic: https://review.openstack.org/#q,topic:bug/1243898,n,z

Addressed by: https://review.openstack.org/54445
    chore: Pass cache into drivers

Addressed by: https://review.openstack.org/54605
    feat: connect sharding manager to control drivers (WIP)

Addressed by: https://review.openstack.org/53127
    feat(shard): queue listing from multiple sources

Addressed by: https://review.openstack.org/54945
    feat(sharding): Cache shard mappings

Addressed by: https://review.openstack.org/54992
    DEBUGGING PATCH - DO NOT MERGE!

Addressed by: https://review.openstack.org/55001
    fix(mongo): Delete claimed message fails

Gerrit topic: https://review.openstack.org/#q,topic:storage-sharding,n,z

Addressed by: https://review.openstack.org/57948
    Remove proxy code

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.