Be able to adjust ring replica counts up and down gradually

Registered by Samuel Merritt on 2012-12-17

This blueprint has been superseded. See the newer blueprint "Make ring's replica count adjustable" for updated plans.

As part of supporting globally-distributed clusters, Swift needs to allow operators to change the replica count for a running cluster so that new regions can be added.

Blueprint information

Status:
Complete
Approver:
None
Priority:
Undefined
Drafter:
None
Direction:
Needs approval
Assignee:
None
Definition:
Superseded
Series goal:
None
Implementation:
Unknown
Milestone target:
None
Completed by
John Dickinson on 2013-01-15

Related branches

Sprints

Whiteboard

As part of supporting globally-distributed clusters, Swift needs to allow operators to change the replica count for a running cluster.

Consider a 3-replica, 3-region cluster. When the owning company grows and wants to add a fourth Swift region, they may want to make the cluster a 4-replica cluster as well, so that there is one replica of each object local to each cluster.

The naive approach of simply adding an entire new replica at once is not adequate to address this requirement. A 3-replica cluster with 30 PB of data in it (10 PB user data) will suffer a very extended period of degraded performance if the replica count is simply incremented.

Instead, a new replica should be able to be added gradually, essentially having fractional replicas. Of course, no one object will have fractional replicas; the idea is nonsensical. Instead, different objects will have different replica counts depending on their hash values. For example, a ring with 3.25 replicas would give 4 replicas to 25% of objects and 3 replicas to the remaining 75%; objects with numerically-smaller hash values would be the ones with more replicas.

A proposed interface:

$ swift-ring-builder foo.builder set-replicas <value>

To gradually add a replica, a cluster operator could run

$ swift-ring-builder foo.builder set-replicas 3.01
$ swift-ring-builder rebalance
<push rings and wait>

$ swift-ring-builder foo.builder set-replicas 3.02
$ swift-ring-builder rebalance
<push rings and wait>

$ swift-ring-builder foo.builder set-replicas 3.03
$ swift-ring-builder rebalance
<push rings and wait>

[...]

$ swift-ring-builder foo.builder set-replicas 4
$ swift-ring-builder rebalance
<push rings>

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.