Using parity to more efficiently store objects.

Registered by David Challoner on 2012-03-22

Swift can provide more efficient storage with tunable resiliently by storing objects in parity encoded chunks. (like an arbitrary cluster wide RAID level). This comes with a cost of higher latency but for longer term archival storage of large amounts of data this is often preferred to replicas (or can be combined with replicas for even more control over space vs. resiliency).

The zfec library (http://pypi.python.org/pypi/zfec) provides fast parity calculation and would allow a tunable amount of control over resiliency for a given resiliency class (i.e store a chunk across 10 systems with 8 chunks needed to reconstruct the original). A 100MB file stored in this way would only require 125 MB(100*10/8) across 10 systems vs 300MB across 3 systems for standard 3x replication.

Blueprint information

Status:
Complete
Approver:
None
Priority:
Not
Drafter:
None
Direction:
Needs approval
Assignee:
None
Definition:
Superseded
Series goal:
None
Implementation:
Unknown
Milestone target:
None
Completed by
John Dickinson on 2013-09-10

Related branches

Sprints

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.