volume in local storage and incremental snapshots are stored in swift

Registered by Rongze Zhu on 2012-09-20

Summary

The goal of this blueprint is to implement a plugin for cinder. This will allow to create volume in local storage and back up point-in-time snapshots of your data to Swift for durable recovery. This snapshots are incremental backups, meaning that only the blocks on the volume that have changed since your last snapshot will be saved.

Even though the snapshots are saved incrementally, when you delete a snapshot, only the data not needed for any other snapshot is removed. So regardless of which prior snapshots have been deleted, all active snapshots will contain all the information needed to restore the volume. In addition, the time to restore the volume is the same for all snapshots, offering the restore time of full backups with the space savings of incremental.

All in a word, our solutions is local storage + qcow2 image + dependent snapshot + swift. This is like http://wiki.cloudstack.org/display/RelOps/Local+storage+for+data+volumes, but we have incremental snapshot than cloundstack.

The name of the plugin is island.

Rationale

    * high speed and low latency in local disk I/O
    * point-in-time snapshot
    * incremental snapshot
    * cost reduction from using inexpensive local disks
    * raid 10 in local storage have a better reliability.
    * user create snapshot in A region, he can create volume in B region from the snapshot.
    * the qcow2 image of volume will really created when the volume was attaching.

Blueprint information

Status:
Not started
Approver:
John Griffith
Priority:
Low
Drafter:
Rongze Zhu
Direction:
Approved
Assignee:
None
Definition:
Approved
Series goal:
None
Implementation:
Not started
Milestone target:
None

Related branches

Sprints

Whiteboard

Is this defined so that each volume driver may (or must) implement it, not just QCOW2?
There exists another approach to achieve the bp. See https://review.openstack.org/#/c/36042/.
But this patch isn't very perfect.
IMO, Nova should give up all the things which are relevant to volume. So Cinder will take over the creation of local volume which now implemented in Nova drivers. You can view my patch's changes to see the details. If Cinder take over the creation of local volume, Cinder should add the configuration. For examle:
local_volume_driver = Cinder.volume.driver.lvm.LVMISCSIDriver
global_volume_driver = Cinder.volume.driver.rbd.RbdDriver
This bp isn't make progress for too long, I think we should push it.
Any options? -- Haomai Updated on 7.16

When we spoke at the summit, there were issues with the eventual consistency model of swift causing issues with working out which snapshot blocks are still needed. Have you made any progress on this? It seems extremely likely that this work would be useful in any incremental or de-duped volume backup too... -- Duncan Thomas

Hi Duncan,all snapshot data and metadata is Write Once, Read Many, so it would not read inconsistent data. -- Rongze

Gerrit topic: https://review.openstack.org/#q,topic:bp/local-storage-volume,n,z

Addressed by: https://review.openstack.org/38468
    local storage volume and store incremental snapshots in other storage

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.