metadata stored with swift object and xbcloud list command

Registered by David Bennett on 2015-01-30

Currently when using innobackupex/xbcloud together to create incremental backups, the checkpoint metadata from the previous backup must be redirected to a separate directory using --extra-lsndir. The next incremental backup must extract the to_lsn from the previous backups metadata in order to proceed. This makes the incremental backup process and the restore process difficult to manage and explain to the user.

In order to make F,i,i,i backups easier to create and restore, meta data can be stored along with each swift object:

1) iso timestamp
2) backup type
3) from_lsn
4) to_lsn
5) host
6) source dir

xbcloud would need to parse this information from the stream at the end of the PUT or maybe an additional block of metadata sent to standard output after the stream.

xbcloud would then use the Swift API Object Metadata POST command to store the time,type lsn's, host and src dir along with the xbstream archive object.

Another possibility would be to integrate xbcloud directly into innobackupex allowing it to manage the swift metadata attachments and retrieve the last to_lsn from the object container.

In addition to storing the metadata, xbcloud could implement a simple show/list feature that would read back the object names and meta data from a container. This will assist in the restore process as well as making it easier to determine the to_lsn from the last backup for the next incremental.

Blueprint information

Status:
Not started
Approver:
None
Priority:
High
Drafter:
David Bennett
Direction:
Needs approval
Assignee:
Sergei Glushchenko
Definition:
New
Series goal:
None
Implementation:
Unknown
Milestone target:
None

Related branches

Sprints

Whiteboard

Another possibility is that we integrate xbcloud directly into innobackupex which may be easier instead of the current pipe implementation. This way, innobackupex could manage the metadata PUT and automatically scan for the incrementals starting LSN.

Additional meta data items that might be nice: host, source_directory

Sergei:
Note there is also an option to store history on server which provides much more convenient way to manage incremental backups. See history_on_server.sh for examples.

I changed the way backup stored while implementing https://blueprints.launchpad.net/percona-xtrabackup/+spec/xbcloud-parse-xbstream and partial downloads. Instead of single SLO there are many smaller files one per xbstream chunk. We can download the contents of xtrabackup_history when querying the metadata.

I think we should entirely replace this BP with the one specifying xbcloud support built into innobackupex.

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.