freezer-metadata

Introduction
============
The proposal is to have information about a backup (referred to as
"backup metadata") stored along the backup data files in unencrypted json
format.
A new structure for the backup files is also proposed.

Problem
=======
Some functionalities are easier to implement if the freezer-agent has direct
access to information pertaining the backup itself.
Not only when the API (where backup metadata is currently stored) is not
available, but also in those cases when the freezer-scheduler is not aware of
the backup information the the freezer-agent is going to need (for example
when the agent has to determine which is the correct backup to restore)

Useful information can be:

    - backup checksum (currently under development)
    - engine used
    - use of encryption
    - presence/use of any engine-metadata file

Proposed solution
=================
The backup storage will generally contain three files for each level:

    - backup metadata (json format)
    - backup binary data
    - backup engine metadata (eg: tar-metadata file)

Another option would be to have all the backup metadata information organized
in a single structured backup-metadata file. The problem in that case would be
that the file needs to be updated every time a new level is created.
That could possibly create problems which don't arise with a write-once file.

So, for example the files in a storage could be:

freezer-data/tar/mybackup_node12/13244321/0_13244321/data
freezer-data/tar/mybackup_node12/13244321/0_13244321/engine-metadata
freezer-meta/tar/mybackup_node12/13244321/1_13244321/backup-metadata
freezer-data/tar/mybackup_node12/13244321/1_13244988/data
freezer-data/tar/mybackup_node12/13244321/1_13244988/engine-metadata
freezer-meta/tar/mybackup_node12/13244321/1_13244988/backup-metadata
freezer-data/tar/mybackup_node12/13244321/2_13249559/data
freezer-data/tar/mybackup_node12/13244321/2_13249559/engine-metadata
freezer-meta/tar/mybackup_node12/13244321/2_13249559/backup-metadata

(see section below for description of new backup structure and naming)

The backup-metadata file would also be uploaded to the API for easy
search and retrieval.

Backup structure and naming
===========================
backup data files are named according to the following criteria:

/engine/backup_name/timestamp_of_level_0/level_timestamp/{data-file engine-metafile freezer-metafile}

This results in pseudocontainer being use in case of object storage (for
example: swift), and in subdirectories when filesystem storage is used (local
storage or over ssh)

Where:

engine = {tar, rsync, nova, cinder}

backup_name = cinder_id (when engine is cinder)
              nova_id (when engine is nova)
              user-provided-name-string_hostname (when engine is tar or rsync)

When using cinder-native backups, freezer will store only two files: the
backup-metadata and a copy of the cinder-backup-metadata.

backup-metadata file content
============================
The content may vary and depend on the actual backup options and engine used.
Some common fields are:

{
    compression-algorithm: string,
    encryption-algorithm: string,
    encryption-key: string,
    backup_uuid: string,
    checksum: string,

    engine-type: string, (redundant)

}

Information about the engine-type is somewhat redundant since it's already
part of the backup name, but it's useful when the backup metadata is uploaded
to the API, since the actual filename is lost.