Backup and restore to multiple storages for data high availability

Objectives:

1) Freezer should have possibility to recover backup even if several of storages are unavailable.

Problems:

1) Storages can be corrupted or have partial (or controversial) information about backups.

It has a big impact on making backup and restore.

2) Storage can be unavailable during backup (or can be unavailable during long period of time).

Old storage with very-very old backups can suddenly became available and it is important to have a way to detect such situation.
Swift uses redundancy to cope with data corruption, but ssh and local have no such trait - and probably it is a good idea to have some storage health checker mechanism (and probably self-recovering mechanism - like reload backups from healthy storage in case of data corruption).

3) Storage should have a way to notify end-user about unsuccessful (or partly successful) backup.

4) Storage should have a way to specify multiple credentials (storages in cli and configuration file) - currently it gets credentials from system variables.

5) We should specify a mechanism of restoring. If we have multiple storages - we should choose health and most up-to-date one and make a restore.

6) We should be able to implement incremental backup even if some of storages are unavailable. And have mechanism of recovering missing incremental backups ( or marking a storage like a broken one)