Store prefix affinity in multibackend setup

Registered by Xavier Lucas on 2017-06-09

This is a proposal to support prefix affinity when using multi backend.

As for now, one can decide to declare a list of stores in a configuration file and pick a mode of operation between stripe and mirror. Supporting an array of stores seen as a single addressable backend makes a lot of sense in these situations :
- When one want to aggregate storage space from mixed sources/providers.
- When one want to maximize data safety by replicating across multiple storage mediums.

However, these storage sources may have :
- Different costs.
- Different sizes.
- Different access latencies ('hot' vs 'cold' storage).

Duplicity allows a user to prefix the manifest, the signatures or the volume by strings he has chosen.

Bringing to duplicity the ability to route file requests to a reduced number of backend based on file prefixes would further leverage this feature. A bit like people use duplicity with S3 & Glacier by prefixing volumes and using transition rules from S3 to Glacier, this would allow for a more generic, fine-grained usage. It would only be compatible with mirror mode though since stripe mode would require to hold a write cursor per matching prefix and lead to usage quirks.

The proposed change to the configuration file consist of adding a new optional key in the multi backend configuration holding a prefix list:
[
    {
        "description": "My store description",
        "url": "abackend://user@host",
        "env": [
            {
                "name": "key1",
                "value": "val1"
            },
            {
                "name": "key2",
                "value": "val2"
            }
        ],
        "prefixes": ["prefix1_", "prefix2_"]
    }
...
]

I'm working at OVH and I'm actually already almost done with this implementation. I have a working version I need to further test that make it possible to store volumes in a cheap cold storage service and both manifests and signatures in a more expensive hot storage offer.

See the offers I'm speaking about, both based on Swift:
https://www.ovh.com/us/public-cloud/storage/cloud-archive/
https://www.ovh.com/us/public-cloud/storage/object-storage/

Blueprint information

Status:
Started
Approver:
duplicity-team
Priority:
Undefined
Drafter:
Xavier Lucas
Direction:
Needs approval
Assignee:
Xavier Lucas
Definition:
Approved
Series goal:
None
Implementation:
Needs Code Review
Milestone target:
None
Started by
Xavier Lucas on 2017-06-09

Sprints

Whiteboard

(?)

Work Items

Work items:
Extend multi backend configuration: DONE
Detect eligible stores: DONE
Update documentation: DONE
More extensive test with other backends than Swift: DONE

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.