Store prefix affinity in multibackend setup

Registered by Xavier Lucas

This is a proposal to support prefix affinity when using multi backend.

As for now, one can decide to declare a list of stores in a configuration file and pick a mode of operation between stripe and mirror. Supporting an array of stores seen as a single addressable backend makes a lot of sense in these situations :
- When one want to aggregate storage space from mixed sources/providers.
- When one want to maximize data safety by replicating across multiple storage mediums.

However, these storage sources may have :
- Different costs.
- Different sizes.
- Different access latencies ('hot' vs 'cold' storage).

Duplicity allows a user to prefix the manifest, the signatures or the volume by strings he has chosen.

Bringing to duplicity the ability to route file requests to a reduced number of backend based on file prefixes would further leverage this feature. A bit like people use duplicity with S3 & Glacier by prefixing volumes and using transition rules from S3 to Glacier, this would allow for a more generic, fine-grained usage. It would only be compatible with mirror mode though since stripe mode would require to hold a write cursor per matching prefix and lead to usage quirks.

The proposed change to the configuration file consist of adding a new optional key in the multi backend configuration holding a prefix list:
        "description": "My store description",
        "url": "abackend://user@host",
        "env": [
                "name": "key1",
                "value": "val1"
                "name": "key2",
                "value": "val2"
        "prefixes": ["prefix1_", "prefix2_"]

I'm working at OVH and I'm actually already almost done with this implementation. I have a working version I need to further test that make it possible to store volumes in a cheap cold storage service and both manifests and signatures in a more expensive hot storage offer.

See the offers I'm speaking about, both based on Swift:

Blueprint information

Xavier Lucas
Needs approval
Xavier Lucas
Series goal:
Milestone target:
Started by
Xavier Lucas
Completed by
Kenneth Loafman




Work Items

Work items:
Extend multi backend configuration: DONE
Detect eligible stores: DONE
Update documentation: DONE
More extensive test with other backends than Swift: DONE

This blueprint contains Public information 
Everyone can see this information.