Store prefix affinity in multibackend setup
This is a proposal to support prefix affinity when using multi backend.
As for now, one can decide to declare a list of stores in a configuration file and pick a mode of operation between stripe and mirror. Supporting an array of stores seen as a single addressable backend makes a lot of sense in these situations :
- When one want to aggregate storage space from mixed sources/providers.
- When one want to maximize data safety by replicating across multiple storage mediums.
However, these storage sources may have :
- Different costs.
- Different sizes.
- Different access latencies ('hot' vs 'cold' storage).
Duplicity allows a user to prefix the manifest, the signatures or the volume by strings he has chosen.
Bringing to duplicity the ability to route file requests to a reduced number of backend based on file prefixes would further leverage this feature. A bit like people use duplicity with S3 & Glacier by prefixing volumes and using transition rules from S3 to Glacier, this would allow for a more generic, fine-grained usage. It would only be compatible with mirror mode though since stripe mode would require to hold a write cursor per matching prefix and lead to usage quirks.
The proposed change to the configuration file consist of adding a new optional key in the multi backend configuration holding a prefix list:
[
{
"url": "abackend:
"env": [
{
},
{
}
],
"prefixes": ["prefix1_", "prefix2_"]
}
...
]
I'm working at OVH and I'm actually already almost done with this implementation. I have a working version I need to further test that make it possible to store volumes in a cheap cold storage service and both manifests and signatures in a more expensive hot storage offer.
See the offers I'm speaking about, both based on Swift:
https:/
https:/
Blueprint information
- Status:
- Complete
- Approver:
- duplicity-team
- Priority:
- Undefined
- Drafter:
- Xavier Lucas
- Direction:
- Needs approval
- Assignee:
- Xavier Lucas
- Definition:
- Approved
- Series goal:
- None
- Implementation:
- Implemented
- Milestone target:
- None
- Started by
- Xavier Lucas
- Completed by
- Kenneth Loafman
Whiteboard
Work Items
Work items:
Extend multi backend configuration: DONE
Detect eligible stores: DONE
Update documentation: DONE
More extensive test with other backends than Swift: DONE