Support for object system metadata

Registered by Alistair Coles

The original system metadata patch (https://review.openstack.org/#/c/51228/) supported only account and container system metadata.

There are now patches in review that store middleware-generated metadata against objects, e.g.:
• on demand migration https://review.openstack.org/#/c/64430/
• server side encryption https://review.openstack.org/#/c/76578/1

Object system metadata should not be stored in the x-object-meta- user metadata namespace because (a) there is a potential name conflict with arbitrarily user metadata and (b) system metadata in the x-object-meta- namespace will be lost if a user sends a POST request to the object.

The goal of this work is to enable system metadata to be persisted with objects in a similar way to user metadata, but with the ability to update individual items of system metadata independently when making a POST request, unlike user metadata which is replaced as a whole by a POST request.

Proposed approach:

Initially enable object system metadata only on PUT requests to support existing use cases:
• System metadata stored in .data file xattrs and copied unchanged to .meta files.
• Use the x-object-sysmeta- namespace already supported by proxy and reserved by gatekeeper middleware.
• Store x-object-sysmeta- headers in object file xattrs alongside other object metadata.
• Store object system metadata attributes as key: (value, timestamp) as per account and container metadata, enabling most-recent-wins rule to be applied to individual attributes in future.

 Follow up with enabling system metadata updates on POST:
• Allow for concurrent POST requests resulting in multiple .meta files with divergent sets of system metadata.
• When handling POST the object server (DiskFile) reads potentially multiple existing .meta files and merges system metadata to a unified set written to a new .meta file – the system metadata key: (value, timestamp) format allows most recent version of each system metadata key to be selected from the set of .meta files.
• When handling GET/HEAD similarly read and merge system metadata from potentially multiple .meta files.
• Only delete .meta files when their system metadata has been merged into a newer .meta file.

Detailed discussion on wiki page: https://wiki.openstack.org/wiki/Swift/ObjectSystemMetadata

Blueprint information

Status:
Not started
Approver:
None
Priority:
Undefined
Drafter:
Alistair Coles
Direction:
Needs approval
Assignee:
Alistair Coles
Definition:
New
Series goal:
None
Implementation:
Unknown
Milestone target:
None

Related branches

Sprints

Whiteboard

Gerrit topic: https://review.openstack.org/#q,topic:bp/object-system-metadata,n,z

Addressed by: https://review.openstack.org/79991
    Enable object system metadata on PUTs

Addressed by: https://review.openstack.org/111508
    Enable object system metadata on PUTs

Gerrit topic: https://review.openstack.org/#q,topic:bug/1351323,n,z

Addressed by: https://review.openstack.org/111866
    Merging master into feature/ec branch

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.