Searchable Metadata

Registered by gholt on 2011-03-18

Goal: Provide a service where objects and containers can be listed by a search key matching names and other metadata.

Blueprint information

Not started
Needs approval
Series goal:
Milestone target:

Related branches



Wiki page:
Include proposed API spec (PDF), Icehouse design summit slides, Etherpad link, and accumulated info briancline and I will collect from community feedback in the openstack-dev email list on this topic.


This was discussed at the Icehouse summit in a design session on 7 November 2013. Lincoln Thomas at HP published the above linked spec to foster discussion and feedback from the community. SoftLayer, who currently uses a custom middleware for its metadata search capabilities, also wishes to participate in moving this blueprint forward.

Based on feedback during the design summit, there is definitely more than enough interest in the community at large in seeing a metadata search API implemented. As mentioned during the session, HP and SoftLayer hope to get broader community feedback and consensus on the proposed API prior to implementation.



There are at least 2 ways to implement this API:

1) an external service (could possible be implemented in swift middleware) that reads metadata in data ingress and stores it in a searchable KV store that includes system and custom metadata for accounts, containers, and objects. This is relatively simple, and it doesn't handle existing data.

2) an internal service that adds functionality to swift to create (and delete) indexes on the account/container DBs, and adds an object DB for its metadata to be searchable too (which could be huge). This is more complicated, but it requires less external dependencies and works with existing data.

This design / impl info will be expanded in the Wiki page.

-gholt / -lincolnt


Work Items

Work items:
Post full proposed API spec from Lincoln: DONE
Post slides from Icehouse design summit from Lincoln: DONE
Agree on the API, from consensus of Stackers' feedback: INPROGRESS
Define method for backfilling/populating index with existing data: TODO
Define reference implementation: TODO
Define the metadata store to be used for the reference implementation, with pluggable interfaces for vendors to supersede with their own: TODO