Provide a base plugin for indexing swift

Registered by Travis Tripp

Summary
=======

We want to be able to search swift objects and containers based on their structured and unstructured metadata.

Motivation
========

Swift does not provide a structured search capability today and does not provide complex querying such as date ranges, geo spatial location, etc on that metadata. Many applications can be enabled with a better metadata search.

Description
=========

IBM softlayer had a presentation on their initial usage of swift and some application capabilities that are possible with an ElasticSearch integration at the Mitaka design summit.

They performed an initial integration as a standalone project that is similar to searchlight concepts and needs to be resolved with searchlight architecture. Their initial presentation is here:

https://www.openstack.org/summit/tokyo-2015/videos/presentation/boosting-the-power-of-swift-using-metadata-search

Here are some keypoints from the Mitaka prioritization meeting:

 * Swift had pain points with creating a query abstraction layer
 * Notifications need to be resolved
 ** Last summit, Swift group preferred Zaqar, but not sure if set in stone. Doubts about the general model and performance at scale (would have to support minimum of 5k-20k notifications/sec without flinching)
 ** SoftLayer uses RabbitMQ, will likely continue to...considering Kafka as well
 ** Search API part of swift? Possibly redirect to SL, still allows custom middleware in swift to handle specific customer cases but translates to searchlight

 * Searchlight needs configurable dynamic template mappings?
 ** Would be very helpful - lack of better mappings and/or templates was pain point for SoftLayer

https://etherpad.openstack.org/p/searchlight-swift-workpad

https://etherpad.openstack.org/p/mitaka-openstackux-swift

Testing
======
Brief instruction for reviewers to exercise the changes, including expected results where non-obvious.

Outside Dependencies
==================
Is this functionality already supported in other services? List the appropriate API calls and if they are extensions or base API functionality.

This should describe any cross project dependencies. This should include:
  * changes to OpenStack services
  * changes to OpenStack services clients (python-*client)
  * changes in external projects
Links to particular patches should be included in the whiteboard area.

Requirements Update Required
========================

Blueprint information

Status:
Complete
Approver:
Travis Tripp
Priority:
High
Drafter:
Travis Tripp
Direction:
Approved
Assignee:
Lakshmi N Sampath
Definition:
Approved
Series goal:
Accepted for mitaka
Implementation:
Implemented
Milestone target:
milestone icon mitaka-rc1
Started by
Travis Tripp
Completed by
Travis Tripp

Related branches

Sprints

Whiteboard

Looks like swift doesn't subscribe to current notifications at all - https://wiki.openstack.org/wiki/SystemUsageData

Etherpad for brainstorming: https://etherpad.openstack.org/p/searchlight-swift-workpad

I've added an experimental patch at https://review.openstack.org/#/c/249471/. I don't expect it will get merged ever since the swift team have said as much, but i'm hoping for some feedback. Otherwise i'll put something together in github maybe, temporarily.

Gerrit topic: https://review.openstack.org/#q,topic:bp/swift-plugin,n,z

Addressed by: https://review.openstack.org/271622
    WIP - Swift plugin Pending RBAC and Code tuning.

Addressed by: https://review.openstack.org/289798
    Add _routing field to ES CRUD for plugins

Gerrit topic: https://review.openstack.org/#q,topic:bug/1541115,n,z

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.