Swift Direct Indexing Library

Registered by Travis Tripp on 2016-03-23

Summary
=======
Provide a library for the Swift team to bypass Messaging middleware and provide direct injection into Searchlight

Problem Statement
===============
The Swift integration uses an experimental middleware patch for sending notifications to searchlight. In discussions with the Swift team, they would like to be able to bypass rabbit messaging because they believe it will be a performance bottleneck.

Description
=========
Incremental indexing for searchlight is typically provided via OSLO messaging. The Swift service currently doesn’t send notifications, but we have discussed with the Swift team a way to investigate more performant ways to achieve indexing. They said they may integrate it deep in the swift service and not in the middleware layer. In the meantime, experimental support for providing notifications via middleware is provided in the following patch:

https://review.openstack.org/#/c/249471

SPEC: https://review.openstack.org/#/c/305404/

See also:

http://docs.openstack.org/developer/searchlight/plugins/swift.html

Blueprint information

Status:
Started
Approver:
None
Priority:
High
Drafter:
Travis Tripp
Direction:
Needs approval
Assignee:
None
Definition:
Review
Series goal:
Accepted for future
Implementation:
Started
Milestone target:
None
Started by
Travis Tripp

Related branches

Sprints

Whiteboard

Potentially this could apply to other services as well. Some investigation is required to determine where such a mechanism could plug in (both to swift and other services). Middleware does have some limitations because it typically is dealing with request and response headers; in the existing experimental swift notifications we simulate a HEAD request to swift to get metadata, but depending on service architecture this might not be practical.

One potential alternative that came to mind to optimize out the rabbitmq middleman is a notification driver that pushes straight to elasticsearch. This still would require services to support notifications, but would give us an existing interface to work from.

It does make sense ot investigate this early in the Newton cycle (and preferably prior to the summit) to give some time for conversations we may need to have with different service teams.

Gerrit topic: https://review.openstack.org/#q,topic:bp/swift-direct-indexing-library,n,z

Addressed by: https://review.openstack.org/305404
    Searchlight Client Library

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.