Swift Direct Indexing Library
Summary
=======
Provide a library for the Swift team to bypass Messaging middleware and provide direct injection into Searchlight
Problem Statement
===============
The Swift integration uses an experimental middleware patch for sending notifications to searchlight. In discussions with the Swift team, they would like to be able to bypass rabbit messaging because they believe it will be a performance bottleneck.
Description
=========
Incremental indexing for searchlight is typically provided via OSLO messaging. The Swift service currently doesn’t send notifications, but we have discussed with the Swift team a way to investigate more performant ways to achieve indexing. They said they may integrate it deep in the swift service and not in the middleware layer. In the meantime, experimental support for providing notifications via middleware is provided in the following patch:
https:/
SPEC: https:/
See also:
http://
Blueprint information
- Status:
- Started
- Approver:
- None
- Priority:
- High
- Drafter:
- Travis Tripp
- Direction:
- Needs approval
- Assignee:
- None
- Definition:
- Review
- Series goal:
- Accepted for future
- Implementation:
-
Started
- Milestone target:
- None
- Started by
- Travis Tripp
- Completed by
Related branches
Related bugs
Sprints
Whiteboard
Potentially this could apply to other services as well. Some investigation is required to determine where such a mechanism could plug in (both to swift and other services). Middleware does have some limitations because it typically is dealing with request and response headers; in the existing experimental swift notifications we simulate a HEAD request to swift to get metadata, but depending on service architecture this might not be practical.
One potential alternative that came to mind to optimize out the rabbitmq middleman is a notification driver that pushes straight to elasticsearch. This still would require services to support notifications, but would give us an existing interface to work from.
It does make sense ot investigate this early in the Newton cycle (and preferably prior to the summit) to give some time for conversations we may need to have with different service teams.
Gerrit topic: https:/
Addressed by: https:/
Searchlight Client Library
