Notification Forwarding (to systems like Zaqar)

Registered by yuntongjin on 2015-11-12

This BP is proposing using zaqar( (messaging-as-a-service)) to translat SearchLight index to zaqar message, and push it to clients.
https://wiki.openstack.org/wiki/Zaqar

Please see the Spec here: https://review.openstack.org/#/c/246220/

SearchLight’s mission is to provide advanced and scalable indexing and search across multi-tenant cloud resources.(https://wiki.openstack.org/wiki/Searchlight)
It use ElasticSearch to build a scabiliable index and will offloading user search queries from existing OS API servers.
But it’s consumers will still use poll mode to get information from it, and this mode is synchronous and not in time.
This BP will add SearchLight has the ablility to push message to it’s consumers, it’s asynchronous and in time.
And this will also offload SearchLight’s user queries.

The main notification/message flow is like below:
OS Internal Service —> Searchlight —> Zaqar —> External app

Use Case:
1. Horizon
Currently, one of the pinpoint in Horizon is that OS resource availability and status is not asynchronous, as such Horizon will have to keeping polling openstack api, for example when boot a instance, horizon will keep polling nova api to get when the instance boot completed.
2. NFV
Telco vendors would like notifications of resource changes .. for example, the CRUD of a VM, CRUD of a glance image, CRUD of a flavor etc. SearchLight today detects such events and indexes them etc.
3. 3rd party APP
3rd party cloud SW like monitor tool needs to get resource availability and status like instance/flavor created, status change, hyper resource usage.

Blueprint information

Status:
Started
Approver:
Travis Tripp
Priority:
Medium
Drafter:
yuntongjin
Direction:
Approved
Assignee:
Lei Zhang
Definition:
Approved
Series goal:
Accepted for ocata
Implementation:
Started
Milestone target:
None
Started by
Travis Tripp on 2016-01-28

Related branches

Sprints

Whiteboard

[TravT] This will require a specification to be submitted in RST format. We put in the request for a spec repo today and should have it shortly. The template for it will look like this:

https://review.openstack.org/#/c/243881/2/doc/source/feature-designs/template.rst

So if you want to get that ready, we should be able to submit it shortly.

Here is some feedback in the meantime to consider: In searchlight, perhaps we could just have a configurable push plugin interface to send along events from the listeners and Zaqar could be configured in. I think Zaqar is definitely a target, but a Zaqar only outbound interface would be too tightly coupled. For example, we might consider a new lightweight process in Searchlight that supports webhooks and provides simple message forwarding without all the bells and whistles of zaqar (persistence).

[Malini]
I do like the idea of notification system being a plugin, particularly also like the option of enabling/disabling persistence of notifications. Might it not be just as easy to have Zaqar offer that option to its subscribers, either across the board, or on a per channel basis?
Why I am asking is should we build in addition a light-weight webhookstyle plugin, would it be necessary to provide as rich a subscription API as Zaqar/messaging system. That might soon make the light weight heavy! Yuntong mentioned to that Zaqar messages have a timeout, which could be made trivially small or a symbol to indicate throw away instantly if no listener.

[TravT]
Yep, we don't want to have to recreate all the Zaqar goodness, but we just can't have a hard dependency on Zaqar, so having an interface for the notification makes the most sense.

FYI, This is the template to use for the spec repo. https://review.openstack.org/#/c/245214/4/specs/template.rst

It has been created now, just waiting to get the initial repo set up. So you'll need to depend on the above patch and add the spec to the mitaka dir.

Gerrit topic: https://review.openstack.org/#q,topic:bp/is,n,z

Addressed by: https://review.openstack.org/246220
    This BP is implementing a notification-forwarding for SearchLight index

[TravT] We will need to move this to Newton. We are sorry that getting some of the core indexing feature improved caused churn. It should settle down in Newton and I still want to see this BP make it in.

Gerrit topic: https://review.openstack.org/#q,topic:bp/to,n,z

Addressed by: https://review.openstack.org/294418
    Move notification-forwarding for SearchLight BP to Newton

Gerrit topic: https://review.openstack.org/#q,topic:bp/zaqar_publisher,n,z

Addressed by: https://review.openstack.org/438384
    (POC) Add Zaqar publisher

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.