OpenStack Searchlight

Optionally disable some listener events

Registered by Steve McLellan

Nova sends a lot of events on power operations - typically two updates and two suspend (or resume) events. We currently do not listen to 'suspend' or 'resume' events (see https://bugs.launchpad.net/searchlight/+bug/1531267events) and when/if we do, we'll receive 4 events per power event.

Processing from notification alone would be fine; hitting the API 2 or 4 times is probably not. The suggestion is to in a general sense allow a configuration option to ignore certain events, and in a more specific case for nova notifications to avoid the transitionary events (such as updates where the state_description indicates a power state change). The general case i think is reasonably simple and useful; the specific case is maybe a little complicated (I don't know if localization can affect things, and making logic decisions based on payload might be a bit dodgy).

Text from related bug that I (TravT) just closed in favor of this BP.

https://bugs.launchpad.net/searchlight/+bug/1577947

Initial indexing is done on demand, but keeping the index up-to-date is dependent on notifications from the various services. Right now there are gaps in the data provided by Nova notifications. We hope to solve this with versioned notifications, but even as we move towards versioned notifications, they may not be supported in legacy deployments.

In best case scenarios, Nova can move through a provisioning request in only a few seconds. This would disable API calls for intermediate states in Nova instance actions, or at least make that a configuration option to disable those callbacks in cases where searchlight indexing callbacks are a problem.

Blueprint information

Status:
Complete
Approver:
Travis Tripp
Priority:
Essential
Drafter:
Steve McLellan
Direction:
Approved
Assignee:
Steve McLellan
Definition:
Approved
Series goal:
Accepted for newton
Implementation:
Implemented
Milestone target:
milestone icon newton-2
Started by
Travis Tripp
Completed by
Travis Tripp

Related branches

Sprints

Whiteboard

[TravT] I think this should be done in conjunction with any efforts to improve notifications, potentially even use versioned notifications from nova: https://blueprints.launchpad.net/searchlight/+spec/nova-versioned-notifications

The full list and payload of events on a successful nova boot (neutron networking) is in http://paste.openstack.org/show/496978/

The relevant events and fields:

# Registers the id, image, flavor etc
compute.instance.update {"state_description": "scheduling", "old_state": "building", "new_task_state": "scheduling", "state": "building"} @ 2016-05-12 20:44:40.352048

# Comes after the scheduler reservation_id is assigned but before a compute host is assigned
compute.instance.update {"state_description": "", "state": "building", "old_state": "building", "old_task_state": "scheduling", new_task_state": null} @ 2016-05-12 20:44:40.867902

# Still no host
compute.instance.create.start {"state_description": "", "state": "building"} @ 2016-05-12 20:44:40.885366

# Host assigned
compute.instance.update {"state_description": "", "state": "building", "old_state": "building", "node": "devstack", "host": "devstack"} @ 2016-05-12 20:44:40.981856

# Networking starts
compute.instance.update {"state_description": "networking", "state": "building", "old_state": "building", "old_task_state": "networking", "new_task_state": "networking"} @ 2016-05-12 20:44:41.082192

# Disks, still no IP information
compute.instance.update {"state_description": "block_device_mapping", "old_task_state": "networking"} @ 2016-05-12 20:44:41.298666

# Actually spin up the VM
compute.instance.update {"state_description": "spawning", "old_task_state": "block_device_mapping", "new_task_state": "spawning"} @ 2016-05-12 20:44:41.355864

# Networking notifications
port.create.start @ 2016-05-12 20:44:41.673135
port.create.end @ 2016-05-12 20:44:42.700366

# Glance notifications
image.send @ 2016-05-12 20:44:42.700366
image.send @ 2016-05-12 20:44:45.444854

# VM finished init. No networking information here
compute.instance.update {"state_description": "", "old_state": "building", "state": "active", "old_task_state": "spawning", "new_task_state": null} @ 2016-05-12 20:44:52.915259

compute.instance.create.end {"state_description": "", "fixed_ips": [ .. ], "state": "active"} @ 2016-05-12 20:44:52.976685

A failed boot (no compute hosts available) is in http://paste.openstack.org/show/496990/. The second compute.update has state:error after which there are two more update events and some scheduler events.

Delete also issues a lot of events: http://paste.openstack.org/show/498248/

Gerrit topic: https://review.openstack.org/#q,topic:bp/reduce-nova-api-impact,n,z

Addressed by: https://review.openstack.org/317100
    Reduce number of API calls on nova boot

(?)

Work Items

Nearby

This blueprint contains Public information 
Everyone can see this information.