Put HostManager._instance_info on a diet

Registered by Matt Riedemann on 2017-04-06

This is a specless blueprint to cleanup some technical debt between the compute manager and host manager in the scheduler.

The default scheduler driver is the filter scheduler, and the affinity/anti-affinity filters/weighers are enabled/loaded by default.

Those rely on the _instance_info cache in the HostManager which is either populated by the ComputeManager sending updates (configured via the CONF.filter_scheduler.track_instance_changes option) or by pulling that information directly from the database if the compute doesn't send it.

The only in-tree filters/weighers that use the _instance_info cache are the affinity ones.

There are a few issues here:

1. By default, the instance info stuff is always sent from compute to scheduler regardless of whether or not the filter scheduler is being used. If I'm using the caching scheduler, then this is just unnecessary RPC traffic.

2. The full instance objects are being sent over RPC but the only thing that the affinity filter/weigher needs is the uuid, because that's what InstanceGroup.members are, a list of instance uuids.

3. There is a _sync_scheduler_instance_info periodic task that runs on all computes by default every 2 minutes. That pulls all non-deleted instances from the database for that host (including unnecessary joins on the info_cache and security_groups tables) and sends the full list of instance objects over rpc to the scheduler (remember, only the uuids are used).

4. As in #1 on the compute, the HostManager in the scheduler by default tracks instance changes, even if not using the FilterScheduler. If mis-configured, and the computes aren't sending the full instance lists on startup, the scheduler will pull them from the database.

This is all obviously problematic and wasteful if (1) you're not even using the filter scheduler or using the affinity filter/weigher, and (2) even if you are, you don't need the full instance object and all of it's sub-objects sent over RPC every 2 minutes from all computes in the deployment. At any reasonably scaled deployment this is a lot of overhead.

In this blueprint we'll tackle the main issues by simply sending uuids insteand of full instance objects, and not actually do this work if not configured for the filter scheduler.

Note that any out of tree filters/weighers relying on the HostManager._instance_info cache of full instance objects to get things from the instances besides uuids would be impacted by this. However, if they are out of tree, they pay the price for changes to Nova out of their control. This isn't an API contract. Having said that, they will still have the instance uuids so if they need they can convert them to full objects from their filter if needed so they won't be totally broken.

Blueprint information

Status:
Started
Approver:
Dan Smith
Priority:
Low
Drafter:
Matt Riedemann
Direction:
Needs approval
Assignee:
Matt Riedemann
Definition:
Pending Approval
Series goal:
None
Implementation:
Started
Milestone target:
None
Started by
Matt Riedemann on 2017-04-06

Whiteboard

Gerrit topic: https://review.openstack.org/#q,topic:bp/put-host-manager-instance-info-on-a-diet,n,z

Addressed by: https://review.openstack.org/453916
    Don't send instance updates from compute if not using filter scheduler

I've hit one snag here, which is the TypeAffinityFilter relies on the instance.instance_type_id, so I guess it's more than just instance.uuid that is needed from the computes. We could still slim down what gets passed from the computes if we just sent a new object (InstanceInfo) or a simple dict with the uuid and instance_type_id keys in it. Will have to think about it.

--

https://review.openstack.org/#/c/457812/ deprecates the TypeAffinityFilter in Pike for removal in Queens. -- mriedem 20170418

Addressed by: https://review.openstack.org/457812
    Deprecate TypeAffinityFilter

I'm going to move this to Queens. If we can deprecate the TypeAffinityFilter in Pike and remove it in Queens, it will make the changes need for this quite a bit easier as all we'll need to pass between nova-compute and nova-scheduler is instance uuids rather than full objects. -- mriedem 20170517

I didn't get the time to work on this in Queens so I'm deferring to Rocky. -- mriedem 20171120

Addressed by: https://review.openstack.org/528099
    Delete the TypeAffinityFilter

Gerrit topic: https://review.openstack.org/#q,topic:bug/1737465,n,z

Addressed by: https://review.openstack.org/569218
    Avoid unnecessary joins in HostManager._get_instances_by_host

Addressed by: https://review.openstack.org/569247
    WIP: Trim the fat on HostState.instances

Addressed by: https://review.openstack.org/570083
    Avoid unnecessary joins in HostManager._get_instances_by_host

Deferring from Rocky but this is being pursued albeit piecemeal. -- mriedem 20180607

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.