Validate target host with scheduler hints for live migration optionally

Registered by Guangya Liu (Jay Lau) on 2013-07-20

This is related to blueprint https://blueprints.launchpad.net/openstack/?searchtext=live-migration-to-conductor. The blueprint is trying to move live migration logic to nova-conductor, this is a good direction.

But for live migration with target host, the current implementation is that once target host was specified for live migration, then the conductor will only validate the host by nova compute. For some cases, there is also a need to validate the host in nova scheduler, as there might be some requirement that customer want to validate the target host for live migration with some scheduler filters.

In the current implementation the user can specify scheduler hints related to the placement of the instance. Those scheduler hints are used when the instance run for the first time and they are ignored afterwards. In order to achieve the validation of the target host we need to persist scheduler hints into the database which makes it possible to verify that the target host passes scheduler filters/hints during live migration.

There are some cases when this verification during live migration will lead the system locked and the user will not be able to migrate an instance, so a new option should be added in order to force the migration without verifying the target host by nova-scheduler.

Blueprint information

Status:
Complete
Approver:
Russell Bryant
Priority:
Undefined
Drafter:
None
Direction:
Needs approval
Assignee:
Guangya Liu (Jay Lau)
Definition:
Obsolete
Series goal:
None
Implementation:
Started
Milestone target:
None
Started by
Guangya Liu (Jay Lau) on 2013-07-20
Completed by
John Garbutt on 2014-05-28

Related branches

Sprints

Whiteboard

Deferred to icehouse-3 as the blueprint was not approved by the icehouse-2 blueprint approval deadline. --russellb

I think this is a tweak to the existing blueprint you referenced, so I'd like to avoid having this one as well. Please add your comments to that blueprint and coordinate with the owners. Thanks. --dansmith

@dansmith, this bp wants to make sure that the target host is a valid host before live migration but not a tweak to the existing blueprint. We just want to make sure that the target host can still obey the filter rules when doing live migration. IMHO, It is scheduler's responsibility to validate/select target host for any VM instance operations. Thanks. --jay-lau-513

I will split the tasks/patches to some small ones for better review later. Thanks. --jay-lau-513

Gerrit topic: https://review.openstack.org/#q,topic:bug/1198977,n,z

Addressed by: https://review.openstack.org/45450
    Validate target host with scheduler hints for live migration

Gerrit topic: https://review.openstack.org/#q,topic:bp/validate-targethost-live-migration,n,z

Addressed by: https://review.openstack.org/62408
    Store scheduler_hints to Instance table when create instance

I think this blueprint is fine to live on its own. This is changing live migration logic, while the other blueprint is just about moving where the flow of control lives. --russellb

The summary of this blueprint says "optionally". What do you intend to make optional about this? --russellb

This is used to give admin some option to decide if they want to validate the host with scheduler hints or not. For some cases, it might be not needed, i.e. force host, if the VM deployed with force host option, then if we validate the host with scheduler hints, the result would be it can only reside on the force host but cannot migration, resize etc, so we give admin this option to ignore scheduler hints for such case. --jay-lau-513

It's good to have a force option to ignore scheduler hints during migration, but will be scheduler_hints validated that they are still correct? I mean, if we have two VMs, vm1 (no hints) and vm2 (hints: same_host=vm1) and then delete vm1. The hints become meaningless and confusing in this case, is it planned to handle it somehow (maybe purge them if they're no longer valid) --novel

Thanks @Novel, really good comments! But I think that what you suggested should deserve another BP and in this BP, I want to limit the scope to force/ignore scheduler hints. Hope it is OK. ;-) --jay-lay-513

Addressed by: https://review.openstack.org/63658
    Validate target host by scheduler

Addressed by: https://review.openstack.org/63853
    Make live migration validation optional

Addressed by: https://review.openstack.org/64046
    WIP: Add scheduler_hints to the output of "nova show"

deferred from icehouse-3 to "next": http://lists.openstack.org/pipermail/openstack-dev/2014-February/026335.html

@Russell, can we do this in Ice-house-3 as all of the patches are now ready for review. Thanks.

Removed from next, as next is now reserved for near misses from the last milestone --johnthetubaguyDeferred to icehouse-3 as the blueprint was not approved by the icehouse-2 blueprint approval deadline. --russellb

I think this is a tweak to the existing blueprint you referenced, so I'd like to avoid having this one as well. Please add your comments to that blueprint and coordinate with the owners. Thanks. --dansmith

@dansmith, this bp wants to make sure that the target host is a valid host before live migration but not a tweak to the existing blueprint. We just want to make sure that the target host can still obey the filter rules when doing live migration. IMHO, It is scheduler's responsibility to validate/select target host for any VM instance operations. Thanks. --jay-lau-513

I will split the tasks/patches to some small ones for better review later. Thanks. --jay-lau-513

Gerrit topic: https://review.openstack.org/#q,topic:bug/1198977,n,z

Addressed by: https://review.openstack.org/45450
    Validate target host with scheduler hints for live migration

Gerrit topic: https://review.openstack.org/#q,topic:bp/validate-targethost-live-migration,n,z

Addressed by: https://review.openstack.org/62408
    Store scheduler_hints to Instance table when create instance

I think this blueprint is fine to live on its own. This is changing live migration logic, while the other blueprint is just about moving where the flow of control lives. --russellb

The summary of this blueprint says "optionally". What do you intend to make optional about this? --russellb

This is used to give admin some option to decide if they want to validate the host with scheduler hints or not. For some cases, it might be not needed, i.e. force host, if the VM deployed with force host option, then if we validate the host with scheduler hints, the result would be it can only reside on the force host but cannot migration, resize etc, so we give admin this option to ignore scheduler hints for such case. --jay-lau-513

It's good to have a force option to ignore scheduler hints during migration, but will be scheduler_hints validated that they are still correct? I mean, if we have two VMs, vm1 (no hints) and vm2 (hints: same_host=vm1) and then delete vm1. The hints become meaningless and confusing in this case, is it planned to handle it somehow (maybe purge them if they're no longer valid) --novel

Thanks @Novel, really good comments! But I think that what you suggested should deserve another BP and in this BP, I want to limit the scope to force/ignore scheduler hints. Hope it is OK. ;-) --jay-lay-513

Addressed by: https://review.openstack.org/63658
    Validate target host by scheduler

Addressed by: https://review.openstack.org/63853
    Make live migration validation optional

Addressed by: https://review.openstack.org/64046
    WIP: Add scheduler_hints to the output of "nova show"

deferred from icehouse-3 to "next": http://lists.openstack.org/pipermail/openstack-dev/2014-February/026335.html

@Russell, can we do this in Ice-house-3 as all of the patches are now ready for review. Thanks.

Removed from next, as next is now reserved for near misses from the last milestone --johnthetubaguy

Marking this blueprint as definition: Drafting. If you are still working on this, please re-submit via nova-specs. If not, please mark as obsolete, and add a quick comment to describe why. --johnthetubaguy (20th April 2014)

Addressed by: https://review.openstack.org/89502
    validate target host for live migration

Spec abandoned, un-targeting blueprint from juno-1 --johnthetubaguy (28th May 2014)

I do like the idea of an API for admins to check if they validate any rules when picking a host, but it in its current form, lets abandon this blueprint.

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.