NUMA-aware live migration

Registered by Artom Lifshitz on 2018-04-04

When an instance with NUMA characteristics is live-migrated, those characteristics are not recalculated on the destination compute host. In the CPU pinning case, using the source host’s pin mappings on the destination can lead to multiple instances being pinned to the same pCPUs. In the case of hugepage-backed instances, which are NUMA-localized, an instance needs to have its NUMA mapping recalculated on the destination compute host during a live migration.

Blueprint information

Status:
Complete
Approver:
Sylvain Bauza
Priority:
Medium
Drafter:
Artom Lifshitz
Direction:
Needs approval
Assignee:
Artom Lifshitz
Definition:
Approved
Series goal:
Accepted for train
Implementation:
Implemented
Milestone target:
None
Started by
Matt Riedemann on 2018-06-26
Completed by
Eric Fried on 2019-09-16

Whiteboard

Approved for Rocky on April 3rd. -- mriedem 20180404

Gerrit topic: https://review.openstack.org/#q,topic:bp/numa-aware-live-migration,n,z

Addressed by: https://review.openstack.org/558842
    Fix typos in NUMA-aware live migration

Addressed by: https://review.openstack.org/566398
    WIP: Add InstanceNUMATopology to LibvirtLiveMigrateData

Addressed by: https://review.openstack.org/566723
    WIP: Service version check for NUMA live migration

Addressed by: https://review.openstack.org/567242
    WIP: NUMA-aware live migration

Addressed by: https://review.openstack.org/574872
    DNM: Depends on intel-nfv-ci-tests to test NUMA LM

Addressed by: https://review.openstack.org/575179
    WIP: Libvirt live migration: update NUMA XML for dest

We're past feature freeze for Rocky, so this must be deferred. Please re-propose the spec for Stein if you'd like to work on it next cycle. -- melwitt 20180727

Addressed by: https://review.openstack.org/599587
    Re-propose numa-aware-live-migration spec

Re-approved for Stein. -- mriedem 20181130

Addressed by: https://review.openstack.org/634605
    [WIP] Add migration param to check_can_live_migrate_destination

Addressed by: https://review.openstack.org/634606
    [WIP] Use claims when live migrating

Addressed by: https://review.openstack.org/634827
    [Very WIP] New objects to transmit NUMA config from dest to source

Addressed by: https://review.openstack.org/634828
    [Very WIP] Send dest NUMA info inside LiveMigrateData

Addressed by: https://review.openstack.org/635229
    [Very WIP] Update NUMA XML during live migration

Addressed by: https://review.openstack.org/635669
    [WIP] Introduce live migration claims

Addressed by: https://review.openstack.org/636210
    Remove _legacy_dict methods

Addressed by: https://review.openstack.org/636261
    Trivial: reorder hashes according to object_hashes.txt

Addressed by: https://review.openstack.org/638653
    Clarify upgrade situation in NUMA live migration

Addressed by: https://review.openstack.org/638654
    [WIP] Drop MoveClaim in rollback_live_migration_at_destination

Addressed by: https://review.openstack.org/638868
    Revert "Fail to live migration if instance has a NUMA topology"

Addressed by: https://review.openstack.org/639428
    Make the use of the CastAsCall fixture configurable

Addressed by: https://review.openstack.org/640021
    CONF.enable_numa_live_migration is not needed >= Stein

This did not complete in the Stein release so I am deferring to the Train release. Remember to re-propose the spec for Train as necessary: https://specs.openstack.org/openstack/nova-specs/readme.html#previously-approved-specifications -- mriedem 20190311

Addressed by: https://review.openstack.org/647581
    re add for train

efried 20190326: reapproved for train

Gerrit topic: https://review.opendev.org/#/q/topic:bp/numa-aware-live-migration

Addressed by: https://review.opendev.org/635669
    [WIP] Introduce live_migration_claim()

Addressed by: https://review.opendev.org/634827
    New objects for NUMA live migration

Addressed by: https://review.opendev.org/634828
    LM: add support for sending NUMAMigrateData to the source

Addressed by: https://review.opendev.org/635229
    LM: add support for updating NUMA-related XML on the source

Addressed by: https://review.opendev.org/634605
    RPC changes to prepare for NUMA live migration

Addressed by: https://review.opendev.org/634606
    NUMA live migration support

Addressed by: https://review.opendev.org/640021
    CONF.enable_numa_live_migration is not needed >= Stein

Addressed by: https://review.opendev.org/672595
    [WIP] Functional tests for NUMA live migration

Addressed by: https://review.opendev.org/678887
    DNM: Run LM integration tests with NUMA flavor

Gerrit topic: https://review.opendev.org/#/q/topic:bp/numa-aware-live-migration-patch27

Addressed by: https://review.opendev.org/679656
    multi numa nfv testing job

Addressed by: https://review.opendev.org/680738
    [DNM] testing with new lable.

Addressed by: https://review.opendev.org/680739
    [DNM] testing with old lable

Addressed by: https://review.opendev.org/680806
    DNM: Run grenade multinode with vcpu_pin_set

Addressed by: https://review.opendev.org/679754
    [DNM] test migration with pinning

Addressed by: https://review.opendev.org/679805
    [DNM] test non overlaping vcpu pin sets.

Addressed by: https://review.opendev.org/681771
    [DNM] numa + pcpus in placment live migration tests

Addressed by: https://review.opendev.org/681807
    [DNM] cpu pinning testing

Addressed by: https://review.opendev.org/681827
    [DNM] test with dedicated cpus only

Addressed by: https://review.opendev.org/681840
    [DNM] legacy vcpu_pin_set pinning with shared emulator threads

[efried 20190916] Marked complete; only functional test patch https://review.opendev.org/#/c/672595 remains

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.