Persist libvirt instance storage metadata (partial)

Registered by Matthew Booth

Libvirt ephemeral storage layout is currently mostly inferred based on the local configuration of the compute node. This is problematic in several cases. In edge cases, it has been the recent cause of several severe security vulnerabilities. It also makes storage configuration hard or impossible to vary between compute nodes in the same installation, or over time after installation. By storing storage metadata of a particular instance explicitly we make its configuration unambiguous and simple to understand, and therefore less vulnerability to security vulnerabilities. We also allow multiple configurations to exist within an installation, and lay a foundation to allow the robust transition between different storage layouts.

Blueprint information

Status:
Complete
Approver:
Matt Riedemann
Priority:
High
Drafter:
Matthew Booth
Direction:
Approved
Assignee:
Matthew Booth
Definition:
Approved
Series goal:
Accepted for newton
Implementation:
Implemented
Milestone target:
milestone icon newton-3
Started by
Matt Riedemann
Completed by
Matt Riedemann

Related branches

Sprints

Whiteboard

Gerrit topic: https://review.openstack.org/#q,topic:bp/libvirt-instance-storage,n,z

Addressed by: https://review.openstack.org/302117
    Persist libvirt instance storage metadata

Addressed by: https://review.openstack.org/279626
    Rename Raw backend to NoBacking

Addressed by: https://review.openstack.org/282432
    Fix signature of copy_image

Addressed by: https://review.openstack.org/267661
    Remove fake_imagebackend.Raw and cleanup dependent tests

Addressed by: https://review.openstack.org/270998
    Rename Image.check_image_exists to Image.exists()

Addressed by: https://review.openstack.org/250872
    Only attempt to inject files if the injection disk exists

Addressed by: https://review.openstack.org/279625
    Add a lock() context manager to image backend

Addressed by: https://review.openstack.org/282434
    Add Image.create_from_func

Addressed by: https://review.openstack.org/278011
    Implement import_file for LVM backend

Addressed by: https://review.openstack.org/267662
    Make import_file do its own cleanup

Addressed by: https://review.openstack.org/279669
    Introduce ImageCacheLocalPool

Addressed by: https://review.openstack.org/265886
    Remove deprecated option libvirt.remove_unused_kernels

Addressed by: https://review.openstack.org/278012
    Implement import_file for ploop backend

Addressed by: https://review.openstack.org/282580
    Update libvirt to use create_from_func and check_backing_from_func

Addressed by: https://review.openstack.org/282433
    Add check_backing_from_func interface

Gerrit topic: https://review.openstack.org/#q,topic:bp/use-libvirt-storage-pools,n,z

Gerrit topic: https://review.openstack.org/#q,topic:libvirt-instance-storage,n,z

Addressed by: https://review.openstack.org/314134
    Implementation of Libvirt Storage Pools

Addressed by: https://review.openstack.org/315582
    Remove unused arguments to images.fetch and images.fetch_to_raw

Addressed by: https://review.openstack.org/316295
    Add RC file for excluding tempest tests for LVM job

Addressed by: https://review.openstack.org/317689
    Ploop: add create_from_image & create_from_func

Addressed by: https://review.openstack.org/317694
    Rbd: add create_from_image & create_from_func

Addressed by: https://review.openstack.org/317701
    Flat: add create_from_image & create_from_func

Addressed by: https://review.openstack.org/318948
    Image: add create_from_image & create_from_func

Addressed by: https://review.openstack.org/320610
    Qcow2: add create_from_image & create_from_func

Addressed by: https://review.openstack.org/320611
    Lvm: add create_from_image & create_from_func

Addressed by: https://review.openstack.org/320910
    Remove image cache image verification

Addressed by: https://review.openstack.org/321579
    Tidy up get_cache_fname

Addressed by: https://review.openstack.org/322285
    Add check_backing_from_func & check_backing_from_image

Addressed by: https://review.openstack.org/322974
    Delete old imagebackend methods & tests

Addressed by: https://review.openstack.org/323038
    Remove the last bits of mox from test_imagebackend

Addressed by: https://review.openstack.org/325496
    Add concurrency tests for ImageCacheLocalDir

Addressed by: https://review.openstack.org/326947
    Remove max_size argument to images.fetch and fetch_to_raw

Addressed by: https://review.openstack.org/329294
    Remove fake_imagebackend.Raw and cleanup dependent tests

Addressed by: https://review.openstack.org/329421
    Only attempt to inject files if the injection disk exists

Addressed by: https://review.openstack.org/332416
    Resize an existing disk

Addressed by: https://review.openstack.org/333244
    Add create_from_image and create_from_func imagebackend methods

Addressed by: https://review.openstack.org/333241
    Use correct uuids and fnames in test_create_image_initrd

Addressed by: https://review.openstack.org/333242
    Replace mox with mock in libvirt rescue tests

Addressed by: https://review.openstack.org/333243
    Pass instance object to _create_images_and_backing in test

Addressed by: https://review.openstack.org/333263
    Pass is_block_dev explicitly from create_from_func

Addressed by: https://review.openstack.org/333271
    libvirt: Pass path to Image base class

Addressed by: https://review.openstack.org/333272
    Rename snapshot() to from_libvirt_path()

Addressed by: https://review.openstack.org/333521
    Rename the import of nova.virt.disk.api from disk to disk_api

Addressed by: https://review.openstack.org/333522
    Add Backend from_image and from_func

Addressed by: https://review.openstack.org/328366
    Fix fake data returned by LibvirtDriverTestCase._disk_info

Addressed by: https://review.openstack.org/333978
    libvirt: Replace some uses of invalid uuids in test_driver

Addressed by: https://review.openstack.org/333979
    libvirt: Remove some unnecessary mocking in test_driver

Addressed by: https://review.openstack.org/333980
    libvirt: Add _create_external_boot_disks

Addressed by: https://review.openstack.org/333981
    libvirt: Move config disk creation into a separate function

Addressed by: https://review.openstack.org/333982
    libvirt: Move local root disk fetch into a separate function

Addressed by: https://review.openstack.org/333983
    libvirt: Move enumeration of ephemerals into a separate function

Addressed by: https://review.openstack.org/333984
    libvirt: Move swap disk fetch into a separate function

Gerrit topic: https://review.openstack.org/#q,topic:libvirt_create_image,n,z

Gerrit topic: https://review.openstack.org/#q,topic:test_rescue,n,z

Addressed by: https://review.openstack.org/337789
    libvirt: Replace _create_images_and_backing in _hard_reboot

Addressed by: https://review.openstack.org/337790
    All the things

Gerrit topic: https://review.openstack.org/#q,topic:libvirt-imagebackend,n,z

Gerrit topic: https://review.openstack.org/#q,topic:storage_policy_2,n,z

Newton is now past feature freeze so the work for this will need to continue in the Ocata release, I've already opened a blueprint for that:

https://blueprints.launchpad.net/nova/+spec/libvirt-instance-storage-ocata

-- mriedem 20160907

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.