VMware: spawn refactor (partial)

Registered by Shawn Hartsock on 2014-03-06

A structured refactor of the vmwareapi spawn method to improve maintainability of the driver. Introduce

Goal: break up spawn method into reusable components

Phase 1: break out the nested inner methods to create components
Phase 2: introduce vmware image objects for validation and convenience
Phase 3: introduce image handler methods for image manipulation

Blueprint information

Status:
Complete
Approver:
John Garbutt
Priority:
High
Drafter:
Shawn Hartsock
Direction:
Approved
Assignee:
Shawn Hartsock
Definition:
Approved
Series goal:
Accepted for juno
Implementation:
Implemented
Milestone target:
milestone icon 2014.2
Started by
Shawn Hartsock on 2014-03-12
Completed by
Michael Still on 2014-09-13

Related branches

Sprints

Whiteboard

See proposal: https://review.openstack.org/#/c/84307/
For design discussion see:
    https://etherpad.openstack.org/p/vmware-spawn-refactor-design

A note on review order:

All patches will be part of one of 3 phases based on their comment. Review in order of phase.

phase 1
    These patches could actually be all independent, but this order is known to work.
    * https://review.openstack.org/#/c/81809/
    * https://review.openstack.org/#/c/81905/
    * https://review.openstack.org/#/c/83207/
    * https://review.openstack.org/#/c/86441/ or https://review.openstack.org/#/c/73865/
    * https://review.openstack.org/#/c/86442/
    * https://review.openstack.org/#/c/86443/

phase 2
    * additional refactoring, mostly of code leading up to gathering
      image information and building a brand new virtual machine. See Phase 2 patches below

phase 3
    The high branching factor in the second half of the spawn code
    is a result of inefficient attempts to address various permutations of the following:

    - different types image to spawn with (sparse, flat, iso,
      streamOptimized, ova, ...)
    - different ways to obtain the image (http download, datastore copy,
      direct use, ...)
    - different ways to prepare the downloaded image bits into a usable
      image in the image cache (replace -flat.vmdk extent, convert from
      sparse, pick from imported VM, ...)
    - different ways to employed the image that has made available (fully
      copy, as base of COW disk, direct use, ...)

    This patch attempts to tease apart into three main areas of
    responsibilites:

    - ImageHandler classes : for obtaining bits of various kinds of images
    - ImageProcessor classes : for processing the bits into a usable form to
                               be cached in the iamge cache
    - Image user callbacks : for dealing with using a prepared/cached image
                             in a VM in various ways.

    The hope is that this will allow for more self-contained additions/changes
    to behaviors in these areas, as well as more avenues to test different
    aspects of the spawn operation in isolation.
---------------------------------------------

Phase 1 patches (all merged)

Gerrit topic: https://review.openstack.org/#q,topic:bp/vmware-spawn-refactor,n,z

Addressed by: https://review.openstack.org/81809
    VMware: spawn refactor - phase 1 - get_vif_info

Addressed by: https://review.openstack.org/84307
    Propose: VMware spawn refactor

Addressed by: https://review.openstack.org/81905
    VMware: spawn refactor - phase 1 - execute_create_vm

Gerrit topic: https://review.openstack.org/#q,topic:bug/1297998,n,z

Addressed by: https://review.openstack.org/83207
    VMWare: spawn refactor - phase 1 - _power_on_vm

Gerrit topic: https://review.openstack.org/#q,topic:bp/and,n,z

Addressed by: https://review.openstack.org/86074
    Propose: VMware ephemeral disk support

Addressed by: https://review.openstack.org/86441
    VMWare: spawn refactor - phase 1 - create_virtual_disk

Addressed by: https://review.openstack.org/86442
    VMware: spawn refactor - phase 1 - fetch_image

Addressed by: https://review.openstack.org/86443
    VMware: spawn refactor - phase 1 - copy_virtual_disk

We are aiming for Juno-1 with this, based on previous conversations, so setting target milestone --johnthetubaguy (28th April 2014)

Now code is upload, target milestone is set, and url is correct, I have approved this --johnthetubaguy (28th April 2014)

Addressed by: https://review.openstack.org/92691
    VMware: spawn refactor - phase 1 - test for spawn

Addressed by: https://review.openstack.org/84713
    VMware: power_off_vm support

Given the code currently up for review, I can't see us getting all the code up for review and in over the next two weeks, so I am moving this to Juno-2. Seems like the coding of this is talking longer than expected? --johnthetubaguy (30th May 2014)

Phase 2 patches:

MERGED:

Addressed by: https://review.openstack.org/52687
    VMware: create utility functions

Addressed by: https://review.openstack.org/98285
    VMware: spawn refactor - DatastorePath class

Gerrit topic: https://review.openstack.org/#q,topic:bp/s,n,z

Addressed by: https://review.openstack.org/98529
    VMware: validation descriptors for VMwareImage

Addressed by: https://review.openstack.org/99238
    VMware: remove unused parameters in imagecache

Addressed by: https://review.openstack.org/99427
    VMware: spawn refactor - Datastore class

Addressed by: https://review.openstack.org/100841
    VMware: remove dsutil.split_datastore_path

Addressed by: https://review.openstack.org/101164
    VMware: refactor get_datastore_ref_and_name

Addressed by: https://review.openstack.org/103948
    VMware: consolidate datastore code

Addressed by: https://review.openstack.org/103949
    VMware: DatastorePath join() and __eq__()

Addressed by: https://review.openstack.org/103950
    VMware: use datastore classes get_allowed_datastores/_sub_folder

Addressed by: https://review.openstack.org/103951
    VMware: use datastore classes in file_move/delete, mkdir

Gerrit topic: https://review.openstack.org/#q,topic:bp_vmware-spawn-refactor,n,z

Addressed by: https://review.openstack.org/104143
    VMware: Pass vm_ref to attach_root_volume if we have it

Addressed by: https://review.openstack.org/104149
    VMware: Trivial indentation cleanups in vmops

Addressed by: https://review.openstack.org/104144
    VMware: Convert vmops to use instance as an object

Addressed by: https://review.openstack.org/106794
    VMware: Consolidate fake_session in test_(vm|ds)_util

Addressed by: https://review.openstack.org/106806
    VMware: Remove unnecessary deepcopy()s in test_configdrive

----

Phase 2 patches still under review

Addressed by: https://review.openstack.org/104145
    VMware: Create VMwareImage object for image metadata

Addressed by: https://review.openstack.org/104146
    VMware: Use ds_util.build_datastore_path() in tests

Addressed by: https://review.openstack.org/104147
    VMware: Remove references to ebs_root from spawn()

Addressed by: https://review.openstack.org/104148
    VMware: refactor spawn() code to build a new VM

Addressed by: https://review.openstack.org/105454
    VMware: test_driver_api: Use local variables in closures

Addressed by: https://review.openstack.org/105736
    VMware: Create fake VM with given datastore

Addressed by: https://review.openstack.org/105737
    VMware: Add _create_array_of_type utility function to fake

Addressed by: https://review.openstack.org/105738
    VMware: Fix type of VM's config.hardware.device in fake

Phase 3 patches:

Addressed by: https://review.openstack.org/98322
    VMware: spawn refactor - image handling

Addressed by: https://review.openstack.org/109752
    VMware: spawn refactor _configure_config_drive

Addressed by: https://review.openstack.org/109753
    VMware: spawn refactor add VirtualMachineConfigInfo

Addressed by: https://review.openstack.org/109754
    VMware: image user functions for spawn()

Addressed by: https://review.openstack.org/109755
    VMware: spawn refactor enlist image

Addressed by: https://review.openstack.org/114817
    VMware: deprecate usage of VMwareVCVMOps

Gerrit topic: https://review.openstack.org/#q,topic:vmware-spawn-refactor,n,z

Addressed by: https://review.openstack.org/115977
    Do_Not_Merge: use oslo.vmware

Addressed by: https://review.openstack.org/115988
    do_not_merge: use oslo.vmware

Gerrit topic: https://review.openstack.org/#q,topic:bp/vmware-vsan-support,n,z

Addressed by: https://review.openstack.org/117467
    VMware: Remove get_copy_virtual_disk_spec from vmops and vm_util

Gerrit topic: https://review.openstack.org/#q,topic:bp/improve-vmware-disk-usage,n,z

Its almost impossible to tell whats happening from the gerrit topics here, need to dig into every change and see whats happening :( --johnthetubaguy 2nd September 2014

Apparently these are now left:
https://review.openstack.org/#/c/109754/
https://review.openstack.org/#/c/109755/
https://review.openstack.org/#/c/117467/
https://review.openstack.org/#/c/114817/
https://review.openstack.org/#/c/98322/

So all but the last patch are bouncing in and out of the gate, so staying in juno for now --johnthetubaguy 2nd September 2014

So, last patch is not making it into juno-3, but the others are in the gate now. --johnthetubaguy 3rd September 2014

Remaining code give -2 to help free up the gate a little bit of FF day. Now marked this as complete. --johnthetubaugy 4th September 2014

Addressed by: https://review.openstack.org/119511
    WIP: VMware: image fetch/cache functions

Addressed by: https://review.openstack.org/119552
    WIP : VMware add tests image fetch/cache fns

Addressed by: https://review.openstack.org/119696
    VMware: add tests for image fetch/cache functions

So last patches for FFE are:
https://review.openstack.org/#/c/98322/
https://review.openstack.org/#/c/119696/
https://review.openstack.org/#/c/117467/
--johnthetubaguy 8th September 2014

Gerrit topic: https://review.openstack.org/#q,topic:rado-vim,n,z

(?)

Work Items

Work items:
get_image_properties (hartsocks): INPROGRESS
get_vif_info (garyk): INPROGRESS
create_vm (maithem): INPROGRESS
_create_virtual_disk (hartsocks): INPROGRESS
_fetch_image_on_esx_datastore (hartsocks): INPROGRESS
_copy_virtual_disk (hartsocks): INPROGRESS
_power_on_vm (tjones): INPROGRESS
Address complex branching (vui): INPROGRESS

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.