VMware nova – Storage optimization for clusters with multiple datastores

This blueprint allows images to launch faster and saves disk space by
creating linked clones of instances using the base disk already present in
the image cache of other datastore.
VMware ESX cluster can have multiple datastores that are shared across all the
hosts of the cluster. Its much more efficient to have a single copy of the
image in only one datastore of the cluster and create linked clones using
this image.

Problem description
===================

ESX clusters can have multiple shared datastores configured.
A datastore is a logical container that stores the virtual machine files.
This includes both the disk (vmdk), configuration (vmx) files and
snapshot files.

A datastore can be created from different types of physical storage
like local direct attached storage, iSCSI, FC SAN and NFS.
The physical storage is presented to all host in the cluster and therefore
the datastore is accessible from all the hosts of the cluster.
Each cluster will typically have its own set of datastores and this is
the recommended configuration. This configuration enable the live migration
of VMs across all hosts of the cluster.

When using the VMware ESX hypervisor nova driver for such clusters with
multiple shared datastores the image can get cached in each datastore.
Additionally a disk of size equal to the flavor of the instance is also
created in the cache. Instance are created as linked clones to this flavor
sized disks.
This approach has the following problems:
* Every time a new datastore is selected by the driver for deployment, a new
  cache is created on the datastore and the image is copied.
* A copy of the image expanded to the size of the flavor is created for each
  instance deployed with a different flavor
This approach leads to loss of actual storage space for creating instances.

In the existing design, when an instance creation is done, the following occur

* Step 1. The driver determines the best datastore to place the instance. It
  does this by selecting the datastore with maximum free space.
* Step 2. The existence of the image is checked in the cache
  (directory named _vmware_base) on the clusters datastore
* Step 3. If the image is not available in the cache, then
  (i) the image is downloaded from glance into nova-compute
  (the VM where the compute service runs)
  (ii) Then the image is transferred from the nova-compute to the datastore
  by vCenter
* Step 4.Instance is spawned using the cached image by first creating a disk
  of the size of the specified in the flavor. Then creating a linked clone
  using the flavor sized disk as the base disk.

The problems exist in Step 1 and Step 2 since the the driver is not utilizing
the cache in other datastores.

Proposed change
===============

* 1. The drivers datastore selection can be modified to select the datastore
  that has the image already cached and if has enough space for the new
  instance (as per Step 4 above)
* 2. If the datastore where the image is cached does not have enough space for
  the new instance, then create a linked clone in a different datastore but
  still using the cached image as the base disk