VMware: Image Cache Management

Registered by Gary Kotton

At the moment there is an image cache. This is not managed at all and in time cached images may use of all of the disk space on the data store. The blueprint will enable one or more nova-compute instances to populate and age the cache.
The blueprint will deal with the following:
1. Refactor the file uploading to the cache: this has two parts. The first is uploading the images to a temporary folder. The second is copying from the temporary folder to the actual cache. In order to simplify the image cache management the image files will be stored in a folder that is named after the image reference id.
2. Aging of cached instances

Blueprint information

Status:
Complete
Approver:
Russell Bryant
Priority:
Undefined
Drafter:
Gary Kotton
Direction:
Needs approval
Assignee:
Gary Kotton
Definition:
Approved
Series goal:
None
Implementation:
Implemented
Milestone target:
None
Started by
Gary Kotton
Completed by
Gary Kotton

Related branches

Sprints

Whiteboard

6 February: Sadly there is no way to track the BP changes and comments. Just an update:
    1. The patch is based on https://review.openstack.org/#/c/59994 - this created a common imagecaching module - this is used by libvirt and the vmware driver
    2. There are no configuration changes. The configuration variables defined in https://github.com/openstack/nova/blob/master/nova/virt/imagecache.py#L21 are used
-- garyk

Deferred to icehouse-3 as the blueprint was not approved by the icehouse-2 blueprint approval deadline. --russellb

This is an important set of patches to merge for Minesweeper. We need to have this soon. --hartsocks

Gerrit topic: https://review.openstack.org/#q,topic:bp/vmware-image-cache-management,n,z

Addressed by: https://review.openstack.org/54096
    VMware: upload images to temporary directory

Addressed by: https://review.openstack.org/52630
    VMware: fix bug when more than one datacenter exists

Addressed by: https://review.openstack.org/56416
    VMware: cache aging

I would really rather see this implemented using common image cache code instead of something vmware specific. Also see https://blueprints.launchpad.net/nova/+spec/multiple-image-cache-handlers -russellb

To approve this I would rather there were not loads of VMware specific config options, we should look to extract existing common image cache properties.

@Russel - please see https://review.openstack.org/#/c/59994/

We are also missing all the config changes, and information that will be needed for operators to use this feature. I don't mind punting on some of the details and adding them later, but we have code for this now, so they should be added before we approve this for merge -- johnthetubaguy

@johnthetubaguy - please see https://review.openstack.org/#/c/59994/

@gary.kotton - I like the re-use of variables. But the blueprint needs enough detail for the docs people to do their magic. I don't think that is currently the case, so I am not approving this just yet -- johnthetubaguy

The generic image cache mechanism is used - please see https://review.openstack.org/#/c/56416/

Addressed by: https://review.openstack.org/57742
    VMware: image aging support

Addressed by: https://review.openstack.org/59994
    Image cache: move all of the variables to a common place

Addressed by: https://review.openstack.org/63933
    VMware: create datastore utility functions

Patches still to merge
=================

Addressed by: https://review.openstack.org/63933
    VMware: create datastore utility functions

This depends on the above:

Addressed by: https://review.openstack.org/56416
    VMware: cache aging

Not sure if this was deferred at some point, but this still seem good for icehouse --johnthetubaguy 26th Feb 2014

Need to look closely at the one remaining patch on this, its a possible FFE --johnthetubaguy
https://review.openstack.org/#/c/56416/

This seems to have lost its FFE now, as the above patch is blocked with some serious concerns. Demoting the priority to reflect this. --johnthetubaguy.

Removed from next, as next is now reserved for near misses from the last milestone --johnthetubaguy6 February: Sadly there is no way to track the BP changes and comments. Just an update:
    1. The patch is based on https://review.openstack.org/#/c/59994 - this created a common imagecaching module - this is used by libvirt and the vmware driver
    2. There are no configuration changes. The configuration variables defined in https://github.com/openstack/nova/blob/master/nova/virt/imagecache.py#L21 are used
-- garyk

Deferred to icehouse-3 as the blueprint was not approved by the icehouse-2 blueprint approval deadline. --russellb

This is an important set of patches to merge for Minesweeper. We need to have this soon. --hartsocks

Gerrit topic: https://review.openstack.org/#q,topic:bp/vmware-image-cache-management,n,z

Addressed by: https://review.openstack.org/54096
    VMware: upload images to temporary directory

Addressed by: https://review.openstack.org/52630
    VMware: fix bug when more than one datacenter exists

Addressed by: https://review.openstack.org/56416
    VMware: cache aging

I would really rather see this implemented using common image cache code instead of something vmware specific. Also see https://blueprints.launchpad.net/nova/+spec/multiple-image-cache-handlers -russellb

To approve this I would rather there were not loads of VMware specific config options, we should look to extract existing common image cache properties.

@Russel - please see https://review.openstack.org/#/c/59994/

We are also missing all the config changes, and information that will be needed for operators to use this feature. I don't mind punting on some of the details and adding them later, but we have code for this now, so they should be added before we approve this for merge -- johnthetubaguy

@johnthetubaguy - please see https://review.openstack.org/#/c/59994/

@gary.kotton - I like the re-use of variables. But the blueprint needs enough detail for the docs people to do their magic. I don't think that is currently the case, so I am not approving this just yet -- johnthetubaguy

The generic image cache mechanism is used - please see https://review.openstack.org/#/c/56416/

Addressed by: https://review.openstack.org/57742
    VMware: image aging support

Addressed by: https://review.openstack.org/59994
    Image cache: move all of the variables to a common place

Addressed by: https://review.openstack.org/63933
    VMware: create datastore utility functions

Patches still to merge
=================

Addressed by: https://review.openstack.org/63933
    VMware: create datastore utility functions

This depends on the above:

Addressed by: https://review.openstack.org/56416
    VMware: cache aging

Not sure if this was deferred at some point, but this still seem good for icehouse --johnthetubaguy 26th Feb 2014

Need to look closely at the one remaining patch on this, its a possible FFE --johnthetubaguy
https://review.openstack.org/#/c/56416/

This seems to have lost its FFE now, as the above patch is blocked with some serious concerns. Demoting the priority to reflect this. --johnthetubaguy.

Removed from next, as next is now reserved for near misses from the last milestone --johnthetubaguy

Marking this blueprint as definition: Drafting. If you are still working on this, please re-submit via nova-specs. If not, please mark as obsolete, and add a quick comment to describe why. --johnthetubaguy (20th April 2014)

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.