OpenStack Compute (Nova)

Preallocate VM images to avoid over provisioning and improve performance

Registered by Pádraig Brady on 2013-02-15

Allocate storage for VM images up front for performance improvements and space availability guarantees. This is implemented by calling fallocate where available to allocate file system blocks efficiently when the VM is initially provisioned. This will give immediate feedback if enough space isn't available. Also it should significantly improve performance on writes to new blocks, and may even improve I/O performance to prewritten blocks due to reduced fragmentation.

A new config option is added: preallocate_images={none, space}
With preallocate_images=space, fallocate is called on the VM images.

References used when implementing this:
  commands and perf tests for using fallocate on vm images:
    http://kashyapc.wordpress.com/2011/12/02/little-more-disk-io-perf-improvement-with-fallocateing-a-qcow2-disk/

== Future work ==
For performance reasons it helps to enable the preallocation=metatdata option in qcow images. In remains to be seen if this helps when done on the backing image, or is only significant for the instance images, in which case they need to be copies and can't have a backing file.

So we may in future also support a preallocate_images=performance option that will do additional processing at VM startup to enable more performant I/O at run time.

Reference info on preallocation=metadata:

  preallocation=metadata significant performance advantages
    http://www.redhat.com/archives/libvir-list/2010-October/msg00946.html
  preallocation=metadata incompat with backing_file
    http://www.gossamer-threads.com/lists/openstack/dev/10592
  preallocation=metadata commands and perf tests
    http://itscblog.tamu.edu/improve-disk-io-performance-in-kvm/
  qemu notes on qcow2 performance
    http://wiki.qemu.org/Qcow2/PerformanceRoadmap
  preallocation=metadata on base image may improve perf?
  I can't see how this is effective, and in my testing it's not.
    http://comments.gmane.org/gmane.comp.emulators.kvm.devel/95270

  reason for qcow -> raw conversion in base
    https://github.com/openstack/nova/commit/ff9d353b2f and
    http://pad.lv/932180

Performance notes for various image types.
Tests done by writing in a VM backed by a local file system, using:
  dd if=/dev/zero of=file bs=1M count=1k conv=notrunc,fdatasync oflag=append
I didn't see gradual degradation as some have seen on NFS at least,
but did see quite different performance depending on the formats used:

  disk performance outside VM = 120MB/s
  raw in $instance_dir/ = 105MB/s
  qcow copy with preallocation=metadata in $instance_dir/ = 100MB/s
  qcow CoW with fallocate full size in $instance_dir/ = 55MB/s
    Note perf a bit more stable than without fallocate
    I didn't test with full host disk where improvements would be more noticeable
  qcow CoW in $instance_dir/ = 52MB/s
  qcow CoW in $instance_dir/ backed by qcow with preallocation=metadata in base = 52MB/s

Another thing to consider in future is having allocation supported as a flavor type rather than as a global setting. Perhaps something along the lines of "Instance disk I/O control" at: https://review.openstack.org/#/c/22105

Blueprint information

Status:
Complete
Approver:
Russell Bryant
Priority:
Low
Drafter:
None
Direction:
Approved
Assignee:
Pádraig Brady
Definition:
Approved
Series goal:
Accepted for grizzly
Implementation:
Implemented
Milestone target:
milestone icon 2013.1
Started by
Pádraig Brady on 2013-02-20
Completed by
Thierry Carrez on 2013-02-20

Related branches

Sprints

Whiteboard

Gerrit topic: https://review.openstack.org/#q,topic:bp/preallocated-images,n,z

Addressed by: https://review.openstack.org/22054
    support preallocated VM images

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.