download images only once per conductor in pxe driver

This bug covers two different things:
 - don't download the same image twice [to the same conductor]
 - download images in parallel.

The former is an efficiency gain, the latter is actually often going to be a pessimisation. Why? because unless there are multiple workers at both ends, parallelisation just means 'context switch more often' - e.g. if we don't read() at wire speed, tcp won't fully open and we'll see slower downloads of each image (particularly visible on faster networks where we can't easily keep up).

So, I'd like to make this bug be 'download any given image once only', and we can use profiling to see whether image download performance is an issue. If it is, we'll need this to be a configurable setting, because a machine with only one hard disk will almost always be better off downloading one thing at a time (less seeks == more throughput) particularly if images exceed the working set on the machine. Long term for performance we should be downloading only the kernel and ramdisk anyway, and using some more efficient (bittorrent/multicast/bitfountain) method for image deployment straight out of glance.

Also refer to discussion had place near 21:00 http://eavesdrop.openstack.org/irclogs/%23openstack-ironic/%23openstack-ironic.2014-02-10.log

Fix proposed to branch: master
Review: https://review.openstack.org/85387

Reviewed: https://review.openstack.org/85387
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=fb771158d0c7b3876e09808b9897f9d64a9fdcf4
Submitter: Jenkins
Branch: master

commit fb771158d0c7b3876e09808b9897f9d64a9fdcf4
Author: Dmitry Tantsur <email address hidden>
Date: Sat Mar 29 16:18:18 2014 +0100

    Implement more robust caching for master images

    This patch introduces caching for master images, which
    are no longer used. This patch also introduces clean up procedure,
    which is triggered every time cache size increases or link
    to master images is deleted. Algorithm is following:
    - Images are eligible for deletion, if their link count equals 1.
      Images that has link count >1 are not considered for deletion,
      because it anyway won't reclaim any disk space.
    - All images older than configurable TTL are deleted
    - Then the oldest images are deleted, until size of cache fits
      into configurable value or no more images eligable for deletion.

    As this change required rethinking how we cache images, I also
    did a refactor and grouped related functions under ImageCache class.

    Change-Id: I1c849fe209c4dbb15e075dbb34c4ea33010a8883
    Closes-Bug: #1199665