libvirt-bin support for hugepages: apparmor needs a rule for hugetlbfs

Firstly, transparent hugepages can be enabled with:

# echo always > /sys/kernel/mm/transparent_hugepage/enabled

they are nice and easy to get working, but do not survive memory pressure on the host. I/O pressure on the host breaks the 2M pages back to 4k pages, and therefore renders transparent hugepages useless for long-lived processes, as nothing will reassemble the 4k pages back to 2M pages after memory pressure has ceased. So, kvm should be set up using explicit hugetlbfs support for now.

Firstly, kvm is very irritating in that it tends to silently use ordinarily backed memory if hugepage support fails to work, so for a while I thought things were fine despite I had just wasted >8 GB of memory on unused hugepages. This is, however, something to fix within kvm itself rather, and it may be that this is how kvm maintainers prefer it to work. I personally disagree, but it is possible to cat /proc/meminfo and observe if kvm is using hugepages or not: HugePages_Free value equals HugePages_Total when nobody is using hugepages.

I chose to mount hugetlbfs at /hugepages, which makes the directory for kvm's memory pages /hugepages/libvirt/qemu. libvirt-bin had to be restarted for it to recognize the hugetlbfs mount location. I need total of 4400 hugetlb pages for my VMs, so I wrote a /etc/sysctl.d/60-custom.conf which says "vm.nr_hugepages=4400" and set that value using sysctl -w. I had to retry that command a bunch of times, each time getting more hugepages than before, until finally I got all 4400.

However, apparmor still denies access to the hugepage mount point by default. I added a line such as "/hugepages/libvirt/qemu/* rw," to every virtual machine's apparmor profie, which are generated from /etc/apparmor.d/libvirt/TEMPLATE by libvirt-aa-helper. I did not discover a way to regenerate the files from the TEMPLATE, so I just manually modified every one after fixing the TEMPLATE file.

I would suggest that libvirt-aa-helper should detect hugepages usage from the domain configuration file (<memoryBacking><hugepages/></memoryBacking>) and then find out where hugepages are mounted and add that rw grant to that location.

$ lsb_release -rd
Description: Ubuntu 12.04 LTS
Release: 12.04

$ apt-cache policy libvirt-bin
libvirt-bin:
  Installed: 0.9.8-2ubuntu17.1
  Candidate: 0.9.8-2ubuntu17.1
  Version table:
 *** 0.9.8-2ubuntu17.1 0
        500 http://fi.archive.ubuntu.com/ubuntu/ precise-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     0.9.8-2ubuntu17 0
        500 http://fi.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages