VM fails to start when vcpu placement='auto'

Default Comment by Bridge

Hi,
libvirt is build --with-numactl on
  amd64 i386 ia64 mips mipsel powerpc ppc64el s390x

You can check effectively if it was built against it by checking for the lib depend like:
ldd /usr/sbin/libvirtd | grep numa
        libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f99e4a39000)

For further debugging until we can reproduce, can you enable more debugging of libvirt and add the more verbose libvirt log here so you and we can check for issues in it?
See:
 http://libvirt.org/guide/html/Application_Development_Guide-Connections-Debug.html

The failure is about numad (vs. libnuma).

I think what's needed here is libvirt to be built with
 '--with-numa=/path/to/numad' (which it isn't).

However, numad is part of universe and libvirt can't depend upon a package
in universe since it's part of main IIUC.

On Fri, Sep 9, 2016 at 12:57 AM, ChristianEhrhardt <
<email address hidden>> wrote:

> Hi,
> libvirt is build --with-numactl on
> amd64 i386 ia64 mips mipsel powerpc ppc64el s390x
>
> You can check effectively if it was built against it by checking for the
> lib depend like:
> ldd /usr/sbin/libvirtd | grep numa
> libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1
> (0x00007f99e4a39000)
>
> For further debugging until we can reproduce, can you enable more
> debugging of libvirt and add the more verbose libvirt log here so you and
> we can check for issues in it?
> See:
> http://libvirt.org/guide/html/Application_Development_Guide-
> Connections-Debug.html
>
> --
> You received this bug notification because you are a member of Ubuntu
> Xen Team, which is subscribed to libvirt in Ubuntu.
> https://bugs.launchpad.net/bugs/1621121
>
> Title:
> VM fails to start when vcpu placement='auto'
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/
> 1621121/+subscriptions
>

Status changed to 'Confirmed' because the bug affects multiple users.

Will investigate this in 17.04, as we have some investigation to do to add the feature. numad isn't part of main, so there will be an MIR assuming we determine that is a pre-req for this feature.

------- Comment From <email address hidden> 2016-10-04 06:04 EDT-------
Any Updates?

Appreciate this is due to be looked at in 17.04 - but - is there a workaround for manually linking numad so that libvirt finds it? From what i can see, it's been built with the library, so, providing numad is operational it should be a case of showing libvirt where it is. Unless i'm missing something?

I have several hosts where numad is installed and operational so if i knew where libvirt was looking for numad then maybe it can be linked in to get it working under 16.04 LTS?

Thanks in advance,

On Tue, Oct 25, 2016 at 6:04 AM, Robert Williams <email address hidden> wrote:

> Appreciate this is due to be looked at in 17.04 - but - is there a
> workaround for manually linking numad so that libvirt finds it? From
> what i can see, it's been built with the library, so, providing numad is
> operational it should be a case of showing libvirt where it is. Unless
> i'm missing something?
>
> I have several hosts where numad is installed and operational so if i
> knew where libvirt was looking for numad then maybe it can be linked in
> to get it working under 16.04 LTS?
>

I don't think this is possible; the feature is a build-time feature and we
cannot
enable it as it requires including numad which is part of Universe for a
package
in main.

As a workaround, you could rebuild the package with the added config flag
in a PPA.

I'll see if I can get such a build into my own PPA and share the debdiff
here.

numad itself will need a MIR[1] and be approved before it can be part of
main.

1. https://wiki.ubuntu.com/MainInclusionProcess

> Thanks in advance,
>
> --
> You received this bug notification because you are a member of Ubuntu
> Virtualisation team, which is subscribed to libvirt in Ubuntu.
> https://bugs.launchpad.net/bugs/1621121
>
> Title:
> VM fails to start when vcpu placement='auto'
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/16211
> 21/+subscriptions
>

Everybody sure that last update by Skymathrix was correct - I didn't see any solution?
Please anybody confirm if there was an alternate solution found, otherwise we should set that back to confirmed.

Hi,
I wanted to at least confirm the suggestion of Ryan.

In the configure of libvirt source this is:
   --with-numad use numad to manage CPU placement dynamically
                           [default=check]

I built it with numad as build time dependency to see if it picks all up correctly.
It does, it ends up with:
  configure: numad: yes

As for the binary dependency I would only choose a "suggests".
Because to quote numad's NEWS file:
  Starting or stopping the 'numad' daemon while running a large virtual
  machine may be dangerous so installing (or removing) this package won't
  start (or stop) it.

Due to its nature numad can cause severe overhead as well as impact an existing system when moving or constraining numa attributed of processes.

So I was a bit afraid it could add a binary dependency via shlibs or so.
But it seems to be ok knowing numad exists - and since it is no lib but only a binary it does not add a dep.

Here in my newly built Deb dir:
for i in $(ls -1 *.deb | sed 's/.deb//'); do apt-cache show $i | grep numad; done
Suggests: libvirt-daemon-system, numad

The diff to get there is trivial, I'll attach it.
But as Ryan suggested the thing needed is a MIR for numad.

Hey,
since I had it available for test builds I created a ppa with it.
=> https://launchpad.net/~paelzer/+archive/ubuntu/bug-1621121-libvirt-numad

@IBM - since I can think of all sorts of negative impacts when enabling numad - could you please take a look at the ppa. If you confirm that with it not only the auto-placement works, but also you didn't happen to run into other new issues we could start a MIR request for numad (which is fortunately not a lot of code).
But that means if possible a full test-suite set or whatever you have defined for that usually.

And as it clearly was not fixed removing that status again.

Default Comment by Bridge

Is there any comment along the xml that bugproxy is still holding back?

@paelzer, so the build-depend can live in universe (since the archive reorg in 16.04), even if libvirt's source is in main -- and if you are manually specifying the resulting dependency to be a suggests (and not a recommends), then is a MIR even needed?

The real question I have is, if libvirt is built with numad support (and a path to the binary is embedded somehow), does libvirt properly handle that binary not being present at runtime?

Hi Nish,
I had the same conclusion and IMO we don't need a MIR.

In fact on merging recent libvirt we get this "for free" without a Delta to be added.

I haven't tested the case of uninstalled numad yet, as I was (I'm still) waiting for the reporters to confirm if the suggested way of inclusion (merging Debian would be about the same) would suit their needs first.

Only then I wanted to go on investigating more to test/verify potential pitfalls from our side.

------- Comment From <email address hidden> 2017-01-12 04:53 EDT-------
reproducible in 16.04.2 , any plans to include the fix in this stream?

#uname -a
Linux powerkvm2-lp1 4.8.0-34-generic #36~16.04.1-Ubuntu SMP Wed Dec 21 18:53:20 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux

dpkg -l|grep libvirt-bin
ii libvirt-bin 1.3.1-1ubuntu10.6 ppc64el programs for the libvirt library

dpkg -l|grep numad
ii numad 0.5+20150602-4 ppc64el User-level daemon that monitors NUMA topology and usage

# virsh start --console 1610
error: Failed to start domain 1610
error: unsupported configuration: numad is not available on this host

Hi Satheera,
the issue is now fixed in Zesty (Development release) and and the Dependency a suggest.
But there is no intention to bring that as a feature back to Xenial.
Especially since the daemon itself is only in universe I'd not consider it all too trustworthy.

I was ok with enabling since it is opt-in and only installed by user (no hard dependency), but putting that into Xenial would take serious convincing efforts to the SRU team.

If you want / need that please give it a try (https://wiki.ubuntu.com/StableReleaseUpdates), but I'm personally not yet convinced it is worth.

FYI - also there was a numad issue see bug 1640783.
In case you are verifying this please consider checking the version of numad to have that included.

------- Comment (attachment only) From <email address hidden> 2017-02-03 08:05 EDT-------

Lacking SRU engangement for Xenial so far I'll set the Xenial task to incomplete for now.

------- Comment (attachment only) From <email address hidden> 2017-02-03 08:05 EDT-------

I'm passing through old bugs, lacking further discussion and any "forcing" argument why that would be needed in Xenial (which would be feature enablement in SRU, so that would have to be a very good argument) I'm closing the Xenial task to clean up.

Also if needed users have the option to get this in Xenial by using the Cloud-Archive Ocata or Pike.