packages dnsmasq and libvirt-bin conflict with each other

I suppose the problem is this :

libvirt provides DHCP service on the host virtual interface (vnet0...) so that guest VMs can get an address. dnsmasq, by default, listens to all available interfaces, so that includes vnet0. Obviously they conflict by both trying to provide the same service on the same interface.

Please test one of the following workarounds:
Enable the following line in /etc/dnsmasq.conf (supposing eth0 is the interfaces you want to use dnsmasq on):
interface=eth0

Alternatively, enable this line in /etc/dnsmasq.conf (supposing vnet0 is the interface you don't want dnsmasq to perturb):
except-interface=vnet0

This should allow dnsmasq and libvirt-bin to run side by side.

The above workarounds only work if you also set "bind-interfaces" in /etc/dnsmasq.conf, otherwise dnsmasq binds to the wildcard address.

dnsmasq is a general purpose dns (and DHCP, but let's ignore this for the purpose of this conversation) server.
The dnsmasq package installs itself and starts listening on all interfaces. That seems like a reasonable default behaviour for a general purpose caching dns server to me.
libvirt uses dnsmasq (the binary, not the service) to provide dns and dhcp services to virtual networks. If you're installing dnsmasq and libvirt-bin alongside each other, you have to work out the conflicts yourself. I'm not convinced there is a reasonable default fix for this case.

How about a more helpful error message? Perhaps libvirtd could notice that dnsmasq is already running and say "please disable dnsmasq on interface xyz". Or maybe dnsmasq itself could be noticing that?

libvirt now depends on dnsmasq-base, which does not install any config files. After doing a default install
dnsmasq starts with:
  dnsmasq --keep-in-foreground --strict-order --bind-interfaces --pid-file --conf-file --listen-address 192.168.122.1 --except-interface lo --dhcp-leasefile=/var/lib/libvirt/dhcp-default.leases --dhcp-range 192.168.122.2,192.168.122.254

I would guess this is a fix?

The problem is not really in libvirt, which calls dnsmasq with the appropriate settings to limit itself to the right interface. It's in dnsmasq, which by default binds to all interfaces, thus preventing the dnsmasq-from-libvirt from serving its addresses.

I tend to agree with Soren and Neal : yes, that's a conflict configuration case (like if you install two web servers that try to run on the same port), and yes, since it's not that obvious that libvirt runs a DHCP server, we should try to make the potential problem (and solution) more discoverable by documentation.

dnsmasq 2.46-1 (soon in an archive near you) implements a /etc/dnsmasq.d directory that allows to drop configuration snippets from other packages. This opens ways for more collaboration between the two packages.

Following advice in http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=505522
We could for example have:

/etc/dnsmasq.d/libvirt
  bind-interfaces
  except-interface=<virtual bridge>

and libvirt would start its dnsmasq using --interface=<virtual-bridge>

Support for dnsmasq.d is now in Jaunty's dnsmasq.
Ready to work with whatever libvirt would drop there.

Is this still a problem with libvirt?

Regards
chuck

Yes, this is still a problem with libvirt (dnsmasq doesn't start in its default config when you install it after libvirt-bin)

I actually checked that except-interface= is additive, so it's fine to ship a snippet listing it.

Implementation of this in libvirt is blocked by a race condition that prevents the use of the --interface argument in libvirt. There is a race condition when using --listen-address that dnsmasq will immediately exit if the interface is not yet visible to userspace, which can often happen with how virbr%d is created. So, dnsmasq needs more work to not do that.

@Simon: what are the options from a dnsmasq perspective ?

Thierry Carrez wrote:
> @Simon: what are the options from a dnsmasq perspective ?
>

Some background: dnsmasq can run in two modes.

Default mode: dnsmasq binds the wildcard address and does network magic
to determine which interface request packets actually come from, so that
the results can be send back with the correct source address. This has
the advantage that network interfaces can come and go and change IP
address and dnsmasq will keep working. It's possible to restrict dnsmasq
to only reply to requests on some interfaces; requests from other
interfaces will be read by dnsmasq and then silently dropped. Telling
dnsmasq to use an interface which doesn't exist but might in the future
will result in a logged warning, but dnsmasq will still start and when
the interface comes up it will work.

Bind-interfaces mode: This is the traditional way to do UDP servers. At
startup dnsmasq enumerates all the extant interfaces and then opens a
socket for each one, listening on the interfaces's IP address.
Interfaces may be skipped if excluded by the --interface and
--except-interface flags, and any interface specified in --interface
which doesn't exist at start-up will generate a fatal error.

In almost all cases, default mode is better: --bind-interfaces is only
there to cope with old platforms which don't support enough socket
options to do default mode.

The only time when --bind-interfaces works better is when it's desirable
to run more than one instance of dnsmasq. This is not possible in
default mode, but it does work in bind-interfaces mode, providing than
_all_ instances are in bind-interfaces mode, and that they listen on a
disjoint set of interfaces. This is what the libvirt package attempts to
do, as I understand it: it starts its own dnsmasq in --bind-interfaces
mode listening on a virtual interface and it forces any "system" dnsmasq
into --bind-interfaces mode and away from the the virtual interface by
dropping a configuration fragment into /etc/dnsmasq.d

This method brings the disadvantages of --bind-interfaces to the private
libvirt dnsmasq instance (the race condition at start-up) and to the
"system" dnsmasq (which will no longer cope with changes to network
interfaces.)

Options.

1) Keep doing what you're doing, work around the race condition and
accept that installing libvirt will limit the functionality of a system
dnsmasq by forcing it into --bind-interfaces mode.

2) Arrange that if libvirt and a "system" dnsmasq co-exist, libvirt uses
the system dnsmasq, rather than starting its own. It should be possible
to drop some configuration to make the system dnsmasq do what is needed
  for libvirt. There's a problem here if libvirt is installed alone and
using a private dnsmasq, and then subsequently a system dnsmasq is
installed.

3) Make some changes to dnsmasq to improve the situation when two
instances are running. This may be possible, but I don't have many clues
what could be done.

4) Look at some sort of partitioning in the kernel so that the virtual
interface is in some sort of separate partition with effectively two
network stacks. A sort of chroot for networking. I don't know if this ...

@Simon, Thanks for the guidance. Of the options:

1) This seems least invasive in the short-term

2) There's a few potential issues here.

a) Install order, because defaults need to be adjusted, and services may need to restart. Some of this can be handled by triggers, but it requires cooperation (perhaps having some way that dnsmasq-base requires users to drop some file that dns-base can use to restart all users when new ones are installed, or similar awkwardnesses).

b) Removal independence: similar to install issues, the packages need to be able to coordinate uninstallation.

c) This interferes with use cases where a user wants to use dnsmasq to provide nameservice inside libvirt and to do dhcp/tftp on a local network simultaneously or similar cases: using a single demon makes having multiple configurations difficult.

3) The big item that struck me looking at this was that dnsmasq exits if unable to bind to every interface in wildcard mode: having this instead check error conditoins from bind failures and either internally drop the interface/address from the list of interfaces on which to listen or poll on the interface being available depending on the nature of the bind failue would significantly ease the difficulties of running multiple instances, especially one wildcard instance and one or more bound instances. It also eases the race condition identified in comment #11.

Another option would be to extend the D-Bus interface to permit D-Bus to be used to remove addresses or interfaces from the wildcard set to reserve them for use by a non-wildcard dnsmasq, preferably in combination with a D-Bus query on startup to automatically enter non-wildcard mode if there is already a wildcard server active and specifically exclude any addresses or interfaces already claimed by another instance.

This seems a lot of longer-term work though.

4) sounds kinda cool, but needing a lot more investigation and code than any of the other options.

    So, my plan is as follows:

1) Add a dpkg trigger subscription to dnsmasq that calls the initscript with force-reload if new configuration snippets are installed

2) Add a configuration snipped to libvirt that switches to bind-interface by default, and excludes virbr0

    This compeltely ignores the race condition between startng virbr0 adn starting dnsmasq, and it doesn't help for any sort of complex configuration (of either libvirt or dnsmasq), but it seems the least impact path to enable simultaneous operation without crippling either tool or needing extensive coding.

Lucid 10.04

With no bind9 installed and apt-get purge libvirt-bin, dnsmasq, dnsmasq-base:

netstat -nap | grep dnsmasq say it still listen on port 53.

sudo apt-get install dnsmasq give me now the error dnsmasq: failed to create listening socket: Address already in use

Why is port 53 not cleaned after purge?

Natty 11.04

does the same as 10.04!

On 10.04

Netstat give me no listen on port 53 for dnsmasq after the reboot.

Installing with apt-get libvirt-bin dnsmasq dnsmasq-base give me no error's. But no virbr0!

So setting up virbr0 with the command virsh net-define default.xml en virsh net-autostart default is ok. But net-start default in de virsh is not possible!

It say's: in virsh internal error exited with non-zero status 2 and signal 0, failed to bind listening socket for

netstat: dnsmasq is listen on 0.0.0.0:53

With uncomment bind-interfaces in /etc/dnsmasq.conf and restart dnsmasq :

netstat -nap | grep dnsmasq:

The 192.168.122.1, 127.0.0.1, 169.254.169.254, 10.0.0.51 are connect on port 53 for tcp and udp. 0.0.0.0:67 for udp.

In virsh now net-start default without error's.

sudo virsh net-list: default active yes

add user eucalyptus in /etc/group

su eucalyptus -c "virsh net-list" works now.

Emmet in #14> So, my plan is as follows:

Should #959037 be taken into consideration here?

@Thomas,
    959037 means that this issue potentially exists for everyone, rather than just those few folk with specialised needs (mine having been running both a virtualisation host and a tftp server on the same machine).

    The main issue being the racy nature of running multiple greedy dnsmasq configurations. There's likely a right way to do it, but it's tricky, especially as most of the potential solutions significantly complicate many of the current user-oriented use cases for dnsmasq.

    That said, I'm probably not the best person to own this bug any longer, having gotten stalled with the project that required tftp a couple years ago, and so not really trying to solve this. I have another project with TFTP coming up soon, and an running libvirt on the laptop I'd like to use as a server: if I happen to find a generic config that works, I'll post a patch (probably based on the plan outlined above, unless the changes to the code in the meantime are sufficiently large to make that no longer workable).

@Thomas,

I was planning on doing a fix like the one Stéphane pushed for lxc+dnsmasq in 928524. Would that address the concerns in this bug? (I believe it would, but maybe there is more to this particular bug that I am missing).

This bug was fixed in the package libvirt - 0.9.12-0ubuntu3

---------------
libvirt (0.9.12-0ubuntu3) quantal; urgency=low

  * install apport hook as right name - libvirt-bin is the binary package,
    the source package name is libvirt. (LP: #1007405)
  * install /etc/dnsmasq.d/libvirt to configure system wide dnsmasq to not
    listen on the libvirt bridge. (Following Stéphane's lxc example)
    (LP: #928524) (LP: #231060)
    - postinst: restart dnsmasq; postrm: remove dnsmasq.d/libvirt file and
      restart dnsmasq; rules, libvirt-bin.dirs and libvirt-bin.install:
      install new debian/libvirt-bin.dnsmasq file.
 -- Serge Hallyn <email address hidden> Fri, 01 Jun 2012 09:36:58 -0500

Oops, I changed the status to "fix released" but then realized that #231060 wasn't addressed by 0.9.12-0ubuntu3. The latter included

    bind-interfaces
    except-interface=virbr0

whereas to fix #231060 the following would be needed.

    bind-interfaces
    except-interface=vnet0

I don't have the power to revert the status change.

@Thomas,

I'm probably wrong, but are you sure that vnet0 needs to be listed? vnet0 is usually a nic bridged to either br0 or virbr0. i would think that dnsmasq would ignore bridged interfaces.

In fact, using just 'except-interface=virbr0' I brought up a kvm guest (which had vnet0 as its nic on the host), and the libvirt dnsmasq was the one to provide it with an ip address.

@Serge: I was just going by what I read in comment #1 where vnet0 was named.

You are right. Adding --bind-interfaces and --exclude-interface=virbr0 allows both the general dnsmasq and the libvirt-bin dnsmasq to start.

Without --bind-interfaces and --exclude-interface=virbr0:

    dnsmasq[5279]: failed to create listening socket for 192.168.122.1: Address already in use
    dnsmasq[5279]: FAILED to start up

With --bind-interfaces and --exclude-interface=virbr0:

    dnsmasq[5737]: started, version 2.59 cachesize 150
    dnsmasq[5737]: compile time options: IPv6 GNU-getopt DBus i18n DHCP TFTP conntrack IDN
    dnsmasq-dhcp[5737]: DHCP, IP range 192.168.122.2 -- 192.168.122.254, lease time 1h
    dnsmasq[5737]: reading /etc/resolv.conf
    dnsmasq[5737]: using nameserver 127.0.0.1#53
    dnsmasq[5737]: read /etc/hosts - 7 addresses

$ ps -elf|grep dnsmasq
5 S dnsmasq 5441 1 0 80 0 - 8253 poll_s 14:02 ? 00:00:00 /usr/sbin/dnsmasq -x /var/run/dnsmasq/dnsmasq.pid -u dnsmasq -r /var/run/dnsmasq/resolv.conf -7 /etc/dnsmasq.d,.dpkg-dist,.dpkg-old,.dpkg-new --bind-interfaces --except-interface=virbr0
5 S 117 5737 1 0 80 0 - 6491 poll_s 14:03 ? 00:00:00 /usr/sbin/dnsmasq -u libvirt-dnsmasq --strict-order --bind-interfaces --pid-file=/var/run/libvirt/network/default.pid --conf-file= --except-interface lo --listen-address 192.168.122.1 --dhcp-range 192.168.122.2,192.168.122.254 --dhcp-leasefile=/var/lib/libvirt/dnsmasq/default.leases --dhcp-lease-max=253 --dhcp-no-override

Great, thanks Thomas!

In that case, the SRU fixes are pending acceptance in the archive for lucid, oneiric and precise.

Thanks for uploading the fix for this bug report to -proposed. However, when reviewing the package in -proposed and the details of this bug report I noticed that the bug description is missing information required for the SRU process. You can find full details at http://wiki.ubuntu.com/StableReleaseUpdates#Procedure but essentially this bug is missing some of the following: a statement of impact, a test case and details regarding the regression potential. Thanks in advance!

Hi Brian,

until recently I wasn't quite sure if there might be a difference between this bug and bug 928524. There is not, so I will mark this bug a duplicate of bug 928524, which has an SRU justification.

Thanks.

Launchpad did NOT like that :) So I added an SRU justification here. thanks.

From the SRU justification:
> Regression potential: There should be none, since we are
> simply telling the system-wide dnsmasq (if any) not to bind
> to the virbr0 which our own dnsmasq instance will bind to.

There is more risk of trouble than that. With this change to libvirt-bin, after libvirt-bin is installed, dnsmasq will operate in bind-interfaces mode whereas it might have been operating in unbound mode before. In bind-interfaces mode dnsmasq will not notice new interfaces, etc.

That's correct, though currently when installing dnsmasq with lxc, it has a 50% chance of not starting at all (the other 50% being lxc not starting at all).

My initial idea was to set bind-interfaces by default in dnsmasq and SRU that as it's default behaviour is breaking many setups and is extremely annoying.

Although I can see some people complaining about bind-interfaces, it's going to improve reliability for currently broken setups and not affect non-broken setups (as we're pushing this change as part of the lxc package, not dnsmasq).

s/lxc/libvirt-bin/ (copy/pasted the reply from an identical comment in bug 928524)

The proposed package (specifically the postrm) breaks Policy - you're unconditionally removing a conffile on upgrade/removal/etc which will destroy any user changes.

Additionally, I'd like to know what the likely impact of adding bind-interfaces to dnsmasq is on users. What (if anything) will break on users' systems?

On 19/06/12 10:10, Chris Halse Rogers wrote:

> Additionally, I'd like to know what the likely impact of adding bind-
> interfaces to dnsmasq is on users. What (if anything) will break on
> users' systems?
>

Three things change.

1) Dnsmasq loses the ability to provide service on dynamically created
interface. If an interface doesn't exist when dnsmasq starts, dnsmasq
will not be listening on that interface when it comes up. Similarly if
an interface changes address.

2) --interface=<non-existent interface> becomes a fatal error.

3) The semantics of -interface changes. Without --bind-interfaces,
--interface filters on packet arrival interface. With --bind-interface,
--interface=<iface> becomes equivalent to --listen-address=<address(es)
of iface>. So, on a multi-homed host, with --interface=<eth0>, a request
sent to the address of eth0 which arrives from eth1 will be answered
with --bind-interfaces, but not answered without it.

HTH

Simon.

http://paste.ubuntu.com/1062735/ is an irc discussion where RAOF and stgraber appear to come to an agreement about the dnsmasq bit. So I will fix the postrm, and resubmit.

As per discussion on irc, I've kept the removal of /etc/dnsmasq.d/libvirt (because otherwise dnsmasq could be kept in bind-interfaces mode even though it didn't need to be), but added a warning comment at the top of the file that any changes will be lost when libvirt-bin is removed.

Package re-dput'ed to precise-proposed.

From the IRC discussion:

23:01 < RAOF> stgraber: I don't suppose there's a way to get the best of both worlds? Have dnsmasq ignore the interfaces you need, *and* pick up new interfaces?
23:01 < RAOF> For example: how disruptive is restarting dnsmasq?

Please note that Simon Kelley is working on enhancing dnsmasq so that it will automatically detect new interfaces and addresses (on LInux, anyway). That's much better than restarting dnsmasq. Once that enhancement is available, forcing dnsmasq into bind-interfaces mode will be less disruptive.

Forcing dnsmasq into bind-interfaces mode now is disruptive, but not terribly so; stgraber argues (persuasively, IMHO) that libvirt and lxc should force standalone dnsmasq into bind-interfaces mode even without the aforemention enhancement.

(Cont'd.) Simon Kelley on enhancing dnsmasq and subsequent discussion in bug #959037 comment #89 (e.g.).

    https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/959037/comments/89

In the same bug report we discuss various ways of getting NetworkManager-controlled dnsmasq ("nm-dnsmasq") and standalone dnsmasq to coexist. The best solution we've found so far also entails forcing standalone dnsmasq into bind-interfaces mode; beyond that it entails changing nm-dnsmasq's listen address to 127.0.0.2. Again, once dnsmasq is enhanced the inconveniences of bind-interfaces mode should be largely eliminated.

Is there anything more to do in dnsmasq for this bug report?

> Is there anything more to do in dnsmasq for this bug report?

I don't believe so. While it'll definately be great to have dnsmasq
in bind-interfaces mode detect new interfaces, that doesn't need to
be tracked here.

Thanks.

Hello Louis-Dominique, or anyone else affected,

Accepted libvirt into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/libvirt/0.9.8-2ubuntu17.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Verified on precise.

Marking verification-done based on previous comment.

This bug was fixed in the package libvirt - 0.9.8-2ubuntu17.2

---------------
libvirt (0.9.8-2ubuntu17.2) precise-proposed; urgency=low

  * debian/libvirt-bin.install, debian/rules: name the apport file
    source_libvirt.py, not source_libvirt-bin.py. (LP: #1007405)
  * install /etc/dnsmasq.d/libvirt to configure system wide dnsmasq to not
    listen on the libvirt bridge. (Following Stéphane's lxc example)
    (LP: #928524) (LP: #231060)
    - postinst: restart dnsmasq; postrm: remove dnsmasq.d/libvirt file and
      restart dnsmasq; rules, libvirt-bin.dirs and libvirt-bin.install:
      install new debian/libvirt-bin.dnsmasq file.
  * Warn user about bad pc-0.12 machine type, and help user transition.
    (LP: #1001625)
    - qemu-warn-on-pc-0.12.patch: When defining or starting a VM which uses the
      pc-0.12 machine type, warn in libvirtd.log.
    - debian/libvirt-migrate-qemu-machinetype: automatically migrate QEMU VMs
      to newest machine type. This is not done automatically as there will
      be some users who have good reason to stay with pc-0.12.
 -- Serge Hallyn <email address hidden> Mon, 11 Jun 2012 21:52:02 -0500