Network Security: VM hosts can SSH to compute node

We could add a default boolean that would be false by default before pushing this to trunk ... The effect of this patch would be the following:

Chain nova-network-FORWARD (1 references)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT tcp -- 0.0.0.0/0 x.x.x.x tcp dpt:8775
DROP all -- 0.0.0.0/0 x.x.x.x

Chain nova-network-INPUT (1 references)
target prot opt source destination
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:67
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:67
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:53
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:53
ACCEPT tcp -- 0.0.0.0/0 x.x.x.x tcp dpt:8775
DROP all -- 0.0.0.0/0 x.x.x.x

Instead of:
Chain nova-network-FORWARD (1 references)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT tcp -- 0.0.0.0/0 10.30.96.8 tcp dpt:8775

Chain nova-network-INPUT (1 references)
target prot opt source destination
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:67
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:67
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:53
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:53

What's missing?

@David, the advisory task is incomplete pending additional details from security reviewers (nova-coresec team).

For your information, the process is described there: https://wiki.openstack.org/wiki/VulnerabilityManagement#Reception

For some reasons, dhcp needs to talk with loopback...

--- linux_net.py.orig 2014-05-06 15:22:13.525362875 +0000
+++ linux_net.py 2014-05-06 22:01:42.914944165 +0000
@@ -808,6 +808,24 @@

 @utils.synchronized('lock_gateway', external=True)
+def isolate_compute_from_guest(network_ref):
+ if not network_ref:
+ return
+
+ iptables_manager.ipv4['filter'].add_rule('INPUT',
+ '-p tcp -d %s --dport 8775 '
+ '-j ACCEPT' % network_ref['dhcp_server'])
+ iptables_manager.ipv4['filter'].add_rule('FORWARD',
+ '-p tcp -d %s --dport 8775 '
+ '-j ACCEPT' % network_ref['dhcp_server'])
+ iptables_manager.ipv4['filter'].add_rule('INPUT',
+ '-d %s ! -i lo '
+ '-j DROP' % network_ref['dhcp_server'])
+ iptables_manager.ipv4['filter'].add_rule('FORWARD',
+ '-d %s ! -i lo '
+ '-j DROP' % network_ref['dhcp_server'])
+ iptables_manager.apply()
+
 def initialize_gateway_device(dev, network_ref):
     if not network_ref:
         return
@@ -1049,6 +1067,7 @@
             try:
                 _execute('kill', '-HUP', pid, run_as_root=True)
                 _add_dnsmasq_accept_rules(dev)
+ isolate_compute_from_guest(network_ref)
                 return
             except Exception as exc: # pylint: disable=W0703
                 LOG.error(_('Hupping dnsmasq threw %s'), exc)
@@ -1101,6 +1120,7 @@

     _add_dnsmasq_accept_rules(dev)

+ isolate_compute_from_guest(network_ref)

 @utils.synchronized('radvd_start')
 def update_ra(context, dev, network_ref):

Nova is using dhcp_release in order to communicate with the DHCP server to release the IP address so ! -i lo addresses this issue.

Can't this be done by implementing EBtables on the Compute host to disallow inappropriate traffic from the nova-network bridge?

I don't think ebtables will work because you'll be able to contact the gateway of another compute node. This patch has been tested and prevent all VMs from all compute nodes from accessing any compute node using nova-network.

(I don't know much of ebtables so I can't be 100% sure of this statement but iptables does the trick)

Didn't test it yet but:

--- a/nova/network/linux_net.py
+++ b/nova/network/linux_net.py
@@ -1631,10 +1631,14 @@ def remove_ebtables_rules(rules, table='filter'):
 def isolate_dhcp_address(interface, address):
     # block arp traffic to address across the interface
     rules = []
+ rules.append('INPUT -p TCP -i %s --dst %s --ip-destination-port 8776 -j ALLOW'
+ % (interface, address))
     rules.append('INPUT -p ARP -i %s --arp-ip-dst %s -j DROP'
                  % (interface, address))
     rules.append('OUTPUT -p ARP -o %s --arp-ip-src %s -j DROP'
                  % (interface, address))
+ rules.append('INPUT -i %s --dst %s -j DROP'
+ % (interface, address))
     # NOTE(vish): the above is not possible with iptables/arptables
     ensure_ebtables_rules(rules)
     # block dhcp broadcast traffic across the interface

The full patch would look like this:

--- a/nova/network/linux_net.py
+++ b/nova/network/linux_net.py
@@ -1631,10 +1631,14 @@ def remove_ebtables_rules(rules, table='filter'):
 def isolate_dhcp_address(interface, address):
     # block arp traffic to address across the interface
     rules = []
+ rules.append('INPUT -p TCP -i %s --dst %s --ip-destination-port 8776 -j ALLOW'
+ % (interface, address))
     rules.append('INPUT -p ARP -i %s --arp-ip-dst %s -j DROP'
                  % (interface, address))
     rules.append('OUTPUT -p ARP -o %s --arp-ip-src %s -j DROP'
                  % (interface, address))
+ rules.append('INPUT -i %s --dst %s -j DROP'
+ % (interface, address))
     # NOTE(vish): the above is not possible with iptables/arptables
     ensure_ebtables_rules(rules)
     # block dhcp broadcast traffic across the interface
@@ -1663,10 +1667,14 @@ def isolate_dhcp_address(interface, address):
 def remove_isolate_dhcp_address(interface, address):
     # block arp traffic to address across the interface
     rules = []
+ rules.append('INPUT -p TCP -i %s --dst %s --ip-destination-port 8776 -j ALLOW'
+ % (interface, address))
     rules.append('INPUT -p ARP -i %s --arp-ip-dst %s -j DROP'
                  % (interface, address))
     rules.append('OUTPUT -p ARP -o %s --arp-ip-src %s -j DROP'
                  % (interface, address))
+ rules.append('INPUT -i %s --dst %s -j DROP'
+ % (interface, address))
     remove_ebtables_rules(rules)
     # NOTE(vish): the above is not possible with iptables/arptables
     # block dhcp broadcast traffic across the interface

I'm not quite sure but I doubt this patch will work because we're not using share_dhcp_address and by default that one is set to false. Looking at the code, the simplest method to get that enforced is to add it to iptables.

Would that be a better patch?

--- a/nova/virt/libvirt/firewall.py
+++ b/nova/virt/libvirt/firewall.py
@@ -76,6 +76,28 @@ class NWFilterFirewall(base_firewall.FirewallDriver):
                   </filter>'''

     @staticmethod
+ def nova_isolate_compte_node():
+ """This filter will disallow all traffic toward the gateway of the guests.
+ """
+
+ return '''<filter name='nova-allow-dhcp-server' chain='ipv4'>
+ <uuid>891e4787-e5c0-d59b-cda6-41bc3c6b36fc</uuid>
+ <rule action='allow' direction='in'
+ priority='100'>
+ <tcp dstipaddr='$DHCPSERVER'
+ dstportstart='8776'/>
+ </rule>
+ <rule action='drop' direction='in'
+ priority='100'>
+ <tcp dstipaddr='$DHCPSERVER' />
+ </rule>
+ <rule action='drop' direction='in'
+ priority='100'>
+ <udp dstipaddr='$DHCPSERVER' />
+ </rule>
+ </filter>'''
+
+ @staticmethod
     def nova_dhcp_filter():
         """The standard allow-dhcp-server filter is an <ip> one, so it uses
            ebtables to allow traffic through. Without a corresponding rule in
@@ -221,6 +243,7 @@ class NWFilterFirewall(base_firewall.FirewallDriver):
         self._define_filter(self._filter_container('nova-vpn',
                                                    ['allow-dhcp-server']))
         self._define_filter(self.nova_dhcp_filter)
+ self._define_filter(self.nova_compute_isolation)

         self.static_filters_configured = True

This one needs to be tested:

--- a/nova/virt/libvirt/firewall.py
+++ b/nova/virt/libvirt/firewall.py
@@ -76,6 +76,28 @@ class NWFilterFirewall(base_firewall.FirewallDriver):
                   </filter>'''

     @staticmethod
+ def nova_dhcp_isolate_filter():
+ """This filter will disallow all traffic toward the gateway of the guests.
+ """
+
+ return '''<filter name='nova-isolate-dhcp-server' chain='ipv4'>
+ <uuid>891e4787-e5c0-d59b-cda6-41bc3c6b36fc</uuid>
+ <rule action='allow' direction='in'
+ priority='100'>
+ <tcp dstipaddr='$DHCPSERVER'
+ dstportstart='8776'/>
+ </rule>
+ <rule action='drop' direction='in'
+ priority='100'>
+ <tcp dstipaddr='$DHCPSERVER' />
+ </rule>
+ <rule action='drop' direction='in'
+ priority='100'>
+ <udp dstipaddr='$DHCPSERVER' />
+ </rule>
+ </filter>'''
+
+ @staticmethod
     def nova_dhcp_filter():
         """The standard allow-dhcp-server filter is an <ip> one, so it uses
            ebtables to allow traffic through. Without a corresponding rule in
@@ -221,6 +243,7 @@ class NWFilterFirewall(base_firewall.FirewallDriver):
         self._define_filter(self._filter_container('nova-vpn',
                                                    ['allow-dhcp-server']))
         self._define_filter(self.nova_dhcp_filter)
+ self._define_filter(self.nova_dhcp_isolate_filter)

         self.static_filters_configured = True

Well, sorry to spam, but I don't know where this could be injected in the code... The easiest place is where it is put in iptables and it also protects the compute node from being access from all the other guests from all the other computes nodes. If it's a ebtable INPUT rule, it must be global and not on a by instance basis. All my previous patches wont work (execpt the first one) as they are on a by instance basis or if share_dhcp_adress is set to true which is not the case in our case.

This patch should be addressing it:

--- a/nova/network/linux_net.py
+++ b/nova/network/linux_net.py
@@ -1447,6 +1447,9 @@ class LinuxBridgeInterfaceDriver(LinuxNetInterfaceDriver):

         if CONF.share_dhcp_address:
             remove_isolate_dhcp_address(iface, network['dhcp_server'])
+ else
+ remove_isolate_compute_node(iface, network['dhcp_server'])
+

         iptables_manager.apply()
         return self.get_dev(network)
@@ -1627,6 +1630,13 @@ def remove_ebtables_rules(rules, table='filter'):
         cmd = ['ebtables', '-t', table, '-D'] + rule.split()
         _execute(*cmd, check_exit_code=False, run_as_root=True)

+def isolate_compute_node(interface, address):
+ rules = []
+ rules.append('INPUT -p TCP -i %s --dst %s --ip-destination-port 8776 -j ALLOW'
+ % (interface, address))
+ rules.append('INPUT -i %s --dst %s -j DROP'
+ % (interface, address))
+ ensure_ebtables_rules(rules)

 def isolate_dhcp_address(interface, address):
     # block arp traffic to address across the interface

--- a/nova/network/linux_net.py
+++ b/nova/network/linux_net.py
@@ -1430,6 +1430,9 @@ class LinuxBridgeInterfaceDriver(LinuxNetInterfaceDriver):

         if CONF.share_dhcp_address:
             isolate_dhcp_address(iface, network['dhcp_server'])
+ else
+ isolate_compute_node(iface, network['dhcp_server'])
+
         # NOTE(vish): applying here so we don't get a lock conflict
         iptables_manager.apply()
         return network['bridge']
@@ -1447,6 +1450,9 @@ class LinuxBridgeInterfaceDriver(LinuxNetInterfaceDriver):

         if CONF.share_dhcp_address:
             remove_isolate_dhcp_address(iface, network['dhcp_server'])
+ else
+ remove_isolate_compute_node(iface, network['dhcp_server'])
+

         iptables_manager.apply()
         return self.get_dev(network)
@@ -1627,6 +1633,13 @@ def remove_ebtables_rules(rules, table='filter'):
         cmd = ['ebtables', '-t', table, '-D'] + rule.split()
         _execute(*cmd, check_exit_code=False, run_as_root=True)

+def isolate_compute_node(interface, address):
+ rules = []
+ rules.append('INPUT -p TCP -i %s --dst %s --ip-destination-port 8776 -j ALLOW'
+ % (interface, address))
+ rules.append('INPUT -i %s --dst %s -j DROP'
+ % (interface, address))
+ ensure_ebtables_rules(rules)

 def isolate_dhcp_address(interface, address):
     # block arp traffic to address across the interface
@@ -1659,6 +1672,13 @@ def isolate_dhcp_address(interface, address):
                           % (interface, address, CONF.iptables_drop_action)),
                          top...

nova coresec: care to comment on this one ?

I'm not working on nova-network currently, but did in a previous life so will add a comment.

One of the better ways to do this is to add a rule to the libvirt xml file to drop all inbound packets to the compute host, something like this in nova/virt/libvirt/firewall.py:

+ def nova_no_my_ip_address(self):
+ # Drop all IPv4 packets going to CONF.my_ip, since the network
+ # stack will loop them back.
+ retval = "<filter name='nova-no-my-ip-address' chain='ipv4'>"
+ retval += """<rule action='drop' direction='out'>
+ <ip dstipaddr='%s' />
+ </rule>""" % CONF.my_ip
+ retval += '</filter>'
+ return retval

Then just put some code in _ensure_static_filters() to define and append that to the existing filter set.

That's untested and based on older code, I see there is a get_host_ip_addr() method now that might be a better choice.

My $.02

What workarounds might be available that could form an OSSN to cover this and previous releases?

For some reason I was thinking there was an OSSN covering this, but the closest I found was the one on poor configuration instructions for live migration exposing libvirt's control interface to tenant instances. I also don't see anything in the security guide specifically addressing network hardening for compute nodes (recommended filter rules for local interfaces, isolating running services onto separate VLANs from instance virtual interfaces, et cetera) though it's possible I've overlooked it.

I think this falls into "hardening" -- it's not an exploitable vulnerability per se, and fixing this would significantly alter behavior and therefore can't really be backported.

It sounds like very welcome hardening though, so it would really be great if (1) we had an OSSN to cover for this and maybe (2) we improved the default situation in future releases.

Hmmm. The default behaviour of a compute system should be to isolate the host from the guests, so I think this is a vulnerability. However if the judgement of the VMT is that this isn't a vulnerability worth releasing an OSSA for, we'd certainly like to keep users informed by issuing an OSSN.

It seems to me that people should be deploying nodes such that this interface isn't used for anything other than the instance's dhcp service. I suspect that many people are already doing this, but perhaps it isn't obvious that it is a necessary step for security reasons?

That would certainly put it square in OSSN territory.

Are there any downsides to enforcing that the interface can only be used for the dhcp service?—
Sent from Mailbox

On Fri, May 30, 2014 at 12:42 PM, Bryan D. Payne <email address hidden> wrote:

> It seems to me that people should be deploying nodes such that this
> interface isn't used for anything other than the instance's dhcp
> service. I suspect that many people are already doing this, but perhaps
> it isn't obvious that it is a necessary step for security reasons?
> --
> You received this bug notification because you are a member of OpenStack
> Security Group, which is subscribed to OpenStack.
> https://bugs.launchpad.net/bugs/1316271
> Title:
> Network Security: VM hosts can SSH to compute node
> Status in OpenStack Compute (Nova):
> New
> Status in OpenStack Security Advisories:
> Incomplete
> Bug description:
> Hi guys,
> We're still using nova-network and we'll be using it for a while
> and we noticed that the VM guests can contact the compute nodes on all
> ports ... The one we're the most preoccupied with is SSH. We've
> written the following patch in order to isolate the VM guests from the
> VM hosts.
> --- linux_net.py.orig 2014-05-05 17:25:10.171746968 +0000
> +++ linux_net.py 2014-05-05 18:42:54.569209220 +0000
> @@ -805,6 +805,24 @@
>
> @utils.synchronized('lock_gateway', external=True)
> +def isolate_compute_from_guest(network_ref):
> + if not network_ref:
> + return
> +
> + iptables_manager.ipv4['filter'].add_rule('INPUT',
> + '-p tcp -d %s --dport 8775 '
> + '-j ACCEPT' % network_ref['dhcp_server'])
> + iptables_manager.ipv4['filter'].add_rule('FORWARD',
> + '-p tcp -d %s --dport 8775 '
> + '-j ACCEPT' % network_ref['dhcp_server'])
> + iptables_manager.ipv4['filter'].add_rule('INPUT',
> + '-d %s '
> + '-j DROP' % network_ref['dhcp_server'])
> + iptables_manager.ipv4['filter'].add_rule('FORWARD',
> + '-d %s '
> + '-j DROP' % network_ref['dhcp_server'])
> + iptables_manager.apply()
> +
> def initialize_gateway_device(dev, network_ref):
> if not network_ref:
> return
> @@ -1046,6 +1064,7 @@
> try:
> _execute('kill', '-HUP', pid, run_as_root=True)
> _add_dnsmasq_accept_rules(dev)
> + isolate_compute_from_guest(network_ref)
> return
> except Exception as exc: # pylint: disable=W0703
> LOG.error(_('Hupping dnsmasq threw %s'), exc)
> @@ -1098,6 +1117,7 @@
> _add_dnsmasq_accept_rules(dev)
> + isolate_compute_from_guest(network_ref)
> @utils.synchronized('radvd_start')
> def update_ra(context, dev, network_ref):
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nova/+bug/1316271/+subscriptions
>...

We're enforcing it at the IPtables level and everything seems to be working but we also implemented iptables rules for isolating further the compute nodes in the network.

OSSN Published, congratulations Stan!
https://wiki.openstack.org/wiki/OSSN/OSSN-0018

Fixing this bug by binding all services to a specific IP adress can be teadious ... Since we're talking about automating everything, I would sugggest to adress this issue by filtering it instead.

Otherwise, each operator will have to end with its own set off solutions in order to secure their network.

Ie: NFS, ssh, xinetd (inetd), ftp, etc

Fix proposed to branch: master
Review: https://review.openstack.org/118476

I created a blueprint in order to further development of logging features for the network layer.

https://blueprints.launchpad.net/nova/+spec/nova-compute-instance-traffic-logging

Related fix proposed to branch: master
Review: https://review.openstack.org/119589

Change abandoned by Daniel Berrange (<email address hidden>) on branch: master
Review: https://review.openstack.org/119589
Reason: You've already got this uploaded under a different review

https://review.openstack.org/#/c/117628/

Soince that review is older and has alot of relevant historical comments I'm marking this one abandoned. Please continue to post new versions to your original review.

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/117628
Reason: This review is > 4 weeks without comment and currently blocked by a core reviewer with a -2. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and contacting the reviewer with the -2 on this review to ensure you address their concerns.

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/118476
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/119589
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Agreed by core team that this is mostly a deployment topology issue, and nova-network was never designed to address it. Will not fix it.