Multiple Interfaces and IPs not detected in AWS VPC

Thanks for taking the time to open a bug and make ubuntu better.

When multiple interfaces and IPs first came out, I looked at this, but yours is the first request for this.

I agree, I'd like to have a udev hook that would fire on interface adaded and then determine if it was in EC2, and if so, hit the EC2 metadata service and configure the device. That'd be smooth.

I'm more than happy to review and help out, but its not something that will be high priority on my plate at the moment.

Scott, thanks for the reply. This is not something I need at the moment, but it does seem like a good way for me to perhaps learn more about Ubuntu network configuration and udev rules. I did a bit of digging and found the following files which seem relevant:

/lib/udev/write_net_rules
/lib/udev/rules.d/75-net-description.rules
/lib/udev/rules.d/75-persistent-net-generator.rules
/etc/udev/rules.d/70-persistent-net.rules

Are there any other starting points I should be aware of?

My current thinking is that a udev rule will need to have a RUN hook that updates /etc/network/interfaces or at a minimum runs the appropriate ip commands based on what is discovered in the EC2 metadata service. It seems like updating /etc/network/interfaces would be good since ifup/ifdown rely on its configuration. According to the VPC docs (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#detach_eni) it appears that eth0 can *not* be removed from a running instance while eth1 may be removed or added while the machine is running. However, it is unclear if additional VIPs can be added to eth0 while the instance is running. I'm not sure how changes to the available IPs would be detected without polling the meta-data service.

Adding notes from IRC:

* use 'manual eth1' to avoid cloud-init blocking on boot waiting for interfaces
* /etc/network/interfaces originates from a hardcoded file in the image
* cloud-init places /etc/cloud/on-ec2 (or similar) flag file
* on interface add/remove have a rule that fires, checks ec2 flag file, then updates /etc/network/interfaces, and ifup/down

I'm in a similar situation: An ec2 instance with just one network interface, to which I have assigned a secondary IP address.

This bit from the AWS docs is interesting:

"If you are using Amazon Linux, the ec2-net-utils package can take care of this step for you. It configures additional network interfaces that you attach while the instance is running, refreshes secondary IP addresses during DHCP lease renewal, and updates the related routing rules."

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-instance-addressing.html#StepTwoConfigOS

Perhaps it would be worthwhile for cloud-init to do the same thing that Amazon's ec2-net-utils package does.

Forest,
  Thanks for the link. I hadn't seen that utility. It does appear to be doing basically what I was suggesting.

Sure. The part that stood out to me was "refreshes secondary IP addresses during DHCP lease renewal." I haven't examined Amazon's code, but that description reads like it solves the problem of automatically configuring additional IP addresses without requiring additional network interfaces.

Hi Forest,

looking at http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#ec2-net-utils

"DHCP script
Whenever the network interface receives a new DHCP lease, this script queries the instance metadata for Elastic IP addresses. For each Elastic IP address, it adds a rule to the routing policy database to ensure that outbound traffic from that address uses the correct network interface. It also adds each private IP address to the network interface as a secondary address."

I think all they do is run a hook script on DHCP lease that configures the interface. And yes they add multiple ip addresses to the same interface.

I am looking at some sort of solution because I want to use some instances as LXC hosts and would love to give each container its own ip. So I think I will have to go down the route of using subinterfaces.

Will keep tracking this bug

Thank you
Alessandro

Hi,

I have actually adapted the files from the Amazon AMI which are working just fine.
There is also an ansible file which tells where the files should be dropped. You might need to adjust a few things but it should all be working

There is only a problem with this bug (https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/1235169) which has actually been solved and I am waiting for the updated package to be available. Otherwise the solution for this is to add the interfaces-eth1 configuration into the main /etc/network/interfaces file.

I am not good at packaging stuff so if you think this is useful it would be great if someone could wrap it up in a nice package. Myself I am ok with using ansible for now :)

Hope this helps
Regards
Alessandro

Re-uploading attachment as solution

Fixed some bugs. New version here
https://github.com/ademaria/ubuntu-ec2net

I'm attaching ec2-utils-0.4-1.19.amzn1.src.rpm as I retrieved it with:
 get_reference_source -p ec2-net-utils
from inside ami-83e4bcea (amazon/amzn-ami-pv-2013.09.1.x86_64-ebs) today.

The license per spec file and headers is 'Apache License 2.0'.

adding also as a tarball of the rpmbuild dir after rpm -Uvh (jsut for convenience).

We're using a variant Alessandro's work to configure out instances.

https://github.com/instacart/ubuntu-ec2net

Reboot doesn't seem to refresh the interfaces list -- maybe there is another event hook that we need? I'm not terribly familiar with udev but would love to see this ironed out and ready for Ubuntu.

Installing the RPMs (via Alien) seemed to have no effect at all.

I'm using cloud-init on CentOS, but would really like this functionality as well. For the time being, I'll be trying to reuse the upstream AWS pieces.

Hi

This is still the same for new instances on AWS ec2 ubuntu 144 . Floating i/p secondary eth0 not coming up automatically. IS there a fix in plan for ubuntu 14,15 ?

Thanks,

NP

Hi,
I made a small patch to support multiple ips on one interface. Should work on several interfaces too even if I din't test it.
I tested it on ec2 with ubuntu 18.04 and netplan export

The attachment "ec2_multiple_ips.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

My initial tests with Gaëtan's patch suggests it disables DHCP for but doesn't set a route or DNS manually.

ci-info: +--------+------+----------------+-----------------+--------+-------------------+
ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
ci-info: +--------+------+----------------+-----------------+--------+-------------------+
ci-info: | eth0 | True | 172.16.255.107 | 255.255.254.0 | global | 06:94:b2:8b:c0:3e |
ci-info: | eth0 | True | 172.16.254.112 | 255.255.254.0 | global | 06:94:b2:8b:c0:3e |
ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . |
ci-info: | lo | True | 172.27.221.33 | 255.255.255.255 | global | . |
ci-info: +--------+------+----------------+-----------------+--------+-------------------+
ci-info: ++++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++
ci-info: +-------+--------------+--------------+---------------+-----------+-------+
ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
ci-info: +-------+--------------+--------------+---------------+-----------+-------+
ci-info: | 0 | 0.0.0.0 | 172.16.254.1 | 0.0.0.0 | eth0 | UG |
ci-info: | 1 | 172.16.254.0 | 0.0.0.0 | 255.255.254.0 | eth0 | U |
ci-info: +-------+--------------+--------------+---------------+-----------+-------+

and

network:
    version: 2
    ethernets:
        eth0:
            addresses:
            - 172.16.255.107/23
            - 172.16.254.112/23
            dhcp6: true
            match:
                macaddress: 06:94:b2:8b:c0:3e
            set-name: eth0

Hi folks, as a heads-up, there's some in-progress work on secondary IP addresses on primary NICs here: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/369792

This was implemented in https://github.com/canonical/cloud-init/pull/114/files

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2361