EC2 cloud-init overwrites 127.0.1.1 in /etc/hosts on every reboot

Bug #890501 reported by Eric Hammond
22
This bug affects 4 people
Affects Status Importance Assigned to Milestone
cloud-init
Fix Released
Medium
Scott Moser
cloud-init (Ubuntu)
Fix Released
Medium
Unassigned

Bug Description

When running an EC2 instance of Ubuntu 11.10 Oneiric, cloud-init sets up /etc/hosts with a default value for 127.0.1.1 looking something like:

    127.0.1.1 ip-10-202-61-233.ec2.internal ip-10-202-61-233

I edit /etc/hosts to change this value to something that makes more sense to my internal software (e.g., Apache). For example:

    127.0.1.1 myhost.example.com myhost

BUG: Whenever I reboot the EC2 instance, cloud-init overwrites my important settings of this value back to the old default. This breaks the startup of my applications on the server as they expect to be able to resolve the names I want set in the /etc/hosts file. My web server never comes up after a simple reboot.

Once the user edits the value for 127.0.1.1 in the /etc/hosts file, it should never be overwritten. Not on a reboot. Not on a stop/start (which assigns new IP addresses). Not even when an AMI is built out of this instance and a new instance is run.

The user changed that value to something they cared about with reasons we can't assume to trump. It should be left alone.

If /etc/hosts does not exist or if 127.0.1.1 is still the same value that cloud-init last set it to, then it *might* be acceptable to overwrite it with a value based on a change in the private IP address, HOWEVER, this might still break the application if software configured itself using the old value.

Either never change the value if it exists, or store elsewhere the value that was initially set and only change it if it is still the same.

See also bug #371936 where a similar bug was fixed back in 2009. Figure out if there is some reason that this part of the system is fragile and prone to breaking.

ProblemType: Bug
DistroRelease: Ubuntu 11.10
Package: cloud-init 0.6.1-0ubuntu22
ProcVersionSignature: User Name 3.0.0-12.20-virtual 3.0.4
Uname: Linux 3.0.0-12-virtual i686
ApportVersion: 1.23-0ubuntu3
Architecture: i386
Date: Tue Nov 15 00:17:46 2011
Ec2AMI: ami-a7f539ce
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: us-east-1d
Ec2InstanceType: t1.micro
Ec2Kernel: aki-805ea7e9
Ec2Ramdisk: unavailable
PackageArchitecture: all
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: cloud-init
UpgradeStatus: No upgrade log present (probably fresh install)

related bugs:
  * bug 871966: FQDN written to /etc/hosts causes problems for clustering systems

Related branches

Revision history for this message
Eric Hammond (esh) wrote :
Eric Hammond (esh)
description: updated
description: updated
description: updated
description: updated
Eric Hammond (esh)
description: updated
Revision history for this message
Scott Moser (smoser) wrote :

Thanks for the bug report.
I believe that you should be able to simply add an entry above the one labeled 'cloud-init'

$ grep -v :: /etc/hosts
127.0.0.1 localhost
127.0.1.2 superman.brickies.net superman

# The following lines are desirable for IPv6 capable hosts
# Added by cloud-init
127.0.1.1 ip-10-212-231-182.ec2.internal ip-10-212-231-182
$ cat /etc/hostname
superman
$ hostname -f
superman.brickies.net
$ ping -c 1 superman.brickies.net -q
PING superman.brickies.net (127.0.1.2) 56(84) bytes of data.

--- superman.brickies.net ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.028/0.028/0.028/0.000 ms

Revision history for this message
Eric Hammond (esh) wrote :

Scott: Learn something new every day. I will give this a try.

Changed in cloud-init (Ubuntu):
status: New → Incomplete
Eric Hammond (esh)
Changed in cloud-init (Ubuntu):
status: Incomplete → Invalid
Revision history for this message
Eric Hammond (esh) wrote :

Scott: Your suggested approach works for me even though I can't find any documentation on how this is supposed to work. Everything I read says that 127.0.1.1 should be the canonical hostname of the instance. Do you happen to have any pointers to the information you are alluding to with IPv6 and 127.0.1.2?

Revision history for this message
Scott Moser (smoser) wrote : Re: [Bug 890501] Re: EC2 cloud-init overwrites 127.0.1.1 in /etc/hosts on every reboot

On Tue, 15 Nov 2011, Eric Hammond wrote:

> Scott: Your suggested approach works for me even though I can't find any
> documentation on how this is supposed to work. Everything I read says
> that 127.0.1.1 should be the canonical hostname of the instance. Do you
> happen to have any pointers to the information you are alluding to with
> IPv6 and 127.0.1.2?

No.
   I actually tried just adding a 127.0.1.1 at the top, and cloud-init
still found it (i was hoping that it would only update the one preceeded
by it's comment, which would have been nicer). I tried that, it didn't
work.
   I don't think this should be "invalid" as I do think we've made things
a bit worse. We were addressing the issue that we saw on openstack where
the ec2metadata's local-hostname did not have a domain portion (bug
854614
).

   Some other fallout of this change has been in services that try to bind
to the response to `hostname -f`. The service will then end up binding to
localhost only and not be available on a real address. Mark Mimms has
raised this issue, but I'm not certain which services in particular, and
I don't think that is a great solution itself, and those services would
have issues with the mechanism that you're using above for /etc/hosts, but
previously we were not writing a fqdn to /etc/hosts, which would allow a
dns-lookup to get the local-ipv4 address.

   I'll talk with some of the people on the foundations team and see if we
can't come up with a final solution for this. I really should have done
some more research before putting this in. Hopefully we can get it right
for 12.04, and possibly SRU for 11.10.

Changed in cloud-init (Ubuntu):
status: Invalid → Confirmed
Scott Moser (smoser)
description: updated
Scott Moser (smoser)
Changed in cloud-init (Ubuntu):
importance: Undecided → Medium
Changed in cloud-init:
importance: Undecided → Medium
Revision history for this message
Scott Moser (smoser) wrote :

This is fix-commited in cloud-init in revision 491 (http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/revision/491).
See the commit message there for more information.

Changed in cloud-init:
assignee: nobody → Scott Moser (smoser)
status: New → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.6.3~bzr497-0ubuntu1

---------------
cloud-init (0.6.3~bzr497-0ubuntu1) precise; urgency=low

  * New upstream snapshot.
    - cloud-config support for configuring apt-proxy
    - selection of local mirror based on presense of 'ubuntu-mirror' dns
      entry in local domain. (LP: #897688)
    - DataSourceEc2: more resilliant to slow metadata service (LP: #894279)
    - close stdin in all programs launched by cloud-init (LP: #903993)
    - revert management of /etc/hosts to 0.6.1 style (LP: #890501, LP: #871966)
    - write full ssh keys to console for easy machine consumption (LP: #893400)
    - put INSTANCE_ID environment variable in bootcmd scripts
    - add 'cloud-init-per' script for easily running things with a given freq
      (this replaced cloud-init-run-module)
    - support configuration of landscape-client via cloud-config (LP: #857366)
    - part-handlers now get base64 decoded content rather than 2xbase64 encoded
      in the payload parameter. (LP: #874342)
 -- Scott Moser <email address hidden> Thu, 22 Dec 2011 04:07:38 -0500

Changed in cloud-init (Ubuntu):
status: Confirmed → Fix Released
Scott Moser (smoser)
Changed in cloud-init:
status: Fix Committed → Fix Released
Revision history for this message
James Falcon (falcojr) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.