Instance without public ip fails reading metadata with separate CC & CLC

Bug #580184 reported by Mike Cook
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
eucalyptus (Ubuntu)
Expired
Medium
Unassigned

Bug Description

With separate CC and CLC, when an instance without a public address (in MANAGED[_NOVLAN] mode) attempts to contact the metadata service to get its ssh key, the CC reroutes the request to the CLC and masquerades the requestor's address. So the CLC appears to get the metadata request from the CC address and doesn't properly respond (how can it identify which instance is asking?), whereas a request from an instance with a public ip shows up as coming from that public ip (since the CC nat's the public/private address and doesn't masquerade it).

For example:
CC+SC: 10.0.0.1
CLC+WALRUS: 10.0.0.2
VNET_MODE=MANAGED_NOVLAN
VNET_SUBNET=172.16.0.0
VNET_CLOUDIP=10.0.0.2

A private instance (no public ip) is created as 172.16.1.1. On boot it queries (in /etc/rc.local) to http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key for it's ssh key. The 169.254.169.254 address is bound on the CC (10.0.0.1), which has a DNAT rule redirecting HTTP to the CLC (10.0.0.2:8773). The CC masquerades the instance's private IP as itself (as it must, since the CLC isn't on the private network) and forwards the request. The CLC then gets the request, but the source IP address is 10.0.0.1 (the CC's address) and it doesn't reply with the key. Without the ssh key you then can't ssh to the private instance (either from the CC or another public/private host in the subnet).

In contrast, a public instance is created as 172.16.1.2 with public ip 10.0.0.3. It makes the same HTTP request which gets redirected through the CC to the CLC. The CC, however, applies SNAT and DNAT rules which make the request appear as coming from 10.0.0.3 (the instance public IP) and the CLC properly responds to the request since it can identify the source instance. And there was much rejoicing...

Mathias Gug (mathiaz)
Changed in eucalyptus (Ubuntu):
importance: Undecided → Medium
Revision history for this message
Scott Moser (smoser) wrote :

Is this easily reproducible for you ? always reproducible ?
It seems that you might be hitting the same bug 566792 .

The symptom of the request to the metadata service coming from the CC is the same.

Revision history for this message
Mike Cook (mikewillcook) wrote :

Indeed, I've never had it work with separate CC & CLC for an instance without a public IP (see my comment #28 in bug 566792).

Revision history for this message
Scott Moser (smoser) wrote :

@Mike,
  Sorry that this has gone so long un-touched.
  Are you still seeing this in Eucalyptus on 11.10 or have you found a solution?
  This not a general issue as most/many Eucalyptus installations do not see this.

Changed in eucalyptus (Ubuntu):
status: New → Incomplete
Revision history for this message
Mike Cook (mikewillcook) wrote :

Sorry, I haven't worked with Eucalyptus since so don't know if this issue was ever resolved. Unless the designed changed somewhat, however, I would have expected it to be a problem for everyone using separate CC & CLC with private addressing and managed mode, as the metadata service keys off public IP. If it's not a general issue then most must be avoiding that set of conditions, or it was addressed.

Revision history for this message
graziano obertelli (graziano.obertelli) wrote :

I just tested this against our partner cloud (multi cluster setup, so CLC-CC-SC on one machine and second CC-SC on a second, ran an instance on the second cluster) and I couldn't reproduce it. Eucalyptus should masquerade all the traffic coming from the private instance (it's a catch-all rule at the end so triggered only if more specific rules do not apply), so the instance, even with only a private address, is able to get to the metadata store.

Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for eucalyptus (Ubuntu) because there has been no activity for 60 days.]

Changed in eucalyptus (Ubuntu):
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.