Peer Discovery through Metadata

(Updated based on ML feedback, and to include proposed API)

When running a clustered service on Nova, typically each node needs to find its peers. In the physical world, this is typically done using multicast. On the cloud, we either can't or don't want to use multicast. The proposed feature would add new functionality to the metadata service to allow bootstrapping cross-instance discovery: instances could POST data to the metadata service that should be shared, and they can GET this data for any other instances in the same project as the current instance. I expect this data will likely be an IP address and port. The clustered service would then use this information to try to connect to suitable peers and start a full gossip protocol.

This can likely be done as an extension to: https://blueprints.launchpad.net/nova/+spec/metadata-service-callbacks We could add either an endpoint for shared data, or just support a 'shared' query parameter or JSON key.

==============================================================

Example API

POST http://169.254.169.254/openstack/latest/metadata
…
request / response as per https://blueprints.launchpad.net/nova/+spec/metadata-service-callbacks
…

A “shared” query parameter can also be passed. If the value is ‘project’ (?), the specified metadata keys passed will be made available to all other instances in the same (keystone) project. Other values are not currently supported and will therefore return a 4xx (?) error.

Note that because entries cannot be changed once set, the sharing on the entries also cannot be changed once set.

Instances can retrieve shared keys by requesting the shared.json (?) endpoint. The shared entries visible to the current instances (including any shared by the instance itself) are returned. For symmetry with the unshared case, the from_server_ prefix is included e.g.

JSON Response
[
{
    "meta": {
        "from_server_key_1": “value1_server1”,
        "from_server_key_2": "value2_server1”,
    },
    "uuid": "d8e02d56-2648-49a3-bf97-6be8f1204f38"
},
{
    "meta": {
        "from_server_key_1": “value1_server2”,
        "from_server_key_2": “value2_server2”,
    },
    "uuid": “12345678-1234-5678-1234-123456780123”
}
]

If no shared values are visible, an empty response is returned:
[
]

==============================================================

Alternatives:

* Expose a per-instance keystone token, and thus make it easy for the instance to query the normal nova api endpoint. That's a good idea (and I'd love to see it implemented), but is a lot more complicated. It would also put the burden on the API user. This blueprint would solves the use case a lot more directly & quickly

* Automatically expose IP addresses of other instances in the project / security group etc. I like this (it was my first inclination), but there are security concerns about exposing information about other instances, even within a project. This BP is only useful if deployed, so we can't go that route.