Support the kubernetes service external-load-balancer feature

Registered by Steven Dake on 2015-01-19

Kubernetes is divided into two network zones. There is the backend, which only the frontend may speak with. There is the frontend which may speak with the backend and be presented traffic from an external IP address.

The problem arises that the frontend doesn't know which IP addresses map to which external services. In order for kubernetes to route the packets properly, a "publicIPs" field must be set for the public IP addresses which Kubernetes is responsible for managing the service on. To keep things simple, I think we can just specify every IP address in the bay when the publicIPs field is set in the yaml.

We will also want to support the createExternalLoadBalancer feature. I think this feature should create 1 IP address per service created and loadbalance traffic to that IP/port to all minions in the cluster via the private network. Then the publicIPs field should be set in the services.yaml to the private network IP list.

https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/services.md

Blueprint information

Status:
Complete
Approver:
Adrian Otto
Priority:
Essential
Drafter:
Steven Dake
Direction:
Approved
Assignee:
Ton Ngo
Definition:
Approved
Series goal:
Accepted for liberty
Implementation:
Implemented
Milestone target:
milestone icon liberty-3
Started by
Steven Dake on 2015-02-17
Completed by
Ton Ngo on 2015-10-20

Related branches

Sprints

Whiteboard

Please add a T-Shirt size estimate for implementation of this feature (S, M, L, XL)

Large - sdake

danehans- From the midcycle, review the URL in the BP as k8s services have undergone a considerable design change.

@ton - haven't seen you in irc meetings - curious of status on this work - sdake
@sdake - I show up as tango in the irc meeting. I have started working on the blueprint, currently studying the template and the bay topology as provisioned.

From chatting on the Google container IRC (Tim Hockin), support for createExternalLoadBalancer in a service is available in Kubernetes for OpenStack. The way this should work is that the apiserver should be provided the argument --cloud_provider and --cloud_config, pointing to OpenStack. When a service is created with "createExternalLoadBalancer: true", Kubernetes will talk to Neutron in OpenStack and set up the load balancer. I am trying this out now, but if this works as described, then we would only need to update Heat template to configure and start kube-apiserver appropriately on the master node. The rest should be transparent.

That's good news indeed, Ton! --adrian_otto

@tango - how are credentials sent? I suspect more R&D work is needed in kubernetes to make it work, but that is indeed good news if it works as described. I am curious how garbage collection works with the load balancer as well. Tim is one of the senior guys in that project, so he would know for sure the details. The fact that there may be built-in support is highly encouraging. -sdake

The configuration file that should be copied to the kube master node looks something like this:
[Global]
auth-url=http://9.30.138.111:5000/v2.0
Username=admin
user-id=
Password=mypassw0rd
api-key=
tenant-id=
tenant-name=admin
domain-id=
domain-name=
Region=RegionOne

[LoadBalancer]
subnet-id=107e831c-4089-4fe0-a0ae-5eaf254cd938
create-monitor=yes
monitor-delay=1m
monitor-timeout=30s
monitor-max-retries=3

So this includes the credential. The Kubernetes code does handle create, update, delete for the Neutron load balancer:
pkg/cloudprovider/openstack/openstack.go OpenStack.TCPLoadBalancer() -- tango

Sweet - that looks really good and simple if it works. -- sdake

We ran into a bug in the Kubernetes code interfacing with Neutron Load Balancer. We reported the bug and Google developers quickly fixed it. The official release does not have this fix yet, not sure when the next one will come, so for now, I am attempting to build Kubernetes locally and deploy with the bay cluster. -- tango

adrian_otto 2015-06-11: Tango, I'd like an update on this blueprint at our upcoming team meeting please. I am placing it on the agenda.

tango 2015-08-04:
I am finding that the backend code in Kubernetes to manage the Neutron load balancer continue to evolve, causing us to have to continually update the Fedora Atomic image to pick up the latest fixes. Kubernetes itself was changing rapidly, with options being added and deprecated, so we had to do some debugging after each release update.

Building the F21 image also ran into problems: the kube-apiserver binary in the image got corrupted in similar way to other binaries like ping. After debugging with the Atomic team (Jason Brooks et al), we found it was a known bug that has to do with the way some app manages the permission. The bug was supposedly fixed in F22, but the process for building the F22 image is not yet well developed and did not work after several attempts. For now I am working around this current problem by doing guestmount on the image and modifying the systemd service. This allows us to bring up a V1 Kubernetes cluster. This also gives us the opportunity to upgrade to the V1 api from the current v1beta3 api.

Looking at the backend code (authored by Angus Lee from Rackspace):
https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/cloudprovider/openstack/openstack.go
This code leverages the gopher cloud package for the actual interface with OpenStack:
https://github.com/rackspace/gophercloud/
In particular, the interface to neutron lbaas is in:
https://github.com/rackspace/gophercloud/tree/master/openstack/networking/v2/extensions/lbaas

The code implements this sequence of interaction with neutron:
1. create lb-pool
2. create lb-member
3. create lb-healthmonitor
4. create lb-vip

The current observation when a service with load balancer is created, is that the Kubernetes backend is able to connect to Neutron and request the lb-pool, but the response from neutron is missed for some reason and the backend keeps retrying creating the lb-pool. On the neutron side, the log shows the lb-pool being requested and created correctly. I posted the problem on the Google container IRC and ML but did not get much help.

I am now going through the go code to see if more logging can be added and attempting a custom build to debug the problem.

8/17/15
   We were able to connect with Angus Lees, the author of the Kubernetes plugin for OpenStack.
Angus pointed out that we need at least the following 3 patches to make it work:
https://github.com/GoogleCloudPlatform/kubernetes/pull/12203
https://github.com/GoogleCloudPlatform/kubernetes/pull/12262
https://github.com/GoogleCloudPlatform/kubernetes/pull/12288
   These are new patches so they have not been picked up by the Kubernetes release yet, so I have done a custom Kubernetes build based on the latest release 1.0.3 to test (also added more logging for debugging).

Update: a custom Kubernetes was built based on version 1.0.3 with the 3 patches and additional logging. The load balancer behavior remains the same. The log output points to problem in the gophercloud code where the response from Neutron does not seem to be picked up correctly.

The upgrade to Kubernetes V1 is now splitted into its own blueprint since we need this independently of the load balancer blueprint. This will help make it easier to set up and test the load balancer. Add a dependency with the new Kubernetes V1 blueprint.

tango 09-14-15:
   After adding lots of logging to Kubernetes v1.0.4 and stepping through the go code, we finally worked out the required set up in the cluster. There are basically 2 problems:
1. The name used to register a minion has to match the nova instance name. This is because the k8s code uses it to look up the IP address for the minion to register it as a member of the pool.
2. The private network in the cluster has to be named "private". This is because the k8s code gets the info for the nova instance and has to find the network and thus the IP. The code looks for "private", "public", "accessIPv4", "accessIPv6"

I tested the following scenario:
-deploy a pod running nginx
-deploy a service for the pod with external load balancer
-verify the pool, member, vip, health monitor are created
-associate a floating IP to the lb-vip
-access nginx through the floating IP

The current patches have been updated to implement the required set up. Manjeet is adding a patch to compute the parameters to set to the templates.

Several issues remain:
1. The lb-vip has an IP in the private subnet, so the user will need to manually associate a floating IP before the load balancer can be accessed externally. I had a discussion with Angus Lees on adding this support in the Kubernetes plugin. Angus mentioned there is some work in progress for the aws plugin and he is following up to see it can be implemented in the plugin for OpenStack.
2. The 3 fixes above are needed for the load balancer to work, but they have not been picked up yet by the current latest release 1.0.6. It is not clear when they will be picked up. Our work-around is to use our own custom Kubernetes build (version 1.0.4 + 3 fixes) until the fixes are released. This is in image fedora-21-atomic-6.qcow2.
3. The backend code in gophercloud still use keystone v2, which is deprecated. At some point they should port to v3, so we need to keep an eye on the port and adjust our code accordingly.
4. The neutron LB resources created by Kubernetes are not managed by Heat, so before deleting the cluster, the user needs to make sure to delete all the services that use LB; otherwise, the cluster delete would fail because the cluster private subnet would still be in use. Alternatively, we can add code in Magnum to delete all Kubernetes services

tango 09-18-15
Discussion on IRC with sdake on a security issue with Kubernetes.
Kubernetes backend needs to talk to OpenStack services and this is done by providing the user credential in a config file stored on the master node: tenant, user name, password. Then on start up, the credential is loaded once and the kube-apiserver service will authenticate with Keystone and make an initial query. Later, the kube-controller-manager service will use the authenticated client to create Neutron LB pool, members, VIP and monitor.
The tenant and user name are extracted from the request context so there is no problem, but the password is problematic because magnum has to handle it and save it in a file.
A more secure approach is to pass the token from the request to Magnum and have Kubernetes backend use it to create Neutron resources. However this will take a fair amount of change in Kubernetes upstream code. We can start the discussion with Angus Lees to see how this can be achieved.
For now, one solution is to use the password approach with the intention to deprecate it later. We need to implement in a way that does not degrade the default security in Magnum and give the user the option to use the load balancer feature.

10/12/15
This issue was discussed on the mailing list and feedback was provided.
The current implementation circumvents this issue by requiring the user to enter the password manually in the Kubernetes configuration file.
A blueprint has been registered to develop a common and secure method to provide credential to COE that needs to make API request to OpenStack services:
https://blueprints.launchpad.net/magnum/+spec/credential-for-coe

Because of this manual step, the functional test is blocked at this time.
A blueprint has been registered for the functional test:
https://blueprints.launchpad.net/magnum/+spec/lb-functional-tests

With these tasks decoupled into separate sub-blueprints, this blueprint can be considered as implemented.

Gerrit topic: https://review.openstack.org/#q,topic:bp/external-lb,n,z

Addressed by: https://review.openstack.org/191878
    WIP: Configure Kubernetes to interface with Neutron

Addressed by: https://review.openstack.org/192425
    WIP: Configure Kubernetes to interface with Neutron for CoreOS

Addressed by: https://review.openstack.org/192426
    WIP: Configure Kubernetes to interface with Neutron for Ironic

Addressed by: https://review.openstack.org/227689
    User guide for Kubernetes external load balancer

(?)

Work Items

Work items:
1. Set up process to build new Fedora Atomic image with latest Kubernetest release: DONE
2. Update configure-kubernetes-master.sh with new options: DONE
3. Update 3 kube master templates: DONE
4. Add 2 new scripts to write config for k8s services: DONE
5. Update template definition to compute parameters for generating config: DONE
6. Resolve handling of password in Magnum: DONE
7. Validate load balancer is created using nginx: DONE
8. Write document on using feature: DONE

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.