Secure Central Data store(etcd)

Registered by yatin

Currently the Data store(etcd) used by COE's is not secure, Attackers can access to this(via public network/private network) and get/store/update/delete keys store there and this can led the COE cluster to misbehave.
So, access to the data store should not be open. IT must be secured by following:
-> etcd ports should be allowed only on private network.
--> tls can be enabled on etcd and then application(kube, flannel, etc) within cluster should communicate with their own etcd cluster using certificates. If applications are running on same node on which etcd is running then they can communicate non secure channel.
Optional: etcd allows provide role based access, if COE's support this it can also be enabled.

Blueprint information

Status:
Complete
Approver:
Adrian Otto
Priority:
Essential
Drafter:
yatin
Direction:
Approved
Assignee:
yatin
Definition:
Review
Series goal:
Accepted for ocata
Implementation:
Implemented
Milestone target:
milestone icon ocata-1
Started by
Adrian Otto
Completed by
yatin

Related branches

Sprints

Whiteboard

TLS can be enabled for etcd. It's required to enable applications to connect etcd over secure channel.
Applications connecting to ETCD for different cluster drivers:-
1) swarm fedora:
swarm manager, swarm agent, flannel

2) mesos:
none

3) k8s coreos:
flannel, k8s_apiserver, k8s_podmaster, kube_proxy

4) k8s atomic
flannel, k8s_apiserver, k8s_podmaster, kube_proxy

Currently i checked it for k8s_apiserver(fedora atomic) and the parameters for etcd(like etcd-cafile, etcd-keyfile and etcd-certfile) are not working(invalid param error). I think reason for this is: image doesn't have correct stable version of kubernetes.
kubernetes-node-1.2.0-0.15.alpha6.gitf0cd09a.fc23.x86_64
kubernetes-1.2.0-0.15.alpha6.gitf0cd09a.fc23.x86_64
kubernetes-client-1.2.0-0.15.alpha6.gitf0cd09a.fc23.x86_64
kubernetes-master-1.2.0-0.15.alpha6.gitf0cd09a.fc23.x86_64

---yatin 14/11/2016

Description and Examples:-
Examples:-
Create a cluster(k8s, swarm) with TLS-enabled or TLS-disabled. This Blueprint just add a security layer over the central data store(ETCD) for the COE, So no functionality for the COE should be impacted, All the COE specific examples given in Magnum Documentation should work as they work before this blueprint.

With TLS-enabled(default for swarm/k8s), etcd will have TLS support, so if one want to access etcd cluster from outside master node should have certificates for authentication.
ETCD Database can be accessed/updated using etcdctl/curl/any other client.
Example:-
etcdctl --endpoint http://<master node ip>:2379 ls
etcdctl --endpoint https://<master node ip>:2379 ls
curl https://<master node ip>:2379/v2/keys
curl http://<master node ip>:2379/v2/keys

curl http://<master node ip>:2379/v2/keys/<any available key> -XDELETE
curl http://<master node ip>:2379/v2/keys/<any available key> -XDELETE

If one has certificates/keys ETCD cluster can be accessed/updated as below:
Since same set of certificates are used for COE/ETCD, following works:-
eval $(magnum cluster-config --force k8s-cluster)
curl --cacert ca.pem --cert cert.pem --key key.pem https://<master node ip>:2379/v2/keys
curl --cacert ca.pem --cert cert.pem --key key.pem https://<master node ip>:2379/v2/keys/<any available key> -XDELETE

---yatin 07/12/2016

Gerrit topic: https://review.openstack.org/#q,topic:bp/secure-etcd-cluster-coe,n,z

Addressed by: https://review.openstack.org/403501
    [swarm] Enable TLS in Etcd cluster

Addressed by: https://review.openstack.org/404782
    [k8s_fedora_atomic] Remove podmaster

Addressed by: https://review.openstack.org/407374
    [k8s_fedora_atomic] Enable TLS in Etcd cluster

Addressed by: https://review.openstack.org/414891
    [k8s_coreos] Enable TLS in Etcd cluster

Addressed by: https://review.openstack.org/420361
    [swarm] Fix cert filename in swarm-agent service

@yatin cam you add release notes for bp?

Addressed by: https://review.openstack.org/432849
    Add reno: bp secure-etcd-cluster-coe

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.