Memcached Resiliency

Currently memcached can be safely run in one instance, running more instances doesn't increase resiliency or distribution of the cached key-value pairs.

Traditionally memcached clients use a consistent hashing algorithm to distribute the keys across a previously defined set of static servers. Memcached forks, like repcached[0] can be used for HA, to decrease the load in an event when a server goes away, however consistency cannot be guaranteed, because of the replication lag. The hashing algorithm and it's configuration on the client side poses a problem when scaling the number of memcached servers inside a cluster. The algorithm can be changed, but configuring it, as well as deploying the change across a number of services is problematic, since they will not have a consistent view of the cache during the reconfiguration and have to be restarted.

The currently preferred solution to these problems is deploying a memcached proxy, a number of implementations are used at Twitter[1], Facebook[2], Netflix[3]. Twitter's "nutcracker" is the most widely deployed one, however it lacks features like replicated pools and online reconfiguration. It also looks abandoned[4]. Netflix's rend is used in very specific scenarios with persistent storage. Mcrouter has an up to date docker image and is actively maintained[5].

I'm proposing the implementation of a new chart for mcrouter and making changes in the default configuration to use it together with one or more memcached servers. The advantage of this is that the operator will be able to scale the cache in a way that best suits them after the initial deployment, without reconfiguring and restarting the services using oslo_cache.

[0] http://repcached.lab.klab.org/
[1] https://github.com/twitter/twemproxy
[2] https://github.com/facebook/mcrouter
[3] https://github.com/Netflix/rend
[4] https://github.com/twitter/twemproxy/pull/321
[5] https://hub.docker.com/r/m3adow/mcrouter/