MongoDb as dogpile caching backend

Proposal :

To provide mongoDb as another backend choice for dogpile caching. MongoDb can be a good alternate solution for distributed caching.

Why MongoDb?

For a distributed caching solution, data consistency, coherent view of updates, partitioning for load balancing, and replication for high availability are critical aspects in various caching schemes.

MongoDb, a NoSql database, has extremely fast reads and already has some of the features built-in e.g. high availability and high read throughput with primary and secondary replica sets, data segmentation via sharding, automatic synchronization among various nodes for coherent view of updates. In addition, MongoDB and mongodb.org supported drivers is open source.

Advantages:
• Extremely fast random reads on large datasets.
     o MongoDb uses memory mapped files and usually it takes only nanoseconds to resolve minor page faults to get file system cached pages mapped into MongoDB’s memory space.
• Cache key hashing (mangling) for optimal read can be taken care by defining index for key field in collection.
• MongoDb scale out very well with increasing size of data.
• MongoDb have built-in support for TTL (time-to-live) data via its TTL type of collections.
• Data can be distributed across multiple shards as a data partition alternative which is usually needed for load balancing.
o With replica setup, cache reads can be directed to secondary nodes to provide improved load balancing of cache calls.
• MongoDb allows a much richer query interface programmatically. Other caching solution typically offers only programmatic/API-based key-based access. Although Search APIs are being added, these are still reasonably primitive in terms of express-ability and run-time performance.
• In MongoDB, a document’s representation in the database is similar to its representation in application memory. This means the database already stores the usable form of data, making the data usable in both the persistent store and in the application cache. Whereas distributed cache implementations are typically guilty of inventing proprietary and opaque persistence formats.

MongoDb backend needs pymongo driver for mongoDb operations. This package dependency and its usage need to be loosely coupled with keystone. If needed persistent data can be encrypted to secure sensitive data. Also MongoDb supports SSL between application and its server nodes.