Caching layer implementation around driver calls.

Registered by Morgan Fainberg on 2013-07-30

There should be a configurable caching layer that lives above the various driver calls within Keystone. This should be configurable (using something akin to dogpile.cache) to allow for varying caching backends (in-memory, file-based, redis, memcache, etc) to alleviate load on the backend storage/mechanism for the differing aspects of keystone.

This system will encompass the feature(s) of https://blueprints.launchpad.net/keystone/+spec/cache-token-revocations and extend to also cover Identity, Token Issuance, Trusts, etc.

In absence of explicit configuration of a caching back-end, the behavior will remain the same as current with no-cache.

Key things to keep in mind are: cache invalidation, ensuring the configuration for keystone is extended to expose a reasonable featureset of the caching layer.

It will be on the developer to ensure that the decorator is added to new methods (as needed) and invalidation will be handled in a consistent manner. The cache interface within keystone will provide a way to be consistent with the implementation of the cache across the different drivers and subsystems within keystone.

The initial discussions via IRC was to provide access to a series of decorators that will decorate the various driver calls, allowing for caching-in-line (via dogpile.cache/memoization) based upon the arguments to the various methods/functions.

An example implementation is outlined: https://review.openstack.org/#/c/38866/

In conjunction with this, caching layer, normalization of the use of kwargs when calling manager methods needs to be completed. The main issue with kwargs, is that the dogpile.cache only supports the use of args (not kwargs) with the default key-generator. If we provide a new generator that can support kwargs, there is a lot more work to ensure that the invalidate() calls (when required) will produce expected results since the arguments will need to generate the same key in the cache. The extra level of introspection of the actual function needed to ensure consistency with kwargs support seems like it could have adverse side-effects, performance-impact, and possibly make the code much harder to maintain/follow.

The kwargs change has the added benefit of making the drivers/managers where kwargs are used more consistent with the rest of keystone (In some cases, notably token drivers/providers, there are extensive uses of kwargs to methods).

Blueprint information

Status:
Complete
Approver:
Adam Young
Priority:
Medium
Drafter:
Morgan Fainberg
Direction:
Approved
Assignee:
Morgan Fainberg
Definition:
Drafting
Series goal:
Accepted for havana
Implementation:
Implemented
Milestone target:
milestone icon 2013.2
Started by
Thierry Carrez on 2013-08-06
Completed by
Dolph Mathews on 2013-09-03

Related branches

Sprints

Whiteboard

morganfainberg:
In conjunction (as a separate review) I am proposing that we also cleanup the token providers to remove the single "issue_token" interface with the kwarg that specifies the version of token to be issued. It would be easier to follow if we simply provided an interface per version to issue tokens.

I am proposing deprecating the <provider>.issue_token(self, version='...', **kwargs) mechanism and exposing issue_v3_token and issue_v2_token methods as the mechanism to issue the types of tokens. Coupled with this change, the move from using **kwargs to an explicit set of expected arguments for each of the issue_<version>_token methods would occur, and it would make the issue_token methods require less argument validation (currently each argument needs to be tested for existence within the kwargs dict).
---------

Gerrit topic: https://review.openstack.org/#q,topic:bp/caching-layer-for-driver-calls,n,z

Addressed by: https://review.openstack.org/39003 (merged)
    Remove kwargs from manager calls where not needed.

Addressed by: https://review.openstack.org/40102 (merged)
    Refactor Token Providers for better version interfaces

Addressed by: https://review.openstack.org/41208 (merged)
    Keystone Caching Layer for Manager Calls

Addressed by: https://review.openstack.org/41598 (merged)
    Remove kwargs from manager calls where not needed.

Addressed by: https://review.openstack.org/43195 (merged)
    Implement caching for Tokens

Addressed by: https://review.openstack.org/43566
    Implement Caching for Token Revocation List (merged)

Addressed by: https://review.openstack.org/43608
    Refactor Token Provider to be aware of expired tokens. (merged)

Addressed by: https://review.openstack.org/43609 (merged)
    Implement basic caching around assignment CRUD

Addressed by: https://review.openstack.org/44211 (merged)
    Add Memory Isolating Cache Proxy

Addressed by: https://review.openstack.org/44408 (merged)
    Cleanup cache layer tests

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.