Keystone's token table grows unconditionally when using SQL backend.

Status changed to 'Confirmed' because the bug affects multiple users.

Moving this issue to a BP: https://blueprints.launchpad.net/keystone/+spec/keystone-manage-token-flush

Fix proposed to branch: master
Review: https://review.openstack.org/28133

Reviewed: https://review.openstack.org/28133
Committed: http://github.com/openstack/keystone/commit/ff76a1b5cd3308cfb0ce936800364e27413ed946
Submitter: Jenkins
Branch: master

commit ff76a1b5cd3308cfb0ce936800364e27413ed946
Author: Jamie Lennox <email address hidden>
Date: Fri May 3 14:04:09 2013 +1000

    Implement Token Flush via keystone-manage.

    Creates a cli entry 'token_flush' which removes all expired tokens.

    Fixes: bug 1032633
    Implements: blueprint keystone-manage-token-flush
    Change-Id: I47eab99b577ff9e9ee74fee08e18fd07c4af5aad

A CLI command is an interesting stopgap, but on a heavily utilized OpenStack installation with automated tools operating against OpenStack, this has a high manual maintenance cost. Surely there is some better default that lies in the middle ground between keeping tokens for ever and ever and requiring a manual removal of tokens?

As a reference point, I wasn't even aware this was an issue, until one of our test deployments of grizzly using a limited IO system started acting horribly (30 second response times). After tracing the problem from nova to keystone to mysql, I found a 442,000 row token table with >440,000 expired tokens. I went and checked our havana test on a somewhat beefier system and found > 1M rows.

This issue is a timebomb for any production OS install.

On 04/05/2014 01:43 PM, Andrew Mann wrote:
> A CLI command is an interesting stopgap, but on a heavily utilized
> OpenStack installation with automated tools operating against OpenStack,
> this has a high manual maintenance cost. Surely there is some better
> default that lies in the middle ground between keeping tokens for ever
> and ever and requiring a manual removal of tokens?
>
> As a reference point, I wasn't even aware this was an issue, until one
> of our test deployments of grizzly using a limited IO system started
> acting horribly (30 second response times). After tracing the problem
> from nova to keystone to mysql, I found a 442,000 row token table with
>> 440,000 expired tokens. I went and checked our havana test on a
> somewhat beefier system and found > 1M rows.
>
> This issue is a timebomb for any production OS install.
>
CRON the CLI job. There is no reason to try and integrate a scheduler
into Keystone.

That seems reasonable. I would suggest adding that as a step to the OpenStack setup documentation. No one has infinite database space, eventually all used OpenStack installations will suffer unless this is done. Regardless of what component (or person) has to complete the task, they'll need to be aware of it.

I would prefer to see some kind of "automatically expire old tokens" configuration option so that the maintenance of keystone stays in keystone - whether that be implemented as a expire-on-access query or cron-derived mechanism, but re-implementing cron is not great and a purge tagged on to other operations could cause a slowdown on those operations.

Since it's not already mentioned in this bug, the long term solution here is to simply not persist tokens at all:

  https://blueprints.launchpad.net/keystone/+spec/ephemeral-pki-tokens

Addressed by the following patch:

https://review.openstack.org/#/c/79105/