Need implementation of purge engine for db.

Registered by Eduard Yuzlikeev on 2013-10-14

There are several points:

1) Nova & other core projects never delete data from the DBs and uses only soft deleting (There is special column in each table that indicates that entry is deleted).

2) To purge soft deleted records you have to get direct access to DB and do it manually by own script. Or use db-archiving feature that allows you to move soft deleted records from main to shadow tables. But db-archiving:
a) works very slow and as there is a complex logic around moving records from one table to another (we are not able to improve this)
b) db-archiving doesn’t take in account relationships between Tables.
c) db-archiving was implemented in Grizzly, during havana we found and fixed a lot of critical bugs, and didn’t see any bug reports form other, seems like nobody use it in production.
d) We didn’t find use cases for db-archiving.

3) Everybody that use OpenStack in production have some db purge script.

Due to these facts we should implement one purge engine that will be available out of box. This purge engine will cleanup considering of tables relationship. Implementation will be based on sql requests that do all work inside DB (without downloading all DB data) to improve deletions speed.

Purge engine's part of tables relationship control could be used next time to improve shadow tables.

This functionality will be available for administrators through the REST api. And probably we should think about periodic task that will cleanup DB.

Blueprint information

Not started
Russell Bryant
Boris Pavlovic
Needs approval
Series goal:
Not started
Milestone target:
milestone icon next

Related branches



This blueprint has been deferred from the icehouse-3 milestone to Juno because it missed the "feature proposal freeze" deadline. --johngarbutt

The patch is abandoned, defer to juno, reseting implementation field.

Signed up core reviewers: johnthetubaguy

Gerrit topic:,topic:bp/db-purge-ingine,n,z

Addressed by:
    Add script for purge db

Gerrit topic:,topic:bp/db-purge-engine,n,z

Related blueprint that is being closed in favor of this one (since this one was filed first):

There is still some debate over the exact approach, but I think the general idea of adding a built-in ability to purge deleted records is straight forward enough to approve the blueprint, and we can work through the details of the approach in the code review. --russellb

Unapproved - please re-submit via nova-spec --johnthetubagy (20th March 2014)

See: -- mriedem (20150710)


Work Items