Common DB purge engine (oslo)

Registered by Boris Pavlovic on 2013-11-20

There are several points:

1) Nova & other core projects never delete data from the DBs and uses only soft deleting (There is special column in each table that indicates that entry is deleted).

2) To purge soft deleted records you have to get direct access to DB and do it manually by own script. Or use db-archiving feature that allows you to move soft deleted records from main to shadow tables. But db-archiving:
a) works very slow and as there is a complex logic around moving records from one table to another (we are not able to improve this)
b) db-archiving doesn’t take in account relationships between Tables.
c) db-archiving was implemented in Grizzly, during havana we found and fixed a lot of critical bugs, and didn’t see any bug reports form other, seems like nobody use it in production.
d) We didn’t find use cases for db-archiving.

3) Everybody that use OpenStack in production have some db purge script.

Due to these facts we should implement one purge engine that will be available out of box. This purge engine will cleanup considering of tables relationship. Implementation will be based on sql requests that do all work inside DB (without downloading all DB data) to improve deletions speed.

Purge engine's part of tables relationship control could be used next time to improve shadow tables.

This functionality will be available for administrators through the REST api. And probably we should think about periodic task that will cleanup DB.

Blueprint information

Status:
Not started
Approver:
Doug Hellmann
Priority:
Medium
Drafter:
Boris Pavlovic
Direction:
Needs approval
Assignee:
Boris Pavlovic
Definition:
Approved
Series goal:
None
Implementation:
Not started
Milestone target:
None

Related branches

Sprints

Whiteboard

Removing from icehouse so we can work out a firm plan. -- dhellmann

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.