Making algorithm of purging soft deleted records older than <date> unbreakable.

Registered by Marcin Kosobucki

Current tool of "cinder-manage -d db purge <days>" is assuming pristine condition of database data. To be specific, assumption that every row in "volume" table, marked as "deleted" and "deleted_at", has all other tables that are using "volume_id" Foreign Key are also marked is far from reality. In practice when operating Cinder for many years and going through different backends, upgrades, outages. There is a high chance of some tables with FK constraint missing "deleted" or "deleted_at".

Proposed change
Change the logic of the "cinder-manage -d db purge <days>" to be equivalent of:

--------------------------------------------------------------------------

delete from backups where volume_id IN (select id from volumes where deleted_at < '<date>');
delete from snapshots where volume_id IN (select id from volumes where deleted_at < '<date>');
delete from transfers where volume_id IN (select id from volumes where deleted_at < '<date>');
delete from volume_admin_metadata where volume_id IN (select id from volumes where deleted_at < '<date>');
delete from volume_admin_metadata_clone where volume_id IN (select id from volumes where deleted_at < '<date>');
delete from volume_attachment where volume_id IN (select id from volumes where deleted_at < '<date>');
delete from volume_glance_metadata where volume_id IN (select id from volumes where deleted_at < '<date>');
delete from volume_metadata where volume_id IN (select id from volumes where deleted_at < '<date>');

# and finally last one that will now always work, making it unbreakable :)

delete from volumes where deleted_at < '<date>'

--------------------------------------------------------------------------

Once this is done, you will never see the following error when you run:

cinder-exec cinder-manage -d db purge 90

2019-10-30 22:33:25.524 1 ERROR cinder.db.sqlalchemy.api [(.....) - - - - -]
DBError detected when purging from volumes: (pymysql.err.IntegrityError)
(1451, u' Cannot delete or update a parent row: a foreign key constraint fails
(`cinder`.`volume_admin_metadata`, CONSTRAINT `volume_admin_metadata_ibfk_1` FOREIGN KEY (`volume_id`) REFERENCES `volumes` (`id`))')
[SQL: u'DELETE FROM volumes WHERE volumes.deleted_at < %(deleted_at_1)s']
[parameters: {u'deleted_at_1': datetime.datetime(2019, 8, 1, 22, 33, 25, 410032)}]
(Background on this error at: http://sqlalche.me/e/gkpj).:
DBReferenceError: (pymysql.err.IntegrityError)
(1451, u'Cannot delete or update a parent row: a foreign key constraint fails
(`cinder`.`volume_admin_metadata`, CONSTRAINT `volume_admin_metadata_ibfk_1` FOREIGN KEY (`volume_id`) REFERENCES `volumes` (`id`))')
[SQL: u'DELETE FROM volumes WHERE volumes.deleted_at < %(deleted_at_1)s'] [parameters: {u'deleted_at_1': datetime.datetime(2019, 8, 1, 22, 33, 25, 410032)}]
(Background on this error at: http://sqlalche.me/e/gkpj)
Purge command failed, check cinder-manage logs for more details.

Blueprint information

Status:
Not started
Approver:
None
Priority:
Undefined
Drafter:
Marcin Kosobucki
Direction:
Needs approval
Assignee:
None
Definition:
New
Series goal:
None
Implementation:
Unknown
Milestone target:
None

Whiteboard

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.