Database strategy for rolling upgrades

Registered by Brian Rosmaita

This spec outlines a database modification strategy for Glance that will facilitate rolling upgrades.

Blueprint information

Brian Rosmaita
Series goal:
Milestone target:
Started by
Brian Rosmaita
Completed by
Brian Rosmaita

Related branches



Gerrit topic:,topic:bp/expand-contract-alembic,n,z

Addressed by:
    Add expand/migrate/contract commands to glance-manage CLI

Gerrit topic:,topic:bp/database-strategy-for-rolling-upgrades,n,z

Addressed by:
    Add expand/migrate/contract migrations for CI

Gerrit topic:,topic:bp/expand-migrate-contract-alembic,n,z

Gerrit topic:,topic:bp/expand-migrate-contract-migrations-for-CI,n,z

Addressed by:
    Db Sync to Use Expand, Migrate & Contract

Appended by Ray Pekowski
Sorry, not sure how best to comment on this, but have you considered an approach whereby the N release reads both the new location and the old location for the data, instead of writing to both places. After all clients and servers have been upgraded to the N release, the data could be migrated instead of expanded and contracted. Once all data is migrated there could be either a switch to disable looking in the old location or another upgrade simply to disable looking in the old location.

(rosmaita) Thanks for the comment, Ray. The problem is that since you have N-1 release nodes running simultaneously with the N release nodes, the old and new locations could get out of sync, and which value should the N release use in that case? (It should honor the most recent write, but it's got no way of knowing what that is with our db design where updated_at has only row granularity.) The way we've addressed this is that the N release reads and writes *only* to the new column (not to both old and new), and database triggers keep the old and new locations synchronized. (The triggers are removed in the contraction phase.) The spec explains this in a lot of detail:

(pekowski) I read that spec and is what pointed me to this blueprint :-)
What I propose might be too complicated to implement in a database lookup, but the general idea is that searches would look in both places for the information. Writes would continue to go to the old location until the point that all N-1 nodes are updated. At that point all nodes would be directed to write to the new location, but still look in both the old and new locations. Then some migration script would move the entries from the old location to the new, one by one, so there is not down time. When the migration is complete, the nodes would be directed to no longer look at the old locations. Maybe that is too complicated, but seems to have an advantage in database size not potentially doubling.


Work Items

This blueprint contains Public information 
Everyone can see this information.