Models should be the same as the state in DB after migration (partial)

Registered by Boris Pavlovic

We are using declarative_base in nova.db.models just for reflection, not for db creation.
For creation we are using migrations.
We don't have any tests that checks, that our models are up-to-date. Also we are testing it only on sqlite, which couldn't test such things as nullable constraints.

So We should:

1) Use always explicit nullable parameter for columns. There is a lot of mistakes in current nova models implementation. (sometimes in schema is nullable=True and in model nullable=False and vise versa). Also at this moment when you see Column description without nullable, you couldn't be sure that is author forget to set nullable='False' or it is really nullable='True'. So the easiest way to track all this and to fix and to avoid such things is to have one rule "always use explicit nullable".
2) Add in __table_args__ indexes and unique constraints
3) Fix all mistakes in models
4) Fix all mistekes in migrations
5) Sync effects of migrations in different backends.
6) Add tests that ensures that models are up-to-date.

This will allow us to find probably some mistake, or missing inexes and make work with db more clean.

Blueprint information

Russell Bryant
Boris Pavlovic
Needs approval
Boris Pavlovic
Series goal:
Slow progress
Milestone target:
milestone icon next
Started by
Boris Pavlovic

Related branches



This blueprint has been deferred from the icehouse-3 milestone to Juno because it missed the "feature proposal freeze" deadline. --johngarbutt

This has lots of abandoned patches, defer this to Juno --johnthetubaguy

List of models:

✓ Certificate
✓ ComputeNode
✓ ComputeNodeStat
✓ Instance
✓ IscsiTarget
✓ KeyPair
✓ Migration
✓ Network
✓ Service

Bugs found that should be fixed by migrations:

*) drop `block_device_mapping_instance_uuid_virtual_name_device_name_idx` index from `block_device_mapping` table (dublicate `block_device_mapping_instance_uuid_device_name_idx`)

*) InstanceAction.instance_uuid, InstanceAction.instance_uuid, InstanceActionEvent.instance_uuid should be nullable=False

*) SecurityGroupIngressRule.protocol and SecurityGroupIngressRule.protocol can have Enum type (it is String today and in comments written, that can be "tcp", "udp", or "icmp" )

*) SecurityGroupIngressRule.parent_group_id and group_id,
 SecurityGroupInstanceAssociation.security_group_id and instance_uuid,
should be nullable=False

*) create fk in migrations for instance_uuid column

*) drop unused tables declared im migrations and skipped in models.

Gerrit topic:,topic:bp/db-sync-models-with-migrations,n,z

Addressed by:
    Fix db.models.Service description

Addressed by:
    Fix db.models.ComputeNode description

Addressed by:
    Fix db.models.ComputeNodeStats description

Addressed by:
    Fix db.models.Certificate description

Addressed by:
    Fix db.models.Instance description

Addressed by:
    Sync models and migrations.

Addressed by:
    Sync db.models.Instance* with migrations.

Addressed by:
    Sync db.models.Quota* with migrations.

Gerrit topic:,topic:db-sync-models-with-migrations,n,z

Addressed by:
    Changed models for AgentBuild, Aggregates, AggregateHost tables.

Addressed by:
    Fix db description for some of Instance* models

Addressed by:
    Sync db.models.Security* with migrations.

Addressed by:
    Sync db.models.Volume* with migrations.

Addressed by:
    Sync db.models with migrations.

Addressed by:
    Sync models and migrations.

Addressed by:
    Sync models and migrations.

Addressed by:
    Sync models.Console* with migrations.

Addressed by:
    Sync db.models and migrations.

Addressed by:
    Sync db.models.Instance* with migrations.

Addressed by:
    Add cinder cleanup to migrations.

Addressed by:
    Check of equal models and migrations (mysql).

Addressed by:
    Sync models with migrations.

Addressed by:
    Add migrations with missing fkeys.

Addressed by:
    Fix migrations index

Gerrit topic:,topic:39685,n,z

Addressed by:
    Change the types of hypervisor_type and launched_on columns

Gerrit topic:,topic:bug/1207750,n,z

Addressed by:
    Add missing indexes back in from 152

Addressed by:
    Sync models with migrations

Addressed by:
    Implement testing of migrations from oslo

Addressed by:
    Sync migration with model PciDevice state

Unapproved - please re-submit via nova-spec --johnthetubagy (20th March 2014)

Addressed by:
    Sync model with migrations

Addressed by:
    Add migration to fix nullable params

Addressed by:
    Sync model SecurityGroup with migrations

Addressed by:
    Add a migration to change index in 'migrations' table.

Addressed by:
    WIP: test model migrations

Addressed by:
    Models should be the same as the state in DB after migration.


Work Items

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.