Models should be the same as the state in DB after migration (partial)

Registered by Boris Pavlovic on 2013-06-08

We are using declarative_base in nova.db.models just for reflection, not for db creation.
For creation we are using migrations.
We don't have any tests that checks, that our models are up-to-date. Also we are testing it only on sqlite, which couldn't test such things as nullable constraints.

So We should:

1) Use always explicit nullable parameter for columns. There is a lot of mistakes in current nova models implementation. (sometimes in schema is nullable=True and in model nullable=False and vise versa). Also at this moment when you see Column description without nullable, you couldn't be sure that is author forget to set nullable='False' or it is really nullable='True'. So the easiest way to track all this and to fix and to avoid such things is to have one rule "always use explicit nullable".
2) Add in __table_args__ indexes and unique constraints
3) Fix all mistakes in models
4) Fix all mistekes in migrations
5) Sync effects of migrations in different backends.
6) Add tests that ensures that models are up-to-date.

This will allow us to find probably some mistake, or missing inexes and make work with db more clean.

Blueprint information

Status:
Started
Approver:
Russell Bryant
Priority:
Low
Drafter:
Boris Pavlovic
Direction:
Needs approval
Assignee:
Boris Pavlovic
Definition:
Review
Series goal:
None
Implementation:
Slow progress
Milestone target:
milestone icon next
Started by
Boris Pavlovic on 2013-06-08

Related branches

Sprints

Whiteboard

This blueprint has been deferred from the icehouse-3 milestone to Juno because it missed the "feature proposal freeze" deadline. https://wiki.openstack.org/wiki/Icehouse_Release_Schedule --johngarbutt

This has lots of abandoned patches, defer this to Juno --johnthetubaguy

List of models:

✓AgentBuild
✓Aggregates
✓AggregateHost
✓AggregateMetadata
✓BandwidthUsage
✓BlockDeviceMapping
✓Cell
✓ Certificate
✓ ComputeNode
✓ ComputeNodeStat
✓Console
✓ConsolePool
✓DNSDomain
✓FixedIp
✓FloatingIp
✓ Instance
✓InstanceAction
✓InstanceActionEvent
✓InstanceFault
✓InstanceGroup
✓InstanceGroupMember
✓InstanceGroupMetadata
✓InstanceGroupPolicy
✓InstanceIdMapping
⚒InstanceInfoCache
✓InstanceMetadata
✓InstanceSystemMetadata
✓InstanceTypeExtraSpecs
✓InstanceTypeProjects
✓InstanceTypes
✓ IscsiTarget
✓ KeyPair
✓ Migration
✓ Network
✓ProviderFirewallRule
✓Quota
✓QuotaClass
✓QuotaUsage
✓Reservation
✓S3Image
✓SecurityGroup
✓SecurityGroupIngressDefaultRule
✓SecurityGroupIngressRule
✓SecurityGroupInstanceAssociation
✓ Service
✓Snapshot
✓SnapshotIdMapping
✓TaskLog
✓VirtualInterface
✓Volume
✓VolumeIdMapping
✓VolumeUsage

Bugs found that should be fixed by migrations:

*) drop `block_device_mapping_instance_uuid_virtual_name_device_name_idx` index from `block_device_mapping` table (dublicate `block_device_mapping_instance_uuid_device_name_idx`)

*) InstanceAction.instance_uuid, InstanceAction.instance_uuid, InstanceActionEvent.instance_uuid should be nullable=False

*) SecurityGroupIngressRule.protocol and SecurityGroupIngressRule.protocol can have Enum type (it is String today and in comments written, that can be "tcp", "udp", or "icmp" )

*) SecurityGroupIngressRule.parent_group_id and group_id,
 SecurityGroupInstanceAssociation.security_group_id and instance_uuid,
SecurityGroupInstanceAssociation.instance_uuid
should be nullable=False

*) create fk in migrations for instance_uuid column

*) drop unused tables declared im migrations and skipped in models.

Gerrit topic: https://review.openstack.org/#q,topic:bp/db-sync-models-with-migrations,n,z

Addressed by: https://review.openstack.org/32264
    Fix db.models.Service description

Addressed by: https://review.openstack.org/32265
    Fix db.models.ComputeNode description

Addressed by: https://review.openstack.org/32266
    Fix db.models.ComputeNodeStats description

Addressed by: https://review.openstack.org/32267
    Fix db.models.Certificate description

Addressed by: https://review.openstack.org/32268
    Fix db.models.Instance description

Addressed by: https://review.openstack.org/33426
    Sync models and migrations.

Addressed by: https://review.openstack.org/33439
    Sync db.models.Instance* with migrations.

Addressed by: https://review.openstack.org/33605
    Sync db.models.Quota* with migrations.

Gerrit topic: https://review.openstack.org/#q,topic:db-sync-models-with-migrations,n,z

Addressed by: https://review.openstack.org/33609
    Changed models for AgentBuild, Aggregates, AggregateHost tables.

Addressed by: https://review.openstack.org/33248
    Fix db description for some of Instance* models

Addressed by: https://review.openstack.org/33627
    Sync db.models.Security* with migrations.

Addressed by: https://review.openstack.org/33629
    Sync db.models.Volume* with migrations.

Addressed by: https://review.openstack.org/33784
    Sync db.models with migrations.

Addressed by: https://review.openstack.org/33788
    Sync models and migrations.

Addressed by: https://review.openstack.org/33791
    Sync models and migrations.

Addressed by: https://review.openstack.org/34373
    Sync models.Console* with migrations.

Addressed by: https://review.openstack.org/34385
    Sync db.models and migrations.

Addressed by: https://review.openstack.org/34394
    Sync db.models.Instance* with migrations.

Addressed by: https://review.openstack.org/35618
    Add cinder cleanup to migrations.

Addressed by: https://review.openstack.org/34212
    Check of equal models and migrations (mysql).

Addressed by: https://review.openstack.org/38522
    Sync models with migrations.

Addressed by: https://review.openstack.org/38525
    Add migrations with missing fkeys.

Addressed by: https://review.openstack.org/38798
    Fix migrations index

Gerrit topic: https://review.openstack.org/#q,topic:39685,n,z

Addressed by: https://review.openstack.org/39685
    Change the types of hypervisor_type and launched_on columns

Gerrit topic: https://review.openstack.org/#q,topic:bug/1207750,n,z

Addressed by: https://review.openstack.org/40548
    Add missing indexes back in from 152

Addressed by: https://review.openstack.org/55416
    Sync models with migrations

Addressed by: https://review.openstack.org/55506
    Implement testing of migrations from oslo

Addressed by: https://review.openstack.org/65968
    Sync migration with model PciDevice state

Unapproved - please re-submit via nova-spec --johnthetubagy (20th March 2014)

Addressed by: https://review.openstack.org/81789
    Sync model with migrations

Addressed by: https://review.openstack.org/81788
    Add migration to fix nullable params

Addressed by: https://review.openstack.org/82130
    Sync model SecurityGroup with migrations

Addressed by: https://review.openstack.org/83733
    Add a migration to change index in 'migrations' table.

Addressed by: https://review.openstack.org/81791
    WIP: test model migrations

Addressed by: https://review.openstack.org/85325
    Models should be the same as the state in DB after migration.

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.