Improve session handling in nova/db/api

Registered by Devananda van der Veen on 2012-11-02

Bring the use of sessions and transactions in nova/db/sqlalchemy/api into a consistent state, based on these goals:
* use explicit transactions only when necessary
* don't pass session objects to public methods
  An easy way to find methods which need to be updated is:
     grep -P 'def [^_].*session=' nova/db/sqlalchemy/api.py

Also, annotate methods that pose a risk but can not be easily addressed right now. Current known risks are:
- with_lockmode may deadlock (or at least create a global bottleneck)
- duplicate-insert race conditions (due to lack of UNIQUE constraints)

Blueprint information

Status:
Complete
Approver:
Russell Bryant
Priority:
High
Drafter:
Devananda van der Veen
Direction:
Approved
Assignee:
Boris Pavlovic
Definition:
Approved
Series goal:
Accepted for havana
Implementation:
Implemented
Milestone target:
milestone icon 2013.2
Started by
Devananda van der Veen on 2012-11-09
Completed by
Boris Pavlovic on 2013-08-07

Related branches

Sprints

Whiteboard

Gerrit topic: https://review.openstack.org/#q,topic:bp/db-session-cleanup,n,z

Addressed by: https://review.openstack.org/15580
    clean up dnsdomain_unregister

Addressed by: https://review.openstack.org/15664
    improve session handling of dnsdomain_list

Addressed by: https://review.openstack.org/15661
    add missing import to test_db_api.py

Addressed by: https://review.openstack.org/15738
    improve session handling around quotas

Addressed by: https://review.openstack.org/15739
    improve sessions for reservation and snapshots

Addressed by: https://review.openstack.org/15756
    improve sessions for reservation

Addressed by: https://review.openstack.org/15833
    improve session handling around virtual_interfaces

Addressed by: https://review.openstack.org/15846
    improve session handling around instance_ methods

Addressed by: https://review.openstack.org/17285
    improve sessions for key_pair_(create,destroy)

Addressed by: https://review.openstack.org/17286
    remove session param from certificate_get

Addressed by: https://review.openstack.org/17287
    remove session param from instance_get_by_uuid

Addressed by: https://review.openstack.org/17288
    remove session param from instance_get

Addressed by: https://review.openstack.org/18439
    Add soft_delete() method in sqlalchemy.orm.query.Query class

Addressed by: https://review.openstack.org/18440
    Using query.soft_delete() method insead of soft deleting by hand

Addressed by: https://review.openstack.org/18555
    Database reservations methods performance optimization

Addressed by: https://review.openstack.org/18587
    db.security_group_rule_destroy() method performance optimization

Addressed by: https://review.openstack.org/18588
    db.network_delete_safe() method performance optimization

Addressed by: https://review.openstack.org/18589
    Database metadata performance optimizations

Addressed by: https://review.openstack.org/18590
    Fix bug in db.instance_test_and_set()

Addressed by: https://review.openstack.org/18714
    Remove session.flush() and session.query() monkey patching

Addressed by: https://review.openstack.org/19035
    NovaBase.delete() rename to NovaBase.soft_delete()

Addressed by: https://review.openstack.org/19095
    Refactor work with TaskLog in sqlalchemy.api

Addressed by: https://review.openstack.org/19097
    Remove restoring soft deleted entries

Gerrit topic: https://review.openstack.org/#q,topic:bp/db-unique-keys,n,z

Addressed by: https://review.openstack.org/19748
    Remove restoring soft deleted entries part 2

Addressed by: https://review.openstack.org/24146
    Add mising tests for db.block_device_mapping_* methods

Addressed by: https://review.openstack.org/24147
    Refactor work with session in db.block_device_mapping_* methods

Gerrit topic: https://review.openstack.org/#q,topic:bug/1157442,n,z

Addressed by: https://review.openstack.org/24849
    Refactor db.service_destroy and db.service_update methods

Addressed by: https://review.openstack.org/26156
    Add missing tests for instance_type_extra_specs_* methods

Addressed by: https://review.openstack.org/26157
    Refactor work with db.instance_type_* methods

Addressed by: https://review.openstack.org/25254
    Remove race condition (in InstanceTypeProjects)

Gerrit topic: https://review.openstack.org/#q,topic:tests_for_inst_extra_specs,n,z

Addressed by: https://review.openstack.org/28235
    Session cleanup for db.floating_ip_* methods

Gerrit topic: https://review.openstack.org/#q,topic:floating_ips_cleanup,n,z

Addressed by: https://review.openstack.org/29509
    Removed session from fixed_ip_*() functions.

Addressed by: https://review.openstack.org/30440
    Removed session from reservation_create()

Gerrit topic: https://review.openstack.org/#q,topic:db-session-cleanup,n,z

Addressed by: https://review.openstack.org/31613
    Add tests for nova.db.api.security_group_rule_* methods. blueprint db-api-tests Change-Id: I20b63683da78bcb99e1f88cc10ca71a64f1f17a9

Addressed by: https://review.openstack.org/31619
    Session cleanup for db.security_group_* methods

Addressed by: https://review.openstack.org/31932
    Session cleanup for db.security_group_rule_* methods

Addressed by: https://review.openstack.org/32130
    Move resource usage sync functions to db backend

Gerrit topic: https://review.openstack.org/#q,topic:db-enforce-unique-keys,n,z

Gerrit topic: https://review.openstack.org/#q,topic:bp/common-quota,n,z

Addressed by: https://review.openstack.org/33962
    Add missing tests for db.api.instance_* methods

Gerrit topic: https://review.openstack.org/#q,topic:bp/bp,n,z

Addressed by: https://review.openstack.org/40125
    Clean up db.sqla.api.instance_* methods

Addressed by: https://review.openstack.org/40126
    Clean up session in db.api.sqla.get_ec2 methods

Addressed by: https://review.openstack.org/40127
    Clean up Session in db.sqla.api.network and sec_groups methods

Addressed by: https://review.openstack.org/40128
    Clean up session in db.sqla.api.migration_* methods

Addressed by: https://review.openstack.org/40246
    Finish DB session clenup

Gerrit topic: https://review.openstack.org/#q,topic:cleanup_db_sessions,n,z

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.